Self-hosted monitoring for homelabs: host and container metrics and uptime checks from all your machines in one dashboard, with an optional AI agent that diagnoses incidents and suggests fixes.
Python ai-agentsaiopsdashboarddockerfastapihomelabincident-managementllmmonitoringopenaireactroot-cause-analysisself-hosteduptime-monitoring
feat(agent): show the investigation steps in reports and chat (#198)
Karlo Kajba Šimanić1d2e52a2986f
diff --git a/ODSTUPANJA.md b/ODSTUPANJA.mdindex d001910..8598eba 100644--- a/ODSTUPANJA.md+++ b/ODSTUPANJA.md@@ -91,7 +91,8 @@ su anotacije dizajna i namjerno nisu ugrađeni.| „Confidence: high · 3 of 3 signals agree" | „3 of 3 checks against the data hold" | Nema modela koji bi procijenio pouzdanost | #93 || Koraci s točnom putanjom (`/opt/gitlab/docker-compose.yml`) i brojkama iz povijesti verzija | Koraci iz fiksnog priručnika po vrsti pravila, s imenom kontejnera i izračunatim limitom | Infradeck ne zna gdje je compose datoteka ni koju verziju slike je kontejner imao prije | – || „Notified ntfy, email" | „sent to the channels people chose" ili „nobody yet" | Ne bilježi se po kanalu | – |-| Chat: odgovori s oznakama izvora („memory 12:30–13:55", „docker events") | Običan tekst odgovora | Model ne vraća izvore | #93 |+| Chat: odgovori s oznakama izvora („memory 12:30–13:55", „docker events") | Običan tekst odgovora; ispod njega sklopljeni „Looked up": pretrage koje je agent napravio, s njegovim pitanjem, što je tražio i redak činjenica iz rezultata | Izvori su pretrage alata, a ne oznake u tekstu; retke činjenica piše Infradeck, ne model | #158 |+| Nema prikaza kako je agent došao do dijagnoze | Ispod signala sklopljeni „How the agent got there · N look-ups" | Prikaz koraka istrage | #158 || Chat: prijedlozi pitanja vezani uz incident | Tri općenita prijedloga | Isto | – || „Resolved 14" po tjednima, s opisom kako je riješeno | Zadnjih 200 riješenih; opis je „Marked resolved by …" ili „Cleared on its own" | Nitko ne upisuje kako je riješeno | – || „Mark resolved" trajno zatvara | Ako je uzrok još tu, na sljedećem javljanju otvara se novi incident | Pravilo i dalje vrijedi | – |diff --git a/SECURITY.md b/SECURITY.mdindex f68508f..36b2858 100644--- a/SECURITY.md+++ b/SECURITY.md@@ -93,7 +93,9 @@ are never returned, so a tool answers everyone the same and shows nothing the daa set number of calls per answer (Admin › Incident agent › Look-ups, 8 per diagnosis and 4 per chat message bydefault, 0 for none); a call past it isn't run and the model then has to answer (`src/agent/investigate.py`). Everycall, run or refused, is written to the audit log with its masked arguments, and users can't forge or delete these-entries.+entries. The page shows each look-up only as a line of facts Infradeck writes from the result, never the raw result,+next to the reason the model gave for the call (masked like everything else it writes); see+[How the agent investigates](docs/agent-investigation.md).**Who sees the agent.** With "Show diagnoses to users" off, the backend leaves the agent's report out of everyincident, Overview line and notification for users who aren't admins. Chat access is "Admins only" by default; withdiff --git a/TESTS.md b/TESTS.mdindex 80d3cc8..7f875fe 100644--- a/TESTS.md+++ b/TESTS.md@@ -4,7 +4,7 @@ Infradeck ships with an automated test suite. This page explains how to run it,covers (and what it doesn't), and what every single test is for, so you can decidewhether a failure matters before you upgrade or change your setup.-- **1574 test cases** from 857 test functions, plus 38 small tests of the frontend helpers (see [Frontend helper tests](#frontend-helper-tests)) (some functions run once per input).+- **1608 test cases** from 867 test functions, plus 45 small tests of the frontend helpers (see [Frontend helper tests](#frontend-helper-tests)) (some functions run once per input).- **About two minutes** to run. No Docker, PostgreSQL, Redis, network or root access needed.- **Safe to run next to a live install.** The tests use a throwaway in-memory databaseand fakes; they never touch your running containers or your data. (Two collector@@ -152,6 +152,7 @@ The chart helper tests for the frontend are separate, in `src/frontend/tests/`.| [`test_agent_tool_calls.py`](#test_agent_tool_callspy) | 30 | Tool calling with Claude, OpenAI (Responses API) and Gemini: calls, several at once, errors, masking, append-only history, forcing an answer, reasoning carried over, nothing stored at OpenAI. || [`test_agent_diagnosis.py`](#test_agent_diagnosispy) | 42 | The agent's diagnosis: the evidence pack, secret masking, dangerous commands, checking the model's answer, saving reports. || [`test_agent_investigation.py`](#test_agent_investigationpy) | 27 | The agent looking things up: the budget of tool calls per diagnosis and chat message, forced final answers, errors, the audit log, citations from tool results. |+| [`test_agent_steps.py`](#test_agent_stepspy) | 34 | The agent's look-ups in plain words: what each tool asked and found, no raw results or secrets, windows left for the page, odd entries never failing. || [`test_agent_tools.py`](#test_agent_toolspy) | 52 | The agent's read-only tools: a closed list with strict schemas, checked arguments, bounded and masked results with fact IDs, no secrets or IP addresses, nothing written. || [`test_agent_watch.py`](#test_agent_watchpy) | 10 | Reusing a recent diagnosis, and watching the fix until the agent resolves the incident. || [`test_agent_worker.py`](#test_agent_workerpy) | 12 | The agent's job queue and worker: one job per opened or worsened incident after the commit, retries, time limit, restarts. |@@ -414,17 +415,35 @@ calls, then the answer) and checks what the loop did with them, on a small datab| `test_a_failing_tool_tells_the_model_nothing_about_why` | A tool that crashes gives the model only "The look-up failed"; the details (here a password) go to the server log. || `test_reaching_the_spending_cap_stops_the_look_ups` | With the monthly budget used up, the model gets no tools and has to answer. || `test_after_the_deadline_the_model_must_answer` | Once the deadline has passed, the model gets no tools and has to answer. |-| `test_every_tool_call_is_written_to_the_audit_log` | Each call, run, failed or refused, is an `agent_tool_call` entry with its user, conversation, question and arguments. |-| `test_secrets_in_the_arguments_are_masked_in_the_trail` | A password in a tool's arguments is masked in the kept arguments and in the error that repeats them. |-| `test_a_diagnosis_may_cite_what_a_tool_found_and_keeps_the_trail` | A signal citing a fact from a tool result is kept; the report stores the look-ups, each model call is counted, the audit entry names the incident, and the model is told its budget. |+| `test_every_tool_call_is_written_to_the_audit_log` | Each call, run, failed or refused, is an `agent_tool_call` entry with its user, conversation, question, the agent's reason and the arguments (without the reason). |+| `test_secrets_in_the_arguments_are_masked_in_the_trail` | A password in a tool's arguments or its reason is masked in the kept arguments, the reason and the error that repeats them. |+| `test_a_diagnosis_may_cite_what_a_tool_found_and_keeps_the_trail` | A signal citing a fact from a tool result is kept; the report stores the look-ups, each model call is counted, the audit entry names the incident, and the model is told its budget. The page gets the look-ups as steps in plain words, without any raw result. || `test_a_cited_id_no_tool_returned_is_still_dropped` | A signal citing an ID neither the pack nor a tool returned is dropped. || `test_a_fix_cannot_be_verified_on_a_metric_only_a_tool_returned` | A "metric stays below" condition on a metric found by a tool becomes manual, because only the pack's metrics can be watched. || `test_an_unusable_answer_is_asked_for_again_without_tools_in_the_same_conversation` | An answer that doesn't parse is asked for again in the same conversation with tools not allowed, so the look-ups aren't repeated. || `test_with_the_diagnosis_budget_at_0_the_agent_gets_no_tools` | With the diagnosis budget at 0 no tools are offered and the model is told so. || `test_a_provider_error_midway_is_counted_once_and_raised` | A provider error after a look-up is recorded once as an error and raised, so the job is retried; no report is saved. |-| `test_a_chat_message_can_look_things_up_within_the_chat_budget` | A chat message runs a tool, every model call is counted, and the call is audited. |+| `test_a_chat_message_can_look_things_up_within_the_chat_budget` | A chat message runs a tool, every model call is counted, the call is audited, and the reply comes back with the step described. || `test_the_chat_remembers_questions_and_replies_but_not_earlier_look_ups` | The next message sends the earlier question and reply, not its look-ups. |+### test_agent_steps.py++Each look-up as people see it (`src/agent/steps.py`), from real results of the tools' own test database (the+fixture of `test_agent_tools.py`). No model is involved.++| Test | What it checks |+|---|---|+| `test_each_tool_is_told_in_plain_words` | For each tool (and events filtered by kind), what it looked up and the line of facts it found, word for word. |+| `test_every_tool_has_a_way_to_say_what_it_asked_and_found` | Every tool in the list has both descriptions, so a new tool can't be added without them. |+| `test_a_step_carries_no_raw_result_and_no_secrets` | For every tool, a step has only its eight fields, no password or IP address, and a short line. |+| `test_the_window_is_left_for_the_page_to_show_in_the_viewers_clock` | The window or moment comes as separate UTC times (a null window as the last 24 hours it resolved to), not inside the text. |+| `test_a_failed_or_refused_call_says_why` | A failed call shows the reason it failed; a refused one "Not run: the look-up budget was used up."; an empty window counts as none. |+| `test_an_odd_entry_gives_a_plain_step_instead_of_failing` | Cut-off arguments, unreadable or oddly shaped results, an unknown tool and an empty entry each give a plain step. |+| `test_an_unusable_result_says_done_rather_than_guessing` | Where a number belongs but isn't one, the line is "Done." rather than a made-up figure. |+| `test_a_disk_that_barely_changes_is_not_growing` | A filesystem changing by less than 0.05 GB a day is "not growing"; one shrinking shows its negative growth. |+| `test_a_shortened_result_says_so` | A result cut to fit says "(the result was shortened to fit)". |+| `test_only_so_many_steps_are_described` | At most 30 steps are described; no trail gives none. |+### test_agent_tools.pyThe tools run on a small database with a fixed clock: two days of check-ins, hourly summaries before the last week,@@ -1249,6 +1268,7 @@ cd src/frontend && node --test # Node 20 or newer; nothing to install|---|---|---|| `tests/chartMath.test.js` | 9 | Histogram bins and weights, the hour-by-weekday grid, donut fractions, the disk-full estimate (clear trends only), axis ticks, gaps staying gaps, weighted summaries, time labels and forecast wording. || `tests/fleetLogic.test.js` | 7 | Container states, top-N rankings, disk ranking and colours, online detection, availability rows, readable durations, and incident columns that fill quiet days with zeros. |+| `tests/stepsLogic.test.js` | 7 | The window of an agent's look-up in the viewer's clock (one day, several days, up to now, one moment, a time without a zone read as UTC, none or unreadable), and the look-up count leaving out refused calls. |The React components themselves (the page layout, tooltips, selection) have no automated tests;they were checked by hand in a real browser.diff --git a/docs/README.md b/docs/README.mdindex 8722ff9..520d492 100644--- a/docs/README.md+++ b/docs/README.md@@ -12,6 +12,7 @@ The [README](../README.md) is the short version: what the project is, how to sta| [Alerts and incidents](alerts.md) | Every rule, its default thresholds, and how incidents open and resolve. || [Accounts, roles and two-factor sign-in](accounts.md) | Passwords, roles, the Account page, authenticator codes, recovery codes, locked-out accounts. || [Incident agent](assistant.md) | The optional agent: diagnoses, watching the fix, notifications, chat, who sees what, and what it costs. |+| [How the agent investigates](agent-investigation.md) | The agent's look-ups end to end: what the model sees, who writes each part of what you see, the budget, what is checked and stored, and the limits. |## Installing and runningdiff --git a/docs/agent-investigation.md b/docs/agent-investigation.mdnew file mode 100644index 0000000..008183c--- /dev/null+++ b/docs/agent-investigation.md@@ -0,0 +1,254 @@+[← Documentation](README.md) · [Incident agent](assistant.md)++# How the agent investigates++This page explains, end to end, what happens between an incident (or a chat question) and the agent's answer: what+the model is given, what it may look up, who writes each part of what you see, what is checked, what is stored, and+where the limits are. [Incident agent](assistant.md) is the short version for people using it; this page is for+anyone who needs to trust it, change it or review it.++The code is in `src/agent/`: `investigate.py` (the loop), `tools/` (the look-ups), `steps.py` (the steps shown to+people), `diagnosis.py` and `chat.py` (the two ways in), `masking.py` and `providers/` (what leaves the machine).++## Contents++- [The short version](#the-short-version)+- [One investigation, step by step](#one-investigation-step-by-step)+- [Who writes what](#who-writes-what)+- [What the model sees and what people see](#what-the-model-sees-and-what-people-see)+- [The budget, and when the model has to answer](#the-budget-and-when-the-model-has-to-answer)+- [What is checked](#what-is-checked)+- [What is stored, and who can see it](#what-is-stored-and-who-can-see-it)+- [The tools, and how each step is described](#the-tools-and-how-each-step-is-described)+- [Limits](#limits)+- [Changing it](#changing-it)++## The short version++1. The model gets the incident's evidence pack (a diagnosis) or the question and the conversation (the chat).+2. It may call read-only **tools** that look things up in Infradeck's database, a few times, within a **budget**.+ Each call names the question it should answer (its **reason**).+3. It writes its answer: a diagnosis that must cite the facts it rests on, or a chat reply.+4. Infradeck checks the answer, saves it with the look-ups behind it, and writes every look-up to the audit log.+5. The page shows the answer and, folded away underneath, each look-up: the agent's reason, what it looked up, and+ one line of facts from the result, written by Infradeck's code, not by the model.++The model never acts: no tool writes anything, runs a command or contacts a monitored machine.++## One investigation, step by step++```+ incident opens someone asks in the chat+ │ │+ ▼ ▼+ evidence pack (evidence.py) question + incident facts + earlier turns+ │ │+ └──────────────┬────────────────────────┘+ ▼+ investigate.py: start the conversation+ (system prompt, the message, the 11 tools)+ │+ ┌────────────►▼+ │ the model takes a step ── answer ──────────────┐+ │ │ │+ │ tool calls │+ │ ▼ │+ │ within budget? ── no ──► "Tool budget used up" │+ │ │ yes │+ │ ▼ │+ │ tools.run: check the arguments, read the DB, │+ │ bound and mask the result │+ │ │ │+ │ audit log entry · trail entry │+ │ │ │+ └── results back to the model (masked again) │+ ▼+ diagnosis: check the answer and save a report+ chat: save the reply with its steps+```++In more detail:++1. **The way in.** A diagnosis starts from the evidence pack (`evidence.py`): the incident, the machine, metrics+ around it, events, the rule's own checks, similar incidents. Every fact in it has an ID such as `metric:main`.+ A chat message starts from the question, the facts of the incident it is about (if any), and the conversation's+ latest 40 turns. Earlier look-ups are not sent again: only questions and replies stay in the conversation.+2. **The conversation starts** (`investigate.py`). The provider gets the system prompt (`DIAGNOSIS_PROMPT.md` or+ `SYSTEM_PROMPT.md`), the message, the list of tools, and the JSON schema the final answer must follow. With+ tools offered the model thinks harder (effort `high`; Claude and OpenAI use it). Optional arguments are plain+ strings where empty means "not given", because providers limit nullable parameters in strict schemas (#196);+ **Test connection** in Admin sends one request of this shape, so a provider that refuses it says so at once.+3. **The model takes a step.** It either answers, or asks for one or more tool calls. Each call is a tool name and+ arguments that follow the tool's schema, including `reason`: one sentence saying what the call should find out,+ written before any result exists.+4. **Each call is checked and run** (`tools/__init__.py`). The name must be one of the 11 tools, and the arguments+ must match the schema exactly (no extra or missing fields, right types, values from the allowed lists). The tool+ reads stored data only. Its result is cut to fit (about 12 KB), masked for secrets, and carries fact IDs.+5. **Each call is recorded**: in the trail kept with the answer, and in the audit log (`agent_tool_call`).+6. **The results go back** to the model, masked once more on the way (`add_tool_results`), and it takes the next+ step. This repeats until it answers or has to (see [the budget](#the-budget-and-when-the-model-has-to-answer)).+7. **The answer is checked and saved.** A diagnosis goes through `check_answer` (see [What is checked](#what-is-checked))+ and becomes a new version in `agent_reports`, with the evidence pack and the trail. A chat reply is saved in+ `chat_messages`, with its steps already described (`steps.py`).+8. **The page shows it.** The incident page and the chat show the answer, and under it "How the agent got there ·+ N look-ups" or "Looked up · N look-ups", folded away until opened.++## Who writes what++Everything you see comes from one of two places, and the page keeps them apart.++| Part | Written by | When | Can it be wrong about the data? |+|---|---|---|---|+| The diagnosis: summary, signals, fix steps, verify condition | the model | after its look-ups | Yes, so it is checked: every signal must cite facts that exist, and commands are checked for danger. |+| A chat reply | the model | after its look-ups | Yes. Chat replies are not checked against the data; treat them as advice. |+| A step's **reason** ("Did memory grow before the restarts?") | the model | when it makes the call, before the result exists | It states an intention, not a finding, so it can be vague but not a wrong claim about the result. |+| A step's **what it looked up** ("Memory on web1 · 1 Nov 00:00 – now") | Infradeck's code (`steps.py`), from the call's arguments | when the page is shown | No: it restates the arguments. |+| A step's **what it found** ("memory 41–96 %, average 63 %") | Infradeck's code (`steps.py`), from the stored result | when the page is shown | No: it copies numbers and names the tool returned, and says "Done." when the result isn't in the expected shape. |+| The evidence chart, events, timeline | Infradeck's code, from stored data | always | No. |++So the model's words are the diagnosis, the reply and the reasons; the facts next to them are Infradeck's. A step+never says what its numbers mean ("memory is the cause"): that is the diagnosis's job, and the diagnosis has to+cite the facts it rests on.++**Why the summaries are not written by the model.** It could summarise each result, but that would cost tokens on+every call and could not be checked against the data. The tools return structured data with the figures already+worked out (minimum, maximum, average, counts, growth), so restating them is plain formatting.++## What the model sees and what people see++| | The model | People on the page |+|---|---|---|+| Evidence pack | all of it, masked | the incident, chart, events and timeline (from stored data, not from the pack) |+| A tool result | all of it: statistics, up to 40 points per series, lists, labels | one line of facts (`found`) |+| A call's arguments | its own | in plain words (`asked`) and the window in their own clock |+| The reason | its own | in quotes, as the agent's |+| Errors | the reason a call was refused, or "The look-up failed" | the same |++Nothing reaches the browser that the dashboard doesn't already show: the raw results stay in the database+(`agent_reports.body`, `investigation`) and are never sent to the page. Check targets inside results are always+without credentials or query string.++**Masking.** Every text that goes to the provider is masked for tokens, keys, passwords and credentials in URLs+(`masking.py`): the evidence pack, each tool result (twice: by the tool and by the provider interface), each+tool error, and each call's arguments and reason as they are stored. The reason is the model's own text, but it+can repeat something it saw, so it is masked too.++## The budget, and when the model has to answer++The budget is a number of tool calls per answer, set in **Admin › Incident agent › Look-ups**:++| | Default | Range |+|---|---|---|+| per diagnosis | 8 | 0–20 |+| per chat message | 4 | 0–10 |++- **0** offers no tools at all: the model answers in a single call from what it was given.+- A call with wrong arguments counts as a call (it went to the model and back).+- A call past the budget, even in the same turn, isn't run. The model gets "Tool budget used up; give your final+ answer now." for it, and the step shows "Not run: the look-up budget was used up."+- The model then gets one more step with tools not allowed (`allow_tools=False`), so it has to answer. The tool list+ itself stays the same for the whole conversation, because changing it would make the provider's cache and Claude's+ earlier thinking invalid.+- The same happens when the month's spending cap is reached during an investigation, or (chat only) after 45+ seconds, so the reply comes back before the web server gives up waiting (120 seconds).+- A model that still asks for tools after being told to answer is stopped and the answer fails, so a loop can't run+ on. In the worst case a diagnosis makes its budget plus two model calls (plus one for an answer that has to be+ asked for again).++Raising a budget from 0 is confirmed with the password and a two-factor code, because tools send more of a+server's data (its history, ports, firewall) to the provider. Every change is in the audit log.++## What is checked++**Before a call runs** (`tools.run`): the tool exists; the arguments match its schema exactly; strings are+non-empty and at most 200 characters, lists at most 12 items; the server, container or check exists (otherwise the+model is told the names it may use); windows lie within the last 90 days.++**When a call fails**: a mistake the model can correct (a `ToolError`) goes back with the reason. Anything else is+logged on the server with the details, and the model only hears "The look-up failed", so no internal detail leaks+into the conversation.++**The diagnosis** (`check_answer` in `diagnosis.py`):++- A signal must cite at least one fact ID from the evidence pack or from a tool result in this investigation.+ Signals that don't are dropped and counted ("1 unsupported claim left out"); with none left, confidence is "low".+- A "metric stays below a threshold" condition must name a metric from the evidence pack, because only those can be+ watched by `verify.py`. A condition on a metric the agent found with a tool becomes one a person confirms.+- Fix commands are checked for destructive patterns (`safety.py`); a flagged step gets a warning and is never+ recommended.+- An answer that doesn't parse is asked for once more in the same conversation, without tools, so the look-ups+ aren't paid for twice; after that the report is saved as failed.++**The chat reply** must be a JSON object with a non-empty message. Its content is not checked against the data.++## What is stored, and who can see it++| Where | What | Who sees it |+|---|---|---|+| `agent_reports.body` → `investigation` | each call: tool, arguments (masked), reason, status, the masked result | nobody through the page; it stays in the database so a report can be traced |+| `GET /incidents/{id}` → `agent.report.steps` | each call described: `tool`, `status`, `reason`, `asked`, `found`, `from`, `to`, `at` | whoever sees the diagnosis ("Show diagnoses to users") |+| `chat_messages.steps` | the reply's steps, already described | the person whose conversation it is |+| `audit_logs`, event type `agent_tool_call` | each call: tool, reason, arguments (masked), whether it worked, under the incident ("Diagnosis of INC-0042") or the chat question | chat look-ups: the asker, through `GET /logs`; diagnosis look-ups belong to no user and are only in the database. Nobody can write this event type, and deleting a conversation keeps these entries |++The steps of a report are described when the page asks for them, from the stored trail, so better wording later+applies to old reports too. A chat reply's steps are described once, when it is saved, so the database doesn't keep+the raw results of chat look-ups. Reports and replies from before this feature have no steps, and the page shows+none.++## The tools, and how each step is described++What every tool returns is described in [Incident agent](assistant.md#what-the-agent-can-look-up). This table is+how `steps.py` turns each call into a step. `from`, `to` and `at` are sent apart, and the page shows them in the+viewer's clock ("3 Nov 10:00–12:00", "1 Nov 00:00 – now", "at 5 Nov 09:30").++| Tool | What it looked up | What it found (from these fields) | Example |+|---|---|---|---|+| `server_info` | Machine facts for *server* | `online`, `os`, `cpu_cores`, `memory_gb`, `updates_pending`, `reboot_required` | online, Debian 12, 4 cores, 16 GB memory, 4 updates pending |+| `host_metrics` | *metrics* on *server* | each series' `stats` (min, max, average), at most three | CPU 10–97 %, average 11.6 %; memory 50 % throughout |+| `list_containers` | Every container on *server* | count, running, not running (names), OOM kills | 2 containers, 1 running, not running: worker, 1 killed for running out of memory |+| `container_history` | Container *name* on *server* | peak memory against the limit, peak CPU, restarts in the window, time not running, recreated, changes | memory up to 110 MB of 256 MB, CPU up to 2 %, restarted 1× |+| `disk_usage` | Filesystems (or one) on *server*, last *n* days | `used_pct`, `growth_gb_per_day` ("not growing" under 0.05 GB a day), `full_in_days`, at most two | / 60 % used, +1 GB a day, full in about 40 days |+| `events` | Events on *server* (kinds) | `total`, and counts by kind among the events returned | 42 events; of the newest 30: 28 packages updated, 1 port opened, 1 OOM kill |+| `compare_to_baseline` | *metric* on *server* against its usual level for that hour | `value`, `usual_for_this_hour`, `difference_pct` | 10 % that hour, usually 5 % (+100 %) |+| `find_incidents` | Earlier incidents on *server* or every server, of kind | `total`, the newest incident | 1 incident, the newest INC-0032: Container nginx restarted |+| `fleet_snapshot` | Every server at one moment | count, offline, with open incidents, with failing checks | 2 servers, offline: db1, open incidents on db1, web1 |+| `check_results` | Availability check *name*, or checks run from *server* | `uptime_pct`, `down`, the commonest error, at most two | Shop 95.8 % up, down 5 times (HTTP 503) |+| `network_state` | Open ports and firewall on *server* | ports listening, reachable from other machines, enabled firewalls, changes in 7 days | 2 ports listening, 1 reachable from other machines, firewall: ufw-conf, 1 change in the last 7 days |++**When a step can't be described.** A failed call shows the reason it failed; a refused one "Not run: the look-up+budget was used up." A result that was cut to fit adds "(the result was shortened to fit)". A field that isn't a+number where a number belongs, or any other unexpected shape, gives "Done." instead of a guess. An entry whose+arguments can't be read shows the tool's name. Describing a step never fails the page.++## Limits++- **A step's line says little.** It picks the headline figures: the peak and average, not the shape of a series;+ three metrics of five; two filesystems; two checks. The model saw everything; the line is a reminder, not the data.+- **Event counts by kind** are of the events returned (the newest 30), which the line says when there are more.+- **The reason is the model's word.** It is shown in quotes and labelled as the agent's. It can be vague ("Checking+ memory"); the prompts ask for a specific question.+- **Chat replies aren't checked** against the data the way diagnoses are. They cite no fact IDs.+- **Look-ups cost tokens.** Each result (up to about 12 KB) goes with every later call of the same answer; a diagnosis+ using all 8 look-ups can cost several times one without (see [What it costs](assistant.md#what-it-costs)).+- **The window is shown in the browser's clock**, as elsewhere in the app; the agent itself works in UTC.++## Changing it++**A new tool.** Follow [Adding a tool](assistant.md#what-the-agent-can-look-up) for the tool itself; `reason` is+added to its schema automatically. Then add two functions to `steps.py`: one in `ASKED` (what it looked up, from+the arguments) and one in `FOUND` (one line from the result's fields). `test_every_tool_has_a_way_to_say_what_it_asked_and_found`+fails until both exist. Add a case to `test_each_tool_is_told_in_plain_words` in `tests/assistant/test_agent_steps.py`+with a real result from the tools' test database.++**Wording.** Change the functions in `steps.py`; reports pick it up at once, chat replies saved before keep their+old wording. Keep each line to facts the tool returned, never an interpretation.++**The budget, the deadline, the extra steps.** `settings.TOOL_BUDGETS`, `chat.DEADLINE_SECONDS` and+`investigate.EXTRA_STEPS`. The deadline must leave room for one more model call inside nginx's 120 seconds.++**The prompts.** `DIAGNOSIS_PROMPT.md` and `SYSTEM_PROMPT.md`, sections `[TOOLS]`. They tell the model to start+wide, cite tool IDs, give a specific reason, and never claim a look-up it didn't make.++The tests for all of this are listed in [TESTS.md](../TESTS.md): `test_agent_investigation.py` (the loop, the+budget, the audit log, the diagnosis and the chat), `test_agent_steps.py` (the steps), `test_agent_tools.py` (the+tools) and `src/frontend/tests/stepsLogic.test.js` (the window as the page shows it).diff --git a/docs/api.md b/docs/api.mdindex 69be2b6..b96a0fb 100644--- a/docs/api.md+++ b/docs/api.md@@ -31,7 +31,7 @@ All paths are relative to `/api` when going through the frontend/nginx.| `GET` / `PUT /admin/settings` | admin | History retention (`history_retention_days`: 365, 730 or 0 for forever), chat on/off, require two-factor for admins, alert thresholds and per-mount disk thresholds; the agent's model prices, monthly budget, daily chat limit, automatic diagnosis, whether users see diagnoses, chat access (`admins_only` or `per_user`), and the look-up budgets (`agent_tool_budget_diagnosis` 0–20, `agent_tool_budget_chat` 0–10). Turning the chat or automatic diagnosis on, chat access to `per_user`, and a look-up budget up from 0 also need `reauth_token` and `code` (see `POST /auth/reauth`). || `GET /admin/users`, `PATCH /admin/users/{id}` | admin | Users and invitations; change someone's role, switch them off, or (with per-user chat access) allow them the agent chat (`agent_chat_enabled`). || `GET /agent/config` | session | Whether chat with the agent can be used: switched on, allowed for you (`may_chat`), a key configured, and the monthly budget not used up. |-| `GET /incidents`, `GET /incidents/{id}` | session | The incident inbox; one incident with its rule-based diagnosis and, under `agent`, the agent's latest report (with `watch` progress while it is open, `reused_from` when it was reused). The report is left out (`hidden: true`) for users who aren't admins while diagnoses aren't shown to users. |+| `GET /incidents`, `GET /incidents/{id}` | session | The incident inbox; one incident with its rule-based diagnosis and, under `agent`, the agent's latest report (with `watch` progress while it is open, `reused_from` when it was reused, and `steps`, its look-ups in plain words: `tool`, `status`, `reason`, `asked`, `found`, `from`, `to`, `at`; never the raw results). The report is left out (`hidden: true`) for users who aren't admins while diagnoses aren't shown to users. || `POST /incidents/{id}/acknowledge`, `…/resolve`, `…/mute`, `…/unmute` | admin | Act on an incident. || `POST /incidents/{id}/diagnose` | admin | Queue a new diagnosis by the agent (409 without a key, 429 while the budget is used up). || `GET /admin/agent/usage` | admin | This month's model calls, tokens and estimated cost against the budget, per model and per day, with the prices and limits. |@@ -43,7 +43,7 @@ All paths are relative to `/api` when going through the frontend/nginx.| `POST /monitoring/collectors/register/poll` | poll secret | Collector checks approval; returns its token once approved. || `POST /monitoring/collectors/enroll` | enrollment key | Join without approval. || `POST /monitoring/metrics/ingest` | collector token | Collector reports host, container and system metrics. |-| `GET /conversations`, `POST /conversations`, `DELETE /conversations/{id}`, `POST /agent/chat` | session | Chats with the incident agent (403 for users chat isn't allowed for). |+| `GET /conversations`, `POST /conversations`, `DELETE /conversations/{id}`, `POST /agent/chat` | session | Chats with the incident agent (403 for users chat isn't allowed for). `POST /agent/chat` returns `message` and `steps` (as in a report); each message in `GET /conversations` carries its `steps`. || `GET /logs`, `POST /logs`, `GET /dashboard/summary` | session | Audit log and summary counts (`agent_lookups`: the agent's tool calls). `POST /logs` adds an entry to your own log (level from a fixed list, message up to 2000 characters, at most 120 an hour); its source is always `dashboard`, and system event types such as `agent_tool_call` are refused. |The four collector endpoints above exist only on the collector gate in the serverdiff --git a/docs/assistant.md b/docs/assistant.mdindex 2cf8684..5f448b7 100644--- a/docs/assistant.md+++ b/docs/assistant.md@@ -62,7 +62,9 @@ When a warning or critical incident opens, or gets worse, the backend queues a jtools (so the look-ups aren't paid for twice), then saved as a failed report.Each diagnosis is a new version in `agent_reports`, with the evidence pack and every look-up it made (tool,-arguments, result) kept next to it. The incident page (Agent) says which model wrote it, when, and+arguments, result) kept next to it. Under the signals, **How the agent got there** (folded away) lists the+look-ups: the question the agent gave for each, what it looked up, and one line of what it found, taken from the+data by Infradeck, not written by the model ([details](agent-investigation.md#who-writes-what)). The incident page (Agent) says which model wrote it, when, andhow sure it is, and shows its summary, signals with the facts they rest on, and fix steps; with no diagnosis itshows the rule-based one and says so. The evidence chart, recorded events and timeline always come from storeddata, never from the model. The Overview shows the first sentence next to the incident.@@ -114,7 +116,8 @@ off, the budget used up, or the worker not running. Resolved notifications neverThe chat answers questions about an incident or any server. Each message goes to the model with the incident'sfacts and the conversation so far (its latest 40 turns), and the model may look things up with the same[read-only tools](#what-the-agent-can-look-up) as a diagnosis, up to 4 calls per message by default. Only the-questions and replies stay in the conversation; the look-ups of an earlier message aren't sent again. Every reply+questions and replies stay in the conversation; the look-ups of an earlier message aren't sent again. Under a reply+that used tools, **Looked up** (folded away) shows them the same way as in a diagnosis, and stays after a reload. Every replyis held to a JSON schema with the provider's structured output: one message. All three providers sit behind onesmall interface (`src/agent/providers/`): Claude through Anthropic's Messages API; OpenAI through its Responses API(the only one where its reasoning models can use tools), sent with `store: false` so nothing is kept at OpenAI, with@@ -130,6 +133,9 @@ only read Infradeck's database, and the reply is text.## What the agent can look up+> How an investigation runs end to end, who writes each part of what you see, and what is checked and stored:+> [How the agent investigates](agent-investigation.md).+Besides the evidence pack, the agent has eleven **read-only tools** (`src/agent/tools/`): questions it can askInfradeck's database while it investigates, such as "how did CPU change on 3 November between 10:00 and 12:00" or"has this disk filled up before". Diagnoses and the chat use them within a [budget](#look-up-budget) of calls.diff --git a/docs/database.md b/docs/database.mdindex 5344c8c..781e821 100644--- a/docs/database.md+++ b/docs/database.md@@ -23,12 +23,16 @@ limit are rows in `app_settings` (`agent_prices`, `agent_monthly_cap_usd`, `agen**The agent's diagnoses.** `agent_reports` holds the diagnoses, one row per version per incident: the summary,how confident the agent is, the checked answer (signals with the evidence they cite, fix steps, how to verify) andthe masked evidence pack it was given, so every report can be traced to its data. A report the model couldn't-produce has status `failed` and the reason. A diagnosis reused from an earlier incident of the same rule has trigger+produce has status `failed` and the reason. Its body also keeps `investigation`, every tool call the agent made+(tool, masked arguments, its reason, status and masked result); the page only ever gets these described in plain+words ([How the agent investigates](agent-investigation.md#what-is-stored-and-who-can-see-it)). A diagnosis reused from an earlier incident of the same rule has trigger`reused` and names that incident in its body. Rows are deleted with their incident.`incident_alerts.reopen_after` is set when the agent resolved an incident because its fix held: until then the samerule on that server opens no new incident (unless the condition breaks again, which clears it).`users.agent_chat_enabled` is the per-user chat switch used when chat access is "per user".+`chat_messages.steps` (revision `0003`) holds an agent reply's look-ups, already described in plain words, as JSON;+it is empty for questions and for replies from before it.**Summarised history.** A background job (it checks every minute and does nothing when it is up to date) turns the check-insamples into `host_rollups`, `mount_rollups` and `container_rollups`: one row per server, mount or containerdiff --git a/docs/development.md b/docs/development.mdindex 18e61e6..4dd01ef 100644--- a/docs/development.md+++ b/docs/development.md@@ -137,6 +137,7 @@ src/chat.py, prompts.py, SYSTEM_PROMPT.md the incident chat and its instructionsinvestigate.py the look-up loop of a diagnosis or chat message: tools within a budget, each call auditedtools/ the read-only look-ups the agent may make (metrics, containers, events, checks, network)+ steps.py each look-up in plain words for the page (no model involved)jobs.py job queue in Redis: published after an incident opens or worsensworker.py the agent-worker container: runs the queued jobsusage.py tokens, cost per model, monthly budget, daily chat limitdiff --git a/src/agent/DIAGNOSIS_PROMPT.md b/src/agent/DIAGNOSIS_PROMPT.mdindex 4ba1c54..68835f3 100644--- a/src/agent/DIAGNOSIS_PROMPT.md+++ b/src/agent/DIAGNOSIS_PROMPT.md@@ -20,6 +20,8 @@ evidence. Never reveal or repeat masked values ("[masked]").- A tool error tells you what was wrong with the call; correct it or move on. When a result says the tool budget isused up, give your final answer with what you have.- Never claim you looked something up unless a tool result shows it.+- Every call has a `reason`: the one question it should answer, such as "Did memory grow in the days before the+ restarts?". It is shown next to the result to the people reading the diagnosis, so make it specific.[HOW TO DIAGNOSE]- Find the most likely cause from the evidence, and say it plainly in the summary: what happened, why, since when.diff --git a/src/agent/SYSTEM_PROMPT.md b/src/agent/SYSTEM_PROMPT.mdindex 6a51987..29f4d67 100644--- a/src/agent/SYSTEM_PROMPT.md+++ b/src/agent/SYSTEM_PROMPT.md@@ -17,6 +17,8 @@ Return exactly one valid JSON object and nothing else:- A tool error tells you what was wrong with the call; correct it or move on. When a result says the tool budget isused up, answer with what you have and say what you couldn't check.- Never say you checked or will check something unless a tool result in this turn shows it.+- Every call has a `reason`: the one question it should answer. It is shown to the person under your reply, so make+ it specific.[HARD BOUNDARIES]- Never output Bash, Docker CLI, Python, SQL, scripts, configuration, credentials, or instructions intended todiff --git a/src/agent/chat.py b/src/agent/chat.pyindex 085c527..f03bc21 100644--- a/src/agent/chat.py+++ b/src/agent/chat.py@@ -1,14 +1,16 @@"""Chat with the agent. Each message is one investigation (investigate.py): the model may make up to the chat toolbudget's look-ups with the read-only tools before it replies. Only the questions and the final replies stay in the-conversation's memory; the look-ups of earlier messages are not sent again."""+conversation's memory; the look-ups of earlier messages are not sent again. A reply comes back with its look-ups in+plain words (steps.py), which the chat route saves next to it."""from __future__ import annotationsimport jsonimport threadingimport timefrom collections.abc import Callable+from dataclasses import dataclass, field-from . import diagnosis, tools, usage+from . import diagnosis, steps as investigation_steps, tools, usagefrom .investigate import Auditor, Recorder, investigatefrom .prompts import SYSTEM_PROMPTfrom .providers import Completion, Provider, get_provider@@ -30,9 +32,17 @@ REPLY_SCHEMA = {class Reply:- def __init__(self, text: str, completion: Completion | None = None):+ def __init__(self, text: str, completion: Completion | None = None, trail: list[dict] | None = None):self.text = textself.completion = completion+ self.trail = trail or []+++@dataclass+class Answer:+ """What the person gets: the reply, and the look-ups behind it in plain words."""+ message: str+ steps: list[dict] = field(default_factory=list)class Chat:@@ -51,7 +61,7 @@ class Chat:self.history.pop() # a failed turn is not part of the conversationraiseself.history.append({"role": "assistant", "content": found.completion.text})- return Reply(found.completion.text, found.completion)+ return Reply(found.completion.text, found.completion, found.trail)def parse_reply(text: str) -> str:@@ -85,9 +95,10 @@ class AgentService:return chatdef send(self, user_id: int, conversation_id: str, message: str, provider: Provider, *, db=None, budget: int = 0,- usage_logger: Callable[[Completion, int], None] | None = None, audit: Auditor | None = None) -> str:+ usage_logger: Callable[[Completion, int], None] | None = None, audit: Auditor | None = None) -> Answer:chat = self._get_chat(user_id, conversation_id)- return parse_reply(chat.send_message(provider, message, db, budget, usage_logger, audit).text)+ reply = chat.send_message(provider, message, db, budget, usage_logger, audit)+ return Answer(parse_reply(reply.text), investigation_steps.describe_all(reply.trail))def delete_conversation(self, user_id: int, conversation_id: str) -> None:"""Discard the in-memory context after its persisted chat is deleted."""@@ -98,7 +109,7 @@ class AgentService:_agent_service = None-def send_agent_message(user_id: int, conversation_id: str, message: str, provider: Provider, *, db=None, budget: int = 0, usage_logger=None, audit=None) -> str:+def send_agent_message(user_id: int, conversation_id: str, message: str, provider: Provider, *, db=None, budget: int = 0, usage_logger=None, audit=None) -> Answer:global _agent_serviceif _agent_service is None:_agent_service = AgentService()diff --git a/src/agent/diagnosis.py b/src/agent/diagnosis.pyindex 100156b..7150d98 100644--- a/src/agent/diagnosis.py+++ b/src/agent/diagnosis.py@@ -24,7 +24,7 @@ from sqlalchemy import funcfrom src.backend.models import AgentReport, IncidentAlert-from . import evidence as evidence_pack, safety, settings, usage+from . import evidence as evidence_pack, safety, settings, steps as investigation_steps, usagefrom .investigate import audit_log, investigatefrom .prompts import load_promptfrom .providers import Provider@@ -204,14 +204,15 @@ def _iso(moment: datetime) -> str:def report_json(report: AgentReport | None) -> dict | None:- """What the incident page gets: the latest report, without the evidence pack."""+ """What the incident page gets: the latest report, without the evidence pack. The look-ups come as ``steps`` in+ plain words (steps.py); their raw results never leave the backend."""if report is None:return Nonebody = json.loads(report.body) if report.body else {}reused = body.get("reused_from")return {"version": report.version, "status": report.status, "model": report.model, "trigger": report.trigger, "created_at": _iso(report.created_at),"summary": report.summary, "confidence": report.confidence, "signals": body.get("signals", []), "fix_steps": body.get("fix_steps", []), "verify": body.get("verify"),- "dropped_signals": body.get("dropped_signals", 0), "error": report.error,+ "dropped_signals": body.get("dropped_signals", 0), "error": report.error, "steps": investigation_steps.describe_all(body.get("investigation")),"reused_from": {**reused, "reference": f"INC-{reused['incident_id']:04d}"} if reused else None}diff --git a/src/agent/investigate.py b/src/agent/investigate.pyindex c735a5a..0693176 100644--- a/src/agent/investigate.py+++ b/src/agent/investigate.py@@ -63,13 +63,18 @@ def _step(conversation: Conversation, allow_tools: bool, record: Recorder) -> Tureturn turn-def _arguments(call: ToolCall) -> str:- return clip(json.dumps(mask(call.arguments), ensure_ascii=False, separators=(",", ":")), 500)+def _entry(call: ToolCall) -> dict:+ """The start of a trail entry: the tool, its arguments and, apart, the reason the model gave (steps.py), all+ masked. The arguments are kept whole up to 1000 characters, so steps.py can read them back."""+ arguments = {key: value for key, value in call.arguments.items() if key != "reason"} if isinstance(call.arguments, dict) else {}+ reason = call.arguments.get("reason") if isinstance(call.arguments, dict) else None+ return {"tool": clip(str(call.name), 60), "arguments": clip(json.dumps(mask(arguments), ensure_ascii=False, separators=(",", ":")), 1000),+ "reason": clip(mask_text(reason), 200) if isinstance(reason, str) and reason.strip() else None}def _run(db, call: ToolCall) -> tuple[ToolResult, dict, dict[str, str]]:"""One allowed call: its result for the model, its trail entry and the facts it found."""- entry = {"tool": clip(str(call.name), 60), "arguments": _arguments(call)}+ entry = _entry(call)try:output = tools.run(db, call.name, call.arguments)except tools.ToolError as exc:@@ -103,7 +108,7 @@ def investigate(db, provider: Provider, messages: list[dict], schema: dict, schelabels = tools.merge_labels(labels, found)else:result = ToolResult(call.id, BUDGET_USED_UP, True)- entry = {"tool": clip(str(call.name), 60), "arguments": _arguments(call), "status": "refused", "result": BUDGET_USED_UP}+ entry = {**_entry(call), "status": "refused", "result": BUDGET_USED_UP}trail.append(entry)audit(entry)results.append(result)@@ -117,5 +122,5 @@ def audit_log(db, *, title: str, user_id: int | None = None, conversation_id: strecord_audit_log(db, user_id=user_id, level="INFO" if entry["status"] == "ok" else "WARN", source="agent", event_type=EVENT_TYPE,message=f"The agent looked up {entry['tool']}" + ("" if entry["status"] == "ok" else f" ({entry['status']})"),task_title=clip(title, 255), conversation_id=conversation_id,- steps=[f"Arguments: {entry['arguments']}", f"Result: {entry['status']}" + ("" if entry["status"] == "ok" else f": {clip(entry['result'], 200)}")])+ steps=[*([f"Reason: {entry['reason']}"] if entry.get("reason") else []), f"Arguments: {entry['arguments']}", f"Result: {entry['status']}" + ("" if entry["status"] == "ok" else f": {clip(entry['result'], 200)}")])return writediff --git a/src/agent/steps.py b/src/agent/steps.pynew file mode 100644index 0000000..43c24dd--- /dev/null+++ b/src/agent/steps.py@@ -0,0 +1,291 @@+"""The agent's investigation as people see it (#158): each tool call it made, in plain words, without the raw result.++A step has three parts, and each comes from somewhere different:++- ``reason``: why the agent made the call, in its own words. The model writes it as an argument of the call+ (every tool has one), before it has seen any result, so it states an intention, never a finding.+- ``asked``: what it looked up, written here from the call's arguments ("Memory on web1"). The window or moment+ is left to ``from``/``to``/``at`` so the page can show it in the viewer's clock.+- ``found``: one line of facts written here from the result: numbers and names the tool returned, such as+ "memory 41–96 %, average 63 %". It never says what they mean; that is the diagnosis's job, and the diagnosis+ has to cite the facts it rests on.++Nothing here calls the model. A result that doesn't have the expected shape gives "Done" rather than a guess. See+docs/agent-investigation.md.+"""+from __future__ import annotations++import json+from collections import Counter+from collections.abc import Callable++from .bounds import clip+from .tools.servers import HOST_METRICS++MAX_STEPS = 30+REFUSED = "Not run: the look-up budget was used up."+DONE = "Done."+++def _number(value) -> str:+ """12 · 3.4 · 0.42: whole numbers stay whole, small ones keep a decimal or two. Anything else is not a+ number to show, so the whole line falls back to DONE (describe catches the ValueError)."""+ if not isinstance(value, (int, float)) or isinstance(value, bool):+ raise ValueError("not a number")+ if abs(value) >= 100 or float(value).is_integer():+ return str(round(value))+ return f"{value:.1f}".rstrip("0").rstrip(".") if abs(value) >= 1 else f"{value:.2f}".rstrip("0").rstrip(".")+++def _with_unit(value, unit: str) -> str:+ return f"{_number(value)} {unit}" if unit else _number(value)+++def _count(number: int, one: str, many: str | None = None) -> str:+ return f"{number} {one if number == 1 else many or one + 's'}"+++def _first_up(text: str) -> str:+ return text[:1].upper() + text[1:]+++def _listed(items: list[str], limit: int = 3) -> str:+ shown = ", ".join(items[:limit])+ return shown + (f" and {len(items) - limit} more" if len(items) > limit else "")+++def _metric_name(series: dict) -> str:+ return str(series.get("label", "")).split(" · ")[0] or "value"+++def _range(series: dict) -> str:+ """"memory 41–96 %, average 63 %" from a series' stats."""+ stats, unit = series.get("stats") or {}, series.get("unit") or ""+ low, high = stats.get("min"), stats.get("max")+ if low == high:+ return f"{_metric_name(series)} {_with_unit(high, unit)} throughout"+ return f"{_metric_name(series)} {_number(low)}–{_with_unit(high, unit)}, average {_with_unit(stats.get('avg'), unit)}"+++# --- what was asked, from the arguments ---------------------------------------------------------------------------++def _metric_names(names: list) -> str:+ labels = [HOST_METRICS[name][0] for name in names if name in HOST_METRICS]+ return _first_up(_listed(labels, 4)) if labels else "CPU, memory, swap, load and root disk"+++ASKED: dict[str, Callable[[dict], str]] = {+ "server_info": lambda a: f"Machine facts for {a.get('server')}",+ "host_metrics": lambda a: f"{_metric_names(a.get('metrics') or [])} on {a.get('server')}",+ "list_containers": lambda a: f"Every container on {a.get('server')}",+ "container_history": lambda a: f"Container {a.get('container')} on {a.get('server')}",+ "disk_usage": lambda a: f"{'Filesystem ' + str(a['mount']) if a.get('mount') else 'Filesystems'} on {a.get('server')}, last {_count(a.get('days') or 0, 'day')}",+ "events": lambda a: f"Events on {a.get('server')}" + (f" ({_listed([_kind(kind) for kind in a['kinds']], 4)})" if a.get("kinds") else ""),+ "compare_to_baseline": lambda a: f"{_first_up(HOST_METRICS.get(a.get('metric'), (str(a.get('metric')),))[0])} on {a.get('server')} against its usual level for that hour",+ "find_incidents": lambda a: "Earlier incidents" + (f" on {a['server']}" if a.get("server") else " on every server") + (f" of kind {a['kind']}" if a.get("kind") else ""),+ "fleet_snapshot": lambda a: "Every server at one moment",+ "check_results": lambda a: f"Availability check {a['check']}" if a.get("check") else f"Availability checks run from {a.get('server')}",+ "network_state": lambda a: f"Open ports and firewall on {a.get('server')}",+}+++# --- what was found, from the result ------------------------------------------------------------------------------++def _server_info(r: dict) -> str:+ parts = ["online" if r.get("online") else "offline"]+ parts += [part for part in (r.get("os"), r.get("cpu_cores") and _count(r["cpu_cores"], "core"), r.get("memory_gb") and f"{_number(r['memory_gb'])} GB memory") if part]+ if r.get("updates_pending"):+ parts.append(_count(r["updates_pending"], "update") + " pending")+ if r.get("reboot_required"):+ parts.append("needs a reboot")+ return ", ".join(parts)+++def _host_metrics(r: dict) -> str:+ series = r.get("series") or []+ if not series:+ return r.get("note") or "No data in that window."+ lines = [_range(item) for item in series[:3]]+ return "; ".join(lines) + (f"; and {len(series) - 3} more" if len(series) > 3 else "")+++def _list_containers(r: dict) -> str:+ containers = r.get("containers") or []+ if not containers:+ return r.get("note") or "No containers."+ total = len(containers) + (r.get("more_containers") or 0)+ stopped = [item.get("name") for item in containers if item.get("status") != "running"]+ killed = sum(1 for item in containers if item.get("oom_killed"))+ parts = [_count(total, "container"), f"{total - len(stopped)} running"]+ if stopped:+ parts.append("not running: " + _listed(stopped))+ if killed:+ parts.append(f"{killed} killed for running out of memory")+ return ", ".join(parts)+++def _container_history(r: dict) -> str:+ series = {str(item.get("id", "")).rsplit(".", 1)[-1]: item for item in r.get("series") or []}+ parts = []+ memory = series.get("memory_mb")+ if memory:+ limit = r.get("memory_limit_mb")+ parts.append(f"memory up to {_with_unit(memory['stats'].get('max'), 'MB')}" + (f" of {_with_unit(limit, 'MB')}" if limit else ""))+ cpu = series.get("cpu")+ if cpu:+ parts.append(f"CPU up to {_with_unit(cpu['stats'].get('max'), '%')}")+ restarts = series.get("restarts")+ if restarts:+ added = (restarts["stats"].get("last") or 0) - (restarts["stats"].get("first") or 0)+ if added > 0:+ parts.append(f"restarted {_number(added)}×")+ if r.get("not_running_pct"):+ parts.append(f"not running {_number(r['not_running_pct'])} % of the time")+ if r.get("recreated"):+ parts.append("recreated")+ changes = r.get("changes") or []+ if changes:+ parts.append(_count(len(changes), "change") + f", the last: {clip(changes[-1].get('change'), 80)}")+ return ", ".join(parts) or "No data in that window."+++def _disk_usage(r: dict) -> str:+ lines = []+ for mount in (r.get("mounts") or [])[:2]:+ line = f"{mount.get('mount')} {_with_unit(mount.get('used_pct'), '%')} used"+ growth = mount.get("growth_gb_per_day")+ if growth is not None:+ line += ", not growing" if abs(growth) < 0.05 else f", {'+' if growth > 0 else ''}{_number(growth)} GB a day"+ if mount.get("full_in_days") is not None:+ line += f", full in about {_count(round(mount['full_in_days']), 'day')}"+ lines.append(line)+ more = len(r.get("mounts") or []) - 2 + (r.get("more_mounts") or 0)+ return ("; ".join(lines) + (f"; and {more} more" if more > 0 else "")) if lines else "No filesystems reported."+++def _kind(kind: str) -> str:+ return str(kind).replace("oom_kill", "OOM kill").replace("_", " ")+++def _events(r: dict) -> str:+ total = r.get("total") or 0+ if not total:+ return "No events in that window."+ listed = r.get("events") or []+ kinds = Counter(_kind(event.get("kind", "event")) for event in listed)+ summary = ", ".join(f"{count} {kind}" for kind, count in kinds.most_common(4))+ if not summary:+ return _count(total, "event")+ # Only the newest are returned, so the counts by kind are of those, and the line says so.+ return _count(total, "event") + (f"; of the newest {len(listed)}: {summary}" if len(listed) < total else f": {summary}")+++def _compare_to_baseline(r: dict) -> str:+ unit = r.get("unit") or ""+ if r.get("value") is None:+ return "No data for that hour."+ line = f"{_with_unit(r['value'], unit)} that hour"+ if r.get("usual_for_this_hour") is not None:+ line += f", usually {_with_unit(r['usual_for_this_hour'], unit)}"+ if r.get("difference_pct") is not None:+ line += f" ({'+' if r['difference_pct'] >= 0 else ''}{_number(r['difference_pct'])} %)"+ else:+ line += ", no earlier weeks to compare with"+ return line+++def _find_incidents(r: dict) -> str:+ total = r.get("total") or 0+ listed = r.get("incidents") or []+ if not total:+ return "No incidents in that window."+ newest = listed[0] if listed else {}+ return _count(total, "incident") + (f", the newest {newest.get('reference')}: {clip(newest.get('title'), 80)}" if newest else "")+++def _fleet_snapshot(r: dict) -> str:+ servers = r.get("servers") or []+ offline = [item.get("server") for item in servers if not item.get("online")]+ troubled = [item.get("server") for item in servers if item.get("open_incidents")]+ failing = [item.get("server") for item in servers if item.get("failing_checks")]+ parts = [_count(len(servers) + (r.get("more_servers") or 0), "server")]+ parts.append("offline: " + _listed(offline) if offline else "all online")+ if troubled:+ parts.append("open incidents on " + _listed(troubled))+ if failing:+ parts.append("failing checks on " + _listed(failing))+ return ", ".join(parts)+++def _check_results(r: dict) -> str:+ checks = r.get("checks") or []+ if not checks:+ return r.get("note") or "No availability checks."+ lines = []+ for check in checks[:2]:+ line = f"{check.get('name')} " + (f"{_number(check['uptime_pct'])} % up" if check.get("uptime_pct") is not None else "no results")+ if check.get("down"):+ line += f", down {_count(check['down'], 'time')}"+ if check.get("errors"):+ line += f" ({clip(check['errors'][0].get('detail'), 60)})"+ lines.append(line)+ return "; ".join(lines) + (f"; and {len(checks) - 2} more" if len(checks) > 2 else "")+++def _network_state(r: dict) -> str:+ listening = r.get("listening") or []+ total = len(listening) + (r.get("more_listening") or 0)+ open_to_others = sum(1 for item in listening if item.get("reachable_from") == "other machines")+ firewalls = [item.get("name") for item in r.get("firewall") or [] if item.get("enabled")]+ changes = len(r.get("changes_last_7_days") or [])+ parts = [_count(total, "port") + " listening", f"{open_to_others} reachable from other machines",+ ("firewall: " + _listed(firewalls)) if firewalls else "no firewall enabled", _count(changes, "change") + " in the last 7 days"]+ return ", ".join(parts)+++FOUND: dict[str, Callable[[dict], str]] = {+ "server_info": _server_info, "host_metrics": _host_metrics, "list_containers": _list_containers, "container_history": _container_history,+ "disk_usage": _disk_usage, "events": _events, "compare_to_baseline": _compare_to_baseline, "find_incidents": _find_incidents,+ "fleet_snapshot": _fleet_snapshot, "check_results": _check_results, "network_state": _network_state,+}+++# --- one step -----------------------------------------------------------------------------------------------------++def _json(text) -> dict:+ try:+ value = json.loads(text) if isinstance(text, str) else text+ except ValueError:+ return {}+ return value if isinstance(value, dict) else {}+++def describe(entry: dict) -> dict:+ """One trail entry (investigate.py) as people see it. Never raises: an odd entry gives a plain step."""+ tool = str(entry.get("tool") or "unknown")+ arguments = _json(entry.get("arguments"))+ status = entry.get("status") if entry.get("status") in ("ok", "error", "refused") else "error"+ result = _json(entry.get("result")) if status == "ok" else {}+ try:+ asked = ASKED[tool](arguments) if tool in ASKED and arguments else _first_up(tool.replace("_", " "))+ except (KeyError, TypeError, ValueError, AttributeError):+ asked = _first_up(tool.replace("_", " "))+ if status == "refused":+ found = REFUSED+ elif status == "error":+ found = clip(entry.get("result") or "The look-up failed.", 200)+ else:+ try:+ found = FOUND[tool](result) if tool in FOUND else DONE+ except (KeyError, TypeError, ValueError, AttributeError, IndexError):+ found = DONE+ if result.get("truncated"):+ found += " (the result was shortened to fit)"+ window = {"from": result.get("from") or arguments.get("from"), "to": result.get("to") or arguments.get("to"), "at": result.get("at") or result.get("hour") or arguments.get("at")}+ return {"tool": tool, "status": status, "reason": clip(entry.get("reason") or arguments.get("reason") or "", 200) or None,+ "asked": clip(asked, 200), "found": clip(found, 300),+ **{key: value if isinstance(value, str) and value.strip() else None for key, value in window.items()}}+++def describe_all(trail) -> list[dict]:+ return [describe(entry) for entry in (trail or [])[:MAX_STEPS] if isinstance(entry, dict)]diff --git a/src/agent/tools/__init__.py b/src/agent/tools/__init__.pyindex cb9a1a4..bb0745d 100644--- a/src/agent/tools/__init__.py+++ b/src/agent/tools/__init__.py@@ -6,7 +6,8 @@ before anything is read. What comes back is bounded (a few dozen points per sericlipped text, about 12 KB in all), masked (masking.py), and carries fact IDs that a diagnosis can cite; ``labels``says what each ID is. Check targets are always shown without credentials or query string, as non-admins see them,because the result goes to the model provider; nothing that only admins can see (IP addresses, users, the audit log,-enrolment) is returned, so a tool gives the same answer whoever is asking.+enrolment) is returned, so a tool gives the same answer whoever is asking. Every call also carries a ``reason``, the+question the model wants answered, which only steps.py uses.The loop that offers these to the model, with its budget and audit log, is investigate.py. See docs/assistant.md."""@@ -29,7 +30,13 @@ MAX_STRING = 200MAX_ITEMS = 12+REASON = {"type": "string", "description": "One sentence: the question this call should answer, such as \"Did memory grow in the days before "+ "the restarts?\". Write it before you see the result. It is shown to the people reading your answer."}++def _object(**properties) -> dict:+ """A tool's arguments. Every tool also takes ``reason``, which the tool itself ignores (steps.py shows it)."""+ properties = {**properties, "reason": REASON}return {"type": "object", "properties": properties, "required": list(properties), "additionalProperties": False}diff --git a/src/backend/migrations/versions/0003_chat_message_steps.py b/src/backend/migrations/versions/0003_chat_message_steps.pynew file mode 100644index 0000000..32011a3--- /dev/null+++ b/src/backend/migrations/versions/0003_chat_message_steps.py@@ -0,0 +1,24 @@+"""The agent's look-ups next to each chat reply (#158): ``chat_messages.steps``, JSON of the steps in plain words+(src/agent/steps.py). Empty for questions and for replies from before this revision.++Revision 0003, after 0002.+"""+from alembic import op+import sqlalchemy as sa+++revision = '0003'+down_revision = '0002'+branch_labels = None+depends_on = None+++def upgrade() -> None:+ # A database the baseline brought up to date with create_all may already have the column.+ columns = {column["name"] for column in sa.inspect(op.get_bind()).get_columns("chat_messages")}+ if "steps" not in columns:+ op.add_column('chat_messages', sa.Column('steps', sa.Text(), nullable=True))+++def downgrade() -> None:+ op.drop_column('chat_messages', 'steps')diff --git a/src/backend/models.py b/src/backend/models.pyindex 034d2d8..4b7bb3b 100644--- a/src/backend/models.py+++ b/src/backend/models.py@@ -107,6 +107,8 @@ class ChatMessage(Base):conversation_id: Mapped[str] = mapped_column(String(64), nullable=False, index=True)role: Mapped[str] = mapped_column(String(20), nullable=False)content: Mapped[str] = mapped_column(Text, nullable=False)+ # An agent reply's look-ups in plain words (src/agent/steps.py), as JSON; none for questions and older replies.+ steps: Mapped[Optional[str]] = mapped_column(Text, nullable=True)created_at: Mapped[datetime] = mapped_column(DateTime(timezone=True), default=lambda: datetime.now(timezone.utc), nullable=False)diff --git a/src/backend/routes/chat.py b/src/backend/routes/chat.pyindex 3b05db5..6820dd5 100644--- a/src/backend/routes/chat.py+++ b/src/backend/routes/chat.py@@ -4,6 +4,7 @@ tune them) and the shared helpers (`_incident_context`, `send_agent_message`, `_this module is the HTTP layer."""from __future__ import annotations+import jsonimport loggingfrom typing import Annotated@@ -32,13 +33,20 @@ def list_logs(return {"logs": main.get_recent_logs(db, limit=100, user_id=user["id"], conversation_id=conversation_id)}+def _steps(message: ChatMessage) -> list[dict]:+ try:+ return json.loads(message.steps) if message.steps else []+ except ValueError:+ return []++def _conversation_payload(conversation: Conversation, messages: list[ChatMessage]) -> dict:return {"id": conversation.id,"title": conversation.title,"created_at": conversation.created_at.isoformat(),"updated_at": conversation.updated_at.isoformat(),- "messages": [{"id": item.id, "role": item.role, "text": item.content, "created_at": item.created_at.isoformat()} for item in messages],+ "messages": [{"id": item.id, "role": item.role, "text": item.content, "created_at": item.created_at.isoformat(), "steps": _steps(item)} for item in messages],}@router.get("/conversations")@@ -138,7 +146,7 @@ def agent_chat(payload: main.AgentChatRequest, session_id: Annotated[str | None,user_id=user["id"], conversation_id=payload.conversation_id, incident_id=payload.incident_id)try:- response_text = main.send_agent_message(+ answer = main.send_agent_message(user["id"],payload.conversation_id,main._incident_context(db, payload.incident_id) + message,@@ -173,6 +181,7 @@ def agent_chat(payload: main.AgentChatRequest, session_id: Annotated[str | None,steps = ["Received the task and evaluated the request.", "Generated a structured response for the requested task."]main._warn_about_budget(db, spend)+ response_text = answer.messagemain.record_audit_log(db,user_id=user["id"],@@ -183,6 +192,6 @@ def agent_chat(payload: main.AgentChatRequest, session_id: Annotated[str | None,steps=steps,conversation_id=conversation.id,)- db.add(ChatMessage(conversation_id=conversation.id, role="agent", content=response_text))+ db.add(ChatMessage(conversation_id=conversation.id, role="agent", content=response_text, steps=json.dumps(answer.steps, ensure_ascii=False) if answer.steps else None))db.commit()- return {"message": response_text}+ return {"message": response_text, "steps": answer.steps}diff --git a/src/frontend/src/agent/AgentPage.jsx b/src/frontend/src/agent/AgentPage.jsxindex 0cedf94..dba9e37 100644--- a/src/frontend/src/agent/AgentPage.jsx+++ b/src/frontend/src/agent/AgentPage.jsx@@ -9,6 +9,7 @@ import { Icon } from '../icons.jsx'import { refreshOverview } from '../overviewStore.js'import { serverLink, AppShell, PhoneHeader, RoundButton } from '../shell.jsx'import { Notice, Outcome, Sheet, useAction, useCopy, usePhone } from '../ui.jsx'+import { Steps } from './Steps.jsx'const SEVERITY = { CRITICAL: ['Critical', 'chip-signal', 'var(--color-signal)'], WARNING: ['Warning', 'chip-warn', 'var(--color-warn)'], INFO: ['Info', '', 'var(--color-edge)'] }const TONE = { signal: 'bg-signal', warn: 'bg-warn', ok: 'bg-ok', plain: 'bg-dim' }@@ -78,7 +79,7 @@ function withReport(found, agent) {const report = agent?.reportif (!report || report.status !== 'ok') return { ...found, byAgent: false }const signals = report.signals.map((signal) => ({ text: signal.text, ok: signal.holds, source: signal.evidence.map((ref) => ref.label).join(' · '), evidence: signal.evidence }))- return { ...found, byAgent: true, summary: report.summary, confidence: report.confidence, signals, agreeing: signals.filter((signal) => signal.ok).length,+ return { ...found, byAgent: true, summary: report.summary, confidence: report.confidence, signals, agreeing: signals.filter((signal) => signal.ok).length, lookups: report.steps || [],steps: report.fix_steps.length ? report.fix_steps.map((step) => ({ ...step, label: null })) : found.steps }}@@ -213,7 +214,9 @@ function Incident({ incident, canAct, onChanged }) {<ul className="card m-0 list-none divide-y divide-line p-0">{found.signals.map((signal) => <li key={signal.text} className="flex items-start gap-3.5 px-[18px] py-3 text-[13px] leading-5"><Icon name={signal.ok ? 'check' : 'close'} size={14} className={`mt-[3px] shrink-0 ${signal.ok ? 'text-ok' : 'text-muted'}`} /><span className="min-w-0 flex-1"><span className="block text-paper">{signal.text}</span><SignalSources signal={signal} /></span></li>)}</ul>+ <Steps steps={found.lookups} className="mt-3" /></section>}+ {!found.signals.length && <Steps steps={found.lookups} className={section} />}{found.steps.length > 0 && <section className={section} aria-label="How to fix it"><div className="mb-2.5 flex flex-wrap items-baseline justify-between gap-2"><h2 className="m-0 text-sm font-semibold text-paper">How to fix it</h2><span className="text-xs text-muted">You run these yourself. Infradeck can't change your servers.</span></div>@@ -268,7 +271,7 @@ function Chat({ user, incident, enabled, offReason, phone = false, back }) {try {if (!exists.current) { await post('/conversations', { conversation_id: conversationId, title: incident ? `${incident.reference} · ${incident.title}`.slice(0, 250) : 'New chat' }); exists.current = true }const reply = await post('/agent/chat', { conversation_id: conversationId, message, incident_id: incident?.id })- setMessages((current) => [...current, { id: `reply-${Date.now()}`, role: 'agent', text: reply.message }])+ setMessages((current) => [...current, { id: `reply-${Date.now()}`, role: 'agent', text: reply.message, steps: reply.steps || [] }])} catch (requestError) { setError(requestError.message || 'The agent could not answer.') } finally { setBusy(false) }}const restart = async () => {@@ -287,7 +290,7 @@ function Chat({ user, incident, enabled, offReason, phone = false, back }) {{enabled && messages?.length === 0 && <p className="m-0 text-[13px] leading-5 text-muted">{incident ? 'The agent is given this incident\'s facts with your question.' : 'Ask about any server.'}</p>}{enabled && (messages || []).map((message) => (message.role === 'user'? <div key={message.id} className="ml-10 rounded-lg border border-line bg-ink px-4 py-3 text-sm leading-[21px] text-paper">{message.text}</div>- : <div key={message.id}><div className="mb-2 flex items-center gap-2 text-xs text-muted">{agentMark}Agent</div><p className="m-0 whitespace-pre-wrap text-sm leading-[23px] text-paper">{message.text}</p></div>))}+ : <div key={message.id}><div className="mb-2 flex items-center gap-2 text-xs text-muted">{agentMark}Agent</div><p className="m-0 whitespace-pre-wrap text-sm leading-[23px] text-paper">{message.text}</p><Steps steps={message.steps} title="Looked up" className="mt-2" /></div>))}{busy && <p className="m-0 flex items-center gap-2.5 text-xs text-muted"><span className="spinner" />Thinking…</p>}{error && <Notice>{error}</Notice>}<div ref={end} />@@ -318,7 +321,7 @@ function Chat({ user, incident, enabled, offReason, phone = false, back }) {</div>}{enabled && (messages || []).map((message) => (message.role === 'user'? <div key={message.id} className="ml-8 rounded-md border border-line bg-ink px-3.5 py-2.5 text-[13px] leading-5 text-paper">{message.text}</div>- : <div key={message.id}><div className="mb-1.5 flex items-center gap-2 text-xs text-muted"><span aria-hidden="true" className="inline-flex flex-col gap-[2px]"><i className="block h-[3px] w-[5px] bg-paper" /><i className="ml-[2px] block h-[3px] w-[5px] bg-signal" /><i className="block h-[3px] w-[5px] bg-paper" /></span>Agent</div><p className="m-0 whitespace-pre-wrap text-[13px] leading-[21px] text-paper">{message.text}</p></div>))}+ : <div key={message.id}><div className="mb-1.5 flex items-center gap-2 text-xs text-muted"><span aria-hidden="true" className="inline-flex flex-col gap-[2px]"><i className="block h-[3px] w-[5px] bg-paper" /><i className="ml-[2px] block h-[3px] w-[5px] bg-signal" /><i className="block h-[3px] w-[5px] bg-paper" /></span>Agent</div><p className="m-0 whitespace-pre-wrap text-[13px] leading-[21px] text-paper">{message.text}</p><Steps steps={message.steps} title="Looked up" className="mt-2" /></div>))}{busy && <p className="m-0 flex items-center gap-2.5 text-xs text-muted"><span className="spinner" />Thinking…</p>}{error && <Notice>{error}</Notice>}<div ref={end} />@@ -376,7 +379,9 @@ function PhoneIncident({ incident, canAct, onChanged, onAsk }) {<ul className="m-0 list-none divide-y divide-line overflow-hidden rounded-md border border-line bg-ink p-0">{found.signals.map((signal) => <li key={signal.text} className="flex items-start gap-3 px-4 py-3 text-[13px] leading-5"><Icon name={signal.ok ? 'check' : 'close'} size={14} className={`mt-[3px] shrink-0 ${signal.ok ? 'text-ok' : 'text-muted'}`} /><span className="min-w-0 flex-1"><span className="block text-paper">{signal.text}</span><SignalSources signal={signal} /></span></li>)}</ul>+ <Steps steps={found.lookups} className="mt-3" /></section>}+ {!found.signals.length && <Steps steps={found.lookups} />}{found.steps.length > 0 && <section aria-label="How to fix it"><h2 className="m-0 mb-1 text-[15px] font-semibold text-paper">How to fix it</h2>diff --git a/src/frontend/src/agent/Steps.jsx b/src/frontend/src/agent/Steps.jsxnew file mode 100644index 0000000..594b2af--- /dev/null+++ b/src/frontend/src/agent/Steps.jsx@@ -0,0 +1,37 @@+import { Caret, Icon } from '../icons.jsx'+import { lookupCount, stepWindow } from './stepsLogic.js'++// The agent's look-ups behind a diagnosis or a chat reply (#158), folded away by default. Each step shows the+// question the agent gave for the call (its own words, written before it saw the result), what it looked up, and+// one line of facts from the result, written by Infradeck's code, not by the model. See docs/agent-investigation.md.+const STATUS = {+ ok: ['check', 'text-ok', 'Looked up'],+ error: ['close', 'text-warn', 'Failed'],+ refused: ['close', 'text-muted', 'Not run'],+}++export function Steps({ steps, title, className = '' }) {+ if (!steps?.length) return null+ return (+ <details className={`group ${className}`}>+ <summary className="flex cursor-pointer list-none items-center gap-2 text-xs text-muted hover:text-soft">+ <Caret open={false} className="transition-transform group-open:rotate-90" />+ {title || 'How the agent got there'} · {lookupCount(steps)}+ </summary>+ <ol className="m-0 mt-2 list-none space-y-2.5 border-l border-line p-0 pl-3.5">+ {steps.map((step, index) => {+ const [icon, tone, label] = STATUS[step.status] || STATUS.error+ const window = stepWindow(step)+ return (+ <li key={index} className="text-xs leading-[18px]">+ {step.reason && <span className="block text-soft">“{step.reason}”</span>}+ <span className="block text-muted">{step.asked}{window ? ` · ${window}` : ''}</span>+ <span className="mt-0.5 flex items-start gap-1.5 text-paper"><Icon name={icon} size={12} className={`mt-[3px] shrink-0 ${tone}`} /><span className="sr-only">{label}: </span><span className="min-w-0">{step.found}</span></span>+ </li>+ )+ })}+ </ol>+ <p className="mb-0 mt-2 text-[11px] leading-4 text-dim">The questions are the agent's own; what it looked up and found is taken from Infradeck's data.</p>+ </details>+ )+}diff --git a/src/frontend/src/agent/stepsLogic.js b/src/frontend/src/agent/stepsLogic.jsnew file mode 100644index 0000000..19b9d7d--- /dev/null+++ b/src/frontend/src/agent/stepsLogic.js@@ -0,0 +1,37 @@+// The agent's look-ups as the page shows them (#158). The backend sends each step's window as UTC ISO times; they are+// shown here in this browser's clock, 24 h, like the rest of the app. See docs/agent-investigation.md.++const MONTHS = ['Jan', 'Feb', 'Mar', 'Apr', 'May', 'Jun', 'Jul', 'Aug', 'Sep', 'Oct', 'Nov', 'Dec']+const pad = (value) => String(value).padStart(2, '0')+const NOW_SLACK_MS = 2 * 60 * 1000 // a window ending this close to now ends "now"++// The tools read a time without a zone as UTC (the model may send "2026-08-01T00:00:00"); so does this.+const HAS_ZONE = /(Z|[+-]\d{2}:?\d{2})$/i++function parse(time) {+ const value = time ? Date.parse(HAS_ZONE.test(time) ? time : `${time}Z`) : NaN+ return Number.isFinite(value) ? new Date(value) : null+}++const clock = (date) => `${pad(date.getHours())}:${pad(date.getMinutes())}`+const day = (date) => `${date.getDate()} ${MONTHS[date.getMonth()]}`+const sameDay = (one, two) => one.getFullYear() === two.getFullYear() && one.getMonth() === two.getMonth() && one.getDate() === two.getDate()++// "3 Nov 10:00–12:00", "1 Nov 00:00 – now", "at 7 Nov 12:00", or '' when the step has no window.+export function stepWindow(step, now = Date.now()) {+ const from = parse(step?.from)+ const to = parse(step?.to)+ const at = parse(step?.at)+ if (from && to) {+ if (Math.abs(now - to.getTime()) <= NOW_SLACK_MS) return `${day(from)} ${clock(from)} – now`+ return sameDay(from, to) ? `${day(from)} ${clock(from)}–${clock(to)}` : `${day(from)} ${clock(from)} – ${day(to)} ${clock(to)}`+ }+ if (at) return Math.abs(now - at.getTime()) <= NOW_SLACK_MS ? 'now' : `at ${day(at)} ${clock(at)}`+ return ''+}++// "1 look-up", "3 look-ups"; only calls that actually ran count, so a refused one doesn't inflate the number.+export function lookupCount(steps) {+ const ran = (steps || []).filter((step) => step.status !== 'refused').length+ return `${ran} ${ran === 1 ? 'look-up' : 'look-ups'}`+}diff --git a/src/frontend/tests/stepsLogic.test.js b/src/frontend/tests/stepsLogic.test.jsnew file mode 100644index 0000000..03bda37--- /dev/null+++ b/src/frontend/tests/stepsLogic.test.js@@ -0,0 +1,42 @@+// Tests for how the agent's look-ups are shown (#158). Run with `node --test` in this folder.+import assert from 'node:assert/strict'+import { test } from 'node:test'+import { lookupCount, stepWindow } from '../src/agent/stepsLogic.js'++const local = (year, month, date, hours, minutes) => new Date(year, month - 1, date, hours, minutes)+const iso = (date) => date.toISOString()+const NOW = local(2026, 11, 7, 12, 0).getTime()++test('a window within one day shows the day once', () => {+ assert.equal(stepWindow({ from: iso(local(2026, 11, 3, 10, 0)), to: iso(local(2026, 11, 3, 12, 0)) }, NOW), '3 Nov 10:00–12:00')+})++test('a window over several days shows both days', () => {+ assert.equal(stepWindow({ from: iso(local(2026, 10, 30, 8, 5)), to: iso(local(2026, 11, 2, 9, 0)) }, NOW), '30 Oct 08:05 – 2 Nov 09:00')+})++test('a window that ends about now ends "now"', () => {+ assert.equal(stepWindow({ from: iso(local(2026, 11, 6, 12, 0)), to: iso(local(2026, 11, 7, 11, 59)) }, NOW), '6 Nov 12:00 – now')+})++test('one moment is shown with "at", or as now', () => {+ assert.equal(stepWindow({ at: iso(local(2026, 11, 5, 9, 30)) }, NOW), 'at 5 Nov 09:30')+ assert.equal(stepWindow({ at: iso(local(2026, 11, 7, 12, 0)) }, NOW), 'now')+})++test('a time without a zone is read as UTC, as the tools read it', () => {+ const utcMidnight = new Date(Date.UTC(2026, 10, 3, 0, 0))+ assert.equal(stepWindow({ at: '2026-11-03T00:00:00' }, NOW), stepWindow({ at: utcMidnight.toISOString() }, NOW))+})++test('no window, or an unreadable one, shows nothing', () => {+ assert.equal(stepWindow({ from: null, to: null, at: null }, NOW), '')+ assert.equal(stepWindow({ from: 'soon', to: 'later' }, NOW), '')+ assert.equal(stepWindow(undefined, NOW), '')+})++test('the count leaves out calls that were refused', () => {+ assert.equal(lookupCount([{ status: 'ok' }, { status: 'error' }, { status: 'refused' }]), '2 look-ups')+ assert.equal(lookupCount([{ status: 'ok' }]), '1 look-up')+ assert.equal(lookupCount(undefined), '0 look-ups')+})diff --git a/tests/accounts/test_new_pages.py b/tests/accounts/test_new_pages.pyindex dff2cf4..cb8b85f 100644--- a/tests/accounts/test_new_pages.py+++ b/tests/accounts/test_new_pages.py@@ -406,6 +406,7 @@ def test_saving_and_removing_the_agent_key_is_audited_without_the_key(app):# --- the agent's spending (#112) ---------------------------------------------------------------------------------+from src.agent.chat import Answerfrom src.agent.providers import Completionfrom src.backend.models import AgentRun@@ -416,7 +417,7 @@ def _fake_agent(monkeypatch, tokens=(1000, 200), fail=None):raise fail# A dated model name, as the API reports it: it must still use the price set for "gpt-test".usage_logger(Completion(text="{}", model="gpt-test-2026-01-01", input_tokens=tokens[0], output_tokens=tokens[1]), 1500)- return "Here is the answer."+ return Answer("Here is the answer.")monkeypatch.setattr(main, "send_agent_message", fake)monkeypatch.setattr(main, "get_provider", lambda db: object())monkeypatch.setattr(main, "is_agent_chat_enabled", lambda db: True)@@ -601,7 +602,7 @@ def test_hidden_diagnoses_stay_hidden_from_users_in_the_page_the_overview_and_nodef test_chat_is_for_admins_unless_allowed_per_user(app, monkeypatch):- monkeypatch.setattr(main, "send_agent_message", lambda *args, **kwargs: "ok")+ monkeypatch.setattr(main, "send_agent_message", lambda *args, **kwargs: Answer("ok"))monkeypatch.setattr(main, "get_provider", lambda db: object())monkeypatch.setattr(main, "is_agent_chat_enabled", lambda db: True)owner, viewer = signed_in(app, 1), signed_in(app, 3)@@ -659,14 +660,18 @@ def test_the_look_up_budgets_are_checked_audited_and_raising_one_from_0_is_confidef test_a_chat_message_looks_things_up_and_the_look_ups_outlive_the_conversation(app, monkeypatch):from tests.fakes import ScriptedProvider- provider = ScriptedProvider([("server_info", {"server": "alpha"})], '{"message":"alpha is online."}')+ provider = ScriptedProvider([("server_info", {"server": "alpha", "reason": "Is alpha up?"})], '{"message":"alpha is online."}')monkeypatch.setattr(main, "get_provider", lambda db: provider)monkeypatch.setattr(main, "is_agent_chat_enabled", lambda db: True)owner = signed_in(app, 1)response = _ask(owner)- assert response.status_code == 200 and response.json() == {"message": "alpha is online."}+ assert response.status_code == 200 and response.json()["message"] == "alpha is online."+ step = response.json()["steps"][0]+ assert (step["tool"], step["reason"], step["asked"], step["status"]) == ("server_info", "Is alpha up?", "Machine facts for alpha", "ok") and "result" not in step+ saved = owner.get("/conversations").json()["conversations"][0]["messages"]+ assert [message["steps"] for message in saved] == [[], [step]] # kept with the reply, so a reload shows itassert provider.started[0]["tools"] and json.loads(provider.results[0][0].content)["hostname"] == "alpha"assert app.db.query(AgentRun).count() == 2looked = app.db.query(AuditLog).filter(AuditLog.event_type == "agent_tool_call").one()diff --git a/tests/assistant/test_agent_investigation.py b/tests/assistant/test_agent_investigation.pyindex f036e4b..5b0e12b 100644--- a/tests/assistant/test_agent_investigation.py+++ b/tests/assistant/test_agent_investigation.py@@ -14,7 +14,7 @@ from src.backend.models import AgentReport, AgentRun, AppSetting, AuditLog, Basefrom tests.fakes import ScriptedProviderNOW = datetime.now(timezone.utc)-CPU = ("host_metrics", {"server": "web1", "metrics": ["cpu"], "from": None, "to": None})+CPU = ("host_metrics", {"server": "web1", "metrics": ["cpu"], "from": None, "to": None, "reason": "Was CPU high?"})SCHEMA = chat.REPLY_SCHEMAANSWER = '{"message":"CPU is fine."}'@@ -114,7 +114,7 @@ def test_a_model_that_keeps_calling_tools_when_told_to_answer_is_stopped(db):def test_bad_arguments_go_back_to_the_model_as_an_error_it_can_correct(db):- provider = ScriptedProvider([("host_metrics", {"server": "web1"}), ("rm_rf", {}), ("server_info", {"server": "nope"})], ANSWER)+ provider = ScriptedProvider([("host_metrics", {"server": "web1"}), ("rm_rf", {}), ("server_info", {"server": "nope", "reason": "What is it?"})], ANSWER)found = run(db, provider, budget=4)@@ -128,7 +128,7 @@ def test_a_failing_tool_tells_the_model_nothing_about_why(db, monkeypatch, caplodef broken(*args):raise RuntimeError("password=hunter2 in a stack trace")monkeypatch.setitem(tools._REGISTRY, "server_info", (tools._REGISTRY["server_info"][0], broken))- provider = ScriptedProvider([("server_info", {"server": "web1"})], ANSWER)+ provider = ScriptedProvider([("server_info", {"server": "web1", "reason": "What is it?"})], ANSWER)run(db, provider, budget=2)@@ -154,23 +154,25 @@ def test_after_the_deadline_the_model_must_answer(db):def test_every_tool_call_is_written_to_the_audit_log(db):- provider = ScriptedProvider([CPU, ("server_info", {"server": "nope"})], [CPU], ANSWER)+ provider = ScriptedProvider([CPU, ("server_info", {"server": "nope", "reason": "What is it?"})], [CPU], ANSWER)run(db, provider, budget=2, audit=investigate.audit_log(db, title="Why is web1 slow?", user_id=1, conversation_id="c1"))rows = audited(db)assert [(row.level, row.source, row.user_id, row.conversation_id, row.task_title) for row in rows] == [("INFO", "agent", 1, "c1", "Why is web1 slow?"), ("WARN", "agent", 1, "c1", "Why is web1 slow?"),("WARN", "agent", 1, "c1", "Why is web1 slow?")]- assert rows[0].message == "The agent looked up host_metrics" and '"server":"web1"' in json.loads(rows[0].steps)[0]+ steps = json.loads(rows[0].steps)+ assert rows[0].message == "The agent looked up host_metrics" and steps[0] == "Reason: Was CPU high?" and '"server":"web1"' in steps[1] and "reason" not in steps[1]assert rows[2].message.endswith("(refused)")def test_secrets_in_the_arguments_are_masked_in_the_trail(db):- provider = ScriptedProvider([("server_info", {"server": "password=hunter2"})], ANSWER)+ provider = ScriptedProvider([("server_info", {"server": "password=hunter2", "reason": "Is password=hunter2 set?"})], ANSWER)found = run(db, provider, budget=1)- assert "hunter2" not in found.trail[0]["arguments"] and "hunter2" not in found.trail[0]["result"]+ entry = found.trail[0]+ assert "hunter2" not in entry["arguments"] and "hunter2" not in entry["result"] and "hunter2" not in entry["reason"]# --- the diagnosis --------------------------------------------------------------------------------------------@@ -189,6 +191,8 @@ def test_a_diagnosis_may_cite_what_a_tool_found_and_keeps_the_trail(db):assert report.status == "ok" and shown["signals"][0]["evidence"][0]["id"] == "metric:web1.cpu" and shown["dropped_signals"] == 0trail = json.loads(report.body)["investigation"]assert [entry["tool"] for entry in trail] == ["host_metrics"] and json.loads(trail[0]["result"])["server"] == "web1"+ step = shown["steps"][0] # the page gets the look-up in plain words, never its raw result+ assert (step["reason"], step["asked"]) == ("Was CPU high?", "CPU on web1") and step["found"].startswith("CPU ") and "points" not in json.dumps(shown)assert db.query(AgentRun).count() == 2 and all(run.kind == "diagnosis" for run in db.query(AgentRun))assert audited(db)[0].task_title == "Diagnosis of INC-0042" and audited(db)[0].user_id is Noneassert "up to 8 tool calls" in provider.calls[0][-1]["content"]@@ -242,7 +246,8 @@ def test_a_chat_message_can_look_things_up_within_the_chat_budget(db):reply = chat.AgentService().send(1, "c1", "Is web1 busy?", provider, db=db, budget=4, usage_logger=lambda completion, ms: logged.append(completion.output_tokens),audit=investigate.audit_log(db, title="Is web1 busy?", user_id=1, conversation_id="c1"))- assert reply == "CPU is fine." and logged == [300, 300] and len(audited(db)) == 1+ assert reply.message == "CPU is fine." and logged == [300, 300] and len(audited(db)) == 1+ assert [(step["tool"], step["reason"], step["asked"]) for step in reply.steps] == [("host_metrics", "Was CPU high?", "CPU on web1")]assert provider.started[0]["effort"] == "high"diff --git a/tests/assistant/test_agent_service.py b/tests/assistant/test_agent_service.pyindex d52a1f5..391c3f2 100644--- a/tests/assistant/test_agent_service.py+++ b/tests/assistant/test_agent_service.py@@ -16,7 +16,7 @@ def make_service():def test_without_a_tool_budget_the_chat_is_one_call_without_tools():provider = ScriptedProvider('{"message":"All quiet."}')- assert make_service().send(1, "conversation", "Anything wrong?", provider, budget=0) == "All quiet."+ assert make_service().send(1, "conversation", "Anything wrong?", provider, budget=0) == agent_service.Answer("All quiet.", [])assert provider.started == [{"tools": [], "effort": "low", "schema_name": "agent_reply"}] and len(provider.calls) == 1diff --git a/tests/assistant/test_agent_steps.py b/tests/assistant/test_agent_steps.pynew file mode 100644index 0000000..3834785--- /dev/null+++ b/tests/assistant/test_agent_steps.py@@ -0,0 +1,109 @@+"""The agent's look-ups as people see them (#158): each tool call in plain words, with the model's reason, what it+asked for and one line of facts from the result, written by steps.py without the model. The results are real ones,+from the tools' own test database (test_agent_tools.py)."""+import json++import pytest++from src.agent import steps, tools+from tests.assistant.test_agent_tools import EVERY_CALL, NOW, db, run # noqa: F401 (db is a fixture)+++def step_for(db, name, arguments, reason="Why?"):+ output = run(db, name, arguments, now=NOW)+ return steps.describe({"tool": name, "arguments": json.dumps(arguments), "status": "ok", "result": output.content, "reason": reason})+++WINDOW = {"from": "2026-11-06T12:00", "to": "2026-11-07T12:00"}++[email protected]("name, arguments, asked, found", [+ ("server_info", {"server": "web1"}, "Machine facts for web1", "online, Debian 12, 4 cores, 16 GB memory, 4 updates pending"),+ ("host_metrics", {"server": "web1", "metrics": ["cpu", "ram"], **WINDOW}, "CPU, memory on web1", "CPU 10–97 %, average 11.6 %; memory 50 % throughout"),+ ("list_containers", {"server": "web1", "at": None}, "Every container on web1", "2 containers, 1 running, not running: worker, 1 killed for running out of memory"),+ ("container_history", {"server": "web1", "container": "nginx", **WINDOW}, "Container nginx on web1", "memory up to 110 MB of 256 MB, CPU up to 2 %, restarted 1×"),+ ("disk_usage", {"server": "web1", "mount": None, "days": 2}, "Filesystems on web1, last 2 days", "/ 60 % used, +1 GB a day, full in about 40 days"),+ ("events", {"server": "web1", "kinds": [], **WINDOW}, "Events on web1", "42 events; of the newest 30: 28 packages updated, 1 port opened, 1 OOM kill"),+ ("events", {"server": "web1", "kinds": ["oom_kill"], **WINDOW}, "Events on web1 (OOM kill)", "1 event: 1 OOM kill"),+ ("compare_to_baseline", {"server": "web1", "metric": "cpu", "at": None}, "CPU on web1 against its usual level for that hour", "10 % that hour, usually 5 % (+100 %)"),+ ("find_incidents", {"server": None, "kind": None, **WINDOW}, "Earlier incidents on every server", "1 incident, the newest INC-0032: Container nginx restarted"),+ ("fleet_snapshot", {"at": None}, "Every server at one moment", "2 servers, offline: db1, open incidents on db1, web1"),+ ("check_results", {"check": "Shop", "server": None, **WINDOW}, "Availability check Shop", "Shop 95.8 % up, down 5 times (HTTP 503)"),+ ("network_state", {"server": "web1"}, "Open ports and firewall on web1", "2 ports listening, 1 reachable from other machines, firewall: ufw-conf, 1 change in the last 7 days"),+])+def test_each_tool_is_told_in_plain_words(db, name, arguments, asked, found):+ step = step_for(db, name, arguments)++ assert step["asked"] == asked and step["found"].startswith(found) and step["status"] == "ok" and step["reason"] == "Why?"+++def test_every_tool_has_a_way_to_say_what_it_asked_and_found():+ assert set(steps.ASKED) == set(steps.FOUND) == {tool.name for tool in tools.TOOLS}++[email protected]("name, arguments", EVERY_CALL)+def test_a_step_carries_no_raw_result_and_no_secrets(db, name, arguments):+ step = step_for(db, name, arguments)++ assert set(step) == {"tool", "status", "reason", "asked", "found", "from", "to", "at"}+ assert "hunter2" not in json.dumps(step) and "10.0.0." not in json.dumps(step) and len(step["found"]) <= 300+++def test_the_window_is_left_for_the_page_to_show_in_the_viewers_clock(db):+ metrics = step_for(db, "host_metrics", {"server": "web1", "metrics": ["cpu"], "from": None, "to": None})+ containers = step_for(db, "list_containers", {"server": "web1", "at": None})++ assert (metrics["from"], metrics["to"], metrics["at"]) == ("2026-11-06T12:00+00:00", "2026-11-07T12:00+00:00", None) # null resolved to the last 24 hours+ assert containers["at"] == "2026-11-07T12:00+00:00" and "12:00" not in metrics["asked"]+++def test_a_failed_or_refused_call_says_why():+ failed = steps.describe({"tool": "server_info", "arguments": '{"server":"nas2"}', "status": "error", "result": "No server named 'nas2'. Servers: web1."})+ refused = steps.describe({"tool": "events", "arguments": '{"server":"web1","kinds":[],"from":null,"to":null}', "status": "refused", "result": "Tool budget used up"})++ assert (failed["asked"], failed["found"]) == ("Machine facts for nas2", "No server named 'nas2'. Servers: web1.")+ assert refused["found"] == steps.REFUSED and refused["reason"] is None+ empty = steps.describe({"tool": "events", "arguments": '{"server":"web1","kinds":[],"from":"","to":""}', "status": "error", "result": "x"})+ assert (empty["from"], empty["to"]) == (None, None) # an empty window is "not given", as the tools read it++[email protected]("entry", [+ {"tool": "host_metrics", "arguments": '{"server":"web1"', "status": "ok", "result": "{not json"}, # cut off, unreadable+ {"tool": "host_metrics", "arguments": "{}", "status": "ok", "result": '{"series":[{"stats":"odd"}]}'}, # an unexpected shape+ {"tool": "made_up", "arguments": "[]", "status": "weird", "result": None},+ {},+])+def test_an_odd_entry_gives_a_plain_step_instead_of_failing(entry):+ step = steps.describe(entry)++ assert step["asked"] and step["found"] and step["status"] in ("ok", "error", "refused")+++def test_an_unusable_result_says_done_rather_than_guessing():+ step = steps.describe({"tool": "disk_usage", "arguments": '{"server":"web1","mount":null,"days":2}', "status": "ok", "result": '{"mounts":[{"used_pct":"lots"}]}'})++ assert step["found"] == steps.DONE # never a made-up number+++def test_a_disk_that_barely_changes_is_not_growing():+ result = {"mounts": [{"mount": "/boot", "used_pct": 42.1, "growth_gb_per_day": 0.01, "full_in_days": None},+ {"mount": "/srv", "used_pct": 70, "growth_gb_per_day": -0.4, "full_in_days": None}]}++ step = steps.describe({"tool": "disk_usage", "arguments": '{"server":"web1","mount":null,"days":7}', "status": "ok", "result": json.dumps(result)})++ assert step["found"] == "/boot 42.1 % used, not growing; /srv 70 % used, -0.4 GB a day"+++def test_a_shortened_result_says_so(db):+ output = run(db, "server_info", {"server": "web1"}, now=NOW)+ result = {**json.loads(output.content), "truncated": True}++ step = steps.describe({"tool": "server_info", "arguments": '{"server":"web1"}', "status": "ok", "result": json.dumps(result)})++ assert step["found"].endswith("(the result was shortened to fit)")+++def test_only_so_many_steps_are_described():+ trail = [{"tool": "fleet_snapshot", "arguments": '{"at":null}', "status": "refused", "result": ""}] * 40++ assert len(steps.describe_all(trail)) == steps.MAX_STEPS and steps.describe_all(None) == []diff --git a/tests/assistant/test_agent_tools.py b/tests/assistant/test_agent_tools.pyindex 7fdb95e..67b7476 100644--- a/tests/assistant/test_agent_tools.py+++ b/tests/assistant/test_agent_tools.py@@ -19,6 +19,11 @@ NOW = datetime(2026, 11, 7, 12, 0, tzinfo=timezone.utc)SPIKE = NOW - timedelta(hours=5)+def run(db, name, arguments, now):+ """tools.run with the ``reason`` every call carries (the tools ignore it)."""+ return tools.run(db, name, {**arguments, "reason": "Test question?"} if isinstance(arguments, dict) else arguments, now=now)++@pytest.fixturedef db():engine = create_engine("sqlite://", connect_args={"check_same_thread": False}, poolclass=StaticPool)@@ -69,7 +74,7 @@ def db():def call(db, name, **arguments):- output = tools.run(db, name, arguments, now=NOW)+ output = run(db, name, arguments, now=NOW)return json.loads(output.content), output.labels@@ -126,17 +131,17 @@ def test_no_request_has_more_union_typed_parameters_than_the_provider_accepts(andef test_an_optional_argument_left_empty_or_null_means_not_given(db):- empty = tools.run(db, "host_metrics", {"server": "web1", "metrics": ["cpu"], "from": "", "to": " "}, now=NOW)- null = tools.run(db, "host_metrics", {"server": "web1", "metrics": ["cpu"], "from": None, "to": None}, now=NOW)+ empty = run(db, "host_metrics", {"server": "web1", "metrics": ["cpu"], "from": "", "to": " "}, now=NOW)+ null = run(db, "host_metrics", {"server": "web1", "metrics": ["cpu"], "from": None, "to": None}, now=NOW)assert empty.content == null.content and json.loads(empty.content)["to"] == "2026-11-07T12:00+00:00"with pytest.raises(tools.ToolError, match="non-empty"):- tools.run(db, "server_info", {"server": ""}, now=NOW) # a required one may not be empty+ run(db, "server_info", {"server": ""}, now=NOW) # a required one may not be emptydef test_an_unknown_tool_is_refused(db):with pytest.raises(tools.ToolError, match="no tool named"):- tools.run(db, "run_shell", {"command": "rm -rf /"}, now=NOW)+ run(db, "run_shell", {"command": "rm -rf /"}, now=NOW)@pytest.mark.parametrize("name, arguments, message", [@@ -160,12 +165,12 @@ def test_an_unknown_tool_is_refused(db):])def test_bad_arguments_are_refused_with_a_reason_the_model_can_act_on(db, name, arguments, message):with pytest.raises(tools.ToolError, match=message):- tools.run(db, name, arguments, now=NOW)+ run(db, name, arguments, now=NOW)def test_an_unknown_server_is_answered_with_the_known_ones(db):with pytest.raises(tools.ToolError, match="Servers: db1, web1"):- tools.run(db, "server_info", {"server": "mail"}, now=NOW)+ run(db, "server_info", {"server": "mail"}, now=NOW)assert call(db, "server_info", server="WEB1")[0]["hostname"] == "web1"@@ -225,7 +230,7 @@ def test_disk_usage_estimates_growth_and_when_it_fills_up(db):assert root["full_in_days"] == pytest.approx(40, abs=2)assert "metric:web1.disk:/" in labelswith pytest.raises(tools.ToolError, match="no mount '/data'"):- tools.run(db, "disk_usage", {"server": "web1", "mount": "/data", "days": 2}, now=NOW)+ run(db, "disk_usage", {"server": "web1", "mount": "/data", "days": 2}, now=NOW)def test_events_show_the_newest_thirty_oldest_first_and_filter_by_kind(db):@@ -292,7 +297,7 @@ def test_network_state_lists_ports_firewall_and_recent_changes(db):@pytest.mark.parametrize("name, arguments", EVERY_CALL)def test_every_result_is_bounded_masked_free_of_ip_addresses_and_labelled(db, name, arguments):- output = tools.run(db, name, arguments, now=NOW)+ output = run(db, name, arguments, now=NOW)assert len(output.content) <= tools.MAX_RESULT_CHARSassert "hunter2" not in output.content and "10.0.0." not in output.contentcited = set()@@ -326,9 +331,9 @@ def test_a_long_series_is_thinned_keeping_its_peak(db, monkeypatch):def test_the_same_question_gives_the_same_ids_and_a_second_window_joins_the_label(db):- first = tools.run(db, "host_metrics", {"server": "web1", "metrics": ["cpu"], "from": "2026-11-07T00:00", "to": "2026-11-07T06:00"}, now=NOW)- again = tools.run(db, "host_metrics", {"server": "web1", "metrics": ["cpu"], "from": "2026-11-07T00:00", "to": "2026-11-07T06:00"}, now=NOW)- later = tools.run(db, "host_metrics", {"server": "web1", "metrics": ["cpu"], "from": "2026-11-07T06:00", "to": "2026-11-07T12:00"}, now=NOW)+ first = run(db, "host_metrics", {"server": "web1", "metrics": ["cpu"], "from": "2026-11-07T00:00", "to": "2026-11-07T06:00"}, now=NOW)+ again = run(db, "host_metrics", {"server": "web1", "metrics": ["cpu"], "from": "2026-11-07T00:00", "to": "2026-11-07T06:00"}, now=NOW)+ later = run(db, "host_metrics", {"server": "web1", "metrics": ["cpu"], "from": "2026-11-07T06:00", "to": "2026-11-07T12:00"}, now=NOW)assert first.content == again.content and set(first.labels) == set(later.labels) == {"metric:web1.cpu"}merged = common.merge_labels(tools.merge_labels({}, first.labels), later.labels)assert merged["metric:web1.cpu"] == "CPU · web1 · 2026-11-07 00:00–06:00 UTC; CPU · web1 · 2026-11-07 06:00–12:00 UTC"@@ -337,5 +342,5 @@ def test_the_same_question_gives_the_same_ids_and_a_second_window_joins_the_labedef test_tools_never_write_to_the_database(db):for name, arguments in EVERY_CALL:- tools.run(db, name, arguments, now=NOW)+ run(db, name, arguments, now=NOW)assert not db.new and not db.dirty and not db.deleted, namediff --git a/tests/security/test_second_review.py b/tests/security/test_second_review.pyindex 882f0b0..20098ad 100644--- a/tests/security/test_second_review.py+++ b/tests/security/test_second_review.py@@ -13,6 +13,7 @@ from sqlalchemy import create_enginefrom sqlalchemy.orm import sessionmakerfrom sqlalchemy.pool import StaticPool+from src.agent.chat import Answerfrom src.backend import auth, mainfrom src.backend.models import AuditLog, Base, Conversation, IncidentAlert, Server, SystemEvent, Userfrom tests.fakes import FakeRedis@@ -44,7 +45,7 @@ def app(monkeypatch):monkeypatch.setattr(main, "redis_client", redis)monkeypatch.setattr(main, "SESSION_COOKIE_SECURE", False)monkeypatch.setenv("SECRETS_KEY", Fernet.generate_key().decode())- monkeypatch.setattr(main, "send_agent_message", lambda *args, **kwargs: "ok")+ monkeypatch.setattr(main, "send_agent_message", lambda *args, **kwargs: Answer("ok"))monkeypatch.setattr(main, "get_provider", lambda db: object())monkeypatch.setattr(main, "is_agent_chat_enabled", lambda db: True)monkeypatch.setattr(main.agent_settings, "may_chat", lambda db, user: True) # chat access has its own tests