Self-hosted monitoring for homelabs: host and container metrics and uptime checks from all your machines in one dashboard, with an optional AI agent that diagnoses incidents and suggests fixes.
Python ai-agentsaiopsdashboarddockerfastapihomelabincident-managementllmmonitoringopenaireactroot-cause-analysisself-hosteduptime-monitoring
fix(agent): mask the chat question and incident context before they reach the provider (#214)
Karlo Kajba Šimanić99ac4884878f
diff --git a/SECURITY.md b/SECURITY.mdindex 36b2858..50b2435 100644--- a/SECURITY.md+++ b/SECURITY.md@@ -70,7 +70,9 @@ rule. An admin can also switch a rule off entirely in Admin › Alert rules, whi**The chat about an incident** sends your question together with that incident's title, descriptionand the facts Infradeck worked out (signals and recorded events, which contain container and host-names) to the model provider. Turn Chat off in Admin › Incident agent if that must not happen.+names) to the model provider. Before it leaves, the whole message is masked for secrets like the+agent's evidence pack (below; `src/agent/masking.py`). Turn Chat off in Admin › Incident agent if that+must not happen.**The agent's diagnosis** (when an API key is saved in Admin › Incident agent) sends an evidence pack to the model provider: theincident, the server's name, OS, cores and memory, metrics around the incident, recorded events, image changes,diff --git a/TESTS.md b/TESTS.mdindex b25fc49..d26d598 100644--- a/TESTS.md+++ b/TESTS.md@@ -147,7 +147,7 @@ The chart helper tests for the frontend are separate, in `src/frontend/tests/`.| [`test_account_security.py`](#test_account_securitypy) | 146 | Two-factor sign-in, recovery codes, encrypted secrets, sessions you can review and end, password change, and the password check before turning two-factor on. || [`test_admin.py`](#test_adminpy) | 18 | Who may do what: admin-only actions, roles, retention cleanup, the chat on/off switch. || [`test_agent_errors.py`](#test_agent_errorspy) | 6 | Helpful, non-leaking error messages from the AI assistant: a plain sentence in the chat, the provider's own words only for admins. |-| [`test_agent_service.py`](#test_agent_servicepy) | 10 | The chat's reply is one message; Test connection trying a real request; the Chat Completions provider and conversation history. |+| [`test_agent_service.py`](#test_agent_servicepy) | 11 | The chat's reply is one message; secrets in the question and incident context are masked; Test connection trying a real request; the Chat Completions provider and conversation history. || [`test_agent_providers.py`](#test_agent_providerspy) | 12 | The Claude provider, and how the chosen provider, model and encrypted API key are stored. || [`test_agent_tool_calls.py`](#test_agent_tool_callspy) | 30 | Tool calling with Claude, OpenAI (Responses API) and Gemini: calls, several at once, errors, masking, append-only history, forcing an answer, reasoning carried over, nothing stored at OpenAI. || [`test_agent_diagnosis.py`](#test_agent_diagnosispy) | 42 | The agent's diagnosis: the evidence pack, secret masking, dangerous commands, checking the model's answer, saving reports. |@@ -324,6 +324,7 @@ shouldn't, or an admin might be locked out.| Test | What it checks ||---|---|| `test_without_a_tool_budget_the_chat_is_one_call_without_tools` | With the chat's look-up budget at 0, a message is one call to the model, with no tools and effort `low`. |+| `test_secrets_in_the_question_and_incident_context_never_reach_the_provider` | Tokens, keys and passwords in a chat question and in the incident context in front of it are masked before the provider sees them, in the first turn and when the history goes back with later turns; the rest of the text stays readable (#207). || `test_a_reply_without_a_message_is_an_error` | A reply whose message is missing or empty is an error, not a blank answer. || `test_test_connection_tries_one_request_shaped_like_a_diagnosis_and_counts_it` | Test connection checks the key, then sends one request exactly as a diagnosis does (every tool, calls allowed, the diagnosis schema), counts it as a `test` call, and raises the provider's refusal (#196, #199). || `test_the_reply_can_only_be_a_message` | The reply schema has one field, the message, and nothing else is allowed. |diff --git a/docs/agent-investigation.md b/docs/agent-investigation.mdindex 70eeca8..f9952ba 100644--- a/docs/agent-investigation.md+++ b/docs/agent-investigation.md@@ -130,8 +130,9 @@ Nothing reaches the browser that the dashboard doesn't already show: the raw reswithout credentials or query string.**Masking.** Every text that goes to the provider is masked for tokens, keys, passwords and credentials in URLs-(`masking.py`): the evidence pack, each tool result (twice: by the tool and by the provider interface), each-tool error, and each call's arguments and reason as they are stored. The reason is the model's own text, but it+(`masking.py`): the evidence pack, each chat question with the incident context in front of it (`chat.py`, once,+before it joins the conversation's history), each tool result (twice: by the tool and by the provider interface),+each tool error, and each call's arguments and reason as they are stored. The reason is the model's own text, but itcan repeat something it saw, so it is masked too.## The budget, and when the model has to answerdiff --git a/src/agent/chat.py b/src/agent/chat.pyindex 8752233..60165c5 100644--- a/src/agent/chat.py+++ b/src/agent/chat.py@@ -12,6 +12,7 @@ from dataclasses import dataclass, fieldfrom . import diagnosis, steps as investigation_steps, tools, usagefrom .investigate import Auditor, Recorder, investigate+from .masking import mask_textfrom .prompts import SYSTEM_PROMPTfrom .providers import Completion, Provider, get_provider@@ -52,7 +53,9 @@ class Chat:self.history: list[dict] = []def send_message(self, provider: Provider, message: str, db=None, budget: int = 0, record: Recorder | None = None, audit: Auditor | None = None) -> Reply:- self.history.append({"role": "user", "content": message})+ # The question and the incident context in front of it (event text from the monitored machines) go to the+ # provider with every later turn too, so they are masked once, here, before they join the history.+ self.history.append({"role": "user", "content": mask_text(message)})messages = [{"role": "system", "content": SYSTEM_PROMPT}, *self.history[-MAX_HISTORY_MESSAGES:]]try:found = investigate(db, provider, messages, REPLY_SCHEMA, "agent_reply", budget=budget, effort="low", record=record or (lambda *_: None),diff --git a/tests/assistant/test_agent_service.py b/tests/assistant/test_agent_service.pyindex 73d007f..b28790e 100644--- a/tests/assistant/test_agent_service.py+++ b/tests/assistant/test_agent_service.py@@ -20,6 +20,22 @@ def test_without_a_tool_budget_the_chat_is_one_call_without_tools():assert provider.started == [{"tools": [], "effort": "low", "schema_name": "agent_reply"}] and len(provider.calls) == 1+def test_secrets_in_the_question_and_incident_context_never_reach_the_provider():+ provider = ScriptedProvider('{"message":"Looking."}', '{"message":"Still looking."}')+ context = ("[INCIDENT INC-0001 on web1, warning, active]\nContainer app exited\n"+ "- GET https://admin:[email protected]/ failed, api_key=ABCDEF123456\n[QUESTION]\n")+ service = make_service()++ service.send(1, "conversation", context + "My token is ghp_" + "a" * 30, provider)+ service.send(1, "conversation", "And now?", provider)++ for sent in provider.calls: # the first turn, and the history sent again with the second+ text = " ".join(message["content"] for message in sent if message["role"] == "user")+ assert "hunter2" not in text and "ABCDEF123456" not in text and "ghp_" + "a" * 30 not in text+ first = provider.calls[0][-1]["content"]+ assert "[masked]" in first and "Container app exited" in first and "INC-0001" in first # the rest stays readable++def test_a_reply_without_a_message_is_an_error():with pytest.raises(ValueError, match="must contain a message"):make_service().send(1, "conversation", "Hello", ScriptedProvider('{"message":" "}'))