Skip to content

armgit / AiSOC

Public Imported from GitHub
0

Open-source AI Security Operations Center: alert fusion, LLM-agent triage, MITRE ATT&CK investigation, and a replayable decision ledger for every agent step. Self-hostable, runs with no API keys, MIT licensed. Ships an MCP server for Claude, Cursor and Continue.

fix(security): bound eval and playbook timeouts (H-7) (#126)

Beenu Arora492e0d0ba59d
diff --git a/CHANGELOG.md b/CHANGELOG.md
index f60e266de..80e4eed61 100644
--- a/CHANGELOG.md
+++ b/CHANGELOG.md
@@ -7,6 +7,36 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
## [Unreleased]
+### Security — bounded eval / playbook timeouts (H-7, batch 10)
+
+- **`POST /detection-proposals/run-eval`** — tightened Pydantic bounds on
+ `RunEvalRequest.timeout_seconds` (`le=900` → `le=600`) and
+ `max_regression_pp` (`le=100` → `le=50`). The synthetic benchmark completes
+ in under 60s on commodity hardware; a 15-minute cap was indistinguishable
+ from "no cap" from a DoS standpoint. 50pp regression tolerance is already
+ generous — anything higher silently disables the gate. `EvalAttachRequest`
+ receives the same `max_regression_pp` cap.
+- **Playbook step bounds** — added strict Pydantic validation to
+ `PlaybookStep.timeout_seconds` (`1 ≤ x ≤ 3600s`) and `retry_max` (`0 ≤ x ≤ 25`)
+ in `services/agents/app/playbook/models.py`. The 1-hour ceiling
+ accommodates human-approval steps; everything else is bounded at runtime.
+- **Runtime parameter clamp** — new module
+ `services/agents/app/playbook/bounds.py` centralises `clamp_timeout()` and
+ `clamp_retries()` helpers. `_handle_osquery_live_query()` and other
+ handlers that read `step.params["timeout_seconds"]` (an untyped dict that
+ bypasses Pydantic) now clamp at runtime against
+ `ABSOLUTE_MAX_PARAM_TIMEOUT_SECONDS = 900s`. This is the layered defence:
+ field validators cap the typed model, runtime clamps cap the dynamic
+ params. Operator overrides via `AISOC_PLAYBOOK_MAX_TIMEOUT_SECONDS` /
+ `AISOC_PLAYBOOK_MAX_RETRIES` are themselves bounded by the absolute
+ ceilings so a typo can't disable the cap entirely.
+- 67 new tests in `services/agents/tests/test_playbook_bounds.py`,
+ `test_playbook_models_bounds.py`, and `test_osquery_live_query_step.py`
+ exercise pathological values (86 400s, negatives, `True`, numeric strings,
+ env-var overrides). 22 new tests in
+ `services/api/tests/api/v1/endpoints/test_detection_proposals_request_bounds.py`
+ pin the API-side caps.
+
### Pydantic v2 configuration migration (H-6)
Eliminates the 63+ `PydanticDeprecatedSince20: Support for class-based
diff --git a/docs/openapi.yaml b/docs/openapi.yaml
index b50d4adc3..e63a50caf 100644
--- a/docs/openapi.yaml
+++ b/docs/openapi.yaml
@@ -16479,10 +16479,12 @@ components:
... --max-regression-pp ...` for the candidate ruleset.
max_regression_pp:
type: number
- maximum: 100.0
+ maximum: 50.0
minimum: 0.0
title: Max Regression Pp
- description: Allowed MITRE accuracy regression vs baseline, in pp.
+ description: Allowed MITRE accuracy regression vs baseline, in pp. Hard-capped
+ at 50 — anything above that is indistinguishable from disabling the regression
+ gate entirely.
default: 1.0
type: object
required:
@@ -21638,18 +21640,21 @@ components:
default: true
max_regression_pp:
type: number
- maximum: 100.0
+ maximum: 50.0
minimum: 0.0
title: Max Regression Pp
description: Allowed MITRE accuracy regression vs the active baseline, in
- percentage points. Forwarded to run_evals.py.
+ percentage points. Forwarded to run_evals.py. Hard-capped at 50 to keep
+ the regression gate meaningful.
default: 1.0
timeout_seconds:
type: integer
- maximum: 900.0
+ maximum: 600.0
minimum: 10.0
title: Timeout Seconds
- description: Hard timeout for the eval subprocess.
+ description: Hard timeout for the eval subprocess. Capped at 10 minutes
+ — the synthetic benchmark completes in under 60s on commodity hardware
+ so anything longer is almost certainly a stuck process.
default: 180
type: object
title: RunEvalRequest
diff --git a/services/agents/app/playbook/bounds.py b/services/agents/app/playbook/bounds.py
new file mode 100644
index 000000000..4978c2024
--- /dev/null
+++ b/services/agents/app/playbook/bounds.py
@@ -0,0 +1,180 @@
+"""Runtime bounds for playbook step execution (H-7 / Batch 10).
+
+Centralises the per-step ``timeout_seconds`` and ``retry_max`` limits so they
+are enforced in *every* place a step parameter is consumed — including
+``step.params.get("timeout_seconds")`` paths that bypass the Pydantic field
+validator on :class:`PlaybookStep`.
+
+Why a runtime clamp on top of Pydantic bounds?
+----------------------------------------------
+Some handlers (e.g. ``_handle_osquery_live_query``) intentionally read
+``timeout_seconds`` out of ``step.params`` instead of the typed field on
+``PlaybookStep``. Pydantic only validates the *field*, so a malicious or
+malformed playbook can sneak ``params.timeout_seconds = 86400`` past the
+model. This module provides a single source of truth that both the model
+defaults and the handlers consult, so the bound holds regardless of where
+the value originates.
+
+Environment overrides
+---------------------
+Operators can raise the ceiling for slow-running on-prem agents via:
+
+``AISOC_PLAYBOOK_MAX_TIMEOUT_SECONDS``
+ Maximum per-step timeout enforced at runtime by :func:`clamp_timeout`.
+ Default 300. Clamped to
+ ``[MIN_TIMEOUT_SECONDS, ABSOLUTE_MAX_PARAM_TIMEOUT_SECONDS]``.
+
+``AISOC_PLAYBOOK_MAX_RETRIES``
+ Maximum ``retry_max`` value any step may declare. Default 10. Clamped
+ to ``[0, ABSOLUTE_MAX_RETRIES]``.
+"""
+
+from __future__ import annotations
+
+import logging
+import os
+from typing import Any, Final
+
+logger = logging.getLogger("aisoc.playbook.bounds")
+
+# ── Hard limits ─────────────────────────────────────────────────────────────
+# These are absolute ceilings — even with env overrides we will never accept
+# a value above ABSOLUTE_MAX_*. They exist to bound the worst case (e.g. a
+# malicious playbook trying to pin a connector thread for hours).
+#
+# Two distinct ceilings:
+#
+# - ``ABSOLUTE_MAX_TIMEOUT_SECONDS`` (1 hour) is the cap enforced by the
+# Pydantic field on :class:`PlaybookStep`. Some step types (notably
+# ``approval``) legitimately wait on a human and need long timeouts —
+# capping at 15 minutes would break existing approval-gated playbooks
+# (e.g. ``phishing-investigation`` declares a 920s step timeout for a
+# 900s approval wait + 20s buffer).
+#
+# - ``ABSOLUTE_MAX_PARAM_TIMEOUT_SECONDS`` (15 minutes) is the cap applied
+# by :func:`clamp_timeout` when a handler reads ``step.params.timeout_seconds``
+# directly (e.g. ``_handle_osquery_live_query``). Those code paths do not
+# wait on humans, so a tight ceiling bounds DoS/amplification regardless
+# of what a malicious playbook declares.
+MIN_TIMEOUT_SECONDS: Final[int] = 1
+ABSOLUTE_MAX_TIMEOUT_SECONDS: Final[int] = 3600 # 1 hour — Pydantic field cap
+ABSOLUTE_MAX_PARAM_TIMEOUT_SECONDS: Final[int] = 900 # 15 min — runtime params cap
+ABSOLUTE_MAX_RETRIES: Final[int] = 25
+
+# ── Default ceilings used by clamp_timeout / clamp_retries ──────────────────
+# Defaults are deliberately tight: detections in this repo never read
+# ``params.timeout_seconds`` above 60s and retry_max above 2, so 300s / 10
+# leaves generous headroom while still being orders of magnitude below
+# pathological values. Operators can raise these via env overrides up to
+# the absolute ceilings above.
+DEFAULT_MAX_TIMEOUT_SECONDS: Final[int] = 300 # 5 minutes
+DEFAULT_MAX_RETRIES: Final[int] = 10
+
+
+def _clamp_int(value: int, lo: int, hi: int) -> int:
+ if value < lo:
+ return lo
+ if value > hi:
+ return hi
+ return value
+
+
+def _env_int(name: str, default: int, lo: int, hi: int) -> int:
+ raw = os.environ.get(name)
+ if raw is None or raw == "":
+ return default
+ try:
+ parsed = int(raw)
+ except (TypeError, ValueError):
+ logger.warning(
+ "Ignoring non-integer %s=%r, falling back to default %d",
+ name,
+ raw,
+ default,
+ )
+ return default
+ clamped = _clamp_int(parsed, lo, hi)
+ if clamped != parsed:
+ logger.warning(
+ "%s=%d clamped to %d (range %d..%d)",
+ name,
+ parsed,
+ clamped,
+ lo,
+ hi,
+ )
+ return clamped
+
+
+def max_timeout_seconds() -> int:
+ """Effective per-step runtime timeout ceiling, honouring env overrides.
+
+ This is the ceiling used by :func:`clamp_timeout` for ``params``-derived
+ values (e.g. ``_handle_osquery_live_query`` reading
+ ``step.params.timeout_seconds``). It is intentionally tighter than the
+ Pydantic field cap on :class:`PlaybookStep.timeout_seconds`, which has
+ to accommodate long human-approval waits.
+ """
+ return _env_int(
+ "AISOC_PLAYBOOK_MAX_TIMEOUT_SECONDS",
+ DEFAULT_MAX_TIMEOUT_SECONDS,
+ MIN_TIMEOUT_SECONDS,
+ ABSOLUTE_MAX_PARAM_TIMEOUT_SECONDS,
+ )
+
+
+def max_retries() -> int:
+ """Effective per-step retry ceiling, honouring env overrides."""
+ return _env_int(
+ "AISOC_PLAYBOOK_MAX_RETRIES",
+ DEFAULT_MAX_RETRIES,
+ 0,
+ ABSOLUTE_MAX_RETRIES,
+ )
+
+
+def clamp_timeout(value: Any, *, default: int | None = None) -> int:
+ """Clamp an arbitrary ``timeout_seconds`` value into the allowed range.
+
+ Non-integer or missing values fall back to ``default`` (or the
+ PlaybookStep default of 30s if unspecified). Bools are rejected because
+ Python treats ``True`` as ``1`` and we never want a step's timeout to be
+ silently coerced from a misconfigured truthy flag.
+ """
+ fallback = 30 if default is None else default
+ if value is None or isinstance(value, bool):
+ candidate: int = fallback
+ else:
+ try:
+ candidate = int(value)
+ except (TypeError, ValueError):
+ logger.warning("Invalid timeout_seconds=%r, using fallback %d", value, fallback)
+ candidate = fallback
+ return _clamp_int(candidate, MIN_TIMEOUT_SECONDS, max_timeout_seconds())
+
+
+def clamp_retries(value: Any, *, default: int = 0) -> int:
+ """Clamp an arbitrary ``retry_max`` value into the allowed range."""
+ if value is None or isinstance(value, bool):
+ candidate: int = default
+ else:
+ try:
+ candidate = int(value)
+ except (TypeError, ValueError):
+ logger.warning("Invalid retry_max=%r, using fallback %d", value, default)
+ candidate = default
+ return _clamp_int(candidate, 0, max_retries())
+
+
+__all__ = [
+ "ABSOLUTE_MAX_PARAM_TIMEOUT_SECONDS",
+ "ABSOLUTE_MAX_RETRIES",
+ "ABSOLUTE_MAX_TIMEOUT_SECONDS",
+ "DEFAULT_MAX_RETRIES",
+ "DEFAULT_MAX_TIMEOUT_SECONDS",
+ "MIN_TIMEOUT_SECONDS",
+ "clamp_retries",
+ "clamp_timeout",
+ "max_retries",
+ "max_timeout_seconds",
+]
diff --git a/services/agents/app/playbook/engine.py b/services/agents/app/playbook/engine.py
index 8fe73572c..042872d6f 100644
--- a/services/agents/app/playbook/engine.py
+++ b/services/agents/app/playbook/engine.py
@@ -23,6 +23,7 @@ from typing import Any
import httpx
+from .bounds import clamp_timeout
from .models import Playbook, PlaybookStep, StepCondition, StepType
from .ssrf_guard import SSRFError, validate_outbound_url
@@ -437,7 +438,10 @@ async def _handle_osquery_live_query(step: PlaybookStep, context: dict[str, Any]
target_hosts: list[str] = step.params.get("target_hosts") or [context.get("host_id") or context.get("host", "")]
template: str = step.params.get("template", "")
template_params: dict[str, Any] = step.params.get("template_params") or {}
- timeout_seconds: int = step.params.get("timeout_seconds", 60)
+ # Runtime clamp — params bypass the Pydantic validator on PlaybookStep, so
+ # a malicious or malformed playbook could otherwise pin the connector
+ # thread for hours. See services/agents/app/playbook/bounds.py.
+ timeout_seconds: int = clamp_timeout(step.params.get("timeout_seconds", 60), default=60)
if not template:
return {"error": "osquery_live_query: 'template' param is required", "partial": True}
diff --git a/services/agents/app/playbook/models.py b/services/agents/app/playbook/models.py
index ba2effb5b..dda894594 100644
--- a/services/agents/app/playbook/models.py
+++ b/services/agents/app/playbook/models.py
@@ -8,6 +8,12 @@ from typing import Annotated, Any, Literal
from pydantic import BaseModel, BeforeValidator, Field
+from .bounds import (
+ ABSOLUTE_MAX_RETRIES,
+ ABSOLUTE_MAX_TIMEOUT_SECONDS,
+ MIN_TIMEOUT_SECONDS,
+)
+
class StepType(str, Enum):
"""Supported step action types."""
@@ -76,8 +82,16 @@ class PlaybookStep(BaseModel):
params: dict[str, Any] = Field(default_factory=dict)
condition: StepConditionField = None
on_failure: Literal["abort", "continue", "retry"] = "abort"
- retry_max: int = 0
- timeout_seconds: int = 30
+ # Hard ceilings — see services/agents/app/playbook/bounds.py for rationale.
+ # Pydantic validates these on the typed field; handlers that read
+ # ``params.timeout_seconds`` directly MUST also pass the value through
+ # ``bounds.clamp_timeout`` at runtime.
+ retry_max: int = Field(default=0, ge=0, le=ABSOLUTE_MAX_RETRIES)
+ timeout_seconds: int = Field(
+ default=30,
+ ge=MIN_TIMEOUT_SECONDS,
+ le=ABSOLUTE_MAX_TIMEOUT_SECONDS,
+ )
# For branching: step IDs to jump to on true / false
next_true: str | None = None
next_false: str | None = None
diff --git a/services/agents/tests/test_osquery_live_query_step.py b/services/agents/tests/test_osquery_live_query_step.py
index aaf2f3d14..6658c4bb5 100644
--- a/services/agents/tests/test_osquery_live_query_step.py
+++ b/services/agents/tests/test_osquery_live_query_step.py
@@ -264,3 +264,190 @@ class TestHostFallback:
call_args = lq.call_args
assert "context-host" in call_args[0][0]
+
+
+# ---------------------------------------------------------------------------
+# H-7 / Batch 10 — runtime clamp tests for params.timeout_seconds.
+#
+# These exercise the *runtime* defence (engine.clamp_timeout) rather than the
+# Pydantic field cap, because params is an untyped dict and a malicious or
+# malformed playbook can bypass the Pydantic validator entirely. See
+# ``test_playbook_models_bounds.py::TestParamsAreNotValidated`` for the
+# matching failure case at the model layer.
+# ---------------------------------------------------------------------------
+
+
+class TestRuntimeTimeoutClamp:
+ @pytest.mark.asyncio
+ async def test_pathological_timeout_is_clamped_to_default_max(self) -> None:
+ """An attacker passes ``timeout_seconds: 86400`` (1 day).
+
+ Without the runtime clamp, the connector thread would pin for 24h.
+ With the clamp, the value collapses to DEFAULT_MAX_TIMEOUT_SECONDS.
+ """
+ from app.playbook.bounds import DEFAULT_MAX_TIMEOUT_SECONDS
+
+ lq = AsyncMock(return_value={"results": {}, "partial": False})
+ _inject_stubs(osctrl_live_query=lq)
+
+ step = _step(
+ {
+ "backend": "osctrl",
+ "template": "running_processes",
+ "target_hosts": ["h1"],
+ "base_url": "http://osc",
+ "api_token": "tok",
+ "timeout_seconds": 86_400, # pathological
+ }
+ )
+ await _handle_osquery_live_query(step, _ctx(), MagicMock())
+
+ # Backend client must be called with the clamped value, not 86400.
+ call_args = lq.call_args
+ forwarded_timeout = call_args[0][3] # 4th positional arg
+ assert forwarded_timeout == DEFAULT_MAX_TIMEOUT_SECONDS
+
+ @pytest.mark.asyncio
+ async def test_negative_timeout_is_clamped_to_min(self) -> None:
+ from app.playbook.bounds import MIN_TIMEOUT_SECONDS
+
+ lq = AsyncMock(return_value={"results": {}, "partial": False})
+ _inject_stubs(osctrl_live_query=lq)
+
+ step = _step(
+ {
+ "backend": "osctrl",
+ "template": "running_processes",
+ "target_hosts": ["h1"],
+ "base_url": "http://osc",
+ "api_token": "tok",
+ "timeout_seconds": -5,
+ }
+ )
+ await _handle_osquery_live_query(step, _ctx(), MagicMock())
+
+ call_args = lq.call_args
+ forwarded_timeout = call_args[0][3]
+ assert forwarded_timeout == MIN_TIMEOUT_SECONDS
+
+ @pytest.mark.asyncio
+ async def test_missing_timeout_uses_handler_default(self) -> None:
+ """When params omits timeout_seconds, the handler default (60) wins."""
+ lq = AsyncMock(return_value={"results": {}, "partial": False})
+ _inject_stubs(osctrl_live_query=lq)
+
+ step = _step(
+ {
+ "backend": "osctrl",
+ "template": "running_processes",
+ "target_hosts": ["h1"],
+ "base_url": "http://osc",
+ "api_token": "tok",
+ # No timeout_seconds in params at all.
+ }
+ )
+ await _handle_osquery_live_query(step, _ctx(), MagicMock())
+
+ call_args = lq.call_args
+ forwarded_timeout = call_args[0][3]
+ assert forwarded_timeout == 60
+
+ @pytest.mark.asyncio
+ async def test_in_range_timeout_passes_through(self) -> None:
+ lq = AsyncMock(return_value={"results": {}, "partial": False})
+ _inject_stubs(osctrl_live_query=lq)
+
+ step = _step(
+ {
+ "backend": "osctrl",
+ "template": "running_processes",
+ "target_hosts": ["h1"],
+ "base_url": "http://osc",
+ "api_token": "tok",
+ "timeout_seconds": 120,
+ }
+ )
+ await _handle_osquery_live_query(step, _ctx(), MagicMock())
+
+ call_args = lq.call_args
+ forwarded_timeout = call_args[0][3]
+ assert forwarded_timeout == 120
+
+ @pytest.mark.asyncio
+ async def test_string_timeout_is_parsed_not_dropped(self) -> None:
+ """YAML may surface numeric strings; the clamp must parse them."""
+ lq = AsyncMock(return_value={"results": {}, "partial": False})
+ _inject_stubs(osctrl_live_query=lq)
+
+ step = _step(
+ {
+ "backend": "osctrl",
+ "template": "running_processes",
+ "target_hosts": ["h1"],
+ "base_url": "http://osc",
+ "api_token": "tok",
+ "timeout_seconds": "90", # string from YAML
+ }
+ )
+ await _handle_osquery_live_query(step, _ctx(), MagicMock())
+
+ call_args = lq.call_args
+ forwarded_timeout = call_args[0][3]
+ assert forwarded_timeout == 90
+
+ @pytest.mark.asyncio
+ async def test_bool_timeout_does_not_become_one(self) -> None:
+ """``timeout_seconds: true`` MUST NOT coerce to 1 — that would silently
+ cause every osquery to time out immediately, which is a sneaky DoS.
+ """
+ lq = AsyncMock(return_value={"results": {}, "partial": False})
+ _inject_stubs(osctrl_live_query=lq)
+
+ step = _step(
+ {
+ "backend": "osctrl",
+ "template": "running_processes",
+ "target_hosts": ["h1"],
+ "base_url": "http://osc",
+ "api_token": "tok",
+ "timeout_seconds": True, # malformed
+ }
+ )
+ await _handle_osquery_live_query(step, _ctx(), MagicMock())
+
+ call_args = lq.call_args
+ forwarded_timeout = call_args[0][3]
+ # Falls back to the handler default, NOT to int(True) == 1.
+ assert forwarded_timeout == 60
+
+ @pytest.mark.asyncio
+ async def test_param_cap_is_runtime_ceiling_not_field_cap(self, monkeypatch: pytest.MonkeyPatch) -> None:
+ """The runtime ceiling is the *param* cap (15 min), not the looser
+ Pydantic field cap (1h). This pins the layered defence: approval
+ steps may declare timeout=3600 on the typed field, but params-driven
+ values cannot exceed the runtime cap.
+ """
+ from app.playbook.bounds import ABSOLUTE_MAX_PARAM_TIMEOUT_SECONDS
+
+ # Operator typo: setting AISOC_PLAYBOOK_MAX_TIMEOUT_SECONDS=100000 must
+ # NOT silently uncap the runtime guard.
+ monkeypatch.setenv("AISOC_PLAYBOOK_MAX_TIMEOUT_SECONDS", "100000")
+
+ lq = AsyncMock(return_value={"results": {}, "partial": False})
+ _inject_stubs(osctrl_live_query=lq)
+
+ step = _step(
+ {
+ "backend": "osctrl",
+ "template": "running_processes",
+ "target_hosts": ["h1"],
+ "base_url": "http://osc",
+ "api_token": "tok",
+ "timeout_seconds": 50_000, # would exceed the param cap
+ }
+ )
+ await _handle_osquery_live_query(step, _ctx(), MagicMock())
+
+ call_args = lq.call_args
+ forwarded_timeout = call_args[0][3]
+ assert forwarded_timeout == ABSOLUTE_MAX_PARAM_TIMEOUT_SECONDS
diff --git a/services/agents/tests/test_playbook_bounds.py b/services/agents/tests/test_playbook_bounds.py
new file mode 100644
index 000000000..af980bc3e
--- /dev/null
+++ b/services/agents/tests/test_playbook_bounds.py
@@ -0,0 +1,221 @@
+"""Unit tests for the playbook runtime bounds module (H-7 / Batch 10).
+
+These tests pin the contract that *both* Pydantic field bounds and runtime
+``clamp_*`` calls must honour, so the security guarantee survives refactors
+to either side of the layered defence (handlers vs. model).
+"""
+
+from __future__ import annotations
+
+import importlib
+
+import pytest
+from app.playbook import bounds
+
+# ---------------------------------------------------------------------------
+# Constants — these are part of the security contract; if anyone tightens or
+# loosens them they should have to update a test.
+# ---------------------------------------------------------------------------
+
+
+class TestConstants:
+ def test_min_timeout_is_positive(self) -> None:
+ assert bounds.MIN_TIMEOUT_SECONDS >= 1
+
+ def test_pydantic_field_cap_above_param_cap(self) -> None:
+ """Pydantic field cap must be >= runtime param cap.
+
+ Pydantic field covers approval-step human-waits (≤1h). Param cap is
+ the tighter ceiling enforced at runtime for handler-driven values
+ (≤15min). Inverting these would either break approval playbooks or
+ defeat the runtime guard.
+ """
+ assert bounds.ABSOLUTE_MAX_TIMEOUT_SECONDS >= bounds.ABSOLUTE_MAX_PARAM_TIMEOUT_SECONDS
+
+ def test_param_cap_is_tighter_than_field_cap(self) -> None:
+ # If these ever become equal, delete one — the layering loses meaning.
+ assert bounds.ABSOLUTE_MAX_PARAM_TIMEOUT_SECONDS < bounds.ABSOLUTE_MAX_TIMEOUT_SECONDS
+
+ def test_defaults_within_absolute_bounds(self) -> None:
+ assert bounds.MIN_TIMEOUT_SECONDS <= bounds.DEFAULT_MAX_TIMEOUT_SECONDS <= bounds.ABSOLUTE_MAX_PARAM_TIMEOUT_SECONDS
+ assert 0 <= bounds.DEFAULT_MAX_RETRIES <= bounds.ABSOLUTE_MAX_RETRIES
+
+
+# ---------------------------------------------------------------------------
+# clamp_timeout
+# ---------------------------------------------------------------------------
+
+
+class TestClampTimeout:
+ def test_in_range_value_passes_through(self) -> None:
+ assert bounds.clamp_timeout(60) == 60
+
+ def test_value_at_default_max_passes_through(self) -> None:
+ # No env override → ceiling is DEFAULT_MAX_TIMEOUT_SECONDS.
+ assert bounds.clamp_timeout(bounds.DEFAULT_MAX_TIMEOUT_SECONDS) == (bounds.DEFAULT_MAX_TIMEOUT_SECONDS)
+
+ def test_value_above_default_max_is_clamped(self) -> None:
+ result = bounds.clamp_timeout(86400) # 1 day — pathological
+ assert result == bounds.DEFAULT_MAX_TIMEOUT_SECONDS
+
+ def test_negative_value_is_clamped_to_min(self) -> None:
+ assert bounds.clamp_timeout(-1) == bounds.MIN_TIMEOUT_SECONDS
+
+ def test_zero_is_clamped_to_min(self) -> None:
+ assert bounds.clamp_timeout(0) == bounds.MIN_TIMEOUT_SECONDS
+
+ def test_none_uses_fallback(self) -> None:
+ assert bounds.clamp_timeout(None, default=45) == 45
+
+ def test_none_without_fallback_uses_step_default(self) -> None:
+ # Mirrors PlaybookStep.timeout_seconds default (30).
+ assert bounds.clamp_timeout(None) == 30
+
+ def test_bool_true_does_not_become_one(self) -> None:
+ """Bools must NOT be coerced to int.
+
+ ``int(True) == 1`` would otherwise silently set timeout=1 for any
+ truthy misconfiguration, which is both surprising and DoS-prone
+ because it short-circuits long-running queries.
+ """
+ result = bounds.clamp_timeout(True, default=60)
+ assert result == 60
+
+ def test_bool_false_uses_fallback(self) -> None:
+ result = bounds.clamp_timeout(False, default=45)
+ assert result == 45
+
+ def test_non_numeric_string_uses_fallback(self) -> None:
+ assert bounds.clamp_timeout("forever", default=42) == 42
+
+ def test_numeric_string_is_parsed(self) -> None:
+ # Operators sometimes pass numeric strings from YAML.
+ assert bounds.clamp_timeout("90") == 90
+
+ def test_float_is_truncated_to_int(self) -> None:
+ assert bounds.clamp_timeout(12.7) == 12
+
+ def test_env_override_raises_ceiling(self, monkeypatch: pytest.MonkeyPatch) -> None:
+ monkeypatch.setenv("AISOC_PLAYBOOK_MAX_TIMEOUT_SECONDS", "600")
+ # 500 should now pass through (was previously clamped to 300).
+ assert bounds.clamp_timeout(500) == 500
+
+ def test_env_override_cannot_exceed_param_absolute_max(self, monkeypatch: pytest.MonkeyPatch) -> None:
+ """Env override must itself be clamped to ABSOLUTE_MAX_PARAM_*.
+
+ Otherwise an operator typo (``AISOC_PLAYBOOK_MAX_TIMEOUT_SECONDS=86400``)
+ would silently uncap the runtime guard and re-introduce the very DoS
+ this module exists to prevent.
+ """
+ monkeypatch.setenv("AISOC_PLAYBOOK_MAX_TIMEOUT_SECONDS", "100000")
+ # The effective ceiling should be ABSOLUTE_MAX_PARAM_TIMEOUT_SECONDS,
+ # not 100000 or the Pydantic field cap.
+ result = bounds.clamp_timeout(50000)
+ assert result == bounds.ABSOLUTE_MAX_PARAM_TIMEOUT_SECONDS
+
+ def test_env_override_garbage_falls_back_to_default(self, monkeypatch: pytest.MonkeyPatch) -> None:
+ monkeypatch.setenv("AISOC_PLAYBOOK_MAX_TIMEOUT_SECONDS", "not-a-number")
+ # 500 must still be clamped to DEFAULT_MAX_TIMEOUT_SECONDS since the
+ # env value is unusable.
+ assert bounds.clamp_timeout(500) == bounds.DEFAULT_MAX_TIMEOUT_SECONDS
+
+ def test_env_override_below_min_is_clamped_up(self, monkeypatch: pytest.MonkeyPatch) -> None:
+ monkeypatch.setenv("AISOC_PLAYBOOK_MAX_TIMEOUT_SECONDS", "-10")
+ # Ceiling clamped to MIN_TIMEOUT_SECONDS, so any value above that
+ # collapses to MIN.
+ assert bounds.clamp_timeout(100) == bounds.MIN_TIMEOUT_SECONDS
+
+
+# ---------------------------------------------------------------------------
+# clamp_retries
+# ---------------------------------------------------------------------------
+
+
+class TestClampRetries:
+ def test_in_range_value_passes_through(self) -> None:
+ assert bounds.clamp_retries(3) == 3
+
+ def test_zero_is_allowed(self) -> None:
+ # 0 means "no retries"; must not be coerced to MIN of 1.
+ assert bounds.clamp_retries(0) == 0
+
+ def test_negative_value_is_clamped_to_zero(self) -> None:
+ assert bounds.clamp_retries(-5) == 0
+
+ def test_value_above_default_max_is_clamped(self) -> None:
+ assert bounds.clamp_retries(9999) == bounds.DEFAULT_MAX_RETRIES
+
+ def test_none_uses_default(self) -> None:
+ assert bounds.clamp_retries(None) == 0
+ assert bounds.clamp_retries(None, default=2) == 2
+
+ def test_bool_uses_fallback(self) -> None:
+ # Same anti-coercion guarantee as clamp_timeout.
+ assert bounds.clamp_retries(True, default=4) == 4
+
+ def test_non_numeric_string_uses_fallback(self) -> None:
+ assert bounds.clamp_retries("nope", default=1) == 1
+
+ def test_env_override_cannot_exceed_absolute_max(self, monkeypatch: pytest.MonkeyPatch) -> None:
+ monkeypatch.setenv("AISOC_PLAYBOOK_MAX_RETRIES", "1000")
+ # Ceiling is hard-clamped to ABSOLUTE_MAX_RETRIES.
+ assert bounds.clamp_retries(500) == bounds.ABSOLUTE_MAX_RETRIES
+
+
+# ---------------------------------------------------------------------------
+# Effective ceilings
+# ---------------------------------------------------------------------------
+
+
+class TestEffectiveCeilings:
+ def test_max_timeout_seconds_default(self, monkeypatch: pytest.MonkeyPatch) -> None:
+ monkeypatch.delenv("AISOC_PLAYBOOK_MAX_TIMEOUT_SECONDS", raising=False)
+ assert bounds.max_timeout_seconds() == bounds.DEFAULT_MAX_TIMEOUT_SECONDS
+
+ def test_max_timeout_seconds_clamped_to_param_absolute(self, monkeypatch: pytest.MonkeyPatch) -> None:
+ monkeypatch.setenv(
+ "AISOC_PLAYBOOK_MAX_TIMEOUT_SECONDS",
+ str(bounds.ABSOLUTE_MAX_PARAM_TIMEOUT_SECONDS + 10_000),
+ )
+ # Even with a huge env value, the function clamps to the param cap,
+ # NOT the looser Pydantic field cap.
+ assert bounds.max_timeout_seconds() == bounds.ABSOLUTE_MAX_PARAM_TIMEOUT_SECONDS
+
+ def test_max_retries_default(self, monkeypatch: pytest.MonkeyPatch) -> None:
+ monkeypatch.delenv("AISOC_PLAYBOOK_MAX_RETRIES", raising=False)
+ assert bounds.max_retries() == bounds.DEFAULT_MAX_RETRIES
+
+ def test_empty_env_value_falls_back_to_default(self, monkeypatch: pytest.MonkeyPatch) -> None:
+ # Empty string (e.g. ``AISOC_PLAYBOOK_MAX_TIMEOUT_SECONDS=``) must be
+ # treated as unset, not as invalid int.
+ monkeypatch.setenv("AISOC_PLAYBOOK_MAX_TIMEOUT_SECONDS", "")
+ assert bounds.max_timeout_seconds() == bounds.DEFAULT_MAX_TIMEOUT_SECONDS
+
+
+# ---------------------------------------------------------------------------
+# Module-level invariants
+# ---------------------------------------------------------------------------
+
+
+def test_module_reload_is_safe() -> None:
+ """The module must not have any startup side effects that break reloads."""
+ importlib.reload(bounds)
+ assert callable(bounds.clamp_timeout)
+ assert callable(bounds.clamp_retries)
+
+
+def test_all_export_is_complete() -> None:
+ """``__all__`` should expose every public symbol used by callers."""
+ required = {
+ "ABSOLUTE_MAX_PARAM_TIMEOUT_SECONDS",
+ "ABSOLUTE_MAX_RETRIES",
+ "ABSOLUTE_MAX_TIMEOUT_SECONDS",
+ "DEFAULT_MAX_RETRIES",
+ "DEFAULT_MAX_TIMEOUT_SECONDS",
+ "MIN_TIMEOUT_SECONDS",
+ "clamp_retries",
+ "clamp_timeout",
+ "max_retries",
+ "max_timeout_seconds",
+ }
+ assert required.issubset(set(bounds.__all__))
diff --git a/services/agents/tests/test_playbook_models_bounds.py b/services/agents/tests/test_playbook_models_bounds.py
new file mode 100644
index 000000000..3b620334d
--- /dev/null
+++ b/services/agents/tests/test_playbook_models_bounds.py
@@ -0,0 +1,159 @@
+"""Pydantic validation tests for PlaybookStep bounds (H-7 / Batch 10).
+
+The Pydantic field cap is the *first* line of defence — well-formed playbooks
+that exceed the cap should fail to load instead of silently running.
+"""
+
+from __future__ import annotations
+
+import pytest
+from app.playbook.bounds import (
+ ABSOLUTE_MAX_RETRIES,
+ ABSOLUTE_MAX_TIMEOUT_SECONDS,
+ MIN_TIMEOUT_SECONDS,
+)
+from app.playbook.models import PlaybookStep, StepType
+from pydantic import ValidationError
+
+# ---------------------------------------------------------------------------
+# timeout_seconds bounds
+# ---------------------------------------------------------------------------
+
+
+class TestTimeoutSecondsField:
+ def test_default_is_30_seconds(self) -> None:
+ step = PlaybookStep(name="x", type=StepType.NOTIFY)
+ assert step.timeout_seconds == 30
+
+ def test_in_range_value_is_accepted(self) -> None:
+ step = PlaybookStep(name="x", type=StepType.NOTIFY, timeout_seconds=120)
+ assert step.timeout_seconds == 120
+
+ def test_minimum_is_inclusive(self) -> None:
+ step = PlaybookStep(
+ name="x",
+ type=StepType.NOTIFY,
+ timeout_seconds=MIN_TIMEOUT_SECONDS,
+ )
+ assert step.timeout_seconds == MIN_TIMEOUT_SECONDS
+
+ def test_below_minimum_is_rejected(self) -> None:
+ with pytest.raises(ValidationError) as exc:
+ PlaybookStep(name="x", type=StepType.NOTIFY, timeout_seconds=0)
+ # Validation message should mention the bound to aid debugging.
+ assert "greater than or equal" in str(exc.value).lower()
+
+ def test_negative_is_rejected(self) -> None:
+ with pytest.raises(ValidationError):
+ PlaybookStep(name="x", type=StepType.NOTIFY, timeout_seconds=-1)
+
+ def test_maximum_is_inclusive(self) -> None:
+ # 920s approval-wait timeouts (e.g. phishing-investigation playbook)
+ # MUST still load — that scenario is exactly why the field cap is 1h
+ # rather than the tighter 15-min runtime cap.
+ step = PlaybookStep(
+ name="approval",
+ type=StepType.APPROVAL,
+ timeout_seconds=920,
+ )
+ assert step.timeout_seconds == 920
+
+ def test_at_absolute_max_is_accepted(self) -> None:
+ step = PlaybookStep(
+ name="x",
+ type=StepType.APPROVAL,
+ timeout_seconds=ABSOLUTE_MAX_TIMEOUT_SECONDS,
+ )
+ assert step.timeout_seconds == ABSOLUTE_MAX_TIMEOUT_SECONDS
+
+ def test_above_absolute_max_is_rejected(self) -> None:
+ with pytest.raises(ValidationError) as exc:
+ PlaybookStep(
+ name="x",
+ type=StepType.NOTIFY,
+ timeout_seconds=ABSOLUTE_MAX_TIMEOUT_SECONDS + 1,
+ )
+ assert "less than or equal" in str(exc.value).lower()
+
+ def test_pathologically_large_is_rejected(self) -> None:
+ # 1 day — the classic DoS payload.
+ with pytest.raises(ValidationError):
+ PlaybookStep(name="x", type=StepType.NOTIFY, timeout_seconds=86_400)
+
+
+# ---------------------------------------------------------------------------
+# retry_max bounds
+# ---------------------------------------------------------------------------
+
+
+class TestRetryMaxField:
+ def test_default_is_zero(self) -> None:
+ step = PlaybookStep(name="x", type=StepType.NOTIFY)
+ assert step.retry_max == 0
+
+ def test_zero_is_accepted(self) -> None:
+ step = PlaybookStep(name="x", type=StepType.NOTIFY, retry_max=0)
+ assert step.retry_max == 0
+
+ def test_in_range_value_is_accepted(self) -> None:
+ step = PlaybookStep(name="x", type=StepType.NOTIFY, retry_max=3)
+ assert step.retry_max == 3
+
+ def test_negative_is_rejected(self) -> None:
+ with pytest.raises(ValidationError):
+ PlaybookStep(name="x", type=StepType.NOTIFY, retry_max=-1)
+
+ def test_at_absolute_max_is_accepted(self) -> None:
+ step = PlaybookStep(
+ name="x",
+ type=StepType.NOTIFY,
+ retry_max=ABSOLUTE_MAX_RETRIES,
+ )
+ assert step.retry_max == ABSOLUTE_MAX_RETRIES
+
+ def test_above_absolute_max_is_rejected(self) -> None:
+ with pytest.raises(ValidationError):
+ PlaybookStep(
+ name="x",
+ type=StepType.NOTIFY,
+ retry_max=ABSOLUTE_MAX_RETRIES + 1,
+ )
+
+ def test_pathologically_large_is_rejected(self) -> None:
+ # Without a cap, retry_max=1_000_000 with a retry-on-failure step
+ # would effectively never abort.
+ with pytest.raises(ValidationError):
+ PlaybookStep(name="x", type=StepType.NOTIFY, retry_max=1_000_000)
+
+
+# ---------------------------------------------------------------------------
+# Params bypass — Pydantic does NOT validate these. This pins the contract
+# that the runtime clamp (in bounds.clamp_timeout) is the only defence.
+# ---------------------------------------------------------------------------
+
+
+class TestParamsAreNotValidated:
+ """Document the field-vs-params split so anyone refactoring sees the gap."""
+
+ def test_params_timeout_seconds_is_not_validated_by_pydantic(self) -> None:
+ # An attacker could declare params.timeout_seconds=86400. Pydantic
+ # cannot stop them because ``params`` is an untyped ``dict[str, Any]``.
+ # This is the exact scenario that the runtime clamp in
+ # ``engine._handle_osquery_live_query`` must catch.
+ step = PlaybookStep(
+ name="x",
+ type=StepType.OSQUERY_LIVE_QUERY,
+ params={"timeout_seconds": 86_400}, # pathological value
+ )
+ # No exception — Pydantic happily accepts arbitrary params content.
+ assert step.params["timeout_seconds"] == 86_400
+ # ...so the runtime clamp is load-bearing. See
+ # ``test_osquery_live_query_step.py`` for the matching enforcement test.
+
+ def test_params_retry_max_is_not_validated_by_pydantic(self) -> None:
+ step = PlaybookStep(
+ name="x",
+ type=StepType.NOTIFY,
+ params={"retry_max": 1_000_000},
+ )
+ assert step.params["retry_max"] == 1_000_000
diff --git a/services/api/app/api/v1/endpoints/detection_proposals.py b/services/api/app/api/v1/endpoints/detection_proposals.py
index c78e7b7c2..cd51e47ac 100644
--- a/services/api/app/api/v1/endpoints/detection_proposals.py
+++ b/services/api/app/api/v1/endpoints/detection_proposals.py
@@ -156,8 +156,12 @@ class EvalAttachRequest(BaseModel):
max_regression_pp: float = Field(
default=1.0,
ge=0.0,
- le=100.0,
- description="Allowed MITRE accuracy regression vs baseline, in pp.",
+ le=50.0,
+ description=(
+ "Allowed MITRE accuracy regression vs baseline, in pp. Hard-capped "
+ "at 50 — anything above that is indistinguishable from disabling "
+ "the regression gate entirely."
+ ),
)
@@ -185,14 +189,22 @@ class RunEvalRequest(BaseModel):
max_regression_pp: float = Field(
default=1.0,
ge=0.0,
- le=100.0,
- description=("Allowed MITRE accuracy regression vs the active baseline, in percentage points. Forwarded to run_evals.py."),
+ le=50.0,
+ description=(
+ "Allowed MITRE accuracy regression vs the active baseline, in "
+ "percentage points. Forwarded to run_evals.py. Hard-capped at 50 "
+ "to keep the regression gate meaningful."
+ ),
)
timeout_seconds: int = Field(
default=180,
ge=10,
- le=900,
- description="Hard timeout for the eval subprocess.",
+ le=600,
+ description=(
+ "Hard timeout for the eval subprocess. Capped at 10 minutes — the "
+ "synthetic benchmark completes in under 60s on commodity hardware "
+ "so anything longer is almost certainly a stuck process."
+ ),
)
diff --git a/services/api/tests/api/v1/endpoints/test_detection_proposals_request_bounds.py b/services/api/tests/api/v1/endpoints/test_detection_proposals_request_bounds.py
new file mode 100644
index 000000000..3e4c116f1
--- /dev/null
+++ b/services/api/tests/api/v1/endpoints/test_detection_proposals_request_bounds.py
@@ -0,0 +1,152 @@
+"""Pydantic bounds tests for ``EvalAttachRequest`` / ``RunEvalRequest`` (H-7).
+
+These pin the *first* line of defence against runaway eval subprocesses:
+- ``max_regression_pp`` is capped at 50pp. Anything above that is
+ indistinguishable from disabling the regression gate, so it MUST be
+ rejected at the API boundary.
+- ``timeout_seconds`` is capped at 600s (10 min). The synthetic benchmark
+ completes in under 60s on commodity hardware; anything longer is almost
+ certainly a stuck process and we don't want callers pinning a worker for
+ 15 minutes.
+
+The matching server-side defence is in ``scripts/run_evals.py``, which also
+imposes a wall-clock timeout via ``subprocess.run(timeout=...)``.
+"""
+
+from __future__ import annotations
+
+import pytest
+from app.api.v1.endpoints.detection_proposals import (
+ EvalAttachRequest,
+ RunEvalRequest,
+)
+from pydantic import ValidationError
+
+# ---------------------------------------------------------------------------
+# max_regression_pp bounds (shared by both request models)
+# ---------------------------------------------------------------------------
+
+
+class TestMaxRegressionPpBounds:
+ def test_default_is_one_pp(self) -> None:
+ req = RunEvalRequest()
+ assert req.max_regression_pp == 1.0
+
+ def test_zero_is_inclusive(self) -> None:
+ req = RunEvalRequest(max_regression_pp=0.0)
+ assert req.max_regression_pp == 0.0
+
+ def test_negative_is_rejected(self) -> None:
+ with pytest.raises(ValidationError):
+ RunEvalRequest(max_regression_pp=-0.1)
+
+ def test_in_range_value_is_accepted(self) -> None:
+ req = RunEvalRequest(max_regression_pp=5.0)
+ assert req.max_regression_pp == 5.0
+
+ def test_at_cap_is_inclusive(self) -> None:
+ req = RunEvalRequest(max_regression_pp=50.0)
+ assert req.max_regression_pp == 50.0
+
+ def test_above_cap_is_rejected_run_eval(self) -> None:
+ with pytest.raises(ValidationError) as exc:
+ RunEvalRequest(max_regression_pp=50.1)
+ assert "less than or equal" in str(exc.value).lower()
+
+ def test_above_cap_is_rejected_eval_attach(self) -> None:
+ with pytest.raises(ValidationError):
+ EvalAttachRequest(eval_report={"x": 1}, max_regression_pp=99.9)
+
+ def test_pathological_value_is_rejected(self) -> None:
+ # 1000pp would effectively disable the gate entirely.
+ with pytest.raises(ValidationError):
+ RunEvalRequest(max_regression_pp=1000.0)
+
+ def test_eval_attach_in_range_value_is_accepted(self) -> None:
+ req = EvalAttachRequest(eval_report={"score": 0.9}, max_regression_pp=2.5)
+ assert req.max_regression_pp == 2.5
+
+
+# ---------------------------------------------------------------------------
+# timeout_seconds bounds (RunEvalRequest only)
+# ---------------------------------------------------------------------------
+
+
+class TestTimeoutSecondsBounds:
+ def test_default_is_180_seconds(self) -> None:
+ req = RunEvalRequest()
+ assert req.timeout_seconds == 180
+
+ def test_minimum_is_ten_seconds(self) -> None:
+ req = RunEvalRequest(timeout_seconds=10)
+ assert req.timeout_seconds == 10
+
+ def test_below_minimum_is_rejected(self) -> None:
+ # 1s is unreasonable — the runner can't even import its deps in 1s.
+ with pytest.raises(ValidationError):
+ RunEvalRequest(timeout_seconds=5)
+
+ def test_zero_is_rejected(self) -> None:
+ with pytest.raises(ValidationError):
+ RunEvalRequest(timeout_seconds=0)
+
+ def test_negative_is_rejected(self) -> None:
+ with pytest.raises(ValidationError):
+ RunEvalRequest(timeout_seconds=-1)
+
+ def test_in_range_value_is_accepted(self) -> None:
+ req = RunEvalRequest(timeout_seconds=300)
+ assert req.timeout_seconds == 300
+
+ def test_at_cap_is_inclusive(self) -> None:
+ req = RunEvalRequest(timeout_seconds=600)
+ assert req.timeout_seconds == 600
+
+ def test_above_cap_is_rejected(self) -> None:
+ with pytest.raises(ValidationError) as exc:
+ RunEvalRequest(timeout_seconds=601)
+ assert "less than or equal" in str(exc.value).lower()
+
+ def test_pathological_value_is_rejected(self) -> None:
+ # 1 day — the classic DoS payload against a subprocess endpoint.
+ with pytest.raises(ValidationError):
+ RunEvalRequest(timeout_seconds=86_400)
+
+ def test_legacy_900s_cap_is_now_rejected(self) -> None:
+ """The pre-H-7 cap was 900s; this test pins the new tighter cap.
+
+ Anyone bumping the cap back up needs to update this test and
+ explain why in the PR — the eval suite completes in <60s, so
+ anything longer than 10 min is almost certainly a stuck process.
+ """
+ with pytest.raises(ValidationError):
+ RunEvalRequest(timeout_seconds=900)
+
+
+# ---------------------------------------------------------------------------
+# Combined: both fields at extremes
+# ---------------------------------------------------------------------------
+
+
+class TestCombined:
+ def test_both_at_cap_accepted(self) -> None:
+ req = RunEvalRequest(max_regression_pp=50.0, timeout_seconds=600)
+ assert req.max_regression_pp == 50.0
+ assert req.timeout_seconds == 600
+
+ def test_both_above_cap_rejected(self) -> None:
+ with pytest.raises(ValidationError):
+ RunEvalRequest(max_regression_pp=51.0, timeout_seconds=601)
+
+ def test_eval_attach_does_not_accept_timeout_seconds(self) -> None:
+ """EvalAttachRequest is the *report-attach* path — there is no
+ subprocess to time out, so it should not accept timeout_seconds.
+
+ This is a regression guard: if someone refactors the two request
+ models into a shared base, they MUST consciously decide whether
+ EvalAttachRequest should also be timeout-bounded.
+ """
+ # Pydantic v2 ignores unknown fields by default unless extra='forbid'.
+ # Either behaviour is fine; what matters is the field isn't *active*.
+ req = EvalAttachRequest(eval_report={"score": 0.9})
+ assert not hasattr(req, "timeout_seconds")