:hedgehog: PostHog is the leading platform for building self-driving products. Our developer tools – AI observability, analytics, session replay, flags, experiments, error tracking, logs, and more – capture all the context agents need to diagnose problems, uncover opportunities, and ship fixes. Steer it all from Slack, web, desktop, or the MCP.
Python ab-testingai-analyticsanalyticscdpdata-warehouseexperimentsfeature-flagsjavascriptproduct-analyticspythonreactsession-replaysurveystypescriptweb-analytics
feat(alerts): hold the insight clickhouse slot only while the query runs (#125257)
Jimmy Zhucb114fb9b273
diff --git a/docs/internal/alerts-platform-temporal.md b/docs/internal/alerts-platform-temporal.mdindex 76fd8ec94b2a..4319a9367ef9 100644--- a/docs/internal/alerts-platform-temporal.md+++ b/docs/internal/alerts-platform-temporal.md@@ -535,8 +535,14 @@ A full logs backfill hit ClickHouse's per-user concurrent query limit and had to1. Copy one internal team with `--team-id`, then a small sample with `--sample-percent`, for example 5. The sample is chosen by alert id, so a rerun copies the same alerts and a larger percentage only adds alerts.2. Watch the parallel run's ClickHouse cost in `query_log`: its `client_query_id` starts with `alerts-platform-insight:`.3. Watch scheduler lag for `source=insight`, and the `capacity` skip reason on the platform's skipped-check counter. Capacity skips mean ClickHouse refused the query for load.-4. Widen the sample only while both stay flat. `ALERTS_PLATFORM_INSIGHT_MAX_INFLIGHT_EVALUATIONS` caps the concurrent checks whatever the sample size. A batch refills the slots its finished checks free, in rounds within its tick, so the cap limits concurrency rather than checks per minute.+4. Widen the sample only while both stay flat. Two pools bound the parallel run, whatever the sample size:+ - `ALERTS_PLATFORM_INSIGHT_MAX_INFLIGHT_EVALUATIONS` caps the queries that run in ClickHouse at once. A check holds one of these slots only while its query runs. A check that gets no slot within 45 seconds records nothing and stays due for a later tick.+ - `ALERTS_PLATFORM_INSIGHT_MAX_INFLIGHT_CHECKS` caps the checks admitted at once, and defaults to the query cap. An admitted check also locks itself, so a later run never starts the same check while it runs. An admitted check holds an evaluation worker slot until it ends, including while it waits for a query slot, so keep this cap inside the workers' activity slots.++ Each worker reads both caps from its own environment, so set them to the same values on every worker that runs insight checks, the AI detector worker included.+ A batch refills the check slots its finished checks free, in rounds within its tick, so the caps limit concurrency rather than checks per minute.`ALERTS_PLATFORM_INSIGHT_KEYS_PER_TICK` bounds how many teams' batches start each minute, so a larger sample can need it raised. Sixty times its value must cover the batch keys due per hour. Otherwise the checks due at the top of the hour start later and later, while ClickHouse stays idle.+5. Raise that cap from its default of 10 only while the daily count of refused queries, `exception_code = 202` in `query_log`, stays flat for both `alerts_platform_insight` and the user that production insight alerts query as. The first shows the parallel run's own contention. The second shows whether it reaches production through the server-wide limit. Do not size it from per-second concurrency, which overcounts because short queries that run back to back inside one second read as concurrent. Code 202 also covers the server-wide limit, so a rise is a reason to look rather than proof that the cap caused it.To stop either parallel run, logs or insight, pass `--disable` to its backfill, with `--team-id` to stop one team.diff --git a/posthog/settings/temporal.py b/posthog/settings/temporal.pyindex 37635fba6a92..32d85ba7250a 100644--- a/posthog/settings/temporal.py+++ b/posthog/settings/temporal.py@@ -411,8 +411,8 @@ ALERTS_PLATFORM_SHARED_ORCHESTRATION_TASK_QUEUE = _set_isolated_temporal_task_qu)ALERTS_PLATFORM_EVALUATION_TASK_QUEUE = _set_isolated_temporal_task_queue("alerts-platform-evaluation-task-queue")ALERTS_PLATFORM_DELIVERY_TASK_QUEUE = _set_isolated_temporal_task_queue("alerts-platform-delivery-task-queue")-# The platform's parallel insight checks allowed to run at once, across every team. A pool of its-# own, so the parallel run never takes a slot from `ALERTS_MAX_INFLIGHT_EVALUATIONS`.+# The platform's insight queries allowed to run in ClickHouse at once, across every team. A pool of+# its own, so the parallel run never takes a slot from `ALERTS_MAX_INFLIGHT_EVALUATIONS`.ALERTS_PLATFORM_INSIGHT_MAX_INFLIGHT_EVALUATIONS: int = get_from_env("ALERTS_PLATFORM_INSIGHT_MAX_INFLIGHT_EVALUATIONS", 10, type_cast=int)@@ -420,6 +420,15 @@ if ALERTS_PLATFORM_INSIGHT_MAX_INFLIGHT_EVALUATIONS <= 0:raise ImproperlyConfigured("ALERTS_PLATFORM_INSIGHT_MAX_INFLIGHT_EVALUATIONS must be a positive integer, or no platform insight check starts")+# A check holds a worker slot while it waits for a query slot, so keep this inside the evaluation+# workers' activity slots. A value below the query pool is raised to it rather than refused, because+# every process loads these settings and a platform value must not stop other workers from starting.+ALERTS_PLATFORM_INSIGHT_MAX_INFLIGHT_CHECKS: int = max(+ get_from_env(+ "ALERTS_PLATFORM_INSIGHT_MAX_INFLIGHT_CHECKS", ALERTS_PLATFORM_INSIGHT_MAX_INFLIGHT_EVALUATIONS, type_cast=int+ ),+ ALERTS_PLATFORM_INSIGHT_MAX_INFLIGHT_EVALUATIONS,+)# Insight batch keys the platform starts per tick, across every team. A key is one team's checks due# in one slot. With a tick a minute, 60 times this must cover the keys due per hour, or hourly alerts# fall behind their cadence. The pool above bounds concurrent queries, not this. A key that finds thediff --git a/products/alerts/backend/evaluation/dispatcher.py b/products/alerts/backend/evaluation/dispatcher.pyindex 57d0bf1a7ff7..e56a7b9abf3a 100644--- a/products/alerts/backend/evaluation/dispatcher.py+++ b/products/alerts/backend/evaluation/dispatcher.py@@ -1,5 +1,5 @@-from collections.abc import Iterator-from contextlib import contextmanager+from collections.abc import Callable, Iterator+from contextlib import AbstractContextManager, contextmanager, nullcontextfrom dataclasses import replacefrom typing import Any@@ -85,12 +85,18 @@ def owner_sql_errors(query: object) -> Iterator[None]:raise AlertExtractionError(str(err)) from err+# Entered around the ClickHouse query alone, so a caller can bound its concurrent queries without+# counting the insight load, the evaluation or a detector's model call.+QuerySlot = Callable[[], AbstractContextManager[object]]++def run_extractor(extractor: Extractor,alert: AlertConfiguration,insight: Insight,query: object,execution_mode: ExecutionMode,+ query_slot: QuerySlot = nullcontext,) -> ExtractionResult:"""Run an extractor, routing a query the runner refuses to the auto-disable path.@@ -109,7 +115,7 @@ def run_extractor((see ``owner_sql_errors``)."""try:- with owner_sql_errors(query):+ with query_slot(), owner_sql_errors(query):return extractor.extract(alert, insight, query, execution_mode)except DRFValidationError as err:if getattr(err, "served_from_query_failure_cache", False):@@ -118,7 +124,12 @@ def run_extractor(def check_detector_alert(- alert: AlertConfiguration, insight: Insight, query: object, *, evaluation_id: str | None = None+ alert: AlertConfiguration,+ insight: Insight,+ query: object,+ *,+ evaluation_id: str | None = None,+ query_slot: QuerySlot = nullcontext,) -> AlertEvaluationResult:"""Route a detector (anomaly) alert to its kind's detector extractor, then score the series.@@ -132,7 +143,9 @@ def check_detector_alert(detector_extractor = DETECTOR_EXTRACTORS.get(kind)if detector_extractor is None:raise NotImplementedError(f"AlertCheckError: Detector alerts for {kind} are not supported yet")- result = run_extractor(detector_extractor, alert, insight, query, _resolve_execution_mode(alert, kind, query))+ result = run_extractor(+ detector_extractor, alert, insight, query, _resolve_execution_mode(alert, kind, query), query_slot+ )return describe_delayed_evaluation(evaluate_with_detector(result, detector_config, insight=insight, alert=alert, evaluation_id=evaluation_id),result,@@ -171,7 +184,9 @@ def _resolved_query(alert: AlertConfiguration) -> Iterator[_ResolvedQuery]:yield _ResolvedQuery(insight=insight, query=query, kind=kind)-def _check_alert_for_insight(alert: AlertConfiguration, *, evaluation_id: str | None = None) -> AlertEvaluationResult:+def _check_alert_for_insight(+ alert: AlertConfiguration, *, evaluation_id: str | None = None, query_slot: QuerySlot = nullcontext+) -> AlertEvaluationResult:"""Dispatch an alert to its insight-kind extractor, then run the shared comparator.If ``detector_config`` is set, routes through the anomaly-detector registry (one extractor per@@ -181,9 +196,11 @@ def _check_alert_for_insight(alert: AlertConfiguration, *, evaluation_id: str |"""with _resolved_query(alert) as resolved:if alert.detector_config:- return check_detector_alert(alert, resolved.insight, resolved.query, evaluation_id=evaluation_id)+ return check_detector_alert(+ alert, resolved.insight, resolved.query, evaluation_id=evaluation_id, query_slot=query_slot+ )- extracted = _extract_for_threshold(alert, resolved.insight, resolved.query, resolved.kind)+ extracted = _extract_for_threshold(alert, resolved.insight, resolved.query, resolved.kind, query_slot)if extracted is None:return AlertEvaluationResult(value=0, breaches=[])return describe_delayed_evaluation(@@ -199,7 +216,7 @@ class _ThresholdExtraction:def _extract_for_threshold(- alert: AlertConfiguration, insight: Insight, query: object, kind: NodeKind+ alert: AlertConfiguration, insight: Insight, query: object, kind: NodeKind, query_slot: QuerySlot = nullcontext) -> _ThresholdExtraction | None:"""Run a threshold alert's query. None when the alert has no bounds, so there is nothing to breach."""extractor = EXTRACTORS.get(kind)@@ -212,7 +229,7 @@ def _extract_for_threshold(return Nonecondition = AlertCondition.model_validate(alert.condition)- result = run_extractor(extractor, alert, insight, query, _resolve_execution_mode(alert, kind, query))+ result = run_extractor(extractor, alert, insight, query, _resolve_execution_mode(alert, kind, query), query_slot)return _ThresholdExtraction(result=result, condition=condition, threshold=threshold)@@ -242,7 +259,9 @@ class PerSeriesEvaluation:truncated: bool = False-def check_alert_per_series(alert: AlertConfiguration) -> PerSeriesEvaluation | AlertEvaluationResult:+def check_alert_per_series(+ alert: AlertConfiguration, *, query_slot: QuerySlot = nullcontext+) -> PerSeriesEvaluation | AlertEvaluationResult:"""Each series' own verdict for a threshold alert, rather than one verdict for the insight.`check_alert_for_insight` stops at the first series that breaches, because one alert holds one@@ -254,7 +273,7 @@ def check_alert_per_series(alert: AlertConfiguration) -> PerSeriesEvaluation | Araise AlertExtractionError("Per-series evaluation supports threshold alerts only")try:with _resolved_query(alert) as resolved:- extracted = _extract_for_threshold(alert, resolved.insight, resolved.query, resolved.kind)+ extracted = _extract_for_threshold(alert, resolved.insight, resolved.query, resolved.kind, query_slot)except DelayedEvaluationUnavailable as err:return AlertEvaluationResult(value=None, breaches=[], skipped_reason=str(err))if extracted is None:@@ -280,9 +299,11 @@ def check_alert_per_series(alert: AlertConfiguration) -> PerSeriesEvaluation | A)-def check_alert_for_insight(alert: AlertConfiguration, *, evaluation_id: str | None = None) -> AlertEvaluationResult:+def check_alert_for_insight(+ alert: AlertConfiguration, *, evaluation_id: str | None = None, query_slot: QuerySlot = nullcontext+) -> AlertEvaluationResult:try:- return _check_alert_for_insight(alert, evaluation_id=evaluation_id)+ return _check_alert_for_insight(alert, evaluation_id=evaluation_id, query_slot=query_slot)except DelayedEvaluationUnavailable as err:if is_llm_detector_config(alert.detector_config):raise LLMDetectorUnavailableError(str(err)) from errdiff --git a/products/alerts/backend/evaluation/test/test_dispatcher.py b/products/alerts/backend/evaluation/test/test_dispatcher.pyindex de4d9c53fc6f..e108591532a0 100644--- a/products/alerts/backend/evaluation/test/test_dispatcher.py+++ b/products/alerts/backend/evaluation/test/test_dispatcher.py@@ -132,14 +132,16 @@ def test_per_series_judges_every_breakdown_value_where_the_alert_stops_at_the_fi)alert.insight.query = {"kind": "TrendsQuery"}extractor = _Extractor(ExtractionResult(series=series, is_breakdown=True))+ query_slot = MagicMock(return_value=nullcontext())with (patch.dict(EXTRACTORS, {NodeKind.TRENDS_QUERY: extractor}),patch("products.alerts.backend.evaluation.dispatcher.upgrade_insight", return_value=nullcontext()),patch("products.alerts.backend.evaluation.dispatcher.validate_evaluation_delay"),patch("products.alerts.backend.evaluation.dispatcher._resolve_execution_mode", return_value=IF_STALE),):- evaluated = check_alert_per_series(alert)+ evaluated = check_alert_per_series(alert, query_slot=query_slot)+ query_slot.assert_called_once_with()assert isinstance(evaluated, PerSeriesEvaluation)assert [(s.label, s.result.value, bool(s.result.breaches)) for s in evaluated.series] == [("Chrome", 150.0, True),diff --git a/products/alerts/backend/platform_source_cycle.py b/products/alerts/backend/platform_source_cycle.pyindex 8b240166c0fa..ad0fef4bbeb2 100644--- a/products/alerts/backend/platform_source_cycle.py+++ b/products/alerts/backend/platform_source_cycle.py@@ -18,9 +18,10 @@ same set of checks.import timeimport asyncio-from collections.abc import Collection, Sequence+from collections.abc import Collection, Iterator, Sequence+from contextlib import contextmanagerfrom dataclasses import replace-from datetime import UTC, datetime+from datetime import UTC, datetime, timedeltafrom typing import Any, Finalfrom django.conf import settings@@ -28,12 +29,14 @@ from django.conf import settingsimport structlogfrom posthog.clickhouse.client.connection import ClickHouseUser+from posthog.clickhouse.client.limit import ConcurrencyLimitExceeded, ConcurrencySlot, RateLimitfrom posthog.clickhouse.query_tagging import Feature, Product, tag_queriesfrom posthog.errors import CH_TRANSIENT_ERRORS, QueryErrorCategory, classify_query_errorfrom posthog.schema_enums import AlertCalculationIntervalfrom posthog.tasks.alerts.schedule_restriction import is_utc_datetime_blockedfrom posthog.tasks.alerts.utils import AlertEvaluationResult, render_alert_insight_chart, skip_because_of_weekendfrom posthog.temporal.alerts.admission import admit_evaluation_slots, release_evaluation_slot+from posthog.temporal.alerts.retry_policy import alert_timeoutsfrom products.alerts.backend.evaluation import check_alert_for_insightfrom products.alerts.backend.evaluation.contract import AlertExtractionError@@ -101,8 +104,33 @@ from products.cdp.backend.internal_events import LEGACY_INSIGHT_ALERT_EVENTlogger = structlog.get_logger(__name__)+# A member also locks its check, so a later run cannot start the same check while it runs.INFLIGHT_KEY = "alerts:platform:insight:evaluations:inflight"+QUERY_INFLIGHT_KEY = "alerts:platform:insight:queries:inflight"CHART_RENDER_INFLIGHT_KEY = "alerts:platform:insight:chart-renders:inflight"+# Under the one-minute tick, so a check that gave up is not still waiting when the next tick admits it.+QUERY_SLOT_WAIT_SECONDS = 45.0+QUERY_SLOT_POLL_SECONDS = 0.5+# Outlasts any query, so only a slot whose worker died expires before its release.+QUERY_SLOT_LEASE_SECONDS = int((alert_timeouts(None).evaluate_start_to_close + timedelta(minutes=2)).total_seconds())+++class ClickHouseBudgetFull(Exception):+ """No ClickHouse slot came free in time. The check records nothing, so it stays due."""+++# The size comes from the setting on each call, so the limit follows the live value.+_QUERY_LIMIT = RateLimit(+ max_concurrency=1,+ limit_name="alerts_platform_insight",+ get_task_name=lambda *_, **__: QUERY_INFLIGHT_KEY,+ get_task_id=lambda holder, **_: holder,+ ttl=QUERY_SLOT_LEASE_SECONDS,+ retry=QUERY_SLOT_POLL_SECONDS,+ retry_timeout=QUERY_SLOT_WAIT_SECONDS,+ retry_jitter=True,+ allow_team_bypass=False,+)class ChartRenderPoolFull(Exception):@@ -238,13 +266,13 @@ def plan_insight_batch(slot,cutoff,exclude=exclude,- limit=settings.ALERTS_PLATFORM_INSIGHT_MAX_INFLIGHT_EVALUATIONS,+ limit=settings.ALERTS_PLATFORM_INSIGHT_MAX_INFLIGHT_CHECKS,)if not candidates:return ()admitted = admit_evaluation_slots(list(candidates),- limit=settings.ALERTS_PLATFORM_INSIGHT_MAX_INFLIGHT_EVALUATIONS,+ limit=settings.ALERTS_PLATFORM_INSIGHT_MAX_INFLIGHT_CHECKS,expires_at=expires_at,key=INFLIGHT_KEY,)@@ -278,17 +306,54 @@ def readmit_retried_check(the two attempts as one check that either can release."""_stop_query(team_id, configuration_id, previous_query_id)+ # The query ID names the attempt that held this slot, so its slot frees now rather than at its lease.+ _QUERY_LIMIT.release(ConcurrencySlot(running_tasks_key=QUERY_INFLIGHT_KEY, task_id=previous_query_id))release_evaluation_slot(configuration_id, held_until=attempt_slot_token(held_until, attempt - 1), key=INFLIGHT_KEY)token = attempt_slot_token(held_until, attempt)admitted = admit_evaluation_slots([configuration_id],- limit=settings.ALERTS_PLATFORM_INSIGHT_MAX_INFLIGHT_EVALUATIONS,+ limit=settings.ALERTS_PLATFORM_INSIGHT_MAX_INFLIGHT_CHECKS,expires_at=token,key=INFLIGHT_KEY,)return token if admitted else None+class _ClickHouseSlot:+ """Holds a ClickHouse slot around one attempt's query, and counts how long it waited for one.++ Raises `ClickHouseBudgetFull` when no slot comes free in time. `holder` names one attempt of one+ check, so a retry never takes over the slot of the attempt before it.+ """++ def __init__(self, holder: str) -> None:+ self.holder = holder+ self.waited_ms = 0++ @contextmanager+ def __call__(self) -> Iterator[None]:+ waiting_since = time.monotonic()+ try:+ slot = _QUERY_LIMIT.use(self.holder, limit=settings.ALERTS_PLATFORM_INSIGHT_MAX_INFLIGHT_EVALUATIONS)+ except ConcurrencyLimitExceeded as error:+ raise ClickHouseBudgetFull() from error+ except Exception as error:+ # Redis is unreachable. The check stays due rather than recording a failure that is not the alert's.+ logger.exception("Could not take a ClickHouse slot for a platform insight check", holder=self.holder)+ raise ClickHouseBudgetFull() from error+ finally:+ self.waited_ms += int((time.monotonic() - waiting_since) * 1000)+ try:+ yield+ finally:+ if slot is not None:+ try:+ _QUERY_LIMIT.release(slot)+ except Exception:+ # Best effort, so a Redis failure never replaces the query's result. The lease frees the slot.+ logger.exception("Could not release a ClickHouse slot", holder=self.holder)++def _stop_query(team_id: int, configuration_id: str, query_id: str) -> None:# Imported here: at module level it changes how many queries the access-control tests count.from posthog.clickhouse.cancel import cancel_query_on_cluster # noqa: PLC0415@@ -334,7 +399,8 @@ def evaluate_insight_check(Returns the outcome with its delivery, if it asks for one. Returns None when the configurationis no longer due, because an earlier attempt already recorded it or it was edited out of the- batch, or when its detector no longer matches the worker it was routed to. Writes nothing.+ batch, when its detector no longer matches the worker it was routed to, or when no ClickHouse+ slot came free in time. Writes nothing.`evaluation_id` names the check across its attempts, so the AI detector reuses a verdict anearlier attempt paid for. An AI detector that reaches no verdict raises@@ -564,11 +630,14 @@ def _decide(alert_calculation_interval=alert.calculation_interval,alert_config_type=(alert.config or {}).get("type"),)+ query_slot = _ClickHouseSlot(f"{evaluation_id}:{attempt}")if check.grouping.mode != GroupingMode.SINGLE:- return _decide_grouped(check, snapshot, alert, now=now)+ return _decide_grouped(check, snapshot, alert, now=now, query_slot=query_slot)started_at = time.monotonic()try:- result = check_alert_for_insight(alert, evaluation_id=evaluation_id)+ result = check_alert_for_insight(alert, evaluation_id=evaluation_id, query_slot=query_slot)+ except ClickHouseBudgetFull:+ return Noneexcept LLMDetectorUnavailableError:if not final_attempt:# As production: a judge that reached no verdict gets another attempt, and only the@@ -588,7 +657,7 @@ def _decide(return _broken(check, snapshot, alert, now=now, message=str(error))except Exception as error:return _query_failed(check, snapshot, alert, error, now=now)- duration_ms = int((time.monotonic() - started_at) * 1000)+ duration_ms = int((time.monotonic() - started_at) * 1000) - query_slot.waited_msif result.skipped_reason is not None:return _skipped(check, snapshot, now=now)@@ -705,8 +774,13 @@ GROUPED_INSIGHT_ALERT_POLICY: Final = replace(INSIGHT_ALERT_POLICY, mute_gates_ndef _decide_grouped(- check: PlatformAlertCheckInput, snapshot: AlertSnapshot, alert: AlertConfiguration, *, now: datetime-) -> PlatformAlertOutcome:+ check: PlatformAlertCheckInput,+ snapshot: AlertSnapshot,+ alert: AlertConfiguration,+ *,+ now: datetime,+ query_slot: _ClickHouseSlot,+) -> PlatformAlertOutcome | None:"""One check of a breakdown alert, judged per breakdown value through the platform's decision.An open value the query no longer returns is judged as not breaching, so it can resolve. When@@ -719,14 +793,16 @@ def _decide_grouped()started_at = time.monotonic()try:- evaluated = check_alert_per_series(alert)+ evaluated = check_alert_per_series(alert, query_slot=query_slot)+ except ClickHouseBudgetFull:+ return Noneexcept AlertExtractionError as error:return _broken(check, snapshot, alert, now=now, message=str(error))except Exception as error:return _query_failed(check, snapshot, alert, error, now=now)if not isinstance(evaluated, PerSeriesEvaluation):return _skipped(check, snapshot, now=now)- duration_ms = int((time.monotonic() - started_at) * 1000)+ duration_ms = int((time.monotonic() - started_at) * 1000) - query_slot.waited_msdecision = decide_groups(check,diff --git a/products/alerts/backend/test/test_platform_source_cycle.py b/products/alerts/backend/test/test_platform_source_cycle.pyindex 18b60905d340..3701a3eab886 100644--- a/products/alerts/backend/test/test_platform_source_cycle.py+++ b/products/alerts/backend/test/test_platform_source_cycle.py@@ -1,4 +1,6 @@import time+from collections.abc import Callable+from contextlib import AbstractContextManagerfrom datetime import UTC, datetime, timedeltafrom typing import Anyfrom uuid import UUID@@ -42,6 +44,7 @@ from products.alerts.backend.platform_source_cycle import (MAX_DETECTOR_POINTS,MAX_RATIONALE_CHARS,QUERY_ID_PREFIX,+ QUERY_INFLIGHT_KEY,ChartRenderPoolFull,evaluate_insight_check,fail_insight_check,@@ -79,8 +82,8 @@ CUTOFF = datetime(2026, 9, 16, 10, tzinfo=UTC)class TestPlatformInsightEvaluation(APIBaseTest):def setUp(self) -> None:super().setUp()- get_client().delete(INFLIGHT_KEY, CHART_RENDER_INFLIGHT_KEY)- self.addCleanup(get_client().delete, INFLIGHT_KEY, CHART_RENDER_INFLIGHT_KEY)+ get_client().delete(INFLIGHT_KEY, QUERY_INFLIGHT_KEY, CHART_RENDER_INFLIGHT_KEY)+ self.addCleanup(get_client().delete, INFLIGHT_KEY, QUERY_INFLIGHT_KEY, CHART_RENDER_INFLIGHT_KEY)self.organization.available_product_features = [{"key": AvailableFeature.HIGH_FREQUENCY_ALERTS, "name": "High-frequency alerts"},{"key": AvailableFeature.REAL_TIME_ALERTS, "name": "Real-time alerts"},@@ -248,6 +251,40 @@ class TestPlatformInsightEvaluation(APIBaseTest):assert render.call_count == 2+ @override_settings(ALERTS_PLATFORM_INSIGHT_MAX_INFLIGHT_EVALUATIONS=1)+ def test_a_check_holds_a_clickhouse_slot_only_while_its_query_runs(self) -> None:+ configuration = self._copy(self._alert())+ slot = slot_of(configuration.next_check_at, CUTOFF)+ expires_at = time.time() + 3600+ (admitted,) = plan_insight_batch(self.team.id, slot, CUTOFF, expires_at=expires_at)+ other_query_until = time.time() + 60+ admit_evaluation_slots(["other-query"], limit=1, expires_at=other_query_until, key=QUERY_INFLIGHT_KEY)+ held_during_query: list[set[str]] = []++ def query(+ *args: Any, query_slot: Callable[[], AbstractContextManager[object]], **kwargs: Any+ ) -> AlertEvaluationResult:+ with query_slot():+ held_during_query.append(inflight_alert_ids(QUERY_INFLIGHT_KEY))+ return AlertEvaluationResult(value=1.0, breaches=[])++ with (+ patch(f"{_MODULE}.check_alert_for_insight", side_effect=query),+ patch(f"{_MODULE}._QUERY_LIMIT.retry_timeout", 0),+ ):+ starved = evaluate_insight_check(+ self.team.id, slot, CUTOFF, admitted, held_until=expires_at, evaluation_id="test"+ )+ release_evaluation_slot("other-query", held_until=other_query_until, key=QUERY_INFLIGHT_KEY)+ evaluation = evaluate_insight_check(+ self.team.id, slot, CUTOFF, admitted, held_until=expires_at, evaluation_id="test"+ )++ assert starved is None+ assert evaluation is not None and evaluation.outcomes[0].failure is None+ assert held_during_query == [{"test:1"}]+ assert inflight_alert_ids(QUERY_INFLIGHT_KEY) == set()+@parameterized.expand([(@@ -662,13 +699,14 @@ class TestPlatformInsightEvaluation(APIBaseTest):(self.team.id, f"{QUERY_ID_PREFIX}run:1:2"),]- @override_settings(ALERTS_PLATFORM_INSIGHT_MAX_INFLIGHT_EVALUATIONS=1)+ @override_settings(ALERTS_PLATFORM_INSIGHT_MAX_INFLIGHT_CHECKS=1)def test_a_retry_stops_the_timed_out_query_and_runs_only_inside_the_pool(self) -> None:held = self._copy(self._alert(), id=UUID(int=1))waiting = self._copy(self._alert(), id=UUID(int=2))slot = slot_of(held.next_check_at, CUTOFF)expires_at = time.time() + 3600assert plan_insight_batch(self.team.id, slot, CUTOFF, expires_at=expires_at) == (str(held.id),)+ admit_evaluation_slots(["run:1:1"], limit=1, expires_at=expires_at, key=QUERY_INFLIGHT_KEY)with patch("posthog.clickhouse.cancel.cancel_query_on_cluster") as cancel:retried = readmit_retried_check(@@ -682,9 +720,10 @@ class TestPlatformInsightEvaluation(APIBaseTest):assert retried is not None and over_the_limit is Noneassert inflight_alert_ids(INFLIGHT_KEY) == {str(held.id)}+ assert inflight_alert_ids(QUERY_INFLIGHT_KEY) == set()cancel.assert_any_call(self.team.id, "alerts-platform-insight:run:1:1")- @override_settings(ALERTS_PLATFORM_INSIGHT_MAX_INFLIGHT_EVALUATIONS=1)+ @override_settings(ALERTS_PLATFORM_INSIGHT_MAX_INFLIGHT_CHECKS=1)def test_the_pool_defers_what_it_cannot_hold_until_a_check_frees_its_slot(self) -> None:first = self._copy(self._alert(), id=UUID(int=1))second = self._copy(self._alert(calculation_interval=AlertCalculationInterval.REAL_TIME.value), id=UUID(int=2))