:hedgehog: PostHog is the leading platform for building self-driving products. Our developer tools – AI observability, analytics, session replay, flags, experiments, error tracking, logs, and more – capture all the context agents need to diagnose problems, uncover opportunities, and ship fixes. Steer it all from Slack, web, desktop, or the MCP.
Python ab-testingai-analyticsanalyticscdpdata-warehouseexperimentsfeature-flagsjavascriptproduct-analyticspythonreactsession-replaysurveystypescriptweb-analytics
fix(autoresearch): bound the daily sweep fan-out and back off on capacity errors (#119220)
posthog[bot]6b1a2fcdc5b8
diff --git a/products/autoresearch/backend/inference/AGENTS.md b/products/autoresearch/backend/inference/AGENTS.mdindex e1e6c5024e74..ee151f199045 100644--- a/products/autoresearch/backend/inference/AGENTS.md+++ b/products/autoresearch/backend/inference/AGENTS.md@@ -42,15 +42,16 @@ The cardinal rule: **inference never refits a persisted model.** A bundle is fitEvery HogQL query goes through `run_hogql()` in `../query.py` with a `QueryContext`: a limit context, which sets `max_execution_time`, and a ClickHouse workload. The entry point chooses it and passes it down explicitly. No code decides it from where it runs.-| Path | Context | Limit and workload |-| ----------------------------------------------------------------------------------------------------------------------- | --------------------------------- | ---------------------------------------------------------------------- |-| Scoring from `activity_run_inference` (sweep and manual `/score`), both routes, with their anchor and population counts | `BATCH_QUERY` | `QUERY_ASYNC` (`HOGQL_INCREASED_MAX_EXECUTION_TIME`, 600 s), `OFFLINE` |-| Online validation from `activity_run_validation` | `BATCH_QUERY` | `QUERY_ASYNC`, `OFFLINE` |-| `check_scorability()` after the champion fit at training completion | `BATCH_QUERY` | `QUERY_ASYNC`, `OFFLINE` |-| `fit_champion_model()` at training completion, `materialize-features`, `/validate`, `/validate_online` | `INTERACTIVE_QUERY` (the default) | `QUERY` (60 s), `DEFAULT` |-| Management commands | `INTERACTIVE_QUERY` (the default) | `QUERY`, `DEFAULT` |--Scoring and validation are batch work that nobody waits on, so they get the batch limit. A path inside a web request keeps the interactive limit, because a 600 s query would outlive the request.+| Path | Context | Limit and workload |+| --------------------------------------------------------------------------------------------------------------------------------------------------------- | --------------------------------- | ---------------------------------------------------------------------------------------- |+| Scoring from `activity_run_inference` (sweep and manual `/score`) and `activity_run_shadow_scoring`, both routes, with their anchor and population counts | `SCORING_QUERY` | `QUERY_ASYNC` (`HOGQL_INCREASED_MAX_EXECUTION_TIME`, 600 s), `OFFLINE`, capacity backoff |+| Online validation from `activity_run_validation` | `BATCH_QUERY` | `QUERY_ASYNC`, `OFFLINE` |+| `check_scorability()` after the champion fit at training completion | `BATCH_QUERY` | `QUERY_ASYNC`, `OFFLINE` |+| `fit_champion_model()` at training completion, `materialize-features`, `/validate`, `/validate_online` | `INTERACTIVE_QUERY` (the default) | `QUERY` (60 s), `DEFAULT` |+| Management commands | `INTERACTIVE_QUERY` (the default) | `QUERY`, `DEFAULT` |++Scoring and validation are batch work that nobody waits on, so they get the batch limit.+`SCORING_QUERY` is `BATCH_QUERY` plus a capacity backoff: `run_hogql()` runs a query that ClickHouse refuses as `RATE_LIMITED` (`ClickHouseAtCapacity`, `ConcurrencyLimitExceeded`, `ClickHouseClusterMemoryLimitExceeded`) again after about 30 s, 60 s, and 120 s, with jitter and with `ClickHouseAtCapacity.wait` as the floor, and then raises. The activity's `HeartbeaterSync` heartbeats during the waits. The other contexts do not retry, because a request path must not wait minutes. A path inside a web request keeps the interactive limit, because a 600 s query would outlive the request.Training stays interactive on purpose: feature SQL that materializes within 60 s at training has ten times that budget at scoring, so the stricter bar is at the gate. When a request path moves to Temporal, give it `BATCH_QUERY`.The activity timeouts in `../temporal/workflows.py` cover several batch queries in sequence at the full limit. Keep them that way when you add a query to either path.@@ -71,8 +72,10 @@ After a live champion run completes, `_score_shadow_set()` in `scoring.py` score- **One feature query per distinct `features.sql`.** The digest of the bundle's SQL (`features_sql_digest()`) keys the materialized rows. The champion's rows serve a shadow model with the same SQL, and each other SQL runs once. A failed materialization is cached too. A result is released after the last model that uses it, so the phase does not hold one result per distinct SQL.- **Person-less events.** A shadow event has `$autoresearch_model_role` `shadow`, no `$set`, and no person processing, and its `distinct_id` is unique per model and person (`_shadow_distinct_id()`), because ingestion deduplicates on timestamp, distinct_id, token, and event name, not on the UUID. Validation keys on `$autoresearch_person_id`. A shadow score never reaches the output person property. Right before its batch goes out, a shadow model checks that it is still in the set (`_require_still_in_shadow_set()`), in place of `_require_still_champion()`.- **Isolated runs.** Each shadow model records its own inference run, with `metrics["shadow"]` set and `scheduled` false, so `find_unscorable_champion()` never reads it, even after the model becomes the champion. A shadow failure fails only that run. Repeatable shadow failures on `UNSCORABLE_AFTER_FAILED_DAYS` scoring days remove the model from the shadow set (`models_failing_repeatably_in_shadow()`). The champion's run records `metrics["shadow_models"]`: the models that completed, failed, or were skipped.+- **Deferred in the daily sweep.** The sweep scores the champion with `defer_shadows`, and scores the shadow set later in its own activity, through `score_shadow_set_for_run()`, after every champion of the sweep has finished. A live run records `metrics["scored_persons_digest"]` (`persons_digest()` of the people it scored), and the deferred phase pairs each feature result against that digest, so the person list never crosses the Temporal payload. The deferred phase queries the champion's SQL again, because the champion's rows are gone with its activity. It scores nothing when the prediction date has passed. A model promoted since the champion's run does not score as a shadow. Manual scoring and `autoresearch_score` still score the shadow set in the same call.- **The rolling ranking ignores shadow events.** `build_inference_anchors_sql()` counts only non-shadow predictions as scores, so shadow scoring does not change who the champion scores next.-- **Cost bound.** No new shadow model starts after `SHADOW_TIME_BUDGET_S` (20 minutes). The rest are skipped for that cadence. The inference activity timeout covers the champion's worst case, the budget, and one more model.+- **Cost bound.** No new shadow model starts after `SHADOW_TIME_BUDGET_S` (20 minutes). The rest are skipped for that cadence. The inference activity timeout covers the champion's worst case, the budget, and one more model. The shadow activity timeout covers the budget and one more model.+- **Timing.** A live champion run records `metrics["scoring_seconds"]`, completed or failed. The deferred phase records `metrics["shadow_seconds"]` on the champion's run. A sweep run also records `metrics["coordinator_concurrency"]`.**Scores are prior-corrected once, in `score_population()`.** A model fitted on a case-control sample (see `../dataset/AGENTS.md`) overstates the odds by `1 / r`, so `corrected_probability()` applies `logit(p) + log(r)`. `$autoresearch_p_y` and the output person property carry the corrected value, `$autoresearch_p_y_raw` the model's own score, and the run records `r` in `AutoresearchRun.negative_sample_rate`.A bundle's `r` is `AutoresearchModel.negative_sample_rate`, which `fit_champion_model()` saves before it writes `model.pkl`. A recipe-only champion measures its sample at each fit and uses that rate. The stub formula never saw a sample, so its `r` is 1.0. Never correct in a scoring route; a second correction would shrink every probability again.@@ -92,7 +95,7 @@ The prediction event is itself an event on the person, so every live cadence add3. If the champion has an `artifact_prefix`, materialize features into a sandbox and run `predict.py`. Otherwise compile the recorded recipe and score in-process.4. Map `person_id` → `distinct_id` and emit one event each.5. Record an `AutoresearchRun` for the execution. A run the daily sweep starts has `scheduled` set. A failed run records `metrics["failure_kind"]` from `classify_failure()` in `failures.py`: `limit_exceeded`, `query_failed` and `model_load_failed` fail again with the same champion, and `other` (transport, capture, cluster capacity, anything unknown) can pass on a retry. Promotion reads the scheduled runs to decide that a champion cannot score.-6. On a live run that completed, score the other shadow-set models on the same people, each with its own run (see Shadow scoring above).+6. On a live run that completed, score the other shadow-set models on the same people, each with its own run (see Shadow scoring above). The daily sweep does this in a later activity.Both model shapes are live and must stay that way: bundle-backed champions take the sandbox path, older recipe-only champions take the in-process path.diff --git a/products/autoresearch/backend/inference/scoring.py b/products/autoresearch/backend/inference/scoring.pyindex 7211b5a0e2ed..cced66ee6538 100644--- a/products/autoresearch/backend/inference/scoring.py+++ b/products/autoresearch/backend/inference/scoring.py@@ -9,7 +9,8 @@ autoresearch_prediction event per person.the scores without the events.- After a live champion run completes, every other model in the shadow set scores thesame people (``_score_shadow_set()``). A shadow model emits person-less events with the- ``shadow`` role and no ``$set``, and records its own inference run.+ ``shadow`` role and no ``$set``, and records its own inference run. The daily sweep defers+ this to a later activity (``score_shadow_set_for_run()``), so every champion scores first.- A live champion event carries ``$set: {output_person_property: p_y}`` only while thepipeline's ``write_person_property`` is on. ``person_property.py`` removes the property again.@@ -39,7 +40,7 @@ import hashlibimport inspectimport importlibimport dataclasses-from collections.abc import Callable+from collections.abc import Callable, Iterable, Mappingfrom datetime import UTC, date, datetimefrom typing import Any@@ -282,6 +283,8 @@ def run_inference_for_pipeline(run: AutoresearchRun | None = None,query_context: QueryContext = INTERACTIVE_QUERY,scheduled: bool = False,+ defer_shadows: bool = False,+ extra_metrics: Mapping[str, Any] | None = None,) -> AutoresearchRun:"""Top-level inference entry point. Creates an AutoresearchRun, scores users,@@ -306,8 +309,13 @@ def run_inference_for_pipeline(A live run that completes then scores the rest of the shadow set (``_score_shadow_set()``).Shadow scoring never fails this run or changes its outcome; the run records which shadow- models completed, failed, or were skipped in ``metrics["shadow_models"]``.+ models completed, failed, or were skipped in ``metrics["shadow_models"]``. With+ ``defer_shadows`` the run stops before the shadow set, and ``score_shadow_set_for_run()``+ scores it later against the people this run recorded in ``metrics["scored_persons_digest"]``.++ ``extra_metrics`` are merged into the run's metrics before scoring starts."""+ started = time.monotonic()window = ScoringWindow.for_date(prediction_date)if run is None:run = create_inference_run(pipeline=pipeline, model=model, window=window, scheduled=scheduled)@@ -318,6 +326,9 @@ def run_inference_for_pipeline(run.metrics.pop("failure_kind", None)run.completed_at = Nonerun.save(update_fields=["model", "status", "error", "metrics", "completed_at"])+ if extra_metrics:+ run.metrics.update(extra_metrics)+ run.save(update_fields=["metrics"])try:team = pipeline.team@@ -340,10 +351,12 @@ def run_inference_for_pipeline("holdout_auc": scored.holdout_auc,# The coverage of a rolling run: rows_scored of rows_eligible were scored today."rows_eligible": scored.rows_eligible,+ "scoring_seconds": _elapsed_s(started),})if not window.is_backfill:run.metrics.update(_likely_segment(team_id=team.pk, pipeline=pipeline, scored=scored))+ run.metrics["scored_persons_digest"] = persons_digest(row["distinct_id"] for row in scored.rows)run.completed_at = django_timezone.now()run.save(update_fields=["status", "rows_scored", "negative_sample_rate", "metrics", "completed_at"])@@ -365,6 +378,7 @@ def run_inference_for_pipeline(run.status = AutoresearchRun.Status.FAILEDrun.error = str(exc)[:2000]run.metrics["failure_kind"] = classify_failure(exc)+ run.metrics["scoring_seconds"] = _elapsed_s(started)run.completed_at = django_timezone.now()run.save(update_fields=["status", "error", "metrics", "completed_at"])logger.exception(@@ -374,12 +388,14 @@ def run_inference_for_pipeline(if not window.is_backfill:_record_coverage(team=team, pipeline=pipeline, run=run, scored=scored, window=window, user=acting_user)+ if not window.is_backfill and not defer_shadows:try:outcome = _score_shadow_set(team=team,pipeline=pipeline,champion=model,- scored=scored,+ persons=run.metrics["scored_persons_digest"] if scored.rows else None,+ features=scored.features,window=window,user=acting_user,query_context=query_context,@@ -394,6 +410,15 @@ def run_inference_for_pipeline(return run+def persons_digest(person_ids: Iterable[Any]) -> str:+ """An order-free digest of a set of person ids, so a later query can prove it selected the same people."""+ return hashlib.sha256("\n".join(sorted({str(person_id) for person_id in person_ids})).encode("utf-8")).hexdigest()+++def _elapsed_s(started: float) -> float:+ return round(time.monotonic() - started, 1)++def _likely_segment(*, team_id: int, pipeline: AutoresearchPipeline, scored: ScoredPopulation) -> dict[str, Any]:"""The people a live champion run scored at or above the current Likely cut point, and that cut point.@@ -777,7 +802,8 @@ def _score_shadow_set(team: Team,pipeline: AutoresearchPipeline,champion: AutoresearchModel,- scored: ScoredPopulation,+ persons: str | None,+ features: MaterializedFeatures | None,window: ScoringWindow,user: User,query_context: QueryContext = INTERACTIVE_QUERY,@@ -785,20 +811,25 @@ def _score_shadow_set() -> ShadowScoringOutcome | None:"""Score every shadow-set model other than the champion against the people the champion's run- scored, so online validation compares the models on the same people on the same date.+ scored (``persons``, their ``persons_digest()``), so online validation compares the models on+ the same people on the same date.- One feature query runs per distinct ``features.sql``: the champion's rows are reused when a- shadow model shares its SQL, and each other SQL materializes once for every model that uses- it. Each model records its own inference run, and its failure fails only that run. No new- model starts after ``SHADOW_TIME_BUDGET_S``. Returns None when there is nothing to score.+ One feature query runs per distinct ``features.sql``: the champion's rows (``features``) are+ reused when a shadow model shares its SQL, and each other SQL materializes once for every model+ that uses it. Each model records its own inference run, and its failure fails only that run. No+ new model starts after ``SHADOW_TIME_BUDGET_S``. Returns None when there is nothing to score."""- members = [model for model in shadow_set(pipeline) if model.pk != champion.pk]- if not members or not scored.rows:+ # A model promoted since the champion's run is not a shadow, so it does not score as one.+ members = [+ model+ for model in shadow_set(pipeline)+ if model.pk != champion.pk and model.role != AutoresearchModel.Role.CHAMPION+ ]+ if not members or persons is None:return None- persons = {str(row["distinct_id"]) for row in scored.rows}materialized: dict[str, InferenceRows | Exception] = {}- if scored.features is not None:- materialized[scored.features.sql_digest] = scored.features.data+ if features is not None:+ materialized[features.sql_digest] = features.datacompleted: list[str] = []failed: list[str] = []@@ -835,6 +866,53 @@ def _score_shadow_set(return ShadowScoringOutcome(completed=completed, failed=failed, skipped=skipped)+def score_shadow_set_for_run(+ run: AutoresearchRun, query_context: QueryContext = INTERACTIVE_QUERY+) -> ShadowScoringOutcome | None:+ """+ Score the shadow set against the people a completed live champion run scored, after that run's+ activity has ended. The daily sweep calls this once every champion has scored, so a champion's+ feature rows are freed before any shadow model loads its own.++ The run's ``scored_persons_digest`` stands for the people, so the person list never crosses the+ Temporal payload. Each feature query binds the run's cutoff and must select exactly those people.+ Returns None, and scores nothing, when the run did not complete, scored nobody, or its prediction+ date has passed: a shadow score emitted after that date would not pair with the champion's.+ """+ started = time.monotonic()+ metrics = run.metrics or {}+ digest = metrics.get("scored_persons_digest")+ if (+ run.status != AutoresearchRun.Status.COMPLETED+ or metrics.get("shadow")+ or not digest+ or not run.rows_scored+ or run.model is None+ ):+ return None+ window = ScoringWindow.for_date(date.fromisoformat(metrics["prediction_date"]))+ if window.is_backfill:+ logger.warning("autoresearch_shadow_scoring_after_prediction_date", run_id=str(run.pk))+ return None+ pipeline = run.pipeline+ team = pipeline.team+ outcome = _score_shadow_set(+ team=team,+ pipeline=pipeline,+ champion=run.model,+ persons=digest,+ features=None,+ window=window,+ user=_acting_user(team=team, pipeline=pipeline, user=None),+ query_context=query_context,+ )+ if outcome is not None:+ run.metrics["shadow_models"] = outcome.as_metrics()+ run.metrics["shadow_seconds"] = _elapsed_s(started)+ run.save(update_fields=["metrics"])+ return outcome++def _read_shadow_bundle(model: AutoresearchModel) -> ArtifactBundle | Exception:"""The model's runnable bundle, or the error that fails its run. The error never stops the other models."""try:@@ -851,7 +929,7 @@ def _score_shadow_model(pipeline: AutoresearchPipeline,model: AutoresearchModel,bundle: ArtifactBundle | Exception,- persons: set[str],+ persons: str,materialized: dict[str, InferenceRows | Exception],window: ScoringWindow,user: User,@@ -934,7 +1012,7 @@ def _shadow_score_data(team: Team,pipeline: AutoresearchPipeline,feature_sql: str,- persons: set[str],+ persons: str,materialized: dict[str, InferenceRows | Exception],window: ScoringWindow,user: User,@@ -959,7 +1037,7 @@ def _shadow_score_data(user=user,query_context=query_context,)- if {str(row["distinct_id"]) for row in data.rows} != persons:+ if persons_digest(row["distinct_id"] for row in data.rows) != persons:raise InferenceRunError("The shadow feature query did not select the people the champion scored; ""refusing to emit scores that do not pair with the champion's"diff --git a/products/autoresearch/backend/inference/test_inference.py b/products/autoresearch/backend/inference/test_inference.pyindex 3e347b7f602b..85df1e366ea0 100644--- a/products/autoresearch/backend/inference/test_inference.py+++ b/products/autoresearch/backend/inference/test_inference.py@@ -15,11 +15,13 @@ from posthog.hogql.errors import QueryErrorfrom posthog.api.capture import CaptureInternalResultfrom posthog.dataclasses import frozenfrom posthog.exceptions import (+ ClickHouseAtCapacity,ClickHouseClusterMemoryLimitExceeded,ClickHouseQueryMemoryLimitExceeded,ClickHouseQueryTimeOut,)+from products.autoresearch.backend import query as autoresearch_queryfrom products.autoresearch.backend.dataset.labeling import PREDICTION_EVENT_NAME, ROLLING_SCORE_LIMITfrom products.autoresearch.backend.inference import (sandbox as sandbox_inference,@@ -47,9 +49,10 @@ from products.autoresearch.backend.inference.scoring import (_score_rows,run_inference_for_pipeline,score_population,+ score_shadow_set_for_run,)from products.autoresearch.backend.models import AutoresearchModel, AutoresearchPipeline, AutoresearchRun-from products.autoresearch.backend.query import BATCH_QUERY, HogQLResult+from products.autoresearch.backend.query import BATCH_QUERY, SCORING_QUERY, HogQLResultfrom products.autoresearch.backend.testing import TeamScopedTestMixinfrom products.autoresearch.backend.training.artifacts import ArtifactBundle@@ -1127,7 +1130,9 @@ class TestShadowScoring(TeamScopedTestMixin, BaseTest):metrics={"model_fitted": True},)- def _run(self, *, failing: dict[str, Exception] | None = None, shadow_rows=_SHADOW_ROWS) -> _Cadence:+ def _run(+ self, *, failing: dict[str, Exception] | None = None, shadow_rows=_SHADOW_ROWS, deferred: bool = False+ ) -> _Cadence:failing = failing or {}materialize = MagicMock(return_value=InferenceRows(rows=shadow_rows, eligible=len(shadow_rows)))@@ -1160,25 +1165,32 @@ class TestShadowScoring(TeamScopedTestMixin, BaseTest):patch.object(scoring, "_resolve_distinct_ids", return_value={"user-1": "real-1", "user-2": "real-2"}),patch.object(scoring, "capture_batch_internal", capture),):- run = run_inference_for_pipeline(pipeline=self.pipeline, model=self.champion)+ run = run_inference_for_pipeline(pipeline=self.pipeline, model=self.champion, defer_shadows=deferred)+ if deferred:+ assert capture.call_count == 1+ score_shadow_set_for_run(AutoresearchRun.objects.get(pk=run.pk))+ run.refresh_from_db()return _Cadence(run=run, capture=capture, materialize=materialize)def _shadow_runs(self, model: AutoresearchModel) -> list[AutoresearchRun]:return list(AutoresearchRun.objects.filter(model=model, run_type=AutoresearchRun.RunType.INFERENCE))- def test_each_shadow_model_scores_the_champions_people_person_less(self):+ # In process, the champion's rows serve the model with its SQL. A deferred phase queries that SQL+ # again. Either way the two models with the other SQL share one query, although the shadow set+ # scores same_sql between them.+ @parameterized.expand(+ [("in_process", False, [_OTHER_FEATURE_SQL]), ("deferred", True, [_ANCHORS_FEATURE_SQL, _OTHER_FEATURE_SQL])]+ )+ def test_each_shadow_model_scores_the_champions_people_person_less(self, _name, deferred, expected_queries):other_a = self._model("other_a", _OTHER_FEATURE_SQL)same_sql = self._model("same_sql", _ANCHORS_FEATURE_SQL)other_b = self._model("other_b", _OTHER_FEATURE_SQL)- cadence = self._run()+ cadence = self._run(deferred=deferred)run, capture, materialize = cadence.run, cadence.capture, cadence.materializeassert run.status == AutoresearchRun.Status.COMPLETED- # The champion's rows serve the model with its SQL; the two models with the other SQL share one- # query, although the shadow set scores same_sql between them.- assert materialize.call_count == 1- assert materialize.call_args.kwargs["feature_sql"] == _OTHER_FEATURE_SQL+ assert sorted(c.kwargs["feature_sql"] for c in materialize.call_args_list) == sorted(expected_queries)assert sorted(run.metrics["shadow_models"]["completed"]) == sorted(str(m.pk) for m in (same_sql, other_a, other_b))@@ -1208,17 +1220,25 @@ class TestShadowScoring(TeamScopedTestMixin, BaseTest):@parameterized.expand([- ("model_load_failure", ModelLoadError("predict.py failed"), _SHADOW_ROWS, "model_load_failed"),- ("unpaired_rows", None, _SHADOW_ROWS[:1], "other"),+ ("model_load_failure", ModelLoadError("predict.py failed"), _SHADOW_ROWS, "model_load_failed", False),+ ("unpaired_rows", None, _SHADOW_ROWS[:1], "other", False),+ (+ "deferred_model_load_failure",+ ModelLoadError("predict.py failed"),+ _SHADOW_ROWS,+ "model_load_failed",+ True,+ ),+ ("deferred_unpaired_rows", None, _SHADOW_ROWS[:1], "other", True),])- def test_a_failing_shadow_model_never_touches_the_champion(self, _name, error, shadow_rows, failure_kind):+ def test_a_failing_shadow_model_never_touches_the_champion(self, _name, error, shadow_rows, failure_kind, deferred):challenger = self._model("challenger", _OTHER_FEATURE_SQL)failing = {str(challenger.pk): error} if error else {}for day in ("2026-09-11T12:00:00Z", "2026-09-12T12:00:00Z"):with time_machine.travel(day, tick=False):- run = self._run(failing=failing, shadow_rows=shadow_rows).run+ run = self._run(failing=failing, shadow_rows=shadow_rows, deferred=deferred).runassert run.status == AutoresearchRun.Status.COMPLETEDassert run.metrics["shadow_models"]["failed"] == [str(challenger.pk)]@@ -1244,3 +1264,60 @@ class TestShadowScoring(TeamScopedTestMixin, BaseTest):assert run.metrics["shadow_models"] == {"completed": [], "failed": [], "skipped": [str(challenger.pk)]}assert self._shadow_runs(challenger) == []assert capture.call_count == 1+++class TestQueryCapacityBackoff(SimpleTestCase):+ @parameterized.expand(+ [+ ("capacity_error_retries_then_succeeds", SCORING_QUERY, ClickHouseAtCapacity, 2, True),+ (+ "cluster_memory_pressure_retries_then_succeeds",+ SCORING_QUERY,+ ClickHouseClusterMemoryLimitExceeded,+ 2,+ True,+ ),+ ("other_error_does_not_retry", SCORING_QUERY, ClickHouseQueryTimeOut, 1, False),+ ("context_without_backoff_does_not_retry", BATCH_QUERY, ClickHouseAtCapacity, 1, False),+ ]+ )+ def test_query_backs_off_on_capacity_errors_only(self, _name, query_context, error, expected_calls, succeeds):+ response = MagicMock(columns=["distinct_id"], results=[["person-1"]], hasMore=False)+ with (+ patch.object(autoresearch_query, "HogQLQueryRunner") as runner,+ patch.object(autoresearch_query, "sleep") as sleep,+ ):+ runner.return_value.run.side_effect = [error(), response]+ if succeeds:+ result = scoring._query(+ team=MagicMock(), sql="SELECT 1", values={}, user=None, what="Feature", query_context=query_context+ )+ assert result.rows == [["person-1"]]+ else:+ with self.assertRaises(InferenceRunError):+ scoring._query(+ team=MagicMock(),+ sql="SELECT 1",+ values={},+ user=None,+ what="Feature",+ query_context=query_context,+ )+ assert runner.return_value.run.call_count == expected_calls+ assert sleep.call_count == expected_calls - 1+ if sleep.called:+ assert 24 <= sleep.call_args.args[0] <= 36++ def test_query_fails_once_the_backoff_runs_out(self):+ with (+ patch.object(autoresearch_query, "HogQLQueryRunner") as runner,+ patch.object(autoresearch_query, "sleep") as sleep,+ ):+ runner.return_value.run.side_effect = ClickHouseAtCapacity()+ with self.assertRaises(InferenceRunError):+ scoring._query(+ team=MagicMock(), sql="SELECT 1", values={}, user=None, what="Feature", query_context=SCORING_QUERY+ )+ assert runner.return_value.run.call_count == 4+ waits = [c.args[0] for c in sleep.call_args_list]+ assert [0.8 * step <= wait <= 1.2 * step for wait, step in zip(waits, (30, 60, 120), strict=True)] == [True] * 3diff --git a/products/autoresearch/backend/query.py b/products/autoresearch/backend/query.pyindex c9281a418e1b..d910552db713 100644--- a/products/autoresearch/backend/query.py+++ b/products/autoresearch/backend/query.py@@ -1,7 +1,10 @@+import randomfrom collections.abc import Callable-from time import perf_counter+from time import perf_counter, sleepfrom typing import Any, TypeVar+import structlog+from posthog.schema import CacheMissResponse, HogQLQuery, QueryStatusResponsefrom posthog.hogql.constants import LimitContext@@ -9,11 +12,14 @@ from posthog.hogql.query_stats import query_stats_scopefrom posthog.clickhouse.workload import Workloadfrom posthog.dataclasses import frozen+from posthog.errors import QueryErrorCategory, classify_query_errorfrom posthog.hogql_queries.hogql_query_runner import HogQLQueryRunnerfrom posthog.hogql_queries.query_runner import ExecutionModefrom posthog.models.team.team import Teamfrom posthog.models.user import User+logger = structlog.get_logger(__name__)+T = TypeVar("T")@@ -27,6 +33,8 @@ class QueryContext:limit_context: LimitContextworkload: Workload+ # The seconds to wait before each retry of a query that ClickHouse refuses for capacity. Empty means no retry.+ capacity_backoff_s: tuple[float, ...] = ()# A query inside a web request keeps the 60 s interactive limit, because a longer query@@ -35,6 +43,14 @@ INTERACTIVE_QUERY = QueryContext(limit_context=LimitContext.QUERY, workload=Work# Scoring and online validation run in a Temporal activity that nobody waits on, so they# get the increased batch limit and run on the offline workload.BATCH_QUERY = QueryContext(limit_context=LimitContext.QUERY_ASYNC, workload=Workload.OFFLINE)+# Scheduled scoring is batch work that also retries a capacity refusal, because the daily sweep starts+# many pipelines together and an immediate retry lands in the same burst. The activity heartbeats+# from a background thread, so a wait does not look like a lost worker.+SCORING_QUERY = QueryContext(+ limit_context=LimitContext.QUERY_ASYNC, workload=Workload.OFFLINE, capacity_backoff_s=(30, 60, 120)+)+# Each wait is the backoff step times a random factor in this range, so concurrent retries do not stay in step.+_BACKOFF_JITTER = (0.8, 1.2)@frozen@@ -93,7 +109,52 @@ def run_hogql(a blocking mode never produces. Raising on those keeps every caller off the union,and keeps "the query returned nothing" distinct from "the query did not run",which callers here read as a real zero.++ A query that ClickHouse refuses for capacity (``RATE_LIMITED``) runs again after each wait in+ ``query_context.capacity_backoff_s``, and then raises as before."""+ backoff = list(query_context.capacity_backoff_s)+ while True:+ try:+ return _run_hogql_once(+ team=team, query=query, user=user, execution_mode=execution_mode, query_context=query_context+ )+ except Exception as exc:+ capacity_error = _capacity_error(exc)+ if not backoff or capacity_error is None:+ raise+ # ClickHouseAtCapacity carries the shortest wait it expects to recover in.+ wait = max(backoff.pop(0) * random.uniform(*_BACKOFF_JITTER), getattr(capacity_error, "wait", 0))+ logger.warning(+ "autoresearch_query_capacity_backoff",+ team_id=team.pk,+ error=type(capacity_error).__name__,+ wait_s=round(wait, 1),+ retries_left=len(backoff),+ )+ sleep(wait)+++def _capacity_error(exc: BaseException) -> Exception | None:+ """The error in the cause chain that ClickHouse raised for capacity, or None."""+ current: BaseException | None = exc+ seen: set[int] = set()+ while current is not None and id(current) not in seen:+ seen.add(id(current))+ if isinstance(current, Exception) and classify_query_error(current) == QueryErrorCategory.RATE_LIMITED:+ return current+ current = current.__cause__+ return None+++def _run_hogql_once(+ *,+ team: Team,+ query: HogQLQuery,+ user: User | None,+ execution_mode: ExecutionMode,+ query_context: QueryContext,+) -> HogQLResult:response = HogQLQueryRunner(query=query,team=team,diff --git a/products/autoresearch/backend/temporal/AGENTS.md b/products/autoresearch/backend/temporal/AGENTS.mdindex 0d3caaa129b8..49de8787cdb8 100644--- a/products/autoresearch/backend/temporal/AGENTS.md+++ b/products/autoresearch/backend/temporal/AGENTS.md@@ -1,14 +1,15 @@# Temporal workflows-The scheduled half of autoresearch. Three workflows and four activities, registered on `settings.AUTORESEARCH_TASK_QUEUE` by `start_temporal_worker`, which imports `WORKFLOWS` and `ACTIVITIES` from `__init__.py`.+The scheduled half of autoresearch. Four workflows and their activities, registered on `settings.AUTORESEARCH_TASK_QUEUE` by `start_temporal_worker`, which imports `WORKFLOWS` and `ACTIVITIES` from `__init__.py`.Scoring, validation and training live in `../inference/`, `../evaluation/`, and `../training/` — the same functions the management commands and the API reach — and the activities call them, so the scheduled path cannot drift from the headless one. What stays here is the sweep's own logic: discovery (cadence, rollout, creator access) and the kickoff's eligibility gates, research schedule and row lock.## What lives here- `workflows.py`- - `AutoresearchCoordinatorWorkflow` (`autoresearch-coordinator`) — the daily tick. Loads active pipelines (`activity_load_active_pipelines`) and fans out inference, validation, and training kickoff per pipeline.- - `AutoresearchInferenceWorkflow` (`autoresearch-inference`) — `activity_run_inference` for one pipeline. The activity reads the current champion itself, so a retry after a mid-scoring promotion scores with the new one.+ - `AutoresearchCoordinatorWorkflow` (`autoresearch-coordinator`) — the daily tick. Loads active pipelines (`activity_load_active_pipelines`) and runs inference, training kickoff, and validation per pipeline, with at most `CoordinatorWorkflowInput.max_concurrency` pipelines in flight (`DEFAULT_COORDINATOR_CONCURRENCY`, 4). After every champion has scored, it runs `activity_run_shadow_scoring` per completed champion run, under the same bound.+ - `AutoresearchInferenceWorkflow` (`autoresearch-inference`) — `activity_run_inference` for one pipeline. The activity reads the current champion itself, so a retry after a mid-scoring promotion scores with the new one. The sweep sets `defer_shadows`, so the activity scores the champion only.+ - `activity_run_shadow_scoring` — scores the shadow set against the people a champion's completed run scored (`score_shadow_set_for_run()` in `../inference/scoring.py`). It takes only the run id. It has one attempt, because a retry would open a second shadow run per model, and its failure never fails the champion's run or counts as a pipeline error.- `AutoresearchValidationWorkflow` (`autoresearch-validation`) — `activity_run_validation` for one pipeline.- `activity_kickoff_training` — launches a training run for a pipeline that is due.- `AutoresearchPropertyCleanupWorkflow` (`autoresearch-property-cleanup`) — `activity_remove_person_property` removes a pipeline's output person property from persons (`../inference/person_property.py`), then runs a reconcile pass an hour later for scores that were still ingesting during the first pass. The run row stays `running` until the reconcile pass ends, and a retry adds to the count on the row. When the activity runs out of retries, the workflow fails the run with `activity_fail_cleanup_run`. The facade starts it, not the coordinator, with one workflow id per cleanup request.@@ -29,15 +30,20 @@ Scoring, validation and training live in `../inference/`, `../evaluation/`, anddaily schedule└─ AutoresearchCoordinatorWorkflow├─ activity_load_active_pipelines- └─ per pipeline:- ├─ AutoresearchInferenceWorkflow → ../inference/- ├─ AutoresearchValidationWorkflow → ../evaluation/- └─ activity_kickoff_training → ../training/+ ├─ per pipeline, at most max_concurrency at once, one step at a time:+ │ ├─ AutoresearchInferenceWorkflow → ../inference/ (champion only)+ │ ├─ activity_kickoff_training → ../training/+ │ └─ AutoresearchValidationWorkflow → ../evaluation/+ └─ after every pipeline above, per completed champion run, under the same bound:+ └─ activity_run_shadow_scoring → ../inference/```+The bound exists because each scoring activity holds a feature matrix in worker memory and runs batch queries. An unbounded sweep started every pipeline at the same moment, which overloaded ClickHouse and got the shared worker OOM-killed. A pipeline holds its slot through all its steps, so validation never runs next to its own scoring. Discovery lists the due pipelines that did not score yesterday first, so they get the first slots. The ordering and the bound come from recorded history (the activity result and the workflow input), so they are deterministic.+The bounded sweep is behind `workflow.patched("autoresearch-bounded-sweep-2026-10")`. A history without the marker replays the old unbounded sweep in `_run_pipeline()`. Remove that path once no coordinator that started before the patch can still be running.+The coordinator decides _whether_ each pipeline is due; the child workflows do one pipeline's work. Keep that split — per-pipeline logic in the coordinator is what makes a fan-out impossible to reason about.Validation runs on every daily sweep, because a matured date should not wait for the next scoring day.-Scoring and training kickoff run on a cadence day only, compared by calendar day, and kickoff starts after scoring ends.+Scoring and training kickoff run on a cadence day only, compared by calendar day, and kickoff starts after scoring ends. Validation starts after kickoff.Discovery pauses a pipeline whose creator has lost access to the team, and skips one outside the flag rollout.Discovery also fails every inference run still RUNNING after `INFERENCE_WORKFLOW_TIMEOUT` (`fail_stale_inference_runs()` in `../inference/failures.py`), with `failure_kind` `other`, because a lost worker never records an outcome.A retry of a scheduled scoring attempt fills the RUNNING row of the attempt before it, so one date keeps one row.diff --git a/products/autoresearch/backend/temporal/__init__.py b/products/autoresearch/backend/temporal/__init__.pyindex 0e2df6276c4c..17bda4978e7e 100644--- a/products/autoresearch/backend/temporal/__init__.py+++ b/products/autoresearch/backend/temporal/__init__.py@@ -15,6 +15,7 @@ from .workflows import (activity_load_active_pipelines,activity_remove_person_property,activity_run_inference,+ activity_run_shadow_scoring,activity_run_validation,)@@ -31,6 +32,7 @@ ACTIVITIES = [activity_load_active_pipelines,activity_remove_person_property,activity_run_inference,+ activity_run_shadow_scoring,activity_run_validation,]diff --git a/products/autoresearch/backend/temporal/test_workflows.py b/products/autoresearch/backend/temporal/test_workflows.pyindex 5247d26b4ddc..ddcfe59ed94a 100644--- a/products/autoresearch/backend/temporal/test_workflows.py+++ b/products/autoresearch/backend/temporal/test_workflows.py@@ -1,4 +1,5 @@import uuid+import asynciofrom datetime import UTC, datetime, timedeltafrom typing import Any, Optional@@ -26,20 +27,28 @@ from products.autoresearch.backend.models import (AutoresearchRun,AutoresearchTrainingRun,)-from products.autoresearch.backend.query import BATCH_QUERY+from products.autoresearch.backend.query import BATCH_QUERY, SCORING_QUERYfrom products.autoresearch.backend.temporal.workflows import (_VALIDATION_ATTEMPT_TIMEOUT,RESCUE_ITERATION_BUDGET,+ ActivePipeline,+ AutoresearchCoordinatorWorkflow,+ AutoresearchInferenceWorkflow,AutoresearchPropertyCleanupWorkflow,+ AutoresearchValidationWorkflow,+ CoordinatorWorkflowInput,FailCleanupRunInput,InferenceWorkflowResult,KickoffTrainingInput,KickoffTrainingResult,LoadActivePipelinesInput,+ LoadActivePipelinesResult,PropertyCleanupWorkflowInput,PropertyCleanupWorkflowResult,RunInferenceInput,RunInferenceResult,+ RunShadowScoringInput,+ RunShadowScoringResult,RunValidationInput,RunValidationResult,ValidationWorkflowResult,@@ -170,6 +179,18 @@ class TestCoordinatorActivities(TeamScopedTestMixin, BaseTest):score_due = {p.pipeline_id: p.score_due for p in result.pipelines}.get(str(pipeline.id))assert score_due == expected_score_due+ def test_load_active_pipelines_lists_pipelines_that_missed_yesterday_first(self) -> None:+ scored_yesterday = self._create_pipeline(last_scored_at=datetime(2026, 9, 10, 3, tzinfo=UTC))+ validates_only = self._create_pipeline(last_scored_at=datetime(2026, 9, 11, 3, tzinfo=UTC))+ missed_yesterday = self._create_pipeline(last_scored_at=datetime(2026, 9, 9, 3, tzinfo=UTC))+ never_scored = self._create_pipeline()++ result = activity_load_active_pipelines(LoadActivePipelinesInput())++ order = [p.pipeline_id for p in result.pipelines]+ assert set(order[:2]) == {str(missed_yesterday.id), str(never_scored.id)}+ assert set(order[2:]) == {str(scored_yesterday.id), str(validates_only.id)}+@parameterized.expand([("creator_lost_access", True, True, {}),@@ -252,7 +273,7 @@ class TestCoordinatorActivities(TeamScopedTestMixin, BaseTest):activity_run_inference,RunInferenceInput(pipeline_id=str(pipeline.id), team_id=self.team.id, prediction_date="2026-09-11"),)- assert mock_inference.call_args.kwargs["query_context"] == BATCH_QUERY+ assert mock_inference.call_args.kwargs["query_context"] == SCORING_QUERYassert mock_inference.call_args.kwargs["scheduled"] is Trueelse:env.run(activity_run_validation, RunValidationInput(pipeline_id=str(pipeline.id), team_id=self.team.id))@@ -614,3 +635,71 @@ async def test_property_cleanup_workflow_reconciles_and_fails_the_run(assert passes == expected_passesassert failed_runs == ([expected_failed_run] if expected_failed_run else [])+++async def test_coordinator_bounds_the_sweep_and_scores_shadows_after_every_champion() -> None:+ pipeline_ids = [f"p{i}" for i in range(5)]+ in_flight = 0+ max_in_flight = 0+ events: list[tuple[str, str]] = []+ inference_inputs: list[RunInferenceInput] = []++ async def heavy_step(name: str, pipeline_id: str) -> None:+ nonlocal in_flight, max_in_flight+ in_flight += 1+ max_in_flight = max(max_in_flight, in_flight)+ await asyncio.sleep(0.05)+ in_flight -= 1+ events.append((name, pipeline_id))++ @activity.defn(name="autoresearch-coordinator.load_active_pipelines")+ async def load(inp: LoadActivePipelinesInput) -> LoadActivePipelinesResult:+ return LoadActivePipelinesResult(+ pipelines=[ActivePipeline(pipeline_id=pid, team_id=1, score_due=True) for pid in pipeline_ids]+ )++ @activity.defn(name="autoresearch-inference.run_inference")+ async def infer(inp: RunInferenceInput) -> RunInferenceResult:+ inference_inputs.append(inp)+ await heavy_step("champion", inp.pipeline_id)+ return RunInferenceResult(run_id=f"run-{inp.pipeline_id}", rows_scored=1, status="completed")++ @activity.defn(name="autoresearch-validation.run_validation")+ async def validate(inp: RunValidationInput) -> RunValidationResult:+ await heavy_step("validation", inp.pipeline_id)+ return RunValidationResult(dates_validated=1, total_rows=1, status="completed")++ @activity.defn(name="autoresearch-coordinator.kickoff_training")+ async def kickoff(inp: KickoffTrainingInput) -> KickoffTrainingResult:+ return KickoffTrainingResult(kicked_off=False, reason="not_due")++ @activity.defn(name="autoresearch-coordinator.run_shadow_scoring")+ async def shadows(inp: RunShadowScoringInput) -> RunShadowScoringResult:+ assert inp.run_id == f"run-{inp.pipeline_id}"+ events.append(("shadow", inp.pipeline_id))+ if inp.pipeline_id == "p0":+ raise ApplicationError("predict sandbox died", non_retryable=True)+ return RunShadowScoringResult(completed=1, failed=0, skipped=0, status="completed")++ async with await WorkflowEnvironment.start_time_skipping() as env:+ async with Worker(+ env.client,+ task_queue="coordinator-test",+ workflows=[AutoresearchCoordinatorWorkflow, AutoresearchInferenceWorkflow, AutoresearchValidationWorkflow],+ activities=[load, infer, validate, kickoff, shadows],+ workflow_runner=UnsandboxedWorkflowRunner(),+ ):+ result = await env.client.execute_workflow(+ AutoresearchCoordinatorWorkflow.run,+ CoordinatorWorkflowInput(run_date="2026-10-18", max_concurrency=2),+ id=str(uuid.uuid4()),+ task_queue="coordinator-test",+ )++ assert max_in_flight == 2+ first_shadow = next(index for index, (name, _) in enumerate(events) if name == "shadow")+ assert {pid for name, pid in events[:first_shadow] if name == "champion"} == set(pipeline_ids)+ assert sorted(pid for name, pid in events if name == "shadow") == pipeline_ids+ assert all(inp.defer_shadows and inp.coordinator_concurrency == 2 for inp in inference_inputs)+ # A shadow failure leaves the champion's pipeline counted as processed.+ assert (result.pipelines_processed, result.pipelines_errored, result.status) == (5, 0, "completed")diff --git a/products/autoresearch/backend/temporal/workflows.py b/products/autoresearch/backend/temporal/workflows.pyindex ae16d8155d35..1a728236dd0a 100644--- a/products/autoresearch/backend/temporal/workflows.py+++ b/products/autoresearch/backend/temporal/workflows.py@@ -10,9 +10,10 @@ for Django ORM + HogQL queries. Workflows are async and only orchestrate activitfrom __future__ import annotations-from collections.abc import Awaitable+from collections.abc import Awaitable, Callablefrom dataclasses import field, replacefrom datetime import date, timedelta+from functools import partialfrom typing import Optional, TypeVar, castimport structlog@@ -42,14 +43,14 @@ with workflow.unsafe.imports_passed_through():)from products.autoresearch.backend.inference.person_property import remove_output_person_propertyfrom products.autoresearch.backend.inference.sandbox import SandboxInferenceError, _resolve_acting_user- from products.autoresearch.backend.inference.scoring import run_inference_for_pipeline+ from products.autoresearch.backend.inference.scoring import run_inference_for_pipeline, score_shadow_set_for_runfrom products.autoresearch.backend.models import (AutoresearchModel,AutoresearchPipeline,AutoresearchRun,AutoresearchTrainingRun,)- from products.autoresearch.backend.query import BATCH_QUERY+ from products.autoresearch.backend.query import BATCH_QUERY, SCORING_QUERYfrom products.autoresearch.backend.training.runner import run_trainingfrom products.autoresearch.backend.training.schedule import research_schedule, schedule_runs_forfrom products.tasks.backend.facade.access import get_desktop_access_decision@@ -80,6 +81,10 @@ class InferenceWorkflowInput:# Set by a manual "Score now": the run row the API already returned, and the user who asked.run_id: Optional[str] = Noneuser_id: Optional[int] = None+ # Set by the daily sweep, which scores the shadow set in its own activity after every champion.+ defer_shadows: bool = False+ # The sweep's bound on pipelines in flight, recorded on the run.+ coordinator_concurrency: Optional[int] = None@frozen@@ -97,6 +102,8 @@ class RunInferenceInput:prediction_date: str # ISO date stringrun_id: Optional[str] = Noneuser_id: Optional[int] = None+ defer_shadows: bool = False+ coordinator_concurrency: Optional[int] = None@frozen@@ -148,8 +155,14 @@ def activity_run_inference(inp: RunInferenceInput) -> RunInferenceResult:prediction_date=date.fromisoformat(inp.prediction_date),user=user,run=manual_run or _open_scheduled_run(pipeline, inp.prediction_date),- query_context=BATCH_QUERY,+ query_context=SCORING_QUERY,scheduled=manual_run is None,+ defer_shadows=inp.defer_shadows,+ extra_metrics=(+ {"coordinator_concurrency": inp.coordinator_concurrency}+ if inp.coordinator_concurrency is not None+ else None+ ),)return RunInferenceResult(run_id=str(run.pk),@@ -257,6 +270,8 @@ class AutoresearchInferenceWorkflow(PostHogWorkflow):prediction_date=inp.prediction_date,run_id=inp.run_id,user_id=inp.user_id,+ defer_shadows=inp.defer_shadows,+ coordinator_concurrency=inp.coordinator_concurrency,),start_to_close_timeout=_SCORE_ATTEMPT_TIMEOUT,heartbeat_timeout=_HEARTBEAT_TIMEOUT,@@ -281,6 +296,58 @@ class AutoresearchInferenceWorkflow(PostHogWorkflow):)+# ── Shadow scoring ────────────────────────────────────────────────────────────+++@frozen+class RunShadowScoringInput:+ pipeline_id: str+ team_id: int+ # The champion's completed run. The activity reads the people it scored from the row.+ run_id: str+++@frozen+class RunShadowScoringResult:+ completed: int+ failed: int+ skipped: int+ status: str+++# SHADOW_TIME_BUDGET_S (20 minutes), plus the last model it starts: an anchor count, a feature query,+# and a predict sandbox (about 30 minutes at worst).+_SHADOW_ATTEMPT_TIMEOUT = timedelta(hours=1)+# One attempt, because a retry would open a second shadow run for each model it already scored.+_SHADOW_RETRY = RetryPolicy(maximum_attempts=1)++[email protected](name="autoresearch-coordinator.run_shadow_scoring")+def activity_run_shadow_scoring(inp: RunShadowScoringInput) -> RunShadowScoringResult:+ """Score the shadow set against the people a champion's completed run scored.++ The daily sweep starts this after every champion has scored. It writes only shadow runs and the+ champion run's ``shadow_models``, so a failure here never fails the champion's run.+ """+ with HeartbeaterSync(), team_scope(inp.team_id):+ run = (+ AutoresearchRun.objects.select_related("pipeline__team__organization", "pipeline__created_by", "model")+ .filter(pk=inp.run_id, pipeline__pk=inp.pipeline_id, run_type=AutoresearchRun.RunType.INFERENCE)+ .first()+ )+ if run is None or run.pipeline.status not in _LIVE_STATUSES or _sweep_access_block(run.pipeline):+ return RunShadowScoringResult(completed=0, failed=0, skipped=0, status="skipped")+ outcome = score_shadow_set_for_run(run, query_context=SCORING_QUERY)+ if outcome is None:+ return RunShadowScoringResult(completed=0, failed=0, skipped=0, status="skipped")+ return RunShadowScoringResult(+ completed=len(outcome.completed),+ failed=len(outcome.failed),+ skipped=len(outcome.skipped),+ status="completed",+ )++# ── Validation workflow I/O ───────────────────────────────────────────────────@@ -550,10 +617,17 @@ class AutoresearchPropertyCleanupWorkflow(PostHogWorkflow):# ── Coordinator workflow I/O ──────────────────────────────────────────────────+# How many pipelines the daily sweep runs at once. Each one holds a feature matrix in worker memory+# and runs batch queries, so an unbounded sweep overloads ClickHouse and the shared worker.+DEFAULT_COORDINATOR_CONCURRENCY = 4++@frozenclass CoordinatorWorkflowInput:# ISO date string; if omitted the workflow uses workflow.now()run_date: Optional[str] = None+ # Read from the input, so an operator can change it on the schedule without a deploy.+ max_concurrency: int = DEFAULT_COORDINATOR_CONCURRENCY@frozen@@ -658,9 +732,13 @@ def activity_load_active_pipelines(inp: LoadActivePipelinesInput) -> LoadActivePreaches it again.It also fails every inference run that its lost worker left RUNNING.++ The result lists the pipelines that are due and did not score yesterday first, so a bounded+ sweep starts the pipelines that missed a day before the others."""fail_stale_inference_runs()today = django_timezone.now().date()+ yesterday = today - timedelta(days=1)# unscoped() because the coordinator's job is exactly to sweep every team. The# pipelines it returns carry their team so the downstream activities can scope.candidates = (@@ -672,7 +750,7 @@ def activity_load_active_pipelines(inp: LoadActivePipelinesInput) -> LoadActiveP.select_related("team", "created_by").order_by("created_at"))- pipelines: list[ActivePipeline] = []+ pipelines: list[tuple[bool, ActivePipeline]] = []for pipeline in candidates:try:creator = _resolve_acting_user(team=pipeline.team, pipeline=pipeline, user=None)@@ -686,14 +764,16 @@ def activity_load_active_pipelines(inp: LoadActivePipelinesInput) -> LoadActiveP# By calendar day, because a run finishes some time after the 02:00 UTC tick and an# elapsed-time comparison would skip the next day's tick.last_scored = pipeline.last_scored_at.date() if pipeline.last_scored_at else None+ score_due = last_scored is None or (today - last_scored).days >= pipeline.cadence_days+ missed_yesterday = score_due and (last_scored is None or last_scored < yesterday)pipelines.append(- ActivePipeline(- pipeline_id=str(pipeline.pk),- team_id=pipeline.team_id,- score_due=last_scored is None or (today - last_scored).days >= pipeline.cadence_days,+ (+ not missed_yesterday,+ ActivePipeline(pipeline_id=str(pipeline.pk), team_id=pipeline.team_id, score_due=score_due),))- return LoadActivePipelinesResult(pipelines=pipelines)+ pipelines.sort(key=lambda entry: entry[0])+ return LoadActivePipelinesResult(pipelines=[pipeline for _, pipeline in pipelines])def _pause_orphaned_pipeline(pipeline: AutoresearchPipeline, *, reason: str) -> None:@@ -834,6 +914,9 @@ def _tasks_gate(pipeline: AutoresearchPipeline) -> str | None:# ── Coordinator workflow ──────────────────────────────────────────────────────+# Histories that started before this marker replay the unbounded sweep.+_BOUNDED_SWEEP_PATCH = "autoresearch-bounded-sweep-2026-10"+@workflow.defn(name="autoresearch-coordinator")class AutoresearchCoordinatorWorkflow(PostHogWorkflow):@@ -849,6 +932,10 @@ class AutoresearchCoordinatorWorkflow(PostHogWorkflow):- activity_kickoff_training — starts a new agent training run if eligible, onceinference ends, so a champion promoted mid-scoring cannot fail the scoring run+ At most ``max_concurrency`` pipelines run at once. Each pipeline scores, then kicks off+ training, then validates. Once every champion has scored, the coordinator scores each+ pipeline's shadow set (``activity_run_shadow_scoring``) under the same bound.+Per-pipeline failures — raised exceptions andactivity results that report a soft failure — are logged and counted inpipelines_errored, but do not prevent other pipelines from running.@@ -873,10 +960,14 @@ class AutoresearchCoordinatorWorkflow(PostHogWorkflow):workflow.logger.info("autoresearch_coordinator_no_active_pipelines")return CoordinatorWorkflowResult(pipelines_processed=0, pipelines_errored=0, status="completed")- outcomes = await _asyncio.gather(- *[self._run_pipeline(pipeline, run_date) for pipeline in active.pipelines],- return_exceptions=True,- )+ outcomes: list[PipelineRunOutcome | BaseException]+ if workflow.patched(_BOUNDED_SWEEP_PATCH):+ outcomes = await self._run_bounded(active.pipelines, run_date, max(1, inp.max_concurrency))+ else:+ outcomes = await _asyncio.gather(+ *[self._run_pipeline(pipeline, run_date) for pipeline in active.pipelines],+ return_exceptions=True,+ )errored = sum(1 for o in outcomes if isinstance(o, BaseException) or not o.succeeded)processed = len(outcomes) - errored@@ -892,17 +983,120 @@ class AutoresearchCoordinatorWorkflow(PostHogWorkflow):status="completed" if errored == 0 else "partial",)- async def _run_pipeline(self, pipeline: ActivePipeline, run_date: str) -> PipelineRunOutcome:- """Run validation, and on a cadence day inference then training kickoff, for one pipeline."""+ async def _run_bounded(+ self, pipelines: list[ActivePipeline], run_date: str, concurrency: int+ ) -> list[PipelineRunOutcome | BaseException]:+ """Run every pipeline with at most ``concurrency`` in flight, then every shadow set under the same bound."""+ slots = _asyncio.Semaphore(concurrency)++ async def bounded(step: Callable[[], Awaitable[_T]]) -> _T:+ async with slots:+ return await step()++ sweeps = await _asyncio.gather(+ *[bounded(partial(self._run_pipeline_in_sequence, p, run_date, concurrency)) for p in pipelines],+ return_exceptions=True,+ )+ # Shadow scoring starts only after every champion has finished, so champions free their memory first.+ shadows = [+ (pipeline, sweep.inference.run_id)+ for pipeline, sweep in zip(pipelines, sweeps)+ if isinstance(sweep, _PipelineSweep)+ and sweep.inference is not None+ and sweep.inference.status == "completed"+ and sweep.inference.run_id+ ]+ shadow_results = await _asyncio.gather(+ *[bounded(partial(self._score_shadows, p, r)) for p, r in shadows],+ return_exceptions=True,+ )+ for (pipeline, _), result in zip(shadows, shadow_results):+ if isinstance(result, BaseException):+ workflow.logger.warning(+ "autoresearch_shadow_scoring_failed",+ extra={"pipeline_id": pipeline.pipeline_id, "error": str(result)},+ )+ return [sweep if isinstance(sweep, BaseException) else sweep.outcome for sweep in sweeps]++ async def _run_pipeline_in_sequence(+ self, pipeline: ActivePipeline, run_date: str, concurrency: int+ ) -> _PipelineSweep:+ """On a cadence day score, then kick off training. Then validate. One step at a time, so a pipeline holds one slot."""pipeline_id = pipeline.pipeline_id- validation = _settle(- workflow.execute_child_workflow(- AutoresearchValidationWorkflow.run,- ValidationWorkflowInput(pipeline_id=pipeline_id, team_id=pipeline.team_id),- id=f"autoresearch-validation-{pipeline_id}-{run_date}",- execution_timeout=_VALIDATION_WORKFLOW_TIMEOUT,+ inference_result: InferenceWorkflowResult | BaseException | None = None+ kickoff_result: KickoffTrainingResult | BaseException | None = None+ if pipeline.score_due:+ inference_result = await _settle(+ workflow.execute_child_workflow(+ AutoresearchInferenceWorkflow.run,+ InferenceWorkflowInput(+ pipeline_id=pipeline_id,+ team_id=pipeline.team_id,+ prediction_date=run_date,+ defer_shadows=True,+ coordinator_concurrency=concurrency,+ ),+ id=inference_workflow_id(pipeline_id, run_date),+ execution_timeout=INFERENCE_WORKFLOW_TIMEOUT,+ ))+ kickoff_result = await _settle(self._kickoff(pipeline))+ validation_result = await _settle(self._validate(pipeline, run_date))+ outcome = self._evaluate(pipeline_id, inference_result, validation_result, kickoff_result)+ return _PipelineSweep(+ outcome=outcome,+ inference=inference_result if isinstance(inference_result, InferenceWorkflowResult) else None,+ )++ async def _score_shadows(self, pipeline: ActivePipeline, run_id: str) -> RunShadowScoringResult:+ return await workflow.execute_activity(+ activity_run_shadow_scoring,+ RunShadowScoringInput(pipeline_id=pipeline.pipeline_id, team_id=pipeline.team_id, run_id=run_id),+ start_to_close_timeout=_SHADOW_ATTEMPT_TIMEOUT,+ heartbeat_timeout=_HEARTBEAT_TIMEOUT,+ retry_policy=_SHADOW_RETRY,)++ def _validate(self, pipeline: ActivePipeline, run_date: str) -> Awaitable[ValidationWorkflowResult]:+ return workflow.execute_child_workflow(+ AutoresearchValidationWorkflow.run,+ ValidationWorkflowInput(pipeline_id=pipeline.pipeline_id, team_id=pipeline.team_id),+ id=f"autoresearch-validation-{pipeline.pipeline_id}-{run_date}",+ execution_timeout=_VALIDATION_WORKFLOW_TIMEOUT,+ )++ def _kickoff(self, pipeline: ActivePipeline) -> Awaitable[KickoffTrainingResult]:+ return workflow.execute_activity(+ activity_kickoff_training,+ KickoffTrainingInput(pipeline_id=pipeline.pipeline_id, team_id=pipeline.team_id),+ start_to_close_timeout=timedelta(minutes=5),+ retry_policy=_KICKOFF_RETRY,+ )++ def _evaluate(+ self,+ pipeline_id: str,+ inference: InferenceWorkflowResult | BaseException | None,+ validation: ValidationWorkflowResult | BaseException,+ kickoff: KickoffTrainingResult | BaseException | None,+ ) -> PipelineRunOutcome:+ outcome = evaluate_pipeline_outcome(+ pipeline_id=pipeline_id, inference=inference, validation=validation, kickoff=kickoff+ )+ for error in outcome.errors:+ workflow.logger.warning(+ "autoresearch_pipeline_step_failed",+ extra={"pipeline_id": pipeline_id, "error": error},+ )+ return outcome++ async def _run_pipeline(self, pipeline: ActivePipeline, run_date: str) -> PipelineRunOutcome:+ """The unbounded sweep, kept for histories that started before ``_BOUNDED_SWEEP_PATCH``.++ Runs validation, and on a cadence day inference then training kickoff, for one pipeline.+ """+ pipeline_id = pipeline.pipeline_id+ validation = _settle(self._validate(pipeline, run_date))inference_result: InferenceWorkflowResult | BaseException | None = Nonekickoff_result: KickoffTrainingResult | BaseException | None = Noneif pipeline.score_due:@@ -919,28 +1113,17 @@ class AutoresearchCoordinatorWorkflow(PostHogWorkflow):)),)- kickoff_result = await _settle(- workflow.execute_activity(- activity_kickoff_training,- KickoffTrainingInput(pipeline_id=pipeline_id, team_id=pipeline.team_id),- start_to_close_timeout=timedelta(minutes=5),- retry_policy=_KICKOFF_RETRY,- )- )+ kickoff_result = await _settle(self._kickoff(pipeline))else:validation_result = await validation- outcome = evaluate_pipeline_outcome(- pipeline_id=pipeline_id,- inference=inference_result,- validation=validation_result,- kickoff=kickoff_result,- )- for error in outcome.errors:- workflow.logger.warning(- "autoresearch_pipeline_step_failed",- extra={"pipeline_id": pipeline_id, "error": error},- )- return outcome+ return self._evaluate(pipeline_id, inference_result, validation_result, kickoff_result)+++@frozen+class _PipelineSweep:+ outcome: PipelineRunOutcome+ # The champion's result, so the shadow phase knows which runs to score against.+ inference: InferenceWorkflowResult | Noneasync def _settle(awaitable: Awaitable[_T]) -> _T | BaseException: