Skip to content

:hedgehog: PostHog is the leading platform for building self-driving products. Our developer tools – AI observability, analytics, session replay, flags, experiments, error tracking, logs, and more – capture all the context agents need to diagnose problems, uncover opportunities, and ship fixes. Steer it all from Slack, web, desktop, or the MCP.

feat(autoresearch): cap live models per project and show the limits (#130498)

posthog[bot]b9ad1c44fa87
diff --git a/products/autoresearch/AGENTS.md b/products/autoresearch/AGENTS.md
index 3eacddd73c1b..8743401d9dec 100644
--- a/products/autoresearch/AGENTS.md
+++ b/products/autoresearch/AGENTS.md
@@ -46,8 +46,15 @@ The daily sweep starts research runs on scoring days until a run fails to promot
Each run takes `iterations_per_run` iterations, and records why it started in `kickoff_reason`.
Per-project limits in `backend/limits.py` bound every sandbox run, however it starts (sweep, UI, API or MCP).
A project starts at most `max_training_runs_per_day` sandbox runs (default 3) in any rolling 24 hours, across all its pipelines, and no run gets more than `max_iterations_per_run` iterations (default 10).
-Both resolve from the `autoresearch` flag payload: the project's `team_configs["<team_id>"]` entry, then `default_team_config`, then the code constants. Edit the payload to change a limit without a deploy.
-Runs that `open_training_run()` opens for an external agent do not count, because the caller's own agent does the work.
+Runs that `open_training_run()` opens for an external agent do not count against the daily limit, because the caller's own agent does the work.
+A project has at most `max_live_pipelines` live pipelines (default 3).
+Live means `bootstrapping`, `running` or `converged`; `draft`, `paused` and `archived` do not count.
+A draft with an open training run counts too, because its promotion makes it live with no check.
+The facade refuses a pipeline going live at the limit with `LivePipelineCapReached` (a 400): starting a draft's first run, opening an external run on a draft, and resuming a paused pipeline.
+Pipelines already over the limit keep running and training.
+All three limits resolve from the `autoresearch` flag payload: the project's `team_configs["<team_id>"]` entry, then `default_team_config`, then the code constants.
+Edit the payload to change a limit without a deploy.
+The pipeline viewset's `limits` action reports each limit with the project's current usage, for the Setup tab.
Each run also records the runtime adapter, model and reasoning effort it was dispatched with (see `backend/training/AGENTS.md`).
`iteration_budget` and `iteration_budget_remaining` are deprecated: nothing scheduled reads them.
diff --git a/products/autoresearch/backend/facade/api.py b/products/autoresearch/backend/facade/api.py
index 1ec658bd2351..c0a658e44379 100644
--- a/products/autoresearch/backend/facade/api.py
+++ b/products/autoresearch/backend/facade/api.py
@@ -58,7 +58,15 @@ from ..evaluation.segment_thresholds import (
training_base_rate,
)
from ..inference.failures import INFERENCE_WORKFLOW_TIMEOUT
-from ..limits import check_training_run_cap, limits_for_team
+from ..limits import (
+ LIVE_PIPELINE_STATUSES,
+ TeamLimits,
+ check_live_pipeline_cap,
+ check_training_run_cap,
+ limits_for_team,
+ live_pipeline_count,
+ training_run_usage,
+)
from ..models import (
RESEARCH_INTERVAL_DAYS_CHOICES as _RESEARCH_INTERVAL_DAYS_CHOICES,
AutoresearchIteration,
@@ -102,6 +110,7 @@ from .contracts import (
InvalidTarget,
Iteration,
IterationTrailEntry,
+ LivePipelineCapReached,
LiveTrainingRun,
MaterializedFeatures,
Model,
@@ -113,6 +122,7 @@ from .contracts import (
PipelineWrite,
PredictionCoverage,
PredictionSegmentThresholds,
+ ProjectLimits,
RealizedAucPoint,
ResolvedTemplate,
Run,
@@ -826,10 +836,14 @@ def set_pipeline_status(team_id: int, pipeline_id: str | UUID, *, status: str, u
The row lock serializes this with a concurrent lifecycle change and with ``start_training``,
so a stale read cannot revive an archived pipeline, and archival cannot race a new run.
Archiving a pipeline that writes its output property starts a cleanup.
+
+ Raises ``LivePipelineCapReached`` when resuming would take the project over its live pipeline limit.
"""
pipeline_uuid = _as_uuid(pipeline_id)
if pipeline_uuid is None:
raise PipelineNotFound("Pipeline not found.")
+ # The flag read can call out, so it runs before the row lock is taken.
+ limits = limits_for_team(team_id) if status in LIVE_PIPELINE_STATUSES else None
with transaction.atomic():
try:
row = (
@@ -846,6 +860,8 @@ def set_pipeline_status(team_id: int, pipeline_id: str | UUID, *, status: str, u
if status == AutoresearchPipeline.Status.ARCHIVED and _has_live_training_run(team_id, row):
# Archival would leave the sandbox writing to, and promoting on, a pipeline nobody sees.
raise AutoresearchConflict("A training run is in progress. Wait for it to finish before archiving.")
+ if limits is not None and row.status not in LIVE_PIPELINE_STATUSES:
+ _require_live_pipeline_slot(team_id, limits)
row.status = status
row.save(update_fields=["status", "updated_at"])
if status == AutoresearchPipeline.Status.ARCHIVED and row.write_person_property:
@@ -853,6 +869,26 @@ def set_pipeline_status(team_id: int, pipeline_id: str | UUID, *, status: str, u
return _pipeline_with_champion(row)
+def _require_live_pipeline_slot(team_id: int, limits: TeamLimits) -> None:
+ """Refuse a pipeline going live when the project is at its limit. Call inside ``transaction.atomic()``."""
+ if cap := check_live_pipeline_cap(team_id, limits):
+ raise LivePipelineCapReached(used=cap.used, limit=cap.limit)
+
+
+def get_project_limits(team_id: int) -> ProjectLimits:
+ """The project's limits and current usage, for display. Takes no lock, so it can be stale by one run."""
+ limits = limits_for_team(team_id)
+ training = training_run_usage(team_id, limits)
+ return ProjectLimits(
+ max_training_runs_per_day=limits.max_training_runs_per_day,
+ training_runs_used=training.used,
+ next_slot_at=training.next_slot_at,
+ max_iterations_per_run=limits.max_iterations_per_run,
+ max_live_pipelines=limits.max_live_pipelines,
+ live_pipelines=live_pipeline_count(team_id),
+ )
+
+
def _queue_property_cleanup(
row: AutoresearchPipeline, *, output_property: str, user_id: int | None, record_run: bool = True
) -> None:
@@ -1510,7 +1546,8 @@ def start_training(
reads the locked row, so a concurrent edit of the target cannot slip past it.
Raises ``TrainingRunCapReached`` when the project has used its sandbox training runs for the
- last 24 hours, across all its pipelines.
+ last 24 hours, across all its pipelines, and ``LivePipelineCapReached`` when a draft's first
+ run would take the project over its live pipeline limit.
"""
# The runner and the sandbox import pandas and pyarrow, so the router path loads them only here.
from ..inference.sandbox import SandboxInferenceError # noqa: PLC0415
@@ -1526,6 +1563,9 @@ def start_training(
pipeline = _claim_pipeline_for_training(team_id, pipeline_id)
if not allow_action_target and pipeline.target_definition.get("type") == "action":
raise InvalidTarget("An action target needs the action:read scope.")
+ # The run moves a draft to bootstrapping, which is live.
+ if pipeline.status == AutoresearchPipeline.Status.DRAFT:
+ _require_live_pipeline_slot(team_id, limits)
if cap := check_training_run_cap(team_id, limits):
raise TrainingRunCapReached(used=cap.used, limit=cap.limit, next_slot_at=cap.next_slot_at)
budget = iteration_budget or pipeline.iterations_per_run
@@ -1552,14 +1592,21 @@ def open_training_run(
``allow_action_target`` works as in ``start_training``: materializing features on the run
labels on the action's steps, so the scope check reads the locked row.
+
+ A draft goes live when this run promotes its first champion, so opening one on a draft checks
+ the project's live pipeline limit, and ``LivePipelineCapReached`` refuses it at the limit.
"""
pipeline = _pipeline_row(team_id, pipeline_id)
if pipeline.status == AutoresearchPipeline.Status.ARCHIVED:
raise AutoresearchConflict("Cannot open a training run on an archived pipeline.")
+ # The flag read can call out, so it runs before the row lock is taken.
+ limits = limits_for_team(team_id) if pipeline.status == AutoresearchPipeline.Status.DRAFT else None
with transaction.atomic():
pipeline = _claim_pipeline_for_training(team_id, pipeline.pk)
if not allow_action_target and pipeline.target_definition.get("type") == "action":
raise InvalidTarget("An action target needs the action:read scope.")
+ if limits is not None and pipeline.status == AutoresearchPipeline.Status.DRAFT:
+ _require_live_pipeline_slot(team_id, limits)
row = AutoresearchTrainingRun.objects.create(
pipeline=pipeline,
status=AutoresearchTrainingRun.Status.RUNNING,
diff --git a/products/autoresearch/backend/facade/contracts.py b/products/autoresearch/backend/facade/contracts.py
index 33ccb8f7f535..45a487704640 100644
--- a/products/autoresearch/backend/facade/contracts.py
+++ b/products/autoresearch/backend/facade/contracts.py
@@ -65,6 +65,25 @@ class TrainingRunCapReached(Exception):
super().__init__(message)
+class LivePipelineCapReached(AutoresearchConflict):
+ """The project already has its limit of live pipelines, so another cannot go live.
+
+ An ``AutoresearchConflict``, so the viewsets map it to a 400 with the message as-is, and the
+ message is user-facing copy.
+ """
+
+ def __init__(self, *, used: int, limit: int) -> None:
+ self.used = used
+ self.limit = limit
+ if limit == 0:
+ message = "Active models are turned off for this project. Contact support to turn them on."
+ else:
+ message = (
+ f"This project has reached its limit of {limit} active models. Pause or delete one to start another."
+ )
+ super().__init__(message)
+
+
class InvalidTarget(ValueError):
"""The target names something the labeler cannot honor.
@@ -361,6 +380,19 @@ class AgentModelOptions:
models: list[AgentModelOption]
+@dataclass(frozen=True)
+class ProjectLimits:
+ """The project's autoresearch limits, and how much of each it is using now."""
+
+ max_training_runs_per_day: int
+ training_runs_used: int
+ # When the next training run can start, while the project is at its daily limit.
+ next_slot_at: datetime | None
+ max_iterations_per_run: int
+ max_live_pipelines: int
+ live_pipelines: int
+
+
# ── Write contracts ────────────────────────────────────────────────────────
diff --git a/products/autoresearch/backend/limits.py b/products/autoresearch/backend/limits.py
index 693ac88256b4..b1396669162d 100644
--- a/products/autoresearch/backend/limits.py
+++ b/products/autoresearch/backend/limits.py
@@ -7,8 +7,8 @@ fleet-wide `default_team_config`, then the code constant here. Edit the flag pay
limit without a deploy:
{
- "default_team_config": {"max_training_runs_per_day": 3, "max_iterations_per_run": 10},
- "team_configs": {"2": {"max_training_runs_per_day": 20}}
+ "default_team_config": {"max_training_runs_per_day": 3, "max_iterations_per_run": 10, "max_live_pipelines": 3},
+ "team_configs": {"2": {"max_training_runs_per_day": 20, "max_live_pipelines": 10}}
}
"""
@@ -18,7 +18,7 @@ import json
from datetime import datetime, timedelta
from django.db import connection
-from django.db.models import Q, QuerySet
+from django.db.models import Exists, OuterRef, Q, QuerySet
from django.utils import timezone
import structlog
@@ -28,7 +28,7 @@ from posthog.dataclasses import frozen
from posthog.exceptions_capture import capture_exception
from .access import AUTORESEARCH_FLAG
-from .models import AutoresearchTrainingRun
+from .models import AutoresearchPipeline, AutoresearchTrainingRun
logger = structlog.get_logger(__name__)
@@ -37,9 +37,19 @@ _PAYLOAD_DISTINCT_ID = "internal_autoresearch_team_limits"
DEFAULT_MAX_TRAINING_RUNS_PER_DAY = 3
DEFAULT_MAX_ITERATIONS_PER_RUN = 10
+DEFAULT_MAX_LIVE_PIPELINES = 3
TEAM_CONFIG_MAX_TRAINING_RUNS_PER_DAY = "max_training_runs_per_day"
TEAM_CONFIG_MAX_ITERATIONS_PER_RUN = "max_iterations_per_run"
+TEAM_CONFIG_MAX_LIVE_PIPELINES = "max_live_pipelines"
+
+# Each of these adds scoring, shadow scoring and validation to the daily sweep, and competes for
+# the project's training runs. Paused, draft and archived pipelines do none of that.
+LIVE_PIPELINE_STATUSES = (
+ AutoresearchPipeline.Status.BOOTSTRAPPING,
+ AutoresearchPipeline.Status.RUNNING,
+ AutoresearchPipeline.Status.CONVERGED,
+)
# The daily cap counts runs over a rolling window, so a burst just after midnight UTC cannot
# spend two days of runs in one sitting.
@@ -47,6 +57,7 @@ TRAINING_RUN_WINDOW = timedelta(hours=24)
# Namespace for the per-team advisory lock, so it cannot collide with another product's key.
_TRAINING_CAP_LOCK_NAMESPACE = 0x41525452 # "ARTR"
+_LIVE_PIPELINE_CAP_LOCK_NAMESPACE = 0x41524C50 # "ARLP"
@frozen
@@ -54,6 +65,8 @@ class TeamLimits:
# 0 stops every sandbox training run for the project.
max_training_runs_per_day: int
max_iterations_per_run: int
+ # 0 stops every pipeline from going live.
+ max_live_pipelines: int
@frozen
@@ -64,6 +77,21 @@ class TrainingCapReached:
next_slot_at: datetime | None
+@frozen
+class TrainingRunUsage:
+ used: int
+ limit: int
+ # When the next run can start while the project is at its limit. None while a slot is free,
+ # and when the limit is 0.
+ next_slot_at: datetime | None
+
+
+@frozen
+class LiveCapReached:
+ used: int
+ limit: int
+
+
def limits_for_team(team_id: int) -> TeamLimits:
"""The project's effective limits. A missing or unreadable payload resolves to the code constants."""
payload = _read_flag_payload()
@@ -74,6 +102,9 @@ def limits_for_team(team_id: int) -> TeamLimits:
max_iterations_per_run=_resolve_int(
payload, team_id, TEAM_CONFIG_MAX_ITERATIONS_PER_RUN, DEFAULT_MAX_ITERATIONS_PER_RUN, minimum=1
),
+ max_live_pipelines=_resolve_int(
+ payload, team_id, TEAM_CONFIG_MAX_LIVE_PIPELINES, DEFAULT_MAX_LIVE_PIPELINES, minimum=0
+ ),
)
@@ -85,6 +116,15 @@ def check_training_run_cap(team_id: int, limits: TeamLimits) -> TrainingCapReach
"""
with connection.cursor() as cursor:
cursor.execute("SELECT pg_advisory_xact_lock(%s, %s)", [_TRAINING_CAP_LOCK_NAMESPACE, team_id])
+ usage = training_run_usage(team_id, limits)
+ if usage.used < usage.limit:
+ return None
+ logger.info("autoresearch_training_cap_reached", team_id=team_id, used=usage.used, limit=usage.limit)
+ return TrainingCapReached(used=usage.used, limit=usage.limit, next_slot_at=usage.next_slot_at)
+
+
+def training_run_usage(team_id: int, limits: TeamLimits) -> TrainingRunUsage:
+ """The project's sandbox training runs in the window, and when the next one can start if it is at its limit."""
limit = limits.max_training_runs_per_day
window_start = timezone.now() - TRAINING_RUN_WINDOW
created = list(
@@ -93,11 +133,44 @@ def check_training_run_cap(team_id: int, limits: TeamLimits) -> TrainingCapReach
.order_by("created_at")
.values_list("created_at", flat=True)
)
- if len(created) < limit:
+ at_limit = limit > 0 and len(created) >= limit
+ next_slot_at = created[len(created) - limit] + TRAINING_RUN_WINDOW if at_limit else None
+ return TrainingRunUsage(used=len(created), limit=limit, next_slot_at=next_slot_at)
+
+
+def check_live_pipeline_cap(team_id: int, limits: TeamLimits) -> LiveCapReached | None:
+ """Whether the project already has its limit of live pipelines, or None when another can go live.
+
+ Call inside the ``transaction.atomic()`` that makes a pipeline live, before the pipeline being
+ changed counts as live. The advisory lock is held until that transaction ends, so two
+ pipelines of one team cannot both take the last slot. Pipelines already over the limit keep
+ running: the cap only stops another one from going live.
+ """
+ with connection.cursor() as cursor:
+ cursor.execute("SELECT pg_advisory_xact_lock(%s, %s)", [_LIVE_PIPELINE_CAP_LOCK_NAMESPACE, team_id])
+ used = live_pipeline_count(team_id)
+ if used < limits.max_live_pipelines:
return None
- next_slot_at = created[len(created) - limit] + TRAINING_RUN_WINDOW if limit > 0 else None
- logger.info("autoresearch_training_cap_reached", team_id=team_id, used=len(created), limit=limit)
- return TrainingCapReached(used=len(created), limit=limit, next_slot_at=next_slot_at)
+ logger.info("autoresearch_live_pipeline_cap_reached", team_id=team_id, used=used, limit=limits.max_live_pipelines)
+ return LiveCapReached(used=used, limit=limits.max_live_pipelines)
+
+
+def live_pipeline_count(team_id: int) -> int:
+ """Live pipelines, plus drafts with an open training run.
+
+ A draft stays a draft while an external agent's run is open, and its promotion makes it live
+ with no check, so the open run holds its slot from the start.
+ """
+ open_run = AutoresearchTrainingRun.objects.for_team(team_id).filter(
+ pipeline=OuterRef("pk"),
+ status__in=[AutoresearchTrainingRun.Status.PENDING, AutoresearchTrainingRun.Status.RUNNING],
+ )
+ return (
+ AutoresearchPipeline.objects.for_team(team_id)
+ .annotate(has_open_run=Exists(open_run))
+ .filter(Q(status__in=LIVE_PIPELINE_STATUSES) | Q(status=AutoresearchPipeline.Status.DRAFT, has_open_run=True))
+ .count()
+ )
def counted_training_runs(team_id: int) -> QuerySet[AutoresearchTrainingRun]:
diff --git a/products/autoresearch/backend/presentation/AGENTS.md b/products/autoresearch/backend/presentation/AGENTS.md
index 982c6941b0ae..7976b1f784d4 100644
--- a/products/autoresearch/backend/presentation/AGENTS.md
+++ b/products/autoresearch/backend/presentation/AGENTS.md
@@ -11,7 +11,7 @@ The `autoresearch-*` MCP tools that expose this API to the sandbox agent are dec
- `views/views.py`
Five viewsets, registered in `../routes.py` under `project_autoresearch_*` basenames and nested pipeline-first.
- - `AutoresearchPipelineViewSet` — full CRUD plus the lifecycle actions: `train`, `score`, `validate_online`, `archive`, `pause`, `resume`, the pre-create helpers `templates`, `resolve-template`, `validate`, and the read-only `online_performance`, which returns realized metrics per model per validated date from the validation runs.
+ - `AutoresearchPipelineViewSet` — full CRUD plus the lifecycle actions: `train`, `score`, `validate_online`, `archive`, `pause`, `resume`, the pre-create helpers `templates`, `resolve-template`, `validate`, the read-only `online_performance`, which returns realized metrics per model per validated date from the validation runs, and the read-only `limits`, which returns the project's limits from `../limits.py` with its current usage.
- `AutoresearchTrainingRunViewSet` — read plus create, and **the agent's entire write surface**: `iterations`, `materialize-features`, `complete`, `artifacts`, `artifacts/upload`, `artifacts/get`, `artifacts/delete`, plus `history`.
- `AutoresearchModelViewSet`, `AutoresearchRunViewSet` — read-only.
- `AutoresearchSuggestionViewSet` — human or agent hypotheses, plus `respond`.
diff --git a/products/autoresearch/backend/presentation/views/serializers.py b/products/autoresearch/backend/presentation/views/serializers.py
index bdb38776af14..13d8636ccd02 100644
--- a/products/autoresearch/backend/presentation/views/serializers.py
+++ b/products/autoresearch/backend/presentation/views/serializers.py
@@ -27,6 +27,7 @@ from products.autoresearch.backend.facade.contracts import (
Pipeline,
PipelineWrite,
PredictionCoverage,
+ ProjectLimits,
RealizedAucPoint,
Run,
Suggestion,
@@ -2049,6 +2050,46 @@ class TrainingRunCapReachedSerializer(serializers.Serializer):
)
+class ProjectLimitsSerializer(DataclassSerializer):
+ max_training_runs_per_day = serializers.IntegerField(
+ help_text="Sandbox training runs the project may start in any 24 hours, across all its pipelines. 0 means training is turned off."
+ )
+ training_runs_used = serializers.IntegerField(
+ help_text="Sandbox training runs the project started in the last 24 hours, across all its pipelines."
+ )
+ next_slot_at = serializers.DateTimeField(
+ allow_null=True,
+ help_text=(
+ "When the next training run can start, while the project is at its daily limit. "
+ "Null while a run can start now, and when training is turned off."
+ ),
+ )
+ max_iterations_per_run = serializers.IntegerField(
+ help_text="Most experiments any one training run gets. A pipeline's iterations_per_run above this is lowered to it."
+ )
+ max_live_pipelines = serializers.IntegerField(
+ help_text=(
+ "Most pipelines the project may have live (bootstrapping, running or converged, or a draft with an open "
+ "training run, which goes live when the run promotes). Starting a draft's first training run or "
+ "resuming a paused pipeline is refused at this limit."
+ )
+ )
+ live_pipelines = serializers.IntegerField(
+ help_text=("Pipelines that count against max_live_pipelines now. Can exceed it if the limit was lowered.")
+ )
+
+ class Meta:
+ dataclass = ProjectLimits
+ fields = [
+ "max_training_runs_per_day",
+ "training_runs_used",
+ "next_slot_at",
+ "max_iterations_per_run",
+ "max_live_pipelines",
+ "live_pipelines",
+ ]
+
+
# ── Agent-recorded training serializers ────────────────────────────────────
diff --git a/products/autoresearch/backend/presentation/views/views.py b/products/autoresearch/backend/presentation/views/views.py
index 1cc816658ea9..d827e40183e8 100644
--- a/products/autoresearch/backend/presentation/views/views.py
+++ b/products/autoresearch/backend/presentation/views/views.py
@@ -72,6 +72,7 @@ from .serializers import (
OnlinePerformanceQuerySerializer,
OnlinePerformanceSerializer,
OpenTrainingRunSerializer,
+ ProjectLimitsSerializer,
RecordIterationSerializer,
ResolvedTemplateSerializer,
ResolveTemplateRequestSerializer,
@@ -175,6 +176,7 @@ class AutoresearchPipelineViewSet(TeamAndOrgViewSetMixin, viewsets.ModelViewSet)
"resolve_template",
"online_performance",
"agent_models",
+ "limits",
]
scope_object_write_actions = [
"create",
@@ -338,6 +340,20 @@ class AutoresearchPipelineViewSet(TeamAndOrgViewSetMixin, viewsets.ModelViewSet)
def agent_models(self, request: Request, *args: Any, **kwargs: Any) -> Response:
return Response(AgentModelOptionsSerializer(instance=api.list_agent_model_options()).data)
+ @extend_schema(
+ responses={200: ProjectLimitsSerializer},
+ summary="Read the project's limits",
+ description=(
+ "Return the project's autoresearch limits and how much of each it is using: sandbox training runs "
+ "in the last 24 hours against the daily limit, with when the next run can start at the limit, the "
+ "most experiments per training run, and live pipelines against the live pipeline limit. "
+ "Read-only; it runs no queries."
+ ),
+ )
+ @action(detail=False, methods=["get"], url_path="limits", pagination_class=None)
+ def limits(self, request: Request, *args: Any, **kwargs: Any) -> Response:
+ return Response(ProjectLimitsSerializer(instance=api.get_project_limits(self.team_id)).data)
+
@validated_request(
request_serializer=ResolveTemplateRequestSerializer,
responses={
@@ -434,7 +450,8 @@ class AutoresearchPipelineViewSet(TeamAndOrgViewSetMixin, viewsets.ModelViewSet)
400: OpenApiResponse(
description=(
"A training run is already in progress, the pipeline is paused, "
- "or the pipeline's target or creator is no longer valid."
+ "the pipeline's target or creator is no longer valid, "
+ "or the pipeline is a draft and the project already has its limit of live pipelines."
)
),
403: OpenApiResponse(
@@ -670,11 +687,16 @@ class AutoresearchPipelineViewSet(TeamAndOrgViewSetMixin, viewsets.ModelViewSet)
response=AutoresearchPipelineSerializer,
description="The pipeline after resuming.",
),
- 400: OpenApiResponse(description="The pipeline is not paused."),
+ 400: OpenApiResponse(
+ description="The pipeline is not paused, or the project already has its limit of live pipelines."
+ ),
404: OpenApiResponse(description="The pipeline does not exist or is archived."),
},
summary="Resume a pipeline",
- description="Resume a paused pipeline. Daily scoring and training will restart on the next cadence tick.",
+ description=(
+ "Resume a paused pipeline. Daily scoring and training will restart on the next cadence tick. "
+ "Refused when the project already has its limit of live pipelines, which the limits endpoint reports."
+ ),
)
@action(detail=True, methods=["post"], url_path="resume")
def resume(self, request: Request, *args: Any, **kwargs: Any) -> Response:
@@ -830,7 +852,8 @@ class AutoresearchTrainingRunViewSet(TeamAndOrgViewSetMixin, viewsets.ModelViewS
400: OpenApiResponse(
description=(
"The pipeline is archived or paused, a run is already in progress, "
- "or an action target needs the action:read scope."
+ "an action target needs the action:read scope, or the pipeline is a draft, which this run "
+ "would make live, and the project already has its limit of live pipelines."
)
),
},
diff --git a/products/autoresearch/backend/tests/test_api.py b/products/autoresearch/backend/tests/test_api.py
index b79f9ccf12d2..3361c9028175 100644
--- a/products/autoresearch/backend/tests/test_api.py
+++ b/products/autoresearch/backend/tests/test_api.py
@@ -1139,6 +1139,130 @@ class TestAutoresearchPipelineAPI(TeamScopedTestMixin, APIBaseTest):
assert body["next_slot_at"] is None
mock_run_training.assert_not_called()
+ def _make_live_pipelines(self, statuses: list[AutoresearchPipeline.Status]) -> None:
+ for index, pipeline_status in enumerate(statuses):
+ self._make_pipeline(name=f"Existing {index}", status=pipeline_status)
+
+ def _live_cap_payload(self, override: str | None) -> dict | None:
+ if override is None:
+ return None
+ team_id = str(self.team.pk) if override == "this_team" else str(self.team.pk + 1000)
+ return {"default_team_config": {"max_live_pipelines": 3}, "team_configs": {team_id: {"max_live_pipelines": 4}}}
+
+ _AT_DEFAULT_CAP = [
+ AutoresearchPipeline.Status.BOOTSTRAPPING,
+ AutoresearchPipeline.Status.RUNNING,
+ AutoresearchPipeline.Status.CONVERGED,
+ ]
+
+ @parameterized.expand(
+ [
+ ("at_the_default_cap", _AT_DEFAULT_CAP, None, status.HTTP_400_BAD_REQUEST),
+ (
+ "draft_paused_and_archived_do_not_count",
+ [
+ AutoresearchPipeline.Status.RUNNING,
+ AutoresearchPipeline.Status.CONVERGED,
+ AutoresearchPipeline.Status.DRAFT,
+ AutoresearchPipeline.Status.PAUSED,
+ AutoresearchPipeline.Status.ARCHIVED,
+ ],
+ None,
+ status.HTTP_200_OK,
+ ),
+ ("team_override_allows_more", _AT_DEFAULT_CAP, "this_team", status.HTTP_200_OK),
+ ("override_for_another_team_does_not_apply", _AT_DEFAULT_CAP, "other_team", status.HTTP_400_BAD_REQUEST),
+ ]
+ )
+ def test_resume_respects_the_live_pipeline_cap(
+ self, _name: str, existing: list[AutoresearchPipeline.Status], override: str | None, expected: int
+ ):
+ self._make_live_pipelines(existing)
+ pipeline = self._make_pipeline(status=AutoresearchPipeline.Status.PAUSED)
+ with patch(
+ "products.autoresearch.backend.limits.posthoganalytics.get_feature_flag_payload",
+ return_value=self._live_cap_payload(override),
+ ):
+ resp = self.client.post(f"{self.base_url}/{pipeline.id}/resume/")
+
+ assert resp.status_code == expected, resp.content
+ pipeline.refresh_from_db()
+ if expected == status.HTTP_400_BAD_REQUEST:
+ assert "limit of 3 active models" in resp.json()["detail"]
+ assert pipeline.status == AutoresearchPipeline.Status.PAUSED
+ else:
+ assert pipeline.status == AutoresearchPipeline.Status.RUNNING
+
+ @parameterized.expand(
+ [
+ ("first_run_of_a_draft_is_refused", AutoresearchPipeline.Status.DRAFT, status.HTTP_400_BAD_REQUEST),
+ # A pipeline that is already live keeps training at the cap.
+ ("retrain_of_a_live_pipeline", AutoresearchPipeline.Status.RUNNING, status.HTTP_200_OK),
+ ]
+ )
+ def test_start_training_respects_the_live_pipeline_cap(
+ self, _name: str, start: AutoresearchPipeline.Status, expected: int
+ ):
+ self._make_live_pipelines([AutoresearchPipeline.Status.RUNNING] * 3)
+ pipeline = self._make_pipeline(status=start)
+
+ def _fake_run_training(*, pipeline: AutoresearchPipeline, iteration_budget: int, **_: Any):
+ return AutoresearchTrainingRun.objects.create(
+ pipeline=pipeline, status=AutoresearchTrainingRun.Status.RUNNING, iteration_budget=iteration_budget
+ )
+
+ with patch(
+ "products.autoresearch.backend.training.runner.run_training", side_effect=_fake_run_training
+ ) as mock_run_training:
+ resp = self.client.post(f"{self.base_url}/{pipeline.id}/train/")
+
+ assert resp.status_code == expected, resp.content
+ assert mock_run_training.called == (expected == status.HTTP_200_OK)
+
+ @parameterized.expand([("three_live", 3), ("draft_with_an_open_external_run_holds_a_slot", 2)])
+ def test_opening_an_external_run_on_a_draft_at_the_live_pipeline_cap_returns_400(self, _name: str, live: int):
+ self._make_live_pipelines([AutoresearchPipeline.Status.RUNNING] * live)
+ if live < 3:
+ # Opened under the cap, still a draft, and live once the run promotes.
+ other_draft = self._make_pipeline(name="Draft with an open run")
+ AutoresearchTrainingRun.objects.create(
+ pipeline=other_draft, status=AutoresearchTrainingRun.Status.RUNNING, iteration_budget=5
+ )
+ pipeline = self._make_pipeline()
+ resp = self.client.post(f"{self.base_url}/{pipeline.id}/training_runs/", {}, format="json")
+
+ assert resp.status_code == status.HTTP_400_BAD_REQUEST, resp.content
+ assert not AutoresearchTrainingRun.objects.for_team(self.team.pk).filter(pipeline=pipeline).exists()
+
+ def test_limits_report_usage_against_each_limit(self):
+ live = self._make_pipeline(status=AutoresearchPipeline.Status.RUNNING)
+ self._make_pipeline(status=AutoresearchPipeline.Status.DRAFT)
+ self._make_pipeline(status=AutoresearchPipeline.Status.PAUSED)
+ started_at = django_timezone.now() - timedelta(hours=3)
+ AutoresearchTrainingRun.objects.create(
+ pipeline=live, status=AutoresearchTrainingRun.Status.COMPLETED, iteration_budget=5, task_id=uuid.uuid4()
+ )
+ AutoresearchTrainingRun.objects.filter(pipeline=live).update(created_at=started_at)
+ # An externally opened run does not use a sandbox, so it does not count.
+ AutoresearchTrainingRun.objects.create(
+ pipeline=live, status=AutoresearchTrainingRun.Status.COMPLETED, iteration_budget=5
+ )
+ with patch(
+ "products.autoresearch.backend.limits.posthoganalytics.get_feature_flag_payload",
+ return_value={"default_team_config": {"max_training_runs_per_day": 1, "max_live_pipelines": 2}},
+ ):
+ resp = self.client.get(f"{self.base_url}/limits/")
+
+ assert resp.status_code == status.HTTP_200_OK, resp.content
+ assert resp.json() == {
+ "max_training_runs_per_day": 1,
+ "training_runs_used": 1,
+ "next_slot_at": (started_at + timedelta(hours=24)).isoformat().replace("+00:00", "Z"),
+ "max_iterations_per_run": 10,
+ "max_live_pipelines": 2,
+ "live_pipelines": 1,
+ }
+
# ──────────────────────────────────── update restrictions ─────────────────────────────────────
def _make_trained_pipeline(self) -> AutoresearchPipeline:
diff --git a/products/autoresearch/backend/training/test_agent_recorded_training.py b/products/autoresearch/backend/training/test_agent_recorded_training.py
index 171dd3c9ab76..33a79345be78 100644
--- a/products/autoresearch/backend/training/test_agent_recorded_training.py
+++ b/products/autoresearch/backend/training/test_agent_recorded_training.py
@@ -492,6 +492,13 @@ class TestTrainingRunHistory(TeamScopedTestMixin, APIBaseTest):
)
self._flag_patcher.start()
self.addCleanup(self._flag_patcher.stop)
+ # Each completed run promotes its draft to live, and some tests promote more pipelines than the default cap.
+ limits_patcher = patch(
+ "products.autoresearch.backend.limits.posthoganalytics.get_feature_flag_payload",
+ return_value={"default_team_config": {"max_live_pipelines": 10}},
+ )
+ limits_patcher.start()
+ self.addCleanup(limits_patcher.stop)
self.pipeline = AutoresearchPipeline.objects.create(
team=self.team, created_by=self.user, name="Main", target_event="downloaded_file", horizon_days=30
)
diff --git a/products/autoresearch/backend/training/test_training.py b/products/autoresearch/backend/training/test_training.py
index e9e396850072..373e1f87aec3 100644
--- a/products/autoresearch/backend/training/test_training.py
+++ b/products/autoresearch/backend/training/test_training.py
@@ -406,7 +406,9 @@ class TestRunTraining(TeamScopedTestMixin, BaseTest):
{"team_configs": {str(self.team.id): {"max_iterations_per_run": team_ceiling}}} if team_ceiling else None
)
limits = (
- TeamLimits(max_training_runs_per_day=3, max_iterations_per_run=caller_ceiling) if caller_ceiling else None
+ TeamLimits(max_training_runs_per_day=3, max_iterations_per_run=caller_ceiling, max_live_pipelines=3)
+ if caller_ceiling
+ else None
)
with patch(
diff --git a/products/autoresearch/frontend/AGENTS.md b/products/autoresearch/frontend/AGENTS.md
index dddc52ef124e..31c078a0f6cc 100644
--- a/products/autoresearch/frontend/AGENTS.md
+++ b/products/autoresearch/frontend/AGENTS.md
@@ -18,7 +18,7 @@ The product is mostly a backend, and the UI is deliberately thin — it reads st
| `newModel/` | Create-form parts: template chips, definition sentence, Advanced section, readiness checklist, "What happens next". |
| `autoresearchPipelineLogic.ts` | Detail logic. Also maps the old `?tab=` keys to the four tabs, so saved links still open. |
| `pipelineLifecycle.ts` | The five lifecycle steps (question set to checked against reality) and which one is current. |
-| `trainingScheduleLogic.ts` | Setup tab training settings: research interval, experiments per run, research model and thinking effort, their save, and the status line. |
+| `trainingScheduleLogic.ts` | Setup tab training settings and their save, the project's limits and usage, the status line, and why the limits disable Train now. |
| `agentSearch.ts` | The agent's search across training runs: chart points, best score so far, change per experiment within a run, log groups and status brief. |
| `onlinePerformance.ts` | Accuracy tab helpers over the `online_performance` endpoint: the stat tiles, the per-metric accuracy trend, predicted against actual. |
| `predictionCoverage.ts` | Predictions tab coverage from the runs list: the measured summary and the coverage and score age per day. |
diff --git a/products/autoresearch/frontend/AutoresearchPipelineScene.stories.tsx b/products/autoresearch/frontend/AutoresearchPipelineScene.stories.tsx
index b1400e0b9c88..cf8a1d73cf5b 100644
--- a/products/autoresearch/frontend/AutoresearchPipelineScene.stories.tsx
+++ b/products/autoresearch/frontend/AutoresearchPipelineScene.stories.tsx
@@ -15,6 +15,7 @@ import type {
AutoresearchRunApi,
AutoresearchSuggestionApi,
AutoresearchTrainingRunApi,
+ ProjectLimitsApi,
} from './generated/api.schemas'
const PIPELINE_ID = '0190c3f2-0000-7000-8000-000000000001'
@@ -366,6 +367,15 @@ const trainingRuns = [
},
] as unknown as AutoresearchTrainingRunApi[]
+const projectLimits: ProjectLimitsApi = {
+ max_training_runs_per_day: 3,
+ training_runs_used: 1,
+ next_slot_at: null,
+ max_iterations_per_run: 10,
+ max_live_pipelines: 3,
+ live_pipelines: 2,
+}
+
const meta: Meta = {
component: App,
title: 'Products/Autoresearch/Model detail scene',
@@ -391,6 +401,7 @@ const meta: Meta = {
]),
[`/api/projects/:team_id/autoresearch/${PIPELINE_ID}/suggestions/`]: toPaginatedResponse(suggestions),
[`/api/projects/:team_id/autoresearch/${PIPELINE_ID}/online_performance/`]: onlinePerformance,
+ '/api/projects/:team_id/autoresearch/limits/': projectLimits,
},
}),
],
diff --git a/products/autoresearch/frontend/autoresearchPipelineLogic.ts b/products/autoresearch/frontend/autoresearchPipelineLogic.ts
index 8f58d797ab04..aa830c71555a 100644
--- a/products/autoresearch/frontend/autoresearchPipelineLogic.ts
+++ b/products/autoresearch/frontend/autoresearchPipelineLogic.ts
@@ -1716,9 +1716,10 @@ export const autoresearchPipelineLogic = kea<autoresearchPipelineLogicType>([
posthog.capture('autoresearch model resumed', { pipeline_id: props.id })
lemonToast.success('Model resumed')
},
- resumePipelineFailure: () => {
+ resumePipelineFailure: ({ errorObject }) => {
posthog.capture('autoresearch model action failed', { action: 'resume', pipeline_id: props.id })
- lemonToast.error('Could not resume the model')
+ // A project at its active model limit explains it in `detail`.
+ lemonToast.error(errorObject?.detail ?? errorObject?.data?.detail ?? 'Could not resume the model')
},
scoreNowSuccess: ({ scoreResult }) => {
if (!scoreResult) {
diff --git a/products/autoresearch/frontend/generated/api.schemas.ts b/products/autoresearch/frontend/generated/api.schemas.ts
index eda072ac86db..cbd2d45a2d42 100644
--- a/products/autoresearch/frontend/generated/api.schemas.ts
+++ b/products/autoresearch/frontend/generated/api.schemas.ts
@@ -1972,6 +1972,24 @@ export interface AgentModelOptionsApi {
models: AgentModelOptionApi[]
}
+export interface ProjectLimitsApi {
+ /** Sandbox training runs the project may start in any 24 hours, across all its pipelines. 0 means training is turned off. */
+ max_training_runs_per_day: number
+ /** Sandbox training runs the project started in the last 24 hours, across all its pipelines. */
+ training_runs_used: number
+ /**
+ * When the next training run can start, while the project is at its daily limit. Null while a run can start now, and when training is turned off.
+ * @nullable
+ */
+ next_slot_at: string | null
+ /** Most experiments any one training run gets. A pipeline's iterations_per_run above this is lowered to it. */
+ max_iterations_per_run: number
+ /** Most pipelines the project may have live (bootstrapping, running or converged, or a draft with an open training run, which goes live when the run promotes). Starting a draft's first training run or resuming a paused pipeline is refused at this limit. */
+ max_live_pipelines: number
+ /** Pipelines that count against max_live_pipelines now. Can exceed it if the limit was lowered. */
+ live_pipelines: number
+}
+
/**
* * `likely_active_soon` - Likely Active Soon
* * `at_risk_of_inactivity` - At Risk Of Inactivity
diff --git a/products/autoresearch/frontend/generated/api.ts b/products/autoresearch/frontend/generated/api.ts
index 9bb59bb9431d..e8546bf51c13 100644
--- a/products/autoresearch/frontend/generated/api.ts
+++ b/products/autoresearch/frontend/generated/api.ts
@@ -41,6 +41,7 @@ import type {
PaginatedAutoresearchSuggestionListApi,
PaginatedAutoresearchTrainingRunListApi,
PatchedAutoresearchPipelineCreateApi,
+ ProjectLimitsApi,
RecordIterationApi,
ResolveTemplateRequestApi,
ResolvedTemplateApi,
@@ -812,7 +813,7 @@ export const getAutoresearchResumeCreateUrl = (projectId: string, id: string) =>
}
/**
- * Resume a paused pipeline. Daily scoring and training will restart on the next cadence tick.
+ * Resume a paused pipeline. Daily scoring and training will restart on the next cadence tick. Refused when the project already has its limit of live pipelines, which the limits endpoint reports.
* @summary Resume a pipeline
*/
export const autoresearchResumeCreate = async (
@@ -904,6 +905,24 @@ export const autoresearchAgentModelsRetrieve = async (
})
}
+export const getAutoresearchLimitsRetrieveUrl = (projectId: string) => {
+ return `/api/projects/${projectId}/autoresearch/limits/`
+}
+
+/**
+ * Return the project's autoresearch limits and how much of each it is using: sandbox training runs in the last 24 hours against the daily limit, with when the next run can start at the limit, the most experiments per training run, and live pipelines against the live pipeline limit. Read-only; it runs no queries.
+ * @summary Read the project's limits
+ */
+export const autoresearchLimitsRetrieve = async (
+ projectId: string,
+ options?: RequestInit
+): Promise<ProjectLimitsApi> => {
+ return apiMutator<ProjectLimitsApi>(getAutoresearchLimitsRetrieveUrl(projectId), {
+ ...options,
+ method: 'GET',
+ })
+}
+
export const getAutoresearchResolveTemplateCreateUrl = (projectId: string) => {
return `/api/projects/${projectId}/autoresearch/resolve-template/`
}
diff --git a/products/autoresearch/frontend/pipeline/SetupTab.tsx b/products/autoresearch/frontend/pipeline/SetupTab.tsx
index 35f5357c1574..b5a079649b1d 100644
--- a/products/autoresearch/frontend/pipeline/SetupTab.tsx
+++ b/products/autoresearch/frontend/pipeline/SetupTab.tsx
@@ -9,6 +9,7 @@ import {
AgentModelOptionsApi,
AgentReasoningEffortOptionApi,
AutoresearchPipelineApi,
+ ProjectLimitsApi,
ResearchIntervalDaysEnumApi,
} from '../generated/api.schemas'
import {
@@ -108,8 +109,15 @@ export function SetupTab(): JSX.Element {
}
function TrainingScheduleSetting({ pipeline }: { pipeline: AutoresearchPipelineApi }): JSX.Element {
- const { schedule, savingSchedule, saveDisabledReason, agentModelOptions, agentModelOptionsLoading, effortOptions } =
- useValues(trainingScheduleLogic({ id: pipeline.id }))
+ const {
+ schedule,
+ savingSchedule,
+ saveDisabledReason,
+ agentModelOptions,
+ agentModelOptionsLoading,
+ effortOptions,
+ projectLimits,
+ } = useValues(trainingScheduleLogic({ id: pipeline.id }))
const { setResearchIntervalDays, setIterationsPerRun, setAgentModel, setAgentReasoningEffort, saveSchedule } =
useActions(trainingScheduleLogic({ id: pipeline.id }))
return (
@@ -126,7 +134,10 @@ function TrainingScheduleSetting({ pipeline }: { pipeline: AutoresearchPipelineA
/>
</label>
<label className="flex flex-col gap-1">
- <span className="text-sm font-semibold">Experiments per run</span>
+ <span className="text-sm font-semibold">
+ Experiments per run
+ {projectLimits ? ` (up to ${projectLimits.max_iterations_per_run})` : ''}
+ </span>
<LemonInput
size="small"
type="number"
@@ -174,12 +185,23 @@ function TrainingScheduleSetting({ pipeline }: { pipeline: AutoresearchPipelineA
</LemonButton>
</div>
<p className="text-xs text-secondary mb-0">
- {trainingScheduleStatus(pipeline)} A larger model or more thinking effort costs more per training run.
+ {trainingScheduleStatus(pipeline, projectLimits)} A larger model or more thinking effort costs more per
+ training run.
</p>
+ {projectLimits && <ProjectUsage limits={projectLimits} />}
</div>
)
}
+function ProjectUsage({ limits }: { limits: ProjectLimitsApi }): JSX.Element {
+ return (
+ <p className="text-xs text-secondary mb-0" data-attr="autoresearch-project-limits">
+ This project has used {limits.training_runs_used} of {limits.max_training_runs_per_day} training runs in the
+ last 24 hours, and has {limits.live_pipelines} of {limits.max_live_pipelines} active models.
+ </p>
+ )
+}
+
function DetailRow({ label, children }: { label: string; children: React.ReactNode }): JSX.Element {
return (
<div className="flex justify-between items-start gap-4 py-2 border-b last:border-0">
diff --git a/products/autoresearch/frontend/pipeline/SteerAgentBar.tsx b/products/autoresearch/frontend/pipeline/SteerAgentBar.tsx
index c0f91b147200..3383b1b6950c 100644
--- a/products/autoresearch/frontend/pipeline/SteerAgentBar.tsx
+++ b/products/autoresearch/frontend/pipeline/SteerAgentBar.tsx
@@ -1,8 +1,9 @@
-import { useActions, useValues } from 'kea'
+import { useActions, useMountedLogic, useValues } from 'kea'
import { LemonButton, LemonCollapse, LemonTag } from '@posthog/lemon-ui'
import { autoresearchPipelineLogic } from '../autoresearchPipelineLogic'
+import { trainingScheduleLogic } from '../trainingScheduleLogic'
import { SuggestionForm } from './SuggestionForm'
import { SuggestionList } from './SuggestionList'
@@ -11,13 +12,15 @@ export function SteerAgentBar(): JSX.Element {
const { pipeline, startTrainingResultLoading, hasLiveTrainingRun, queuedSuggestionCount } =
useValues(autoresearchPipelineLogic)
const { startTraining, steerOpened } = useActions(autoresearchPipelineLogic)
+ const { props } = useMountedLogic(autoresearchPipelineLogic)
+ const { trainDisabledByLimitReason } = useValues(trainingScheduleLogic(props))
const trainDisabledReason = startTrainingResultLoading
? 'Starting…'
: pipeline?.status === 'paused'
? 'Resume the model to train it'
: hasLiveTrainingRun
? 'A training run is already in progress'
- : undefined
+ : trainDisabledByLimitReason
return (
<div className="flex flex-col @md:flex-row @md:items-start gap-2">
<LemonCollapse
diff --git a/products/autoresearch/frontend/trainingScheduleLogic.test.ts b/products/autoresearch/frontend/trainingScheduleLogic.test.ts
new file mode 100644
index 000000000000..c91d1a86661f
--- /dev/null
+++ b/products/autoresearch/frontend/trainingScheduleLogic.test.ts
@@ -0,0 +1,49 @@
+import { dayjs } from 'lib/dayjs'
+
+import type { AutoresearchPipelineApi, ProjectLimitsApi } from './generated/api.schemas'
+import { trainingScheduleStatus, trainingDisabledByLimitReason } from './trainingScheduleLogic'
+
+const NOW = dayjs('2026-03-01T12:00:00Z')
+
+const LIMITS: ProjectLimitsApi = {
+ max_training_runs_per_day: 3,
+ training_runs_used: 1,
+ next_slot_at: null,
+ max_iterations_per_run: 10,
+ max_live_pipelines: 3,
+ live_pipelines: 1,
+}
+
+const PIPELINE = {
+ status: 'running',
+ research_interval_days: 7,
+ training_phase: 'maintain',
+ next_research_at: '2026-03-05T00:00:00Z',
+} as AutoresearchPipelineApi
+
+describe('trainingScheduleLogic limits', () => {
+ it.each<[string, Partial<ProjectLimitsApi>, string]>([
+ ['a slot is free', {}, 'Next research run on Mar 5.'],
+ [
+ 'the project is at its daily limit',
+ { training_runs_used: 3, next_slot_at: '2026-03-01T15:30:00Z' },
+ `This project is at its daily limit of 3 training runs. The next run can start at ${dayjs('2026-03-01T15:30:00Z').format('MMM D, h:mm A')}.`,
+ ],
+ [
+ 'training is turned off',
+ { max_training_runs_per_day: 0, training_runs_used: 0 },
+ 'Training is turned off for this project. Contact support to turn it on.',
+ ],
+ ])('status line when %s', (_name, overrides, expected) => {
+ expect(trainingScheduleStatus(PIPELINE, { ...LIMITS, ...overrides }, NOW)).toEqual(expected)
+ })
+
+ it.each<[string, AutoresearchPipelineApi['status'], Partial<ProjectLimitsApi>, boolean]>([
+ ['a draft at the active model limit', 'draft', { live_pipelines: 3 }, true],
+ ['a live model retraining at the active model limit', 'running', { live_pipelines: 3 }, false],
+ ['a draft under the active model limit', 'draft', {}, false],
+ ])('train is disabled for %s', (_name, status, overrides, disabled) => {
+ const reason = trainingDisabledByLimitReason({ ...PIPELINE, status }, { ...LIMITS, ...overrides })
+ expect(reason !== undefined).toBe(disabled)
+ })
+})
diff --git a/products/autoresearch/frontend/trainingScheduleLogic.ts b/products/autoresearch/frontend/trainingScheduleLogic.ts
index 710454142fb2..5830fbbf8fb0 100644
--- a/products/autoresearch/frontend/trainingScheduleLogic.ts
+++ b/products/autoresearch/frontend/trainingScheduleLogic.ts
@@ -7,11 +7,13 @@ import { lemonToast } from 'lib/lemon-ui/LemonToast'
import { teamLogic } from 'scenes/teamLogic'
import { AutoresearchPipelineLogicProps, autoresearchPipelineLogic } from './autoresearchPipelineLogic'
-import { autoresearchAgentModelsRetrieve, autoresearchPartialUpdate } from './generated/api'
+import { autoresearchAgentModelsRetrieve, autoresearchLimitsRetrieve, autoresearchPartialUpdate } from './generated/api'
import type {
AgentModelOptionsApi,
AgentReasoningEffortOptionApi,
AutoresearchPipelineApi,
+ AutoresearchTrainingRunApi,
+ ProjectLimitsApi,
ResearchIntervalDaysEnumApi,
} from './generated/api.schemas'
@@ -26,8 +28,49 @@ export interface TrainingScheduleDraft {
agent_reasoning_effort: string | null
}
-/** One line on what training does next, from the saved schedule. */
-export function trainingScheduleStatus(pipeline: AutoresearchPipelineApi, now: dayjs.Dayjs = dayjs()): string {
+/** Why no training run can start now because of the project's daily limit, or undefined while one can. */
+function dailyTrainingLimitReason(limits: ProjectLimitsApi | null): string | undefined {
+ if (!limits) {
+ return undefined
+ }
+ if (limits.max_training_runs_per_day === 0) {
+ return 'Training is turned off for this project. Contact support to turn it on.'
+ }
+ if (!limits.next_slot_at) {
+ return undefined
+ }
+ return `This project is at its daily limit of ${limits.max_training_runs_per_day} training runs. The next run can start at ${dayjs(limits.next_slot_at).format('MMM D, h:mm A')}.`
+}
+
+/** Why a draft cannot start its first training run because of the project's active model limit. */
+function liveModelLimitReason(
+ pipeline: AutoresearchPipelineApi | null,
+ limits: ProjectLimitsApi | null
+): string | undefined {
+ if (!pipeline || !limits || pipeline.status !== 'draft' || limits.live_pipelines < limits.max_live_pipelines) {
+ return undefined
+ }
+ return `This project has reached its limit of ${limits.max_live_pipelines} active models. Pause or delete one to train this model.`
+}
+
+/** Why the project's limits stop this model from training now, or undefined when they do not. */
+export function trainingDisabledByLimitReason(
+ pipeline: AutoresearchPipelineApi | null,
+ limits: ProjectLimitsApi | null
+): string | undefined {
+ return liveModelLimitReason(pipeline, limits) ?? dailyTrainingLimitReason(limits)
+}
+
+/** One line on what training does next, from the saved schedule and the project's daily limit. */
+export function trainingScheduleStatus(
+ pipeline: AutoresearchPipelineApi,
+ limits: ProjectLimitsApi | null = null,
+ now: dayjs.Dayjs = dayjs()
+): string {
+ const limitReason = dailyTrainingLimitReason(limits)
+ if (limitReason) {
+ return limitReason
+ }
if (pipeline.research_interval_days === null || !pipeline.next_research_at) {
return 'Training starts only when you ask for it.'
}
@@ -63,6 +106,9 @@ export interface trainingScheduleLogicValues {
agentModelOptions: AgentModelOptionsApi | null
agentModelOptionsLoading: boolean
effortOptions: AgentReasoningEffortOptionApi[]
+ projectLimits: ProjectLimitsApi | null
+ projectLimitsLoading: boolean
+ trainDisabledByLimitReason: string | undefined
}
export interface trainingScheduleLogicActions {
@@ -82,6 +128,21 @@ export interface trainingScheduleLogicActions {
payload?: any
}
loadAgentModelOptionsFailure: (error: string) => { error: string }
+ loadProjectLimits: () => { value: true }
+ loadProjectLimitsSuccess: (projectLimits: ProjectLimitsApi | null) => {
+ projectLimits: ProjectLimitsApi | null
+ payload?: any
+ }
+ loadProjectLimitsFailure: (error: string) => { error: string }
+ startTrainingSuccess: (
+ startTrainingResult: AutoresearchTrainingRunApi | null,
+ payload?: any
+ ) => { startTrainingResult: AutoresearchTrainingRunApi | null; payload?: any }
+ startTrainingFailure: (error: string, errorObject?: any) => { error: string; errorObject?: any }
+ resumePipelineSuccess: (
+ pipeline: AutoresearchPipelineApi | null,
+ payload?: any
+ ) => { pipeline: AutoresearchPipelineApi | null; payload?: any }
resetDraft: () => { value: true }
saveSchedule: () => { value: true }
saveScheduleFinished: () => { value: true }
@@ -99,7 +160,10 @@ export const trainingScheduleLogic = kea<trainingScheduleLogicType>([
path((key) => ['products', 'autoresearch', 'frontend', 'trainingScheduleLogic', key]),
connect((props: AutoresearchPipelineLogicProps) => ({
values: [teamLogic, ['currentTeamId'], autoresearchPipelineLogic(props), ['pipeline']],
- actions: [autoresearchPipelineLogic(props), ['loadPipelineSuccess']],
+ actions: [
+ autoresearchPipelineLogic(props),
+ ['loadPipelineSuccess', 'startTrainingSuccess', 'startTrainingFailure', 'resumePipelineSuccess'],
+ ],
})),
actions({
setResearchIntervalDays: (days: ResearchIntervalDaysEnumApi | null) => ({ days }),
@@ -143,6 +207,17 @@ export const trainingScheduleLogic = kea<trainingScheduleLogicType>([
},
},
],
+ projectLimits: [
+ null as ProjectLimitsApi | null,
+ {
+ loadProjectLimits: async () => {
+ if (!values.currentTeamId) {
+ return null
+ }
+ return autoresearchLimitsRetrieve(String(values.currentTeamId))
+ },
+ },
+ ],
})),
selectors({
schedule: [
@@ -184,6 +259,11 @@ export const trainingScheduleLogic = kea<trainingScheduleLogicType>([
return undefined
},
],
+ trainDisabledByLimitReason: [
+ (s) => [s.pipeline, s.projectLimits],
+ (pipeline: AutoresearchPipelineApi | null, projectLimits: ProjectLimitsApi | null): string | undefined =>
+ trainingDisabledByLimitReason(pipeline, projectLimits),
+ ],
effortOptions: [
(s) => [s.agentModelOptions, s.schedule],
(
@@ -199,6 +279,10 @@ export const trainingScheduleLogic = kea<trainingScheduleLogicType>([
],
}),
listeners(({ actions, values, props }) => ({
+ // Each changes the project's usage, and a refused start means the shown limits were stale.
+ startTrainingSuccess: () => actions.loadProjectLimits(),
+ startTrainingFailure: () => actions.loadProjectLimits(),
+ resumePipelineSuccess: () => actions.loadProjectLimits(),
saveSchedule: async () => {
const { schedule, currentTeamId } = values
if (!schedule || schedule.iterations_per_run === undefined || !currentTeamId) {
@@ -252,6 +336,7 @@ export const trainingScheduleLogic = kea<trainingScheduleLogicType>([
})),
afterMount(({ actions }) => {
actions.loadAgentModelOptions()
+ actions.loadProjectLimits()
}),
])
diff --git a/products/autoresearch/mcp/tools.yaml b/products/autoresearch/mcp/tools.yaml
index 082a6175c5a2..5e4db165ff51 100644
--- a/products/autoresearch/mcp/tools.yaml
+++ b/products/autoresearch/mcp/tools.yaml
@@ -77,6 +77,24 @@ tools:
- status
- iteration_budget_remaining
feature_flag: autoresearch
+ autoresearch-limits-retrieve:
+ operation: autoresearch_limits_retrieve
+ enabled: false
+ disabled_reason:
+ Feeds the Setup tab. The refusals from autoresearch-train-create already name the limit and when the
+ next run can start, which keeps the family under its tool cap
+ scopes:
+ - autoresearch:read
+ annotations:
+ readOnly: true
+ destructive: false
+ idempotent: true
+ title: Read project limits
+ description: >
+ Read the project's autoresearch limits and current usage: training runs in the last 24 hours against the
+ daily limit and when the next can start, the most experiments per run, and live pipelines against the live
+ pipeline limit.
+ feature_flag: autoresearch
autoresearch-list:
operation: autoresearch_list
enabled: true
@@ -491,16 +509,17 @@ tools:
reaches 'completed' or 'failed'. A pipeline's first run has no champion until it completes and server-side
promotion runs; on a retrain the existing champion keeps scoring until a new one is promoted. Returns 400 if
a training run is already in progress, the pipeline is paused, or its creator or target is no longer valid
- (recreate the pipeline). Returns 403 from inside a sandbox, such as a PostHog AI cloud task: do not retry,
- and ask the user to start training from the pipeline's page in PostHog. Needs PostHog Desktop access,
- because the run is a cloud sandbox. After a run promotes a champion, the daily 02:00 UTC scoring job scores
- the inference population and emits autoresearch_prediction events. To score now instead, call
- autoresearch-score-create once. The run uses the pipeline's iterations_per_run unless you override
- iteration_budget for this run, and the project's iteration limit per run can lower either. Returns 429 when
- the project has started its limit of training runs in the last 24 hours, across all its pipelines: do not
- retry before next_slot_at in the response, and tell the user when the next run can start. A manual run
- resets the clock for the next scheduled research run. A pipeline with an action target also needs the
- action:read scope.
+ (recreate the pipeline). Also returns 400 on a draft pipeline when the project already has its limit of live
+ pipelines (bootstrapping, running or converged): tell the user to pause or delete one first, and do not
+ retry. Returns 403 from inside a sandbox, such as a PostHog AI cloud task: do not retry, and ask the user to
+ start training from the pipeline's page in PostHog. Needs PostHog Desktop access, because the run is a cloud
+ sandbox. After a run promotes a champion, the daily 02:00 UTC scoring job scores the inference population
+ and emits autoresearch_prediction events. To score now instead, call autoresearch-score-create once. The run
+ uses the pipeline's iterations_per_run unless you override iteration_budget for this run, and the project's
+ iteration limit per run can lower either. Returns 429 when the project has started its limit of training
+ runs in the last 24 hours, across all its pipelines: do not retry before next_slot_at in the response, and
+ tell the user when the next run can start. A manual run resets the clock for the next scheduled research
+ run. A pipeline with an action target also needs the action:read scope.
feature_flag: autoresearch
autoresearch-training-runs-artifacts-delete-create:
operation: autoresearch_training_runs_artifacts_delete_create
diff --git a/services/mcp/schema/generated-tool-definitions.json b/services/mcp/schema/generated-tool-definitions.json
index eeecb631d6d6..35c2d39b3f77 100644
--- a/services/mcp/schema/generated-tool-definitions.json
+++ b/services/mcp/schema/generated-tool-definitions.json
@@ -1642,7 +1642,7 @@
"feature_flag": "autoresearch"
},
"autoresearch-train-create": {
- "description": "Start an asynchronous training run for a pipeline. Launches the autoresearch agent in a sandbox and returns the training run immediately with status 'running'. Poll autoresearch-training-runs-list until the run reaches 'completed' or 'failed'. A pipeline's first run has no champion until it completes and server-side promotion runs; on a retrain the existing champion keeps scoring until a new one is promoted. Returns 400 if a training run is already in progress, the pipeline is paused, or its creator or target is no longer valid (recreate the pipeline). Returns 403 from inside a sandbox, such as a PostHog AI cloud task: do not retry, and ask the user to start training from the pipeline's page in PostHog. Needs PostHog Desktop access, because the run is a cloud sandbox. After a run promotes a champion, the daily 02:00 UTC scoring job scores the inference population and emits autoresearch_prediction events. To score now instead, call autoresearch-score-create once. The run uses the pipeline's iterations_per_run unless you override iteration_budget for this run, and the project's iteration limit per run can lower either. Returns 429 when the project has started its limit of training runs in the last 24 hours, across all its pipelines: do not retry before next_slot_at in the response, and tell the user when the next run can start. A manual run resets the clock for the next scheduled research run. A pipeline with an action target also needs the action:read scope.",
+ "description": "Start an asynchronous training run for a pipeline. Launches the autoresearch agent in a sandbox and returns the training run immediately with status 'running'. Poll autoresearch-training-runs-list until the run reaches 'completed' or 'failed'. A pipeline's first run has no champion until it completes and server-side promotion runs; on a retrain the existing champion keeps scoring until a new one is promoted. Returns 400 if a training run is already in progress, the pipeline is paused, or its creator or target is no longer valid (recreate the pipeline). Also returns 400 on a draft pipeline when the project already has its limit of live pipelines (bootstrapping, running or converged): tell the user to pause or delete one first, and do not retry. Returns 403 from inside a sandbox, such as a PostHog AI cloud task: do not retry, and ask the user to start training from the pipeline's page in PostHog. Needs PostHog Desktop access, because the run is a cloud sandbox. After a run promotes a champion, the daily 02:00 UTC scoring job scores the inference population and emits autoresearch_prediction events. To score now instead, call autoresearch-score-create once. The run uses the pipeline's iterations_per_run unless you override iteration_budget for this run, and the project's iteration limit per run can lower either. Returns 429 when the project has started its limit of training runs in the last 24 hours, across all its pipelines: do not retry before next_slot_at in the response, and tell the user when the next run can start. A manual run resets the clock for the next scheduled research run. A pipeline with an action target also needs the action:read scope.",
"category": "Autoresearch",
"feature": "autoresearch",
"summary": "Start training",
diff --git a/services/mcp/schema/tool-definitions-all.json b/services/mcp/schema/tool-definitions-all.json
index c71f39768ff1..974e64034fa9 100644
--- a/services/mcp/schema/tool-definitions-all.json
+++ b/services/mcp/schema/tool-definitions-all.json
@@ -1657,7 +1657,7 @@
"feature_flag": "autoresearch"
},
"autoresearch-train-create": {
- "description": "Start an asynchronous training run for a pipeline. Launches the autoresearch agent in a sandbox and returns the training run immediately with status 'running'. Poll autoresearch-training-runs-list until the run reaches 'completed' or 'failed'. A pipeline's first run has no champion until it completes and server-side promotion runs; on a retrain the existing champion keeps scoring until a new one is promoted. Returns 400 if a training run is already in progress, the pipeline is paused, or its creator or target is no longer valid (recreate the pipeline). Returns 403 from inside a sandbox, such as a PostHog AI cloud task: do not retry, and ask the user to start training from the pipeline's page in PostHog. Needs PostHog Desktop access, because the run is a cloud sandbox. After a run promotes a champion, the daily 02:00 UTC scoring job scores the inference population and emits autoresearch_prediction events. To score now instead, call autoresearch-score-create once. The run uses the pipeline's iterations_per_run unless you override iteration_budget for this run, and the project's iteration limit per run can lower either. Returns 429 when the project has started its limit of training runs in the last 24 hours, across all its pipelines: do not retry before next_slot_at in the response, and tell the user when the next run can start. A manual run resets the clock for the next scheduled research run. A pipeline with an action target also needs the action:read scope.",
+ "description": "Start an asynchronous training run for a pipeline. Launches the autoresearch agent in a sandbox and returns the training run immediately with status 'running'. Poll autoresearch-training-runs-list until the run reaches 'completed' or 'failed'. A pipeline's first run has no champion until it completes and server-side promotion runs; on a retrain the existing champion keeps scoring until a new one is promoted. Returns 400 if a training run is already in progress, the pipeline is paused, or its creator or target is no longer valid (recreate the pipeline). Also returns 400 on a draft pipeline when the project already has its limit of live pipelines (bootstrapping, running or converged): tell the user to pause or delete one first, and do not retry. Returns 403 from inside a sandbox, such as a PostHog AI cloud task: do not retry, and ask the user to start training from the pipeline's page in PostHog. Needs PostHog Desktop access, because the run is a cloud sandbox. After a run promotes a champion, the daily 02:00 UTC scoring job scores the inference population and emits autoresearch_prediction events. To score now instead, call autoresearch-score-create once. The run uses the pipeline's iterations_per_run unless you override iteration_budget for this run, and the project's iteration limit per run can lower either. Returns 429 when the project has started its limit of training runs in the last 24 hours, across all its pipelines: do not retry before next_slot_at in the response, and tell the user when the next run can start. A manual run resets the clock for the next scheduled research run. A pipeline with an action target also needs the action:read scope.",
"category": "Autoresearch",
"feature": "autoresearch",
"summary": "Start training",
diff --git a/services/mcp/src/api/generated.ts b/services/mcp/src/api/generated.ts
index d8a2304888e0..09c7db6a5bc0 100644
--- a/services/mcp/src/api/generated.ts
+++ b/services/mcp/src/api/generated.ts
@@ -93479,6 +93479,24 @@ export namespace Schemas {
web_analytics_pre_aggregated_tables_enabled?: boolean | null;
}
+ export interface ProjectLimits {
+ /** Sandbox training runs the project may start in any 24 hours, across all its pipelines. 0 means training is turned off. */
+ max_training_runs_per_day: number;
+ /** Sandbox training runs the project started in the last 24 hours, across all its pipelines. */
+ training_runs_used: number;
+ /**
+ * When the next training run can start, while the project is at its daily limit. Null while a run can start now, and when training is turned off.
+ * @nullable
+ */
+ next_slot_at: string | null;
+ /** Most experiments any one training run gets. A pipeline's iterations_per_run above this is lowered to it. */
+ max_iterations_per_run: number;
+ /** Most pipelines the project may have live (bootstrapping, running or converged, or a draft with an open training run, which goes live when the run promotes). Starting a draft's first training run or resuming a paused pipeline is refused at this limit. */
+ max_live_pipelines: number;
+ /** Pipelines that count against max_live_pipelines now. Can exceed it if the limit was lowered. */
+ live_pipelines: number;
+ }
+
/**
* The compact envelope returned ahead of the verbose `payload`.
*