:hedgehog: PostHog is the leading platform for building self-driving products. Our developer tools – AI observability, analytics, session replay, flags, experiments, error tracking, logs, and more – capture all the context agents need to diagnose problems, uncover opportunities, and ship fixes. Steer it all from Slack, web, desktop, or the MCP.
Python ab-testingai-analyticsanalyticscdpdata-warehouseexperimentsfeature-flagsjavascriptproduct-analyticspythonreactsession-replaysurveystypescriptweb-analytics
feat(api): add strict_fetch to catch lazy loads in list endpoints (#126317)
Paul D'Ambra0b4e4deb79a6
diff --git a/.agents/skills/improving-drf-endpoints/SKILL.md b/.agents/skills/improving-drf-endpoints/SKILL.mdindex 8f680ac5d5e2..58c3cf8089cb 100644--- a/.agents/skills/improving-drf-endpoints/SKILL.md+++ b/.agents/skills/improving-drf-endpoints/SKILL.md@@ -69,6 +69,7 @@ See [serializer-fields.md](references/serializer-fields.md) for patterns and exa14. **Prefer `@validated_request`** over manual `serializer.is_valid()` + `@extend_schema` — it handles both in one decorator15. **ViewSets outside `products/` need `@extend_schema(extensions={"x-product": "<product>"})`** — ViewSets in `products/<name>/backend/` are auto-attributed via module path; ViewSets in `posthog/api/` or `ee/` aren't and must declare attribution explicitly via the `x-product` extension. Accepts a plain string (`"product_analytics"`) or `ProductKey.X` enum (kebab values are normalized). Don't use `tags=["<product>"]` to influence codegen routing — `tags` is for Swagger UI display only. Without `x-product`, the MCP scaffold and frontend type generator can't route the endpoint to the right product16. **`partial_update` `request=` override must be a superset of runtime write fields** — `extend_schema(request=CustomSerializer)` replaces drf-spectacular's inference from `serializer_class`; omitted fields disappear from OpenAPI, frontend types, and MCP tool schemas even when the runtime serializer still accepts them. After changing the override, run `hogli build:openapi` and verify generated MCP tool schemas still expose every OpenAPI body field+17. **Wrap the list queryset with `strict_fetch()`** from `posthog.models.utils`, in `safely_get_queryset` when `self.action == "list"`. In tests and local dev, a lazy load of a forward relation or deferred field then raises `FieldFetchBlocked`, so a missing `select_related` fails the list test. In production, the missing field costs one extra query for the page, not one per row. Fix a failure with the `select_related` the error names, or stop deferring that field. The guard does not cover reverse FK or M2M managers, so keep `prefetch_related` and an `assertNumQueries` test for those. Reference: `AnnotationsViewSet` in `products/annotations/backend/api/annotation.py`**Streaming endpoints:** For SSE or streaming responses, use `@extend_schema(request=InputSerializer, responses={(200, "text/event-stream"): OpenApiTypes.STR})` to document the request schema even though the response can't be fully typed.diff --git a/posthog/models/utils.py b/posthog/models/utils.pyindex d2f61f0d919a..56a0b4812c18 100644--- a/posthog/models/utils.py+++ b/posthog/models/utils.py@@ -11,12 +11,13 @@ from decimal import Decimalfrom typing import TYPE_CHECKING, Any, Literal, Optional, TypeVar, Unionfrom uuid import UUID+from django.conf import settingsfrom django.contrib.auth.hashers import PBKDF2PasswordHasherfrom django.core.exceptions import ValidationErrorfrom django.db import IntegrityError, connections, models, transactionfrom django.db.backends.ddl_references import Statementfrom django.db.backends.utils import CursorWrapper-from django.db.models import Q, Subquery, UniqueConstraint+from django.db.models import Q, QuerySet, Subquery, UniqueConstraintfrom django.db.models.constraints import BaseConstraintfrom django.utils.text import slugify@@ -27,6 +28,7 @@ if TYPE_CHECKING:from posthog.hogql import astT = TypeVar("T")+ModelT = TypeVar("ModelT", bound=models.Model)BASE62 = string.digits + string.ascii_letters # All lowercase ASCII letters + all uppercase ASCII letters + digitsAMBIGUOUS_CHARS = frozenset("01OIl")@@ -338,6 +340,13 @@ def get_deferred_field_set_for_model(return {f"{field_prefix}{x.name}" for x in model._meta.fields if x.name not in fields_not_deferred}+def strict_fetch(queryset: QuerySet[ModelT]) -> QuerySet[ModelT]:+ # In tests and local dev, a lazy load of a related object or deferred field raises FieldFetchBlocked,+ # so a missing select_related fails loudly. In production, Django loads that field for all rows of the+ # queryset in one query, so a missing select_related costs one extra query and not one per row.+ return queryset.fetch_mode(models.FETCH_RAISE if settings.DEBUG or settings.TEST else models.FETCH_PEERS)++class UniqueConstraintByExpression(BaseConstraint):def __init__(self, *, name: str, expression: str, concurrently=True):self.name = namediff --git a/products/annotations/backend/api/annotation.py b/products/annotations/backend/api/annotation.pyindex 1bee20c539bf..26dae0b58731 100644--- a/products/annotations/backend/api/annotation.py+++ b/products/annotations/backend/api/annotation.py@@ -10,6 +10,7 @@ from posthog.api.routing import TeamAndOrgViewSetMixinfrom posthog.api.scoped_related_fields import TeamScopedPrimaryKeyRelatedFieldfrom posthog.api.shared import UserBasicSerializerfrom posthog.models.user import User+from posthog.models.utils import strict_fetchfrom products.annotations.backend.models.annotation import Annotationfrom products.dashboards.backend.models.dashboard import Dashboard@@ -233,7 +234,7 @@ class AnnotationsViewSet(TeamAndOrgViewSetMixin, ForbidDestroyModel, viewsets.Modef safely_get_queryset(self, queryset) -> QuerySet:if self.action == "list":# `date_marker` alone is not a total order, so limit-offset pages can repeat or drop rows.- queryset = queryset.order_by("-date_marker", "-id")+ queryset = strict_fetch(queryset.order_by("-date_marker", "-id"))if self.action != "partial_update":# We never want deleted items to be included in the queryset… except when we want to restore an annotation# That's because annotations are restored with a PATCH request setting `deleted` to `False`