:hedgehog: PostHog is the leading platform for building self-driving products. Our developer tools – AI observability, analytics, session replay, flags, experiments, error tracking, logs, and more – capture all the context agents need to diagnose problems, uncover opportunities, and ship fixes. Steer it all from Slack, web, desktop, or the MCP.
Python ab-testingai-analyticsanalyticscdpdata-warehouseexperimentsfeature-flagsjavascriptproduct-analyticspythonreactsession-replaysurveystypescriptweb-analytics
feat(data-warehouse): add mighty networks membership and rsvp tables (#122655)
Tom Owers448d802cda68
diff --git a/products/warehouse_sources/backend/temporal/data_imports/sources/COVERAGE_GAPS_APPENDIX.md b/products/warehouse_sources/backend/temporal/data_imports/sources/COVERAGE_GAPS_APPENDIX.mdindex 09b1a039a4dd..9c8ab4b6ecff 100644--- a/products/warehouse_sources/backend/temporal/data_imports/sources/COVERAGE_GAPS_APPENDIX.md+++ b/products/warehouse_sources/backend/temporal/data_imports/sources/COVERAGE_GAPS_APPENDIX.md@@ -8016,16 +8016,16 @@ Note: The Clarity Data Export API has exactly one endpoint: GET https://www.clar## Mighty Networks — gaps-Today (9): `Badges`, `Events`, `Members`, `Plans`, `Posts`, `Purchases`, `Spaces`, `Subscriptions`, `Tags`+Today (14): `Badges`, `EventRsvps`, `Events`, `MemberBadges`, `MemberTags`, `Members`, `PlanMembers`, `Plans`, `Posts`, `Purchases`, `SpaceMembers`, `Spaces`, `Subscriptions`, `Tags`Diffed against: <https://api.mn.co/admin/v1/spec.json>-- [ ] `Space members (GET /admin/v1/networks/{network_id}/spaces/{space_id}/members)` — connect members to spaces to measure community participation and access (high)-- [ ] `Plan members (GET /admin/v1/networks/{network_id}/plans/{plan_id}/members)` — measure plan access, including memberships that payment subscriptions do not represent (high)+- [x] `Space members (GET /admin/v1/networks/{network_id}/spaces/{space_id}/members)` — connect members to spaces to measure community participation and access (high)+- [x] `Plan members (GET /admin/v1/networks/{network_id}/plans/{plan_id}/members)` — measure plan access, including memberships that payment subscriptions do not represent (high)- [ ] `Post comments (GET /admin/v1/networks/{network_id}/posts/{post_id}/comments)` — analyze member discussion activity and engagement on existing posts (medium)- [ ] `Post reactions (GET /admin/v1/networks/{network_id}/posts/{post_id}/reactions)` — measure reactions by member and post to identify engaging content (medium)-- [ ] `Event RSVPs (GET /admin/v1/networks/{network_id}/events/{event_id}/rsvps)` — measure event registration and response status by member (medium)-- [ ] `Member tags and badges (GET /admin/v1/networks/{network_id}/members/{member_id}/tags; GET /admin/v1/networks/{network_id}/members/{member_id}/badges)` — connect members to existing tag and badge catalogs for segmentation and recognition analysis (medium)+- [x] `Event RSVPs (GET /admin/v1/networks/{network_id}/events/{event_id}/rsvps)` — measure event registration and response status by member (medium)+- [x] `Member tags and badges (GET /admin/v1/networks/{network_id}/members/{member_id}/tags; GET /admin/v1/networks/{network_id}/members/{member_id}/badges)` — connect members to existing tag and badge catalogs for segmentation and recognition analysis (medium)- [ ] `Custom field answers (GET /admin/v1/networks/{network_id}/custom_fields/{custom_field_id}/members/{member_id}/answers)` — segment members by profile answers; field definitions and option lists provide labels for those answers (medium)- [ ] `Invites (GET /admin/v1/networks/{network_id}/invites; GET /admin/v1/networks/{network_id}/plans/{plan_id}/invites)` — analyze invitation volume, recipients, and senders for network and plan acquisition (medium)- [ ] `Polls (GET /admin/v1/networks/{network_id}/polls)` — analyze community questions, poll choices, and publishing activity that the post schema does not expose (medium)diff --git a/products/warehouse_sources/backend/temporal/data_imports/sources/mighty_networks/canonical_descriptions.py b/products/warehouse_sources/backend/temporal/data_imports/sources/mighty_networks/canonical_descriptions.pyindex 845bcaaacb90..8abbaded6fcd 100644--- a/products/warehouse_sources/backend/temporal/data_imports/sources/mighty_networks/canonical_descriptions.py+++ b/products/warehouse_sources/backend/temporal/data_imports/sources/mighty_networks/canonical_descriptions.py@@ -154,4 +154,56 @@ CANONICAL_DESCRIPTIONS: CanonicalDescriptions = {"avatar_url": "URL to the badge image.",},},+ "SpaceMembers": {+ "description": "Members of each Space, one row per Space and member.",+ "docs_url": _DOCS_URL,+ "columns": {+ "space_id": "The ID of the Space the member belongs to.",+ "id": "The member's integer ID.",+ "email": "The member's email address.",+ "member_type": "The member's type: 'full' (belongs to the Network) or 'limited' (belongs only to specific spaces).",+ },+ },+ "PlanMembers": {+ "description": "Members with access through each Plan, one row per Plan and member.",+ "docs_url": _DOCS_URL,+ "columns": {+ "plan_id": "The ID of the Plan that gives the member access.",+ "id": "The member's integer ID.",+ "email": "The member's email address.",+ },+ },+ "MemberTags": {+ "description": "Tags assigned to each member, one row per member and tag.",+ "docs_url": _DOCS_URL,+ "columns": {+ "member_id": "The ID of the member the tag is assigned to.",+ "id": "The tag's integer ID.",+ "title": "The title of the tag.",+ "custom_field_id": "The ID of the custom field this tag belongs to.",+ },+ },+ "MemberBadges": {+ "description": "Badges awarded to each member, one row per member and badge.",+ "docs_url": _DOCS_URL,+ "columns": {+ "member_id": "The ID of the member the badge is awarded to.",+ "id": "The badge's integer ID.",+ "title": "The title of the badge.",+ "custom_field_id": "The ID of the custom field this badge belongs to.",+ },+ },+ "EventRsvps": {+ "description": "Member RSVPs to Events.",+ "docs_url": _DOCS_URL,+ "columns": {+ "event_id": "The ID of the Event the RSVP is for.",+ "id": "The RSVP's integer ID.",+ "updated": "The date and time the RSVP was last updated.",+ "status": "The RSVP status: yes, maybe, or no.",+ "event": "The Event being RSVPed to.",+ "member": "The member who set the RSVP.",+ "event_instance": "The specific occurrence of a recurring Event.",+ },+ },}diff --git a/products/warehouse_sources/backend/temporal/data_imports/sources/mighty_networks/mighty_networks.py b/products/warehouse_sources/backend/temporal/data_imports/sources/mighty_networks/mighty_networks.pyindex 497daca66c30..125b891a777e 100644--- a/products/warehouse_sources/backend/temporal/data_imports/sources/mighty_networks/mighty_networks.py+++ b/products/warehouse_sources/backend/temporal/data_imports/sources/mighty_networks/mighty_networks.py@@ -1,21 +1,30 @@import dataclassesfrom collections.abc import Callable-from typing import Any, Optional+from typing import Any, Optional, castfrom products.warehouse_sources.backend.temporal.data_imports.sources.common.http import make_tracked_sessionfrom products.warehouse_sources.backend.temporal.data_imports.sources.common.rest_source import (RESTAPIConfig,rest_api_resource,)+from products.warehouse_sources.backend.temporal.data_imports.sources.common.rest_source.fanout import (+ build_dependent_resource,+)from products.warehouse_sources.backend.temporal.data_imports.sources.common.rest_source.paginators import (PageNumberPaginator,)-from products.warehouse_sources.backend.temporal.data_imports.sources.common.rest_source.typing import ClientConfig+from products.warehouse_sources.backend.temporal.data_imports.sources.common.rest_source.resource import Resource+from products.warehouse_sources.backend.temporal.data_imports.sources.common.rest_source.typing import (+ ClientConfig,+ Endpoint,+)from products.warehouse_sources.backend.temporal.data_imports.sources.common.resumable import ResumableSourceManagerfrom products.warehouse_sources.backend.temporal.data_imports.sources.common.source_helpers import validate_via_probefrom products.warehouse_sources.backend.temporal.data_imports.sources.common.typings import SourceResponsefrom products.warehouse_sources.backend.temporal.data_imports.sources.mighty_networks.settings import (BASE_URL,+ ENDPOINT_CONFIGS,+ FANOUTS,PARTITION_FIELDS,PATHS,PER_PAGE,@@ -31,6 +40,8 @@ USER_AGENT = "PostHog-DataWarehouse/1.0 (+https://posthog.com)"class MightyNetworksResumeConfig:# Next 1-indexed page to fetch. None means "start from page 1".next_page: Optional[int] = None+ # `build_dependent_resource` checkpoint for the per-parent walk of a fan-out endpoint.+ fanout_state: Optional[dict[str, Any]] = Nonedef _network_url(network_id: str, path: str = "") -> str:@@ -63,6 +74,16 @@ _ROW_FIXUPS: dict[str, Callable[[dict[str, Any]], dict[str, Any]]] = {}+# The docs show two different placeholder envelopes for list responses ("data"/meta and+# "items"/links); "data" is the one paired with the concrete meta fields+# (current_page/total_pages/total_count) the paginator relies on, so it's the more likely real+# field name. Fail loud if that's wrong rather than silently syncing zero rows.+_DATA_SELECTOR: Endpoint = {"data_selector": "data", "data_selector_required": True}+++_UNBOUNDED_FANOUT_PARENTS = frozenset({"Members", "Events"})++def _client_config(api_key: str, network_id: str) -> ClientConfig:return {"base_url": _network_url(network_id),@@ -72,14 +93,55 @@ def _client_config(api_key: str, network_id: str) -> ClientConfig:}-def mighty_networks_source(+def _fanout_resource(api_key: str,network_id: str,endpoint: str,team_id: int,job_id: str,resumable_source_manager: ResumableSourceManager[MightyNetworksResumeConfig],-) -> SourceResponse:+) -> Resource:+ fanout = FANOUTS[endpoint]+ resume_hook: Optional[Callable[[Optional[dict[str, Any]]], None]] = None+ resume: Optional[MightyNetworksResumeConfig] = None+ # The fan-out checkpoint lists every completed parent, so it grows without bound when the+ # parent is a network-wide list like Members or Events. Those children are short, so a restart+ # costs little. Spaces and Plans are few, and their member lists are long.+ if fanout.parent_name not in _UNBOUNDED_FANOUT_PARENTS:+ resume = resumable_source_manager.load_state() if resumable_source_manager.can_resume() else None++ def resume_hook(state: Optional[dict[str, Any]]) -> None:+ if state:+ resumable_source_manager.save_state(MightyNetworksResumeConfig(fanout_state=state))++ return cast(+ Resource,+ build_dependent_resource(+ endpoint_configs=ENDPOINT_CONFIGS,+ child_endpoint=endpoint,+ fanout=fanout,+ client_config=_client_config(api_key, network_id),+ path_format_values={},+ team_id=team_id,+ job_id=job_id,+ db_incremental_field_last_value=None,+ parent_endpoint_extra=_DATA_SELECTOR,+ child_endpoint_extra=_DATA_SELECTOR,+ page_size_param="per_page",+ resume_hook=resume_hook,+ initial_paginator_state=resume.fanout_state if resume else None,+ ),+ )+++def _list_resource(+ api_key: str,+ network_id: str,+ endpoint: str,+ team_id: int,+ job_id: str,+ resumable_source_manager: ResumableSourceManager[MightyNetworksResumeConfig],+) -> Resource:rest_config: RESTAPIConfig = {"client": _client_config(api_key, network_id),"resource_defaults": {"write_disposition": "replace"},@@ -90,13 +152,7 @@ def mighty_networks_source("endpoint": {"path": PATHS[endpoint],"params": {"per_page": PER_PAGE},- # The docs show two different placeholder envelopes for list responses- # ("data"/meta and "items"/links); "data" is the one paired with the concrete- # meta fields (current_page/total_pages/total_count) this paginator relies on,- # so it's the more likely real field name. Fail loud if that's wrong rather- # than silently syncing zero rows.- "data_selector": "data",- "data_selector_required": True,+ **_DATA_SELECTOR,},"table_format": "delta",}@@ -127,6 +183,19 @@ def mighty_networks_source(fixup = _ROW_FIXUPS.get(endpoint)if fixup is not None:resource = resource.add_map(fixup)+ return resource+++def mighty_networks_source(+ api_key: str,+ network_id: str,+ endpoint: str,+ team_id: int,+ job_id: str,+ resumable_source_manager: ResumableSourceManager[MightyNetworksResumeConfig],+) -> SourceResponse:+ build = _fanout_resource if endpoint in FANOUTS else _list_resource+ resource = build(api_key, network_id, endpoint, team_id, job_id, resumable_source_manager)partition_key = PARTITION_FIELDS.get(endpoint)@@ -155,6 +224,12 @@ def validate_credentials(api_key: str, network_id: str) -> tuple[bool, int | Non)+def _probe_path(endpoint: str) -> str:+ # A fan-out child has no list path without a parent id, so probe the parent list it walks.+ fanout = FANOUTS.get(endpoint)+ return PATHS[fanout.parent_name] if fanout else PATHS[endpoint]++def check_endpoint_access(api_key: str, network_id: str, endpoint: str) -> Optional[str]:"""Probe a single list endpoint and report why it's inaccessible, or ``None`` if it's fine.@@ -163,7 +238,7 @@ def check_endpoint_access(api_key: str, network_id: str, endpoint: str) -> Optio"""try:response = make_tracked_session(redact_values=(api_key,)).get(- _network_url(network_id, PATHS[endpoint]),+ _network_url(network_id, _probe_path(endpoint)),headers={"Authorization": f"Bearer {api_key}", "User-Agent": USER_AGENT, "Accept": "application/json"},params={"per_page": 1},timeout=10,diff --git a/products/warehouse_sources/backend/temporal/data_imports/sources/mighty_networks/settings.py b/products/warehouse_sources/backend/temporal/data_imports/sources/mighty_networks/settings.pyindex f704c5631198..64c542bb53a6 100644--- a/products/warehouse_sources/backend/temporal/data_imports/sources/mighty_networks/settings.py+++ b/products/warehouse_sources/backend/temporal/data_imports/sources/mighty_networks/settings.py@@ -1,3 +1,10 @@+from typing import Optional++from posthog.dataclasses import frozen++from products.warehouse_sources.backend.temporal.data_imports.sources.common.rest_source.fanout import (+ DependentEndpointConfig,+)from products.warehouse_sources.backend.types import IncrementalFieldBASE_URL = "https://api.mn.co/admin/v1"@@ -15,6 +22,11 @@ ENDPOINTS = ("Purchases","Tags","Badges",+ "SpaceMembers",+ "PlanMembers",+ "MemberTags",+ "MemberBadges",+ "EventRsvps",)# Path under /admin/v1/networks/{network_id}.@@ -28,15 +40,62 @@ PATHS: dict[str, str] = {"Purchases": "/purchases","Tags": "/tags","Badges": "/badges",+ "SpaceMembers": "/spaces/{space_id}/members",+ "PlanMembers": "/plans/{plan_id}/members",+ "MemberTags": "/members/{member_id}/tags",+ "MemberBadges": "/members/{member_id}/badges",+ "EventRsvps": "/events/{event_id}/rsvps",+}+++def _fanout(parent_name: str, resolve_param: str) -> DependentEndpointConfig:+ return DependentEndpointConfig(+ parent_name=parent_name,+ resolve_param=resolve_param,+ resolve_field="id",+ include_from_parent=["id"],+ parent_field_renames={"id": resolve_param},+ # A parent removed between the parent listing and its child fetch 404s; skip it rather+ # than fail the run with the "network not found" non-retryable error.+ child_response_actions=[{"status_code": 404, "action": "ignore"}],+ )+++# These lists are only reachable per parent, so each child row carries its parent id.+FANOUTS: dict[str, DependentEndpointConfig] = {+ "SpaceMembers": _fanout("Spaces", "space_id"),+ "PlanMembers": _fanout("Plans", "plan_id"),+ "MemberTags": _fanout("Members", "member_id"),+ "MemberBadges": _fanout("Members", "member_id"),+ "EventRsvps": _fanout("Events", "event_id"),+}+++@frozen+class MightyNetworksEndpointConfig:+ name: str+ path: str+ incremental_fields: list[IncrementalField]+ default_incremental_field: Optional[str] = None+ page_size: int = PER_PAGE+++ENDPOINT_CONFIGS: dict[str, MightyNetworksEndpointConfig] = {+ name: MightyNetworksEndpointConfig(name=name, path=PATHS[name], incremental_fields=[]) for name in ENDPOINTS}# Subscriptions and Purchases nest the record's real id under a `subscription`/`purchase` object# rather than exposing it at the row root (see mighty_networks.py's flatten helpers), everything# else has a flat `id`.-PRIMARY_KEYS: dict[str, list[str]] = {name: ["id"] for name in ENDPOINTS}+# Fan-out child ids (a member, tag, or badge id) repeat across parents, so the parent id is part+# of the key.+PRIMARY_KEYS: dict[str, list[str]] = {+ name: [FANOUTS[name].resolve_param, "id"] if name in FANOUTS else ["id"] for name in ENDPOINTS+}-# All endpoints expose created_at, and it never changes after creation, unlike updated_at.-PARTITION_FIELDS: dict[str, str] = dict.fromkeys(ENDPOINTS, "created_at")+# created_at never changes after creation, unlike updated_at. RSVPs only expose `updated`, so they+# aren't partitioned.+PARTITION_FIELDS: dict[str, str] = {name: "created_at" for name in ENDPOINTS if name != "EventRsvps"}# No list endpoint documents or accepts an updated_since/created_since filter (checked against the# published OpenAPI spec at api.mn.co/admin/v1/spec.json and docs.mightynetworks.com/admin-api) —diff --git a/products/warehouse_sources/backend/temporal/data_imports/sources/mighty_networks/tests/test_mighty_networks.py b/products/warehouse_sources/backend/temporal/data_imports/sources/mighty_networks/tests/test_mighty_networks.pyindex 0a7958773b2a..4267d22cae13 100644--- a/products/warehouse_sources/backend/temporal/data_imports/sources/mighty_networks/tests/test_mighty_networks.py+++ b/products/warehouse_sources/backend/temporal/data_imports/sources/mighty_networks/tests/test_mighty_networks.py@@ -6,6 +6,7 @@ from products.warehouse_sources.backend.temporal.data_imports.sources.common.tesScriptedResponse,SourceDriver,always,+ route,scripted_network,)from products.warehouse_sources.backend.temporal.data_imports.sources.generated_configs.mightynetworks import (@@ -144,6 +145,80 @@ class TestMightyNetworksSourceNonFanout:assert result.rows[0]["updated_at"] == "2026-01-02T00:00:00Z"+class TestMightyNetworksSourceFanout:+ @parameterized.expand(+ [+ ("SpaceMembers", "spaces", "members", "space_id", True),+ ("PlanMembers", "plans", "members", "plan_id", True),+ ("MemberTags", "members", "tags", "member_id", False),+ ("MemberBadges", "members", "badges", "member_id", False),+ ("EventRsvps", "events", "rsvps", "event_id", False),+ ]+ )+ def test_fans_out_per_parent_and_keys_rows_by_parent(+ self, endpoint: str, parent: str, child: str, parent_key: str, resumable: bool+ ) -> None:+ base = "/admin/v1/networks/1234"+ result = _driver().run(+ endpoint,+ route(+ {+ f"{base}/{parent}": [_response([{"id": 10}, {"id": 20}], total_pages=1)],+ f"{base}/{parent}/10/{child}": [+ _response([{"id": 1}], current_page=1, total_pages=2),+ _response([{"id": 2}], current_page=2, total_pages=2),+ ],+ f"{base}/{parent}/20/{child}": [_response([{"id": 1}], total_pages=1)],+ }+ ),+ )++ assert result.raised is None+ assert result.paths == [+ f"{base}/{parent}",+ f"{base}/{parent}/10/{child}",+ f"{base}/{parent}/10/{child}",+ f"{base}/{parent}/20/{child}",+ ]+ assert [(row[parent_key], row["id"]) for row in result.rows] == [(10, 1), (10, 2), (20, 1)]+ assert all(f"_{parent.capitalize()}_id" not in row for row in result.rows)+ # A checkpoint over network-wide parents would list every member or event it completed.+ assert bool(result.saved_states) is resumable++ def test_skips_parent_deleted_before_its_child_fetch(self) -> None:+ result = _driver().run(+ "SpaceMembers",+ route(+ {+ "/admin/v1/networks/1234/spaces": [_response([{"id": 10}, {"id": 20}], total_pages=1)],+ "/admin/v1/networks/1234/spaces/10/members": [ScriptedResponse(status=404)],+ "/admin/v1/networks/1234/spaces/20/members": [_response([{"id": 1}], total_pages=1)],+ }+ ),+ )++ assert result.raised is None+ assert [(row["space_id"], row["id"]) for row in result.rows] == [(20, 1)]++ def test_resume_skips_parents_already_read(self) -> None:+ result = _driver().run(+ "SpaceMembers",+ route(+ {+ "/admin/v1/networks/1234/spaces": [_response([{"id": 10}, {"id": 20}], total_pages=1)],+ "/admin/v1/networks/1234/spaces/20/members": [_response([{"id": 3}], total_pages=1)],+ }+ ),+ resume_state=MightyNetworksResumeConfig(+ fanout_state={"completed": ["/spaces/10/members"], "current": None, "child_state": None}+ ),+ )++ assert result.raised is None+ assert result.paths == ["/admin/v1/networks/1234/spaces", "/admin/v1/networks/1234/spaces/20/members"]+ assert [(row["space_id"], row["id"]) for row in result.rows] == [(20, 3)]++class TestValidateCredentials:def test_ok(self) -> None:with scripted_network([ScriptedResponse(status=200)]) as network:diff --git a/products/warehouse_sources/backend/temporal/data_imports/sources/mighty_networks/tests/test_mighty_networks_source.py b/products/warehouse_sources/backend/temporal/data_imports/sources/mighty_networks/tests/test_mighty_networks_source.pyindex 59806c6fc42c..6032df51ec5b 100644--- a/products/warehouse_sources/backend/temporal/data_imports/sources/mighty_networks/tests/test_mighty_networks_source.py+++ b/products/warehouse_sources/backend/temporal/data_imports/sources/mighty_networks/tests/test_mighty_networks_source.py@@ -85,14 +85,22 @@ class TestMightyNetworksSource:def test_get_endpoint_permissions_probes_every_requested_endpoint(self) -> None:with scripted_network(- [ScriptedResponse(status=200), ScriptedResponse(status=403, json={"message": "missing scope"})]+ [+ ScriptedResponse(status=200),+ ScriptedResponse(status=403, json={"message": "missing scope"}),+ ScriptedResponse(status=200),+ ]) as network:- permissions = self.source.get_endpoint_permissions(self.config, self.team_id, ["Members", "Purchases"])+ permissions = self.source.get_endpoint_permissions(+ self.config, self.team_id, ["Members", "Purchases", "SpaceMembers"]+ )- assert permissions == {"Members": None, "Purchases": "missing scope"}+ assert permissions == {"Members": None, "Purchases": "missing scope", "SpaceMembers": None}+ # A fan-out table probes the parent list it walks, since its own path needs a parent id.assert [request.path for request in network.requests_log] == ["/admin/v1/networks/1234/members","/admin/v1/networks/1234/purchases",+ "/admin/v1/networks/1234/spaces",]- assert [request.param("per_page") for request in network.requests_log] == ["1", "1"]+ assert [request.param("per_page") for request in network.requests_log] == ["1", "1", "1"]assert all(request.headers["authorization"] == "Bearer key" for request in network.requests_log)