:hedgehog: PostHog is the leading platform for building self-driving products. Our developer tools – AI observability, analytics, session replay, flags, experiments, error tracking, logs, and more – capture all the context agents need to diagnose problems, uncover opportunities, and ship fixes. Steer it all from Slack, web, desktop, or the MCP.
Python ab-testingai-analyticsanalyticscdpdata-warehouseexperimentsfeature-flagsjavascriptproduct-analyticspythonreactsession-replaysurveystypescriptweb-analytics
feat(warehouse): add missing northpass lms endpoint tables (#123160)
Tom Owersdeaa976c3616
diff --git a/products/warehouse_sources/backend/temporal/data_imports/sources/COVERAGE_GAPS_APPENDIX.md b/products/warehouse_sources/backend/temporal/data_imports/sources/COVERAGE_GAPS_APPENDIX.mdindex 6a9c8076022a..5eb506cea54b 100644--- a/products/warehouse_sources/backend/temporal/data_imports/sources/COVERAGE_GAPS_APPENDIX.md+++ b/products/warehouse_sources/backend/temporal/data_imports/sources/COVERAGE_GAPS_APPENDIX.md@@ -8472,16 +8472,16 @@ Today (12): `activity_events`, `categories`, `course_activities`, `course_enrollDiffed against: <https://developers.northpass.com/reference/get_v2-courses>- [x] `GET /v2/events` — the learner activity event stream (lesson views, completions, logins) - the fact table behind every engagement metric (high)-- [ ] `GET /v2/groups/{group_uuid}/memberships` — the people-to-groups join table; groups and people are both synced but nothing links them (high)-- [ ] `GET /v2/credentials/{credential_uuid}/achievements` — certificates/credentials actually earned by learners - the completion outcome enrollments only hint at (high)-- [ ] `GET /v2/submissions and /v2/assignments/{assignment_uuid}/submissions` — learner assignment submissions with grades/status, the assessment counterpart to quizzes (high)+- [x] `GET /v2/groups/{group_uuid}/memberships` — the people-to-groups join table; groups and people are both synced but nothing links them (high)+- [x] `GET /v2/credentials/{credential_uuid}/achievements` — certificates/credentials actually earned by learners - the completion outcome enrollments only hint at (high)+- [x] `GET /v2/submissions and /v2/assignments/{assignment_uuid}/submissions` — learner assignment submissions with grades/status, the assessment counterpart to quizzes (high)- [x] `GET /v2/courses/{course_uuid}/activities` — the lesson/activity structure inside a course - required to break enrollment progress down below course level (high)-- [ ] `GET /v2/credentials` — lookup table resolving the credential definitions that achievements reference (medium)+- [x] `GET /v2/credentials` — lookup table resolving the credential definitions that achievements reference (medium)- [ ] `GET /v2/assignments` — assignment definitions, the lookup that submissions join back to (medium)- [ ] `GET /v2/transcripts/{learner_id}` — per-learner transcript rolling up all course and learning-path completions (medium)- [ ] `GET /v2/groups/{group_uuid}/courses` — course assignments per group - which cohorts were given which curriculum (medium)- [ ] `GET /v2/communications/deliveries` — email delivery records for course invitations and reminders, for nudge-effectiveness analysis (medium)-- [ ] `GET /v2/properties/people and /v2/properties/courses` — custom properties on people and courses plus /v2/properties/property-definitions as the schema lookup - the segmentation dimensions customers configure (medium)+- [x] `GET /v2/properties/people and /v2/properties/courses` — custom properties on people and courses plus /v2/properties/property-definitions as the schema lookup - the segmentation dimensions customers configure (medium)- [ ] `GET /v2/quiz-attempts/{quiz_attempt_uuid}/answers and /v2/question-banks` — per-question answer detail and the question bank lookup; quizzes are synced but nothing about how learners answered (low)Note: Enumerated every /reference/get_v2-\* route from the ReadMe-hosted reference page (no public OpenAPI file is served). PostHog covers the 6 core objects plus 2 enrollment fan-outs; the learner-activity and assessment halves of the API are entirely absent.diff --git a/products/warehouse_sources/backend/temporal/data_imports/sources/northpass_lms/canonical_descriptions.py b/products/warehouse_sources/backend/temporal/data_imports/sources/northpass_lms/canonical_descriptions.pyindex 1127306f9964..1ed57bafd458 100644--- a/products/warehouse_sources/backend/temporal/data_imports/sources/northpass_lms/canonical_descriptions.py+++ b/products/warehouse_sources/backend/temporal/data_imports/sources/northpass_lms/canonical_descriptions.py@@ -176,4 +176,95 @@ CANONICAL_DESCRIPTIONS: CanonicalDescriptions = {"created_at": "Timestamp when the answer was recorded.",},},+ "group_memberships": {+ "description": "Memberships linking people to groups, one row per person per group. Fanned out per group.",+ "docs_url": "https://developers.northpass.com/reference/get_v2-groups-group-uuid-memberships",+ "columns": {+ "id": "Unique identifier (UUID) for the membership.",+ "type": "JSON:API resource type, always 'memberships'.",+ "group_id": "UUID of the group the membership belongs to.",+ "person_id": "UUID of the person who is a member of the group.",+ "created_at": "Timestamp when the person was added to the group.",+ },+ },+ "credentials": {+ "description": "Credential (certificate) definitions in your Northpass school. One row per credential.",+ "docs_url": "https://developers.northpass.com/reference/get_v2-credentials",+ "columns": {+ "id": "Unique identifier (UUID) for the credential.",+ "type": "JSON:API resource type, always 'credentials'.",+ "name": "The credential name.",+ "managed_by": "Name of the admin who manages the credential.",+ "public": "Whether the credential is public.",+ "shareable_to_linkedin": "Whether learners can share the credential to LinkedIn.",+ "expiration_enabled": "Whether issued credentials expire.",+ "recertification_enabled": "Whether learners can recertify when the credential expires.",+ "created_at": "Timestamp when the credential was created.",+ "updated_at": "Timestamp when the credential was last updated.",+ },+ },+ "credential_achievements": {+ "description": "Credentials issued to learners, one row per achievement. Fanned out per credential.",+ "docs_url": "https://developers.northpass.com/reference/get_v2-credentials-credential-uuid-achievements",+ "columns": {+ "id": "Unique identifier (UUID) for the achievement.",+ "type": "JSON:API resource type, always 'credential_achievements'.",+ "credential_id": "UUID of the credential that was issued.",+ "person_email": "Email address of the learner who earned the credential.",+ "status": "Status of the achievement (e.g. 'Certified').",+ "criteria_type": "Type of the record that earned the credential (e.g. 'CourseAttempt').",+ "criteria_uuid": "UUID of the record that earned the credential.",+ "criteria_display_name": "Display name of the record that earned the credential.",+ "issued_at": "Timestamp when the credential was issued.",+ "expires_at": "Timestamp when the credential expires, if it expires.",+ "is_valid": "Whether the credential is currently valid.",+ },+ },+ "assignment_submissions": {+ "description": "Learner submissions for every assignment, one row per submission.",+ "docs_url": "https://developers.northpass.com/reference/get_v2-submissions",+ "columns": {+ "id": "Unique identifier (UUID) for the submission.",+ "type": "JSON:API resource type, always 'assignment_submissions'.",+ "created_at": "Timestamp when the submission was made.",+ "assignment_id": "UUID of the assignment the submission is for.",+ "course_id": "UUID of the course the assignment belongs to.",+ "person_id": "UUID of the learner who made the submission.",+ "course_attempt_id": "UUID of the learner's course attempt the submission belongs to.",+ },+ },+ "person_properties": {+ "description": "Property values for every person, one row per person, including custom properties.",+ "docs_url": "https://developers.northpass.com/reference/get_v2-properties-people",+ "columns": {+ "id": "UUID of the person the properties belong to.",+ "type": "JSON:API resource type, always 'person_properties'.",+ "properties": "Every property of the person, default and custom, keyed by property name.",+ "customized_properties": "Only the custom properties of the person, keyed by property name.",+ },+ },+ "course_properties": {+ "description": "Property values for every course, one row per course, including custom properties.",+ "docs_url": "https://developers.northpass.com/reference/get_v2-properties-courses",+ "columns": {+ "id": "UUID of the course the properties belong to.",+ "type": "JSON:API resource type, always 'course_properties'.",+ "properties": "Every property of the course, default and custom, keyed by property name.",+ "customized_properties": "Only the custom properties of the course, keyed by property name.",+ },+ },+ "property_definitions": {+ "description": "Custom property definitions, one row per property. The schema for person_properties and course_properties.",+ "docs_url": "https://developers.northpass.com/reference/get_v2-properties-property-definitions",+ "columns": {+ "id": "Unique identifier (UUID) for the property definition.",+ "type": "Data type of the property values (e.g. 'string', 'integer', 'boolean').",+ "namespace": "What the property applies to (e.g. 'person', 'course').",+ "name": "Property name, the key used in the properties columns.",+ "display_name": "Human-readable property name.",+ "default_value": "Default value of the property.",+ "initial_value": "Initial value of the property.",+ "status": "Status of the property definition (e.g. 'ready').",+ },+ },}diff --git a/products/warehouse_sources/backend/temporal/data_imports/sources/northpass_lms/northpass_lms.py b/products/warehouse_sources/backend/temporal/data_imports/sources/northpass_lms/northpass_lms.pyindex 7a1d6034536c..af86b3103eb4 100644--- a/products/warehouse_sources/backend/temporal/data_imports/sources/northpass_lms/northpass_lms.py+++ b/products/warehouse_sources/backend/temporal/data_imports/sources/northpass_lms/northpass_lms.py@@ -204,16 +204,18 @@ def _make_quiz_attempt_flattener(seen: set[str]) -> Callable[[dict[str, Any]], dreturn _flatten-def _make_quiz_answer_flattener(parent_name: str, parent_id_field: str) -> Callable[[dict[str, Any]], dict[str, Any]]:- """Child flattener for quiz answers: standard child flattening plus a promoted ``question_id``.+def _make_child_relationship_flattener(+ parent_name: str, parent_id_field: str, relationship_id_fields: dict[str, str]+) -> Callable[[dict[str, Any]], dict[str, Any]]:+ """Child flattener that also promotes related-resource ids to root columns.- An answer references its question only inside JSON:API ``relationships``; promoting the id to a- root column keeps answers joinable to their question.+ Quiz answers reference their question, and group memberships their person, only inside JSON:API+ ``relationships``; promoting those ids keeps the child rows joinable."""flatten_child = _make_child_flattener(parent_name, parent_id_field)def _flatten(item: dict[str, Any]) -> dict[str, Any]:- return _promote_relationship_ids(flatten_child(item), {"question": "question_id"})+ return _promote_relationship_ids(flatten_child(item), relationship_id_fields)return _flatten@@ -232,10 +234,10 @@ def _collection_data_map(def _child_data_map(- endpoint: str, parent_name: str, parent_id_field: str+ config: NorthpassEndpointConfig, parent_name: str, parent_id_field: str) -> Callable[[dict[str, Any]], dict[str, Any]]:- if endpoint == "quiz_attempt_answers":- return _make_quiz_answer_flattener(parent_name, parent_id_field)+ if config.relationship_id_fields:+ return _make_child_relationship_flattener(parent_name, parent_id_field, config.relationship_id_fields)return _make_child_flattener(parent_name, parent_id_field)@@ -358,7 +360,7 @@ def _fan_out_source(# as a valid empty page and move on to the next parent rather than failing the sync."response_actions": [{"status_code": 404, "action": "ignore"}],},- "data_map": _child_data_map(endpoint, parent_name, config.parent_id_field),+ "data_map": _child_data_map(config, parent_name, config.parent_id_field),},],}diff --git a/products/warehouse_sources/backend/temporal/data_imports/sources/northpass_lms/settings.py b/products/warehouse_sources/backend/temporal/data_imports/sources/northpass_lms/settings.pyindex 3c7cb7d94abd..c21587990216 100644--- a/products/warehouse_sources/backend/temporal/data_imports/sources/northpass_lms/settings.py+++ b/products/warehouse_sources/backend/temporal/data_imports/sources/northpass_lms/settings.py@@ -8,7 +8,7 @@ from products.warehouse_sources.backend.types import IncrementalField# cursor. See `northpass_lms.py` for the pagination transport.-@dataclass+@dataclass(frozen=True)class NorthpassEndpointConfig:name: str# JSON:API collection path relative to the API base. Fan-out children use a `{parent_id}`@@ -28,7 +28,7 @@ class NorthpassEndpointConfig:parent_id_field: Optional[str] = None# For endpoints whose rows reference other resources only inside JSON:API `relationships`# (e.g. `/events` rows carry no `id` of their own): relationship name -> column name under- # which the related resource's id is promoted to the row root.+ # which the related resource's id is promoted to the row root. Applies to fan-out children too.relationship_id_fields: Optional[dict[str, str]] = None# Human-readable note surfaced in the schema picker / docs.description: Optional[str] = None@@ -148,6 +148,7 @@ NORTHPASS_ENDPOINTS: dict[str, NorthpassEndpointConfig] = {primary_keys=["quiz_attempt_id", "id"],fan_out_parent="quiz_attempts",parent_id_field="quiz_attempt_id",+ relationship_id_fields={"question": "question_id"},should_sync_default=False,description=("Per-question learner answers for every completed quiz attempt. Fans out one request per "@@ -155,6 +156,62 @@ NORTHPASS_ENDPOINTS: dict[str, NorthpassEndpointConfig] = {"table is off by default. Full refresh only."),),+ # Group membership rows only reference their person inside `relationships`; the person id is+ # promoted so the table joins `people` to `groups`.+ "group_memberships": NorthpassEndpointConfig(+ name="group_memberships",+ path="/groups/{parent_id}/memberships",+ partition_key="created_at",+ primary_keys=["group_id", "id"],+ fan_out_parent="groups",+ parent_id_field="group_id",+ relationship_id_fields={"person": "person_id"},+ description="People in every group, one row per membership. Fans out one request per group. Full refresh only.",+ ),+ "credentials": NorthpassEndpointConfig(+ name="credentials",+ path="/credentials",+ partition_key="created_at",+ ),+ "credential_achievements": NorthpassEndpointConfig(+ name="credential_achievements",+ path="/credentials/{parent_id}/achievements",+ partition_key="issued_at",+ primary_keys=["credential_id", "id"],+ fan_out_parent="credentials",+ parent_id_field="credential_id",+ description="Credentials (certificates) issued to learners, one row per achievement. Fans out one request per credential. Full refresh only.",+ ),+ # `/submissions` lists submissions across every assignment, so the per-assignment+ # `/assignments/{uuid}/submissions` endpoint would only re-fetch the same rows.+ "assignment_submissions": NorthpassEndpointConfig(+ name="assignment_submissions",+ path="/submissions",+ partition_key="created_at",+ relationship_id_fields={+ "assignment": "assignment_id",+ "course": "course_id",+ "person": "person_id",+ "course_attempt": "course_attempt_id",+ },+ ),+ # The `/properties` endpoints document no page-size param, so they page at the server default.+ # Their rows carry no timestamps, so they are unpartitioned.+ "person_properties": NorthpassEndpointConfig(+ name="person_properties",+ path="/properties/people",+ params={},+ ),+ "course_properties": NorthpassEndpointConfig(+ name="course_properties",+ path="/properties/courses",+ params={},+ ),+ "property_definitions": NorthpassEndpointConfig(+ name="property_definitions",+ path="/properties/property_definitions",+ params={},+ ),}ENDPOINTS = tuple(NORTHPASS_ENDPOINTS.keys())diff --git a/products/warehouse_sources/backend/temporal/data_imports/sources/northpass_lms/tests/test_northpass_lms.py b/products/warehouse_sources/backend/temporal/data_imports/sources/northpass_lms/tests/test_northpass_lms.pyindex 5a1507b92c99..276cb628f200 100644--- a/products/warehouse_sources/backend/temporal/data_imports/sources/northpass_lms/tests/test_northpass_lms.py+++ b/products/warehouse_sources/backend/temporal/data_imports/sources/northpass_lms/tests/test_northpass_lms.py@@ -93,6 +93,14 @@ class TestTopLevelPagination:# The first page is skipped entirely — resume starts at the saved URL.assert result.urls == [COURSES_P2]+ def test_properties_endpoint_sends_no_page_size(self) -> None:+ # The /properties endpoints document no `limit`, so the request must not carry the default one.+ result = _driver().run("person_properties", [_page([{"id": "p1", "attributes": {"properties": {"a": 1}}}])])++ assert result.raised is None+ assert result.urls == ["https://api.northpass.com/v2/properties/people"]+ assert result.rows[0]["properties"] == {"a": 1}+@parameterized.expand([("attacker_host", "https://evil.example.com/steal?limit=100"),@@ -143,6 +151,29 @@ class TestFanOut:# c1's enrollments must not be re-fetched when resuming past it.assert "https://api.northpass.com/v2/courses/c1/enrollments?limit=100" not in result.urls+ def test_group_memberships_link_each_person_to_their_group(self) -> None:+ def membership(membership_id: str, person_id: str) -> dict[str, Any]:+ return {+ "id": membership_id,+ "type": "memberships",+ "attributes": {"created_at": "2024-10-08T08:37:24Z"},+ "relationships": {"person": {"data": {"type": "people", "id": person_id}}},+ }++ result = _driver().run(+ "group_memberships",+ route(+ {+ "/v2/groups": [_page([{"id": "g1"}, {"id": "g2"}])],+ "/v2/groups/g1/memberships": [_page([membership("m1", "p1")])],+ "/v2/groups/g2/memberships": [_page([membership("m2", "p1")])],+ }+ ),+ )++ assert result.raised is None+ assert [(row["group_id"], row["person_id"]) for row in result.rows] == [("g1", "p1"), ("g2", "p1")]+def test_reraises_non_404_child_error(self) -> None:pages = self._parent_and_children()pages["/v2/courses/c1/enrollments"] = [_resp({}, status=400)]