Skip to content

:hedgehog: PostHog is the leading platform for building self-driving products. Our developer tools – AI observability, analytics, session replay, flags, experiments, error tracking, logs, and more – capture all the context agents need to diagnose problems, uncover opportunities, and ship fixes. Steer it all from Slack, web, desktop, or the MCP.

feat(warehouse): add missing northpass lms endpoint tables (#123160)

Tom Owersdeaa976c3616
diff --git a/products/warehouse_sources/backend/temporal/data_imports/sources/COVERAGE_GAPS_APPENDIX.md b/products/warehouse_sources/backend/temporal/data_imports/sources/COVERAGE_GAPS_APPENDIX.md
index 6a9c8076022a..5eb506cea54b 100644
--- a/products/warehouse_sources/backend/temporal/data_imports/sources/COVERAGE_GAPS_APPENDIX.md
+++ b/products/warehouse_sources/backend/temporal/data_imports/sources/COVERAGE_GAPS_APPENDIX.md
@@ -8472,16 +8472,16 @@ Today (12): `activity_events`, `categories`, `course_activities`, `course_enroll
Diffed against: <https://developers.northpass.com/reference/get_v2-courses>
- [x] `GET /v2/events` — the learner activity event stream (lesson views, completions, logins) - the fact table behind every engagement metric (high)
-- [ ] `GET /v2/groups/{group_uuid}/memberships` — the people-to-groups join table; groups and people are both synced but nothing links them (high)
-- [ ] `GET /v2/credentials/{credential_uuid}/achievements` — certificates/credentials actually earned by learners - the completion outcome enrollments only hint at (high)
-- [ ] `GET /v2/submissions and /v2/assignments/{assignment_uuid}/submissions` — learner assignment submissions with grades/status, the assessment counterpart to quizzes (high)
+- [x] `GET /v2/groups/{group_uuid}/memberships` — the people-to-groups join table; groups and people are both synced but nothing links them (high)
+- [x] `GET /v2/credentials/{credential_uuid}/achievements` — certificates/credentials actually earned by learners - the completion outcome enrollments only hint at (high)
+- [x] `GET /v2/submissions and /v2/assignments/{assignment_uuid}/submissions` — learner assignment submissions with grades/status, the assessment counterpart to quizzes (high)
- [x] `GET /v2/courses/{course_uuid}/activities` — the lesson/activity structure inside a course - required to break enrollment progress down below course level (high)
-- [ ] `GET /v2/credentials` — lookup table resolving the credential definitions that achievements reference (medium)
+- [x] `GET /v2/credentials` — lookup table resolving the credential definitions that achievements reference (medium)
- [ ] `GET /v2/assignments` — assignment definitions, the lookup that submissions join back to (medium)
- [ ] `GET /v2/transcripts/{learner_id}` — per-learner transcript rolling up all course and learning-path completions (medium)
- [ ] `GET /v2/groups/{group_uuid}/courses` — course assignments per group - which cohorts were given which curriculum (medium)
- [ ] `GET /v2/communications/deliveries` — email delivery records for course invitations and reminders, for nudge-effectiveness analysis (medium)
-- [ ] `GET /v2/properties/people and /v2/properties/courses` — custom properties on people and courses plus /v2/properties/property-definitions as the schema lookup - the segmentation dimensions customers configure (medium)
+- [x] `GET /v2/properties/people and /v2/properties/courses` — custom properties on people and courses plus /v2/properties/property-definitions as the schema lookup - the segmentation dimensions customers configure (medium)
- [ ] `GET /v2/quiz-attempts/{quiz_attempt_uuid}/answers and /v2/question-banks` — per-question answer detail and the question bank lookup; quizzes are synced but nothing about how learners answered (low)
Note: Enumerated every /reference/get_v2-\* route from the ReadMe-hosted reference page (no public OpenAPI file is served). PostHog covers the 6 core objects plus 2 enrollment fan-outs; the learner-activity and assessment halves of the API are entirely absent.
diff --git a/products/warehouse_sources/backend/temporal/data_imports/sources/northpass_lms/canonical_descriptions.py b/products/warehouse_sources/backend/temporal/data_imports/sources/northpass_lms/canonical_descriptions.py
index 1127306f9964..1ed57bafd458 100644
--- a/products/warehouse_sources/backend/temporal/data_imports/sources/northpass_lms/canonical_descriptions.py
+++ b/products/warehouse_sources/backend/temporal/data_imports/sources/northpass_lms/canonical_descriptions.py
@@ -176,4 +176,95 @@ CANONICAL_DESCRIPTIONS: CanonicalDescriptions = {
"created_at": "Timestamp when the answer was recorded.",
},
},
+ "group_memberships": {
+ "description": "Memberships linking people to groups, one row per person per group. Fanned out per group.",
+ "docs_url": "https://developers.northpass.com/reference/get_v2-groups-group-uuid-memberships",
+ "columns": {
+ "id": "Unique identifier (UUID) for the membership.",
+ "type": "JSON:API resource type, always 'memberships'.",
+ "group_id": "UUID of the group the membership belongs to.",
+ "person_id": "UUID of the person who is a member of the group.",
+ "created_at": "Timestamp when the person was added to the group.",
+ },
+ },
+ "credentials": {
+ "description": "Credential (certificate) definitions in your Northpass school. One row per credential.",
+ "docs_url": "https://developers.northpass.com/reference/get_v2-credentials",
+ "columns": {
+ "id": "Unique identifier (UUID) for the credential.",
+ "type": "JSON:API resource type, always 'credentials'.",
+ "name": "The credential name.",
+ "managed_by": "Name of the admin who manages the credential.",
+ "public": "Whether the credential is public.",
+ "shareable_to_linkedin": "Whether learners can share the credential to LinkedIn.",
+ "expiration_enabled": "Whether issued credentials expire.",
+ "recertification_enabled": "Whether learners can recertify when the credential expires.",
+ "created_at": "Timestamp when the credential was created.",
+ "updated_at": "Timestamp when the credential was last updated.",
+ },
+ },
+ "credential_achievements": {
+ "description": "Credentials issued to learners, one row per achievement. Fanned out per credential.",
+ "docs_url": "https://developers.northpass.com/reference/get_v2-credentials-credential-uuid-achievements",
+ "columns": {
+ "id": "Unique identifier (UUID) for the achievement.",
+ "type": "JSON:API resource type, always 'credential_achievements'.",
+ "credential_id": "UUID of the credential that was issued.",
+ "person_email": "Email address of the learner who earned the credential.",
+ "status": "Status of the achievement (e.g. 'Certified').",
+ "criteria_type": "Type of the record that earned the credential (e.g. 'CourseAttempt').",
+ "criteria_uuid": "UUID of the record that earned the credential.",
+ "criteria_display_name": "Display name of the record that earned the credential.",
+ "issued_at": "Timestamp when the credential was issued.",
+ "expires_at": "Timestamp when the credential expires, if it expires.",
+ "is_valid": "Whether the credential is currently valid.",
+ },
+ },
+ "assignment_submissions": {
+ "description": "Learner submissions for every assignment, one row per submission.",
+ "docs_url": "https://developers.northpass.com/reference/get_v2-submissions",
+ "columns": {
+ "id": "Unique identifier (UUID) for the submission.",
+ "type": "JSON:API resource type, always 'assignment_submissions'.",
+ "created_at": "Timestamp when the submission was made.",
+ "assignment_id": "UUID of the assignment the submission is for.",
+ "course_id": "UUID of the course the assignment belongs to.",
+ "person_id": "UUID of the learner who made the submission.",
+ "course_attempt_id": "UUID of the learner's course attempt the submission belongs to.",
+ },
+ },
+ "person_properties": {
+ "description": "Property values for every person, one row per person, including custom properties.",
+ "docs_url": "https://developers.northpass.com/reference/get_v2-properties-people",
+ "columns": {
+ "id": "UUID of the person the properties belong to.",
+ "type": "JSON:API resource type, always 'person_properties'.",
+ "properties": "Every property of the person, default and custom, keyed by property name.",
+ "customized_properties": "Only the custom properties of the person, keyed by property name.",
+ },
+ },
+ "course_properties": {
+ "description": "Property values for every course, one row per course, including custom properties.",
+ "docs_url": "https://developers.northpass.com/reference/get_v2-properties-courses",
+ "columns": {
+ "id": "UUID of the course the properties belong to.",
+ "type": "JSON:API resource type, always 'course_properties'.",
+ "properties": "Every property of the course, default and custom, keyed by property name.",
+ "customized_properties": "Only the custom properties of the course, keyed by property name.",
+ },
+ },
+ "property_definitions": {
+ "description": "Custom property definitions, one row per property. The schema for person_properties and course_properties.",
+ "docs_url": "https://developers.northpass.com/reference/get_v2-properties-property-definitions",
+ "columns": {
+ "id": "Unique identifier (UUID) for the property definition.",
+ "type": "Data type of the property values (e.g. 'string', 'integer', 'boolean').",
+ "namespace": "What the property applies to (e.g. 'person', 'course').",
+ "name": "Property name, the key used in the properties columns.",
+ "display_name": "Human-readable property name.",
+ "default_value": "Default value of the property.",
+ "initial_value": "Initial value of the property.",
+ "status": "Status of the property definition (e.g. 'ready').",
+ },
+ },
}
diff --git a/products/warehouse_sources/backend/temporal/data_imports/sources/northpass_lms/northpass_lms.py b/products/warehouse_sources/backend/temporal/data_imports/sources/northpass_lms/northpass_lms.py
index 7a1d6034536c..af86b3103eb4 100644
--- a/products/warehouse_sources/backend/temporal/data_imports/sources/northpass_lms/northpass_lms.py
+++ b/products/warehouse_sources/backend/temporal/data_imports/sources/northpass_lms/northpass_lms.py
@@ -204,16 +204,18 @@ def _make_quiz_attempt_flattener(seen: set[str]) -> Callable[[dict[str, Any]], d
return _flatten
-def _make_quiz_answer_flattener(parent_name: str, parent_id_field: str) -> Callable[[dict[str, Any]], dict[str, Any]]:
- """Child flattener for quiz answers: standard child flattening plus a promoted ``question_id``.
+def _make_child_relationship_flattener(
+ parent_name: str, parent_id_field: str, relationship_id_fields: dict[str, str]
+) -> Callable[[dict[str, Any]], dict[str, Any]]:
+ """Child flattener that also promotes related-resource ids to root columns.
- An answer references its question only inside JSON:API ``relationships``; promoting the id to a
- root column keeps answers joinable to their question.
+ Quiz answers reference their question, and group memberships their person, only inside JSON:API
+ ``relationships``; promoting those ids keeps the child rows joinable.
"""
flatten_child = _make_child_flattener(parent_name, parent_id_field)
def _flatten(item: dict[str, Any]) -> dict[str, Any]:
- return _promote_relationship_ids(flatten_child(item), {"question": "question_id"})
+ return _promote_relationship_ids(flatten_child(item), relationship_id_fields)
return _flatten
@@ -232,10 +234,10 @@ def _collection_data_map(
def _child_data_map(
- endpoint: str, parent_name: str, parent_id_field: str
+ config: NorthpassEndpointConfig, parent_name: str, parent_id_field: str
) -> Callable[[dict[str, Any]], dict[str, Any]]:
- if endpoint == "quiz_attempt_answers":
- return _make_quiz_answer_flattener(parent_name, parent_id_field)
+ if config.relationship_id_fields:
+ return _make_child_relationship_flattener(parent_name, parent_id_field, config.relationship_id_fields)
return _make_child_flattener(parent_name, parent_id_field)
@@ -358,7 +360,7 @@ def _fan_out_source(
# as a valid empty page and move on to the next parent rather than failing the sync.
"response_actions": [{"status_code": 404, "action": "ignore"}],
},
- "data_map": _child_data_map(endpoint, parent_name, config.parent_id_field),
+ "data_map": _child_data_map(config, parent_name, config.parent_id_field),
},
],
}
diff --git a/products/warehouse_sources/backend/temporal/data_imports/sources/northpass_lms/settings.py b/products/warehouse_sources/backend/temporal/data_imports/sources/northpass_lms/settings.py
index 3c7cb7d94abd..c21587990216 100644
--- a/products/warehouse_sources/backend/temporal/data_imports/sources/northpass_lms/settings.py
+++ b/products/warehouse_sources/backend/temporal/data_imports/sources/northpass_lms/settings.py
@@ -8,7 +8,7 @@ from products.warehouse_sources.backend.types import IncrementalField
# cursor. See `northpass_lms.py` for the pagination transport.
-@dataclass
+@dataclass(frozen=True)
class NorthpassEndpointConfig:
name: str
# JSON:API collection path relative to the API base. Fan-out children use a `{parent_id}`
@@ -28,7 +28,7 @@ class NorthpassEndpointConfig:
parent_id_field: Optional[str] = None
# For endpoints whose rows reference other resources only inside JSON:API `relationships`
# (e.g. `/events` rows carry no `id` of their own): relationship name -> column name under
- # which the related resource's id is promoted to the row root.
+ # which the related resource's id is promoted to the row root. Applies to fan-out children too.
relationship_id_fields: Optional[dict[str, str]] = None
# Human-readable note surfaced in the schema picker / docs.
description: Optional[str] = None
@@ -148,6 +148,7 @@ NORTHPASS_ENDPOINTS: dict[str, NorthpassEndpointConfig] = {
primary_keys=["quiz_attempt_id", "id"],
fan_out_parent="quiz_attempts",
parent_id_field="quiz_attempt_id",
+ relationship_id_fields={"question": "question_id"},
should_sync_default=False,
description=(
"Per-question learner answers for every completed quiz attempt. Fans out one request per "
@@ -155,6 +156,62 @@ NORTHPASS_ENDPOINTS: dict[str, NorthpassEndpointConfig] = {
"table is off by default. Full refresh only."
),
),
+ # Group membership rows only reference their person inside `relationships`; the person id is
+ # promoted so the table joins `people` to `groups`.
+ "group_memberships": NorthpassEndpointConfig(
+ name="group_memberships",
+ path="/groups/{parent_id}/memberships",
+ partition_key="created_at",
+ primary_keys=["group_id", "id"],
+ fan_out_parent="groups",
+ parent_id_field="group_id",
+ relationship_id_fields={"person": "person_id"},
+ description="People in every group, one row per membership. Fans out one request per group. Full refresh only.",
+ ),
+ "credentials": NorthpassEndpointConfig(
+ name="credentials",
+ path="/credentials",
+ partition_key="created_at",
+ ),
+ "credential_achievements": NorthpassEndpointConfig(
+ name="credential_achievements",
+ path="/credentials/{parent_id}/achievements",
+ partition_key="issued_at",
+ primary_keys=["credential_id", "id"],
+ fan_out_parent="credentials",
+ parent_id_field="credential_id",
+ description="Credentials (certificates) issued to learners, one row per achievement. Fans out one request per credential. Full refresh only.",
+ ),
+ # `/submissions` lists submissions across every assignment, so the per-assignment
+ # `/assignments/{uuid}/submissions` endpoint would only re-fetch the same rows.
+ "assignment_submissions": NorthpassEndpointConfig(
+ name="assignment_submissions",
+ path="/submissions",
+ partition_key="created_at",
+ relationship_id_fields={
+ "assignment": "assignment_id",
+ "course": "course_id",
+ "person": "person_id",
+ "course_attempt": "course_attempt_id",
+ },
+ ),
+ # The `/properties` endpoints document no page-size param, so they page at the server default.
+ # Their rows carry no timestamps, so they are unpartitioned.
+ "person_properties": NorthpassEndpointConfig(
+ name="person_properties",
+ path="/properties/people",
+ params={},
+ ),
+ "course_properties": NorthpassEndpointConfig(
+ name="course_properties",
+ path="/properties/courses",
+ params={},
+ ),
+ "property_definitions": NorthpassEndpointConfig(
+ name="property_definitions",
+ path="/properties/property_definitions",
+ params={},
+ ),
}
ENDPOINTS = tuple(NORTHPASS_ENDPOINTS.keys())
diff --git a/products/warehouse_sources/backend/temporal/data_imports/sources/northpass_lms/tests/test_northpass_lms.py b/products/warehouse_sources/backend/temporal/data_imports/sources/northpass_lms/tests/test_northpass_lms.py
index 5a1507b92c99..276cb628f200 100644
--- a/products/warehouse_sources/backend/temporal/data_imports/sources/northpass_lms/tests/test_northpass_lms.py
+++ b/products/warehouse_sources/backend/temporal/data_imports/sources/northpass_lms/tests/test_northpass_lms.py
@@ -93,6 +93,14 @@ class TestTopLevelPagination:
# The first page is skipped entirely — resume starts at the saved URL.
assert result.urls == [COURSES_P2]
+ def test_properties_endpoint_sends_no_page_size(self) -> None:
+ # The /properties endpoints document no `limit`, so the request must not carry the default one.
+ result = _driver().run("person_properties", [_page([{"id": "p1", "attributes": {"properties": {"a": 1}}}])])
+
+ assert result.raised is None
+ assert result.urls == ["https://api.northpass.com/v2/properties/people"]
+ assert result.rows[0]["properties"] == {"a": 1}
+
@parameterized.expand(
[
("attacker_host", "https://evil.example.com/steal?limit=100"),
@@ -143,6 +151,29 @@ class TestFanOut:
# c1's enrollments must not be re-fetched when resuming past it.
assert "https://api.northpass.com/v2/courses/c1/enrollments?limit=100" not in result.urls
+ def test_group_memberships_link_each_person_to_their_group(self) -> None:
+ def membership(membership_id: str, person_id: str) -> dict[str, Any]:
+ return {
+ "id": membership_id,
+ "type": "memberships",
+ "attributes": {"created_at": "2024-10-08T08:37:24Z"},
+ "relationships": {"person": {"data": {"type": "people", "id": person_id}}},
+ }
+
+ result = _driver().run(
+ "group_memberships",
+ route(
+ {
+ "/v2/groups": [_page([{"id": "g1"}, {"id": "g2"}])],
+ "/v2/groups/g1/memberships": [_page([membership("m1", "p1")])],
+ "/v2/groups/g2/memberships": [_page([membership("m2", "p1")])],
+ }
+ ),
+ )
+
+ assert result.raised is None
+ assert [(row["group_id"], row["person_id"]) for row in result.rows] == [("g1", "p1"), ("g2", "p1")]
+
def test_reraises_non_404_child_error(self) -> None:
pages = self._parent_and_children()
pages["/v2/courses/c1/enrollments"] = [_resp({}, status=400)]