:hedgehog: PostHog is the leading platform for building self-driving products. Our developer tools – AI observability, analytics, session replay, flags, experiments, error tracking, logs, and more – capture all the context agents need to diagnose problems, uncover opportunities, and ship fixes. Steer it all from Slack, web, desktop, or the MCP.
Python ab-testingai-analyticsanalyticscdpdata-warehouseexperimentsfeature-flagsjavascriptproduct-analyticspythonreactsession-replaysurveystypescriptweb-analytics
feat(data-warehouse): add 12 missing tables to the JamfPro source (#124188)
Tom Owersde877719bcd7
diff --git a/products/warehouse_sources/backend/temporal/data_imports/sources/COVERAGE_GAPS_APPENDIX.md b/products/warehouse_sources/backend/temporal/data_imports/sources/COVERAGE_GAPS_APPENDIX.mdindex 4f441aed949d..5e49c3a67399 100644--- a/products/warehouse_sources/backend/temporal/data_imports/sources/COVERAGE_GAPS_APPENDIX.md+++ b/products/warehouse_sources/backend/temporal/data_imports/sources/COVERAGE_GAPS_APPENDIX.md@@ -6557,10 +6557,10 @@ Diffed against: <https://developer.jamf.com/jamf-pro/reference/get_v1-buildings>- [x] `GET /api/v1/managed-software-updates/update-statuses (+ /computers/{id}, /computer-groups/{id})` — OS update enforcement status per device — core compliance metric (high)- [x] `GET /api/v2/patch-policies and /api/v2/patch-policies/{id}/logs` — per-device patch deployment outcomes for the packages we sync (medium). Added as `patch_policies` and `patch_policy_logs` (fan-out over `patch_policies`).- ~~`GET /api/v1/computers-inventory/filevault (and /{id}/filevault)`~~ — skipped: `computers` already syncs the `DISK_ENCRYPTION` section with the same compliance fields; the only extra field is the personal recovery key, a secret that does not belong in the warehouse (medium)-- [ ] `GET /api/v1/computer-extension-attributes and /api/v1/mobile-device-extension-attributes` — lookup resolving the custom extension-attribute IDs embedded in inventory records (medium)-- [ ] `GET /api/v1/volume-purchasing-locations/{id}/content and /api/v1/volume-purchasing-subscriptions` — VPP app license inventory and consumption (medium)-- [ ] `GET /api/v1/device-enrollments and /{id}/devices` — ADE/DEP enrollment records and their device rosters (medium)-- [ ] `GET /api/v1/{buildings,categories,departments,packages,scripts}/{id}/history` — object change history for the reference tables we already sync (low)+- [x] `GET /api/v1/computer-extension-attributes and /api/v1/mobile-device-extension-attributes` — lookup resolving the custom extension-attribute IDs embedded in inventory records (medium). Added as `computer_extension_attributes` and `mobile_device_extension_attributes`.+- [x] `GET /api/v1/volume-purchasing-locations/{id}/content and /api/v1/volume-purchasing-subscriptions` — VPP app license inventory and consumption (medium). Added as `volume_purchasing_content` (fan-out over the new `volume_purchasing_locations` table) and `volume_purchasing_subscriptions`.+- [x] `GET /api/v1/device-enrollments and /{id}/devices` — ADE/DEP enrollment records and their device rosters (medium). Added as `device_enrollments` and `device_enrollment_devices`.+- [x] `GET /api/v1/{buildings,categories,departments,packages,scripts}/{id}/history` — object change history for the reference tables we already sync (low). Added as `building_history`, `category_history`, `department_history`, `package_history` and `script_history` (full refresh).Note: Jamf publishes no downloadable OpenAPI on developer.jamf.com (ReadMe-hosted; /openapi.json 404s) — the resource list was extracted from the 408 get\_\* reference slugs embedded in the reference page HTML. The instance-hosted spec at https://<tenant>/api/schema requires auth. Source currently syncs /api/v1/computers-inventory and /api/v2/mobile-devices; note v4/computers-inventory and v2/mobile-devices/detail are the newest versions of those. Classic (XML) API resources such as policies and configuration profiles were not audited here.diff --git a/products/warehouse_sources/backend/temporal/data_imports/sources/jamf_pro/canonical_descriptions.py b/products/warehouse_sources/backend/temporal/data_imports/sources/jamf_pro/canonical_descriptions.pyindex 07e393454c26..db66ec30e282 100644--- a/products/warehouse_sources/backend/temporal/data_imports/sources/jamf_pro/canonical_descriptions.py+++ b/products/warehouse_sources/backend/temporal/data_imports/sources/jamf_pro/canonical_descriptions.py@@ -265,4 +265,168 @@ CANONICAL_DESCRIPTIONS: CanonicalDescriptions = {"mobileDeviceId": "Identifier of the member mobile device.",},},+ "computer_extension_attributes": {+ "description": "Custom computer extension attributes defined in Jamf Pro. Resolves the extension attribute ids in computer inventory records.",+ "docs_url": "https://developer.jamf.com/jamf-pro/reference/get_v1-computer-extension-attributes",+ "columns": {+ "id": "Unique identifier of the extension attribute.",+ "name": "Display name of the extension attribute.",+ "description": "Description of the extension attribute.",+ "dataType": "Type of the collected value: INTEGER, STRING or DATE.",+ "inventoryDisplayType": "Inventory category the extension attribute shows in.",+ "inputType": "How the value is collected: SCRIPT, TEXT, POPUP or DIRECTORY_SERVICE_ATTRIBUTE_MAPPING.",+ "enabled": "Whether the extension attribute is enabled.",+ "scriptContents": "Script that returns the value on each inventory submission, when inputType is SCRIPT.",+ "popupMenuChoices": "Allowed values, when inputType is POPUP.",+ "ldapAttributeMapping": "Directory service attribute that populates the value, when inputType is DIRECTORY_SERVICE_ATTRIBUTE_MAPPING.",+ },+ },+ "mobile_device_extension_attributes": {+ "description": "Custom mobile device extension attributes defined in Jamf Pro. Resolves the extension attribute ids in mobile device inventory records.",+ "docs_url": "https://developer.jamf.com/jamf-pro/reference/get_v1-mobile-device-extension-attributes",+ "columns": {+ "id": "Unique identifier of the extension attribute.",+ "name": "Display name of the extension attribute.",+ "description": "Description of the extension attribute.",+ "dataType": "Type of the collected value: INTEGER, STRING or DATE.",+ "inventoryDisplayType": "Inventory category the extension attribute shows in.",+ "inputType": "How the value is collected: TEXT, POPUP or DIRECTORY_SERVICE_ATTRIBUTE_MAPPING.",+ "popupMenuChoices": "Allowed values, when inputType is POPUP.",+ "ldapAttributeMapping": "Directory service attribute that populates the value, when inputType is DIRECTORY_SERVICE_ATTRIBUTE_MAPPING.",+ },+ },+ "volume_purchasing_locations": {+ "description": "Apple volume purchasing (VPP) locations connected to Jamf Pro, with their license totals.",+ "docs_url": "https://developer.jamf.com/jamf-pro/reference/get_v1-volume-purchasing-locations",+ "columns": {+ "id": "Unique identifier of the volume purchasing location.",+ "name": "Display name of the location.",+ "totalPurchasedLicenses": "Number of licenses purchased in the location.",+ "totalUsedLicenses": "Number of licenses in use in the location.",+ "appleId": "Apple ID of the location.",+ "organizationName": "Organization name of the location.",+ "tokenExpiration": "Expiry date of the location's service token.",+ "countryCode": "Country of the App Store the location buys from.",+ "lastSyncTime": "Timestamp of the last sync with Apple.",+ "siteId": "Identifier of the site the location belongs to.",+ },+ },+ "volume_purchasing_content": {+ "description": "Apps and books bought in each volume purchasing location, with their license counts.",+ "docs_url": "https://developer.jamf.com/jamf-pro/reference/get_v1-volume-purchasing-locations-id-content",+ "columns": {+ "volumePurchasingLocationId": "Identifier of the volume purchasing location.",+ "adamId": "App Store identifier of the content.",+ "name": "Name of the content.",+ "pricingParam": "Pricing tier the licenses were bought at: STDQ (standard) or PLUS (high quality).",+ "contentType": "Type of the content: IOS_APP, MAC_APP, BOOK or UNKNOWN.",+ "deviceTypes": "Device platforms the content runs on.",+ "licenseCountTotal": "Number of licenses bought.",+ "licenseCountInUse": "Number of licenses assigned in Jamf Pro.",+ "licenseCountReported": "Number of licenses Apple reports as in use.",+ },+ },+ "volume_purchasing_subscriptions": {+ "description": "Volume purchasing notification subscriptions: who is told when licenses run out or content leaves the App Store.",+ "docs_url": "https://developer.jamf.com/jamf-pro/reference/get_v1-volume-purchasing-subscriptions",+ "columns": {+ "id": "Unique identifier of the subscription.",+ "name": "Name of the subscription.",+ "enabled": "Whether the subscription is enabled.",+ "triggers": "Events that send a notification: NO_MORE_LICENSES or REMOVED_FROM_APP_STORE.",+ "locationIds": "Identifiers of the volume purchasing locations the subscription watches.",+ "internalRecipients": "Jamf Pro accounts that get the notification.",+ "externalRecipients": "Email recipients outside Jamf Pro that get the notification.",+ "siteId": "Identifier of the site the subscription belongs to.",+ },+ },+ "device_enrollments": {+ "description": "Automated Device Enrollment (ADE, formerly DEP) instances connected to Jamf Pro.",+ "docs_url": "https://developer.jamf.com/jamf-pro/reference/get_v1-device-enrollments",+ "columns": {+ "id": "Unique identifier of the device enrollment instance.",+ "name": "Display name of the instance.",+ "serverName": "Name of the MDM server in Apple Business Manager or Apple School Manager.",+ "serverUuid": "UUID of the MDM server.",+ "orgName": "Organization name from the service token.",+ "tokenExpirationDate": "Expiry date of the service token.",+ "siteId": "Identifier of the site the instance belongs to.",+ },+ },+ "device_enrollment_devices": {+ "description": "Devices assigned to each device enrollment instance in Apple Business Manager or Apple School Manager, with their prestage profile status.",+ "docs_url": "https://developer.jamf.com/jamf-pro/reference/get_v1-device-enrollments-id-devices",+ "columns": {+ "deviceEnrollmentProgramInstanceId": "Identifier of the device enrollment instance.",+ "id": "Identifier of the device record.",+ "prestageId": "Identifier of the prestage enrollment assigned to the device.",+ "serialNumber": "Hardware serial number of the device.",+ "model": "Device model.",+ "profileStatus": "Enrollment profile status: EMPTY, ASSIGNED, PUSHED or REMOVED.",+ "syncState": "Result of the last profile assignment sync with Apple.",+ "profileAssignTime": "Timestamp the enrollment profile was assigned.",+ "profilePushTime": "Timestamp the enrollment profile was pushed.",+ "deviceAssignedDate": "Timestamp the device was assigned to the instance.",+ },+ },+ "building_history": {+ "description": "Change history of each building: one row per history note recorded when the building was created or edited.",+ "docs_url": "https://developer.jamf.com/jamf-pro/reference/get_v1-buildings-id-history",+ "columns": {+ "buildingId": "Identifier of the building the note belongs to.",+ "id": "Identifier of the history note.",+ "username": "Jamf Pro user who made the change.",+ "date": "Timestamp of the change.",+ "note": "Short summary of the change.",+ "details": "Details of the change, if any.",+ },+ },+ "category_history": {+ "description": "Change history of each category: one row per history note recorded when the category was created or edited.",+ "docs_url": "https://developer.jamf.com/jamf-pro/reference/get_v1-categories-id-history",+ "columns": {+ "categoryId": "Identifier of the category the note belongs to.",+ "id": "Identifier of the history note.",+ "username": "Jamf Pro user who made the change.",+ "date": "Timestamp of the change.",+ "note": "Short summary of the change.",+ "details": "Details of the change, if any.",+ },+ },+ "department_history": {+ "description": "Change history of each department: one row per history note recorded when the department was created or edited.",+ "docs_url": "https://developer.jamf.com/jamf-pro/reference/get_v1-departments-id-history",+ "columns": {+ "departmentId": "Identifier of the department the note belongs to.",+ "id": "Identifier of the history note.",+ "username": "Jamf Pro user who made the change.",+ "date": "Timestamp of the change.",+ "note": "Short summary of the change.",+ "details": "Details of the change, if any.",+ },+ },+ "package_history": {+ "description": "Change history of each package: one row per history note recorded when the package was created or edited.",+ "docs_url": "https://developer.jamf.com/jamf-pro/reference/get_v1-packages-id-history",+ "columns": {+ "packageId": "Identifier of the package the note belongs to.",+ "id": "Identifier of the history note.",+ "username": "Jamf Pro user who made the change.",+ "date": "Timestamp of the change.",+ "note": "Short summary of the change.",+ "details": "Details of the change, if any.",+ },+ },+ "script_history": {+ "description": "Change history of each script: one row per history note recorded when the script was created or edited.",+ "docs_url": "https://developer.jamf.com/jamf-pro/reference/get_v1-scripts-id-history",+ "columns": {+ "scriptId": "Identifier of the script the note belongs to.",+ "id": "Identifier of the history note.",+ "username": "Jamf Pro user who made the change.",+ "date": "Timestamp of the change.",+ "note": "Short summary of the change.",+ "details": "Details of the change, if any.",+ },+ },}diff --git a/products/warehouse_sources/backend/temporal/data_imports/sources/jamf_pro/settings.py b/products/warehouse_sources/backend/temporal/data_imports/sources/jamf_pro/settings.pyindex aa111c4b2ad1..ba64816cf0dc 100644--- a/products/warehouse_sources/backend/temporal/data_imports/sources/jamf_pro/settings.py+++ b/products/warehouse_sources/backend/temporal/data_imports/sources/jamf_pro/settings.py@@ -219,6 +219,93 @@ JAMF_PRO_ENDPOINTS: dict[str, JamfProEndpointConfig] = {parent_filter=("isSmartGroup", False),row_fields=["mobileDeviceId"],),+ "computer_extension_attributes": JamfProEndpointConfig(+ name="computer_extension_attributes",+ path="/api/v1/computer-extension-attributes",+ # scriptContents can embed credentials, like scripts.+ capture_samples=False,+ ),+ "mobile_device_extension_attributes": JamfProEndpointConfig(+ name="mobile_device_extension_attributes",+ path="/api/v1/mobile-device-extension-attributes",+ ),+ "volume_purchasing_locations": JamfProEndpointConfig(+ name="volume_purchasing_locations",+ path="/api/v1/volume-purchasing-locations",+ ),+ # Content rows have no id of their own. One app can be bought at both pricing tiers, so the+ # tier is part of the key.+ "volume_purchasing_content": JamfProEndpointConfig(+ name="volume_purchasing_content",+ path="/api/v1/volume-purchasing-locations/{id}/content",+ primary_keys=["volumePurchasingLocationId", "adamId", "pricingParam"],+ sort="name:asc",+ parent="volume_purchasing_locations",+ parent_id_field="volumePurchasingLocationId",+ ),+ "volume_purchasing_subscriptions": JamfProEndpointConfig(+ name="volume_purchasing_subscriptions",+ path="/api/v1/volume-purchasing-subscriptions",+ ),+ "device_enrollments": JamfProEndpointConfig(+ name="device_enrollments",+ path="/api/v1/device-enrollments",+ ),+ # The device roster takes no paging parameters and returns every device of the instance at once.+ "device_enrollment_devices": JamfProEndpointConfig(+ name="device_enrollment_devices",+ path="/api/v1/device-enrollments/{id}/devices",+ paginated=False,+ sort=None,+ primary_keys=["deviceEnrollmentProgramInstanceId", "id"],+ parent="device_enrollments",+ parent_id_field="deviceEnrollmentProgramInstanceId",+ ),+ # Change history of the reference objects. Full refresh only: the `date` filter would not cut the+ # request count (one walk per parent either way), and a run-wide watermark across parents could skip+ # notes written to an earlier parent while later parents are still being read.+ "building_history": JamfProEndpointConfig(+ name="building_history",+ path="/api/v1/buildings/{id}/history",+ primary_keys=["buildingId", "id"],+ sort="date:asc",+ parent="buildings",+ parent_id_field="buildingId",+ ),+ "category_history": JamfProEndpointConfig(+ name="category_history",+ path="/api/v1/categories/{id}/history",+ primary_keys=["categoryId", "id"],+ sort="date:asc",+ parent="categories",+ parent_id_field="categoryId",+ ),+ "department_history": JamfProEndpointConfig(+ name="department_history",+ path="/api/v1/departments/{id}/history",+ primary_keys=["departmentId", "id"],+ sort="date:asc",+ parent="departments",+ parent_id_field="departmentId",+ ),+ "package_history": JamfProEndpointConfig(+ name="package_history",+ path="/api/v1/packages/{id}/history",+ primary_keys=["packageId", "id"],+ sort="date:asc",+ parent="packages",+ parent_id_field="packageId",+ ),+ "script_history": JamfProEndpointConfig(+ name="script_history",+ path="/api/v1/scripts/{id}/history",+ primary_keys=["scriptId", "id"],+ sort="date:asc",+ # This session also fetches the parent scripts list, which carries scriptContents.+ capture_samples=False,+ parent="scripts",+ parent_id_field="scriptId",+ ),}ENDPOINTS = tuple(JAMF_PRO_ENDPOINTS.keys())diff --git a/products/warehouse_sources/backend/temporal/data_imports/sources/jamf_pro/source.py b/products/warehouse_sources/backend/temporal/data_imports/sources/jamf_pro/source.pyindex 39d9bf7ec510..1fa873d329ca 100644--- a/products/warehouse_sources/backend/temporal/data_imports/sources/jamf_pro/source.py+++ b/products/warehouse_sources/backend/temporal/data_imports/sources/jamf_pro/source.py@@ -76,7 +76,7 @@ class JamfProSource(ResumableSource[JamfProSourceConfig, JamfProResumeConfig]):keywords=["jamf", "mdm", "apple", "device management"],caption="""Enter your Jamf Pro instance URL and credentials to pull your device inventory into the PostHog Data warehouse.-The recommended way to connect is an API client: in Jamf Pro go to **Settings > System > API roles and clients**, create an API role with **read** privileges for the objects you want to sync (Computers, Mobile Devices, Buildings, Departments, Categories, Sites, Smart Computer Groups, Static Computer Groups, Smart Mobile Device Groups, Static Mobile Device Groups, Scripts, Packages, Users, Patch Management Software Titles, Patch Policies), plus **View MDM command information in Jamf Pro API** for MDM commands, then create an API client with that role and enable it.+The recommended way to connect is an API client: in Jamf Pro go to **Settings > System > API roles and clients**, create an API role with **read** privileges for the objects you want to sync (Computers, Mobile Devices, Buildings, Departments, Categories, Sites, Smart Computer Groups, Static Computer Groups, Smart Mobile Device Groups, Static Mobile Device Groups, Scripts, Packages, Users, Patch Management Software Titles, Patch Policies, Computer Extension Attributes, Mobile Device Extension Attributes, Volume Purchasing Locations, Device Enrollment Program Instances), plus **View MDM command information in Jamf Pro API** for MDM commands, then create an API client with that role and enable it.Alternatively, connect with a Jamf Pro user account that has read access to those objects.""",diff --git a/products/warehouse_sources/backend/temporal/data_imports/sources/jamf_pro/tests/test_jamf_pro.py b/products/warehouse_sources/backend/temporal/data_imports/sources/jamf_pro/tests/test_jamf_pro.pyindex 39390fcae69a..b32cad8a042c 100644--- a/products/warehouse_sources/backend/temporal/data_imports/sources/jamf_pro/tests/test_jamf_pro.py+++ b/products/warehouse_sources/backend/temporal/data_imports/sources/jamf_pro/tests/test_jamf_pro.py@@ -237,6 +237,8 @@ class TestValidateCredentials:"/api/v2/mdm/commands","page=0&page-size=1&filter=dateSent%3E%3D%221970-01-01T00%3A00%3A00.000Z%22",),+ ("device_enrollment_devices", "/api/v1/device-enrollments", "page=0&page-size=1"),+ ("script_history", "/api/v1/scripts", "page=0&page-size=1"),],)def test_scoped_probe_targets_an_addressable_url(self, schema_name, expected_path, expected_query):@@ -386,6 +388,79 @@ class TestGetRows:assert result.rows[-1] == {"deviceId": "d1", "statusEnum": "COMPLETE", "patchPolicyId": "200"}assert result.raised is None+ @pytest.mark.parametrize(+ "endpoint, parent_path, child_path, parent_id_field",+ [+ ("building_history", "/api/v1/buildings", "/api/v1/buildings/5/history", "buildingId"),+ ("category_history", "/api/v1/categories", "/api/v1/categories/5/history", "categoryId"),+ ("department_history", "/api/v1/departments", "/api/v1/departments/5/history", "departmentId"),+ ("package_history", "/api/v1/packages", "/api/v1/packages/5/history", "packageId"),+ ("script_history", "/api/v1/scripts", "/api/v1/scripts/5/history", "scriptId"),+ (+ "volume_purchasing_content",+ "/api/v1/volume-purchasing-locations",+ "/api/v1/volume-purchasing-locations/5/content",+ "volumePurchasingLocationId",+ ),+ ],+ )+ def test_paginated_fan_out_children_page_each_parent(self, endpoint, parent_path, child_path, parent_id_field):+ parents = ScriptedResponse(json={"totalCount": 1, "results": [{"id": "5"}]})+ child_page_0 = ScriptedResponse(json={"totalCount": 201, "results": [{"id": 1}] * 200})+ child_page_1 = ScriptedResponse(json={"totalCount": 201, "results": [{"id": 2}]})+ result = self._drive([parents, child_page_0, child_page_1], endpoint=endpoint)++ assert result.paths[1:] == [parent_path, child_path, child_path]+ assert result.params("page")[2:] == ["0", "1"]+ assert result.params("sort")[2:] == [JAMF_PRO_ENDPOINTS[endpoint].sort] * 2+ assert len(result.rows) == 201+ assert result.rows[-1] == {"id": 2, parent_id_field: "5"}+ assert result.raised is None++ def test_unpaginated_fan_out_child_reads_results_wrapper(self):+ instances = ScriptedResponse(json={"totalCount": 2, "results": [{"id": "1"}, {"id": "2"}]})+ roster_1 = ScriptedResponse(+ json={+ "totalCount": 2,+ "results": [+ {"id": "10", "serialNumber": "SERIAL0001", "deviceEnrollmentProgramInstanceId": "1"},+ {"id": "11", "serialNumber": "SERIAL0002"},+ ],+ }+ )+ roster_2 = ScriptedResponse(json={"totalCount": 0, "results": []})+ result = self._drive([instances, roster_1, roster_2], endpoint="device_enrollment_devices")++ # The roster endpoint takes no paging or sort parameters.+ assert result.urls[2:] == [+ "https://example.jamfcloud.com/api/v1/device-enrollments/1/devices",+ "https://example.jamfcloud.com/api/v1/device-enrollments/2/devices",+ ]+ assert result.rows == [+ {"id": "10", "serialNumber": "SERIAL0001", "deviceEnrollmentProgramInstanceId": "1"},+ {"id": "11", "serialNumber": "SERIAL0002", "deviceEnrollmentProgramInstanceId": "1"},+ ]+ assert result.raised is None++ @pytest.mark.parametrize(+ "endpoint, expected_capture",+ [+ ("computer_extension_attributes", False),+ ("script_history", False),+ ("mobile_device_extension_attributes", True),+ ("building_history", True),+ ],+ )+ def test_data_session_sample_capture(self, endpoint, expected_capture):+ # Script bodies can embed credentials that the name-based scrubbers cannot see.+ config = JAMF_PRO_ENDPOINTS[endpoint]+ fetched_config = JAMF_PRO_ENDPOINTS[config.parent] if config.parent else config+ payload = {"results": []} if fetched_config.paginated else []+ result = self._drive([ScriptedResponse(json=payload)], endpoint=endpoint)++ assert result.session_options[-1] == {"capture": expected_capture}+ assert result.raised is None+def test_fan_out_refuses_an_unbounded_parent_list(self):endless_parents = ScriptedResponse(json={"totalCount": 10**9, "results": [{"id": "1"}, {"id": "2"}]})with mock.patch.object(jamf_pro_module, "MAX_PARENT_IDS", 3):