:hedgehog: PostHog is the leading platform for building self-driving products. Our developer tools – AI observability, analytics, session replay, flags, experiments, error tracking, logs, and more – capture all the context agents need to diagnose problems, uncover opportunities, and ship fixes. Steer it all from Slack, web, desktop, or the MCP.
Python ab-testingai-analyticsanalyticscdpdata-warehouseexperimentsfeature-flagsjavascriptproduct-analyticspythonreactsession-replaysurveystypescriptweb-analytics
feat(data-warehouse): add 2 missing tables to the SharePoint source (#128635)
Tom Owersea5b59911b75
diff --git a/products/warehouse_sources/backend/temporal/data_imports/sources/COVERAGE_GAPS_APPENDIX.md b/products/warehouse_sources/backend/temporal/data_imports/sources/COVERAGE_GAPS_APPENDIX.mdindex 328094b15fe0..bb059bd10f15 100644--- a/products/warehouse_sources/backend/temporal/data_imports/sources/COVERAGE_GAPS_APPENDIX.md+++ b/products/warehouse_sources/backend/temporal/data_imports/sources/COVERAGE_GAPS_APPENDIX.md@@ -10944,8 +10944,8 @@ Diffed against: <https://learn.microsoft.com/en-us/graph/api/resources/driveitem- [x] `Drive item versions (GET /v1.0/drives/{drive-id}/items/{item-id}/versions)` — track document revisions, editors, and historical file sizes (medium)- [x] `List item versions (GET /v1.0/sites/{site-id}/lists/{list-id}/items/{item-id}/versions)` — analyze changes to business records and the people who edited them (medium)- [ ] `Recent item activities (GET /v1.0/drives/{drive-id}/items/{item-id}/activities)` — analyze recent user actions on documents instead of only their latest modification timestamps (medium)-- [ ] `Drive item permissions (GET /v1.0/drives/{drive-id}/items/{item-id}/permissions)` — analyze document sharing, granted access, and links across document libraries (medium)-- [ ] `Site pages (GET /v1.0/sites/{site-id}/pages/microsoft.graph.sitePage)` — analyze published pages, authorship, and publishing state separately from file metadata (medium)+- [x] `Drive item permissions (GET /v1.0/drives/{drive-id}/items/{item-id}/permissions)` — analyze document sharing, granted access, and links across document libraries (medium)+- [x] `Site pages (GET /v1.0/sites/{site-id}/pages/microsoft.graph.sitePage)` — analyze published pages, authorship, and publishing state separately from file metadata (medium)Note: Diffed Microsoft's v1.0 references for drive items, list items, sites, and content types, including linked operation pages. Checked each reported path and its application permissions; nested collections require requests for each parent. The source already expands list fields, walks drive items through delta, and optionally imports CSV and Excel contents. Excluded uploads, thumbnails, notifications, and broader directory resources.diff --git a/products/warehouse_sources/backend/temporal/data_imports/sources/sharepoint/canonical_descriptions.py b/products/warehouse_sources/backend/temporal/data_imports/sources/sharepoint/canonical_descriptions.pyindex 6ca019923494..3f4ac375f906 100644--- a/products/warehouse_sources/backend/temporal/data_imports/sources/sharepoint/canonical_descriptions.py+++ b/products/warehouse_sources/backend/temporal/data_imports/sources/sharepoint/canonical_descriptions.py@@ -171,4 +171,46 @@ CANONICAL_DESCRIPTIONS: CanonicalDescriptions = {"fields": "Values of the list's columns in this version, keyed by column internal name.",},},+ "drive_item_permissions": {+ "description": "A sharing permission or sharing link on a file or folder in a SharePoint document library.",+ "docs_url": "https://learn.microsoft.com/en-us/graph/api/resources/permission",+ "columns": {+ "site_id": "Identifier of the site the drive belongs to.",+ "drive_id": "Identifier of the drive (document library) the item belongs to.",+ "item_id": "Identifier of the file or folder the permission applies to.",+ "id": "Identifier of the permission. An inherited permission keeps the identifier of the ancestor's permission.",+ "roles": "Type of access the permission grants, such as read, write, or owner.",+ "grantedToV2": "For user permissions, the user, group, or app the permission was granted to.",+ "grantedToIdentitiesV2": "For link permissions, the users and groups that were granted access through the link.",+ "inheritedFrom": "Reference to the ancestor item the permission is inherited from, if any.",+ "link": "For link permissions, the link type and scope. The URL is removed for anonymous links.",+ "invitation": "Details of the sharing invitation for the permission, if any.",+ "expirationDateTime": "Date and time the permission expires, if it has an expiration.",+ "hasPassword": "Whether a password is set on the permission.",+ },+ },+ "site_pages": {+ "description": "A modern page or news post in the site pages library of a SharePoint site.",+ "docs_url": "https://learn.microsoft.com/en-us/graph/api/resources/sitepage",+ "columns": {+ "site_id": "Identifier of the site the page belongs to.",+ "id": "Unique identifier of the page.",+ "name": "File name of the page.",+ "title": "Title of the page.",+ "description": "Description of the page.",+ "webUrl": "URL that opens the page in a browser.",+ "pageLayout": "Layout of the page, such as article or home.",+ "promotionKind": "Whether the page is a regular page or a news post.",+ "publishingState": "Publishing level and version of the page.",+ "createdDateTime": "Date and time the page was created.",+ "lastModifiedDateTime": "Date and time the page was last modified.",+ "createdBy": "Identity of the user who created the page.",+ "lastModifiedBy": "Identity of the user who last modified the page.",+ "showComments": "Whether comments show at the bottom of the page.",+ "showRecommendedPages": "Whether recommended pages show at the bottom of the page.",+ "thumbnailWebUrl": "URL of the page's thumbnail image.",+ "reactions": "Reaction counts for the page, such as likes, comments, and shares.",+ "titleArea": "Title area settings of the page.",+ },+ },}diff --git a/products/warehouse_sources/backend/temporal/data_imports/sources/sharepoint/settings.py b/products/warehouse_sources/backend/temporal/data_imports/sources/sharepoint/settings.pyindex f400da390add..b07a2d0e7414 100644--- a/products/warehouse_sources/backend/temporal/data_imports/sources/sharepoint/settings.py+++ b/products/warehouse_sources/backend/temporal/data_imports/sources/sharepoint/settings.py@@ -44,6 +44,8 @@ class SharePointEndpoint(StrEnum):SITE_ACTIVITY_STATS = "site_activity_stats"DRIVE_ITEM_VERSIONS = "drive_item_versions"LIST_ITEM_VERSIONS = "list_item_versions"+ DRIVE_ITEM_PERMISSIONS = "drive_item_permissions"+ SITE_PAGES = "site_pages"@dataclass(frozen=True)@@ -123,6 +125,22 @@ SHAREPOINT_ENDPOINTS: dict[str, SharePointEndpointConfig] = {),should_sync_default=False,),+ SharePointEndpoint.DRIVE_ITEM_PERMISSIONS: SharePointEndpointConfig(+ name=SharePointEndpoint.DRIVE_ITEM_PERMISSIONS,+ # Inherited permissions keep the id of the ancestor's permission on every item below it.+ primary_keys=[DRIVE_ID_COLUMN, ITEM_ID_COLUMN, "id"],+ description=(+ "Sharing permissions and sharing links of every file and folder in every document library. "+ "Each sync makes one request per file or folder"+ ),+ should_sync_default=False,+ ),+ SharePointEndpoint.SITE_PAGES: SharePointEndpointConfig(+ name=SharePointEndpoint.SITE_PAGES,+ primary_keys=[SITE_ID_COLUMN, "id"],+ description="Modern site pages and news posts in each site, with their publishing state",+ partition_key="createdDateTime",+ ),}ENDPOINTS = tuple(SHAREPOINT_ENDPOINTS.keys())diff --git a/products/warehouse_sources/backend/temporal/data_imports/sources/sharepoint/sharepoint.py b/products/warehouse_sources/backend/temporal/data_imports/sources/sharepoint/sharepoint.pyindex 220e794592d4..b7368a96cd31 100644--- a/products/warehouse_sources/backend/temporal/data_imports/sources/sharepoint/sharepoint.py+++ b/products/warehouse_sources/backend/temporal/data_imports/sources/sharepoint/sharepoint.py@@ -33,7 +33,12 @@ VALIDATE_TIMEOUT_SECONDS = 30_module_logger: FilteringBoundLogger = structlog.get_logger(__name__)LIST_ENDPOINTS = (SharePointEndpoint.LIST_ITEMS, SharePointEndpoint.LIST_ITEM_VERSIONS)-VERSION_ENDPOINTS = (SharePointEndpoint.DRIVE_ITEM_VERSIONS, SharePointEndpoint.LIST_ITEM_VERSIONS)+# Tables read with one request per list item or drive item, keyed by the collection below each item.+ITEM_CHILD_COLLECTIONS = {+ SharePointEndpoint.DRIVE_ITEM_VERSIONS: "versions",+ SharePointEndpoint.LIST_ITEM_VERSIONS: "versions",+ SharePointEndpoint.DRIVE_ITEM_PERMISSIONS: "permissions",+}SITES_DENIED_ERROR = ("Microsoft Graph denied access to SharePoint. Grant the app the Sites.Read.All application permission "@@ -396,8 +401,8 @@ def _iter_grandchildren(def stage(next_url: str, site_id: str = site_id, parent_id: str = parent_id) -> None:manager.save_state(SharePointResumeConfig(site_id=site_id, parent_id=parent_id, next_url=next_url))- if endpoint in VERSION_ENDPOINTS:- pages = _iter_parent_versions(+ if endpoint in ITEM_CHILD_COLLECTIONS:+ pages = _iter_item_children(client, endpoint, quoted_site, site_id, parent_id, start_url, stage, manager.safe_point, logger)else:@@ -464,7 +469,19 @@ def _iter_parent_rows(yield rows-def _iter_parent_versions(+def _strip_access_secrets(row: dict[str, Any]) -> dict[str, Any]:+ # Graph's preauthenticated download URL grants temporary access to the document contents.+ row.pop("@microsoft.graph.downloadUrl", None)+ link = row.get("link")+ if isinstance(link, dict) and link.get("scope") == "anonymous":+ # Anyone with an anonymous sharing link can open the item without signing in.+ link.pop("webUrl", None)+ link.pop("webHtml", None)+ row.pop("shareId", None)+ return row+++def _iter_item_children(client: SharePointClient,endpoint: SharePointEndpoint,quoted_site: str,@@ -475,22 +492,23 @@ def _iter_parent_versions(safe_point: Callable[[], None],logger: FilteringBoundLogger,) -> Iterator[list[dict[str, Any]]]:- """Versions of every item in one list (`list_item_versions`) or every file in one drive (`drive_item_versions`).+ """Rows below every item in one list or drive: versions of list items and files, or permissions of drive items.- The cursor names the page of items in progress, not the next one, because the versions of- that page are still being read. A resume reads the versions of that page again.+ The cursor names the page of items in progress, not the next one, because the rows of+ that page are still being read. A resume reads the rows of that page again."""quoted_parent = quote(parent_id, safe="!-_")is_list = endpoint == SharePointEndpoint.LIST_ITEM_VERSIONS+ collection = ITEM_CHILD_COLLECTIONS[endpoint]if is_list:items_path = f"/sites/{quoted_site}/lists/{quoted_parent}/items"item_prefix = items_path- version_params: Optional[dict[str, str]] = {"expand": "fields"}+ child_params: Optional[dict[str, str]] = {"expand": "fields"}parent_columns = {SITE_ID_COLUMN: site_id, LIST_ID_COLUMN: parent_id}else:items_path = f"/drives/{quoted_parent}/root/delta"item_prefix = f"/drives/{quoted_parent}/items"- version_params = None+ child_params = Noneparent_columns = {SITE_ID_COLUMN: site_id, DRIVE_ID_COLUMN: parent_id}next_url: Optional[str] = None@@ -509,18 +527,17 @@ def _iter_parent_versions(item_id = item.get("id")if item_id is None or item_id in seen:continue- # Folders have no versions, and delta can return an item more than once.- if not is_list and (item.get("deleted") is not None or item.get("file") is None):+ # Delta can return an item more than once. Folders have permissions but no versions.+ if not is_list and item.get("deleted") is not None:+ continue+ if endpoint == SharePointEndpoint.DRIVE_ITEM_VERSIONS and item.get("file") is None:continueseen.add(item_id)- path = f"{item_prefix}/{quote(item_id, safe='!-_')}/versions"+ path = f"{item_prefix}/{quote(item_id, safe='!-_')}/{collection}"item_columns = {**parent_columns, ITEM_ID_COLUMN: item_id}item_yielded = False- for versions in _walk_child_collection(client, path, version_params, None, lambda _: None, logger):- rows = []- for version in versions:- version.pop("@microsoft.graph.downloadUrl", None)- rows.append(_normalize(version, endpoint, **item_columns))+ for children in _walk_child_collection(client, path, child_params, None, lambda _: None, logger):+ rows = [_normalize(_strip_access_secrets(child), endpoint, **item_columns) for child in children]item_yielded = Trueyield rowsif item_yielded:@@ -568,6 +585,7 @@ def _get_rows(SharePointEndpoint.LISTS: "lists",SharePointEndpoint.DRIVES: "drives",SharePointEndpoint.CONTENT_TYPES: "contentTypes",+ SharePointEndpoint.SITE_PAGES: "pages/microsoft.graph.sitePage",}if sharepoint_endpoint == SharePointEndpoint.SITE_ACTIVITY_STATS:collection = _activity_stats_collection(incremental_field_last_value, datetime.now(UTC))diff --git a/products/warehouse_sources/backend/temporal/data_imports/sources/sharepoint/tests/test_sharepoint.py b/products/warehouse_sources/backend/temporal/data_imports/sources/sharepoint/tests/test_sharepoint.pyindex a9d5f1606401..e75e5be5cc69 100644--- a/products/warehouse_sources/backend/temporal/data_imports/sources/sharepoint/tests/test_sharepoint.py+++ b/products/warehouse_sources/backend/temporal/data_imports/sources/sharepoint/tests/test_sharepoint.py@@ -352,6 +352,7 @@ class TestSharePointSiteTables:[("drives", "drives"),("content_types", "contentTypes"),+ ("site_pages", "pages/microsoft.graph.sitePage"),])def test_reads_each_site_collection(self, endpoint: str, collection: str) -> None:@@ -486,6 +487,66 @@ class TestSharePointVersions:assert _g("/drives/b!drive-a/root/delta") not in resumed.urlsassert [(row["item_id"], row["id"]) for row in resumed.rows] == [("f1", "2.0"), ("f1", "1.0"), ("f3", "1.0")]+ def test_drive_item_permissions_cover_folders_and_hide_anonymous_links(self) -> None:+ routes: dict[str, Route] = {+ **SITE_A_ROUTE,+ _g(f"/sites/{SITE_A}/drives"): {"value": [{"id": "b!drive-a"}]},+ _g("/drives/b!drive-a/root/delta"): {+ "value": [+ {"id": "root", "root": {}, "folder": {}},+ {"id": "f1", "file": {}},+ {"id": "gone", "deleted": {}},+ {"id": "f1", "file": {}},+ ]+ },+ _g("/drives/b!drive-a/items/root/permissions"): {+ "value": [{"id": "owners", "roles": ["owner"], "grantedToV2": {"siteGroup": {"id": "3"}}}]+ },+ _g("/drives/b!drive-a/items/f1/permissions"): {+ "value": [+ {"id": "owners", "roles": ["owner"], "inheritedFrom": {"id": "root"}},+ {+ "id": "anon-link",+ "roles": ["read"],+ "shareId": "u!fake-share-token",+ "link": {+ "scope": "anonymous",+ "type": "view",+ "webUrl": "https://contoso.example.com/:x:/anon",+ },+ },+ {+ "id": "org-link",+ "roles": ["write"],+ "shareId": "u!fake-org-token",+ "link": {+ "scope": "organization",+ "type": "edit",+ "webUrl": "https://contoso.example.com/:x:/org",+ },+ },+ ]+ },+ }++ result = _run("drive_item_permissions", routes, site_urls=SITE_A_URL)++ assert result.raised is None+ rows = {(row["drive_id"], row["item_id"], row["id"]): row for row in result.rows}+ assert list(rows) == [+ ("b!drive-a", "root", "owners"),+ ("b!drive-a", "f1", "owners"),+ ("b!drive-a", "f1", "anon-link"),+ ("b!drive-a", "f1", "org-link"),+ ]+ assert rows[("b!drive-a", "f1", "anon-link")]["link"] == {"scope": "anonymous", "type": "view"}+ assert "shareId" not in rows[("b!drive-a", "f1", "anon-link")]+ assert rows[("b!drive-a", "f1", "org-link")]["link"]["webUrl"] == "https://contoso.example.com/:x:/org"+ assert [url for url in result.urls if "/permissions" in url] == [+ _g("/drives/b!drive-a/items/root/permissions"),+ _g("/drives/b!drive-a/items/f1/permissions"),+ ]+def test_list_item_versions_skip_hidden_lists_and_expand_fields(self) -> None:routes: dict[str, Route] = {**SITE_A_ROUTE,