:hedgehog: PostHog is the leading platform for building self-driving products. Our developer tools – AI observability, analytics, session replay, flags, experiments, error tracking, logs, and more – capture all the context agents need to diagnose problems, uncover opportunities, and ship fixes. Steer it all from Slack, web, desktop, or the MCP.
Python ab-testingai-analyticsanalyticscdpdata-warehouseexperimentsfeature-flagsjavascriptproduct-analyticspythonreactsession-replaysurveystypescriptweb-analytics
feat(data-warehouse): add 3 missing tables to the MyHours source (#123766)
Tom Owersde497bd0f869
diff --git a/products/warehouse_sources/backend/temporal/data_imports/sources/COVERAGE_GAPS_APPENDIX.md b/products/warehouse_sources/backend/temporal/data_imports/sources/COVERAGE_GAPS_APPENDIX.mdindex 36a3b7aa8358..a7495df8a75e 100644--- a/products/warehouse_sources/backend/temporal/data_imports/sources/COVERAGE_GAPS_APPENDIX.md+++ b/products/warehouse_sources/backend/temporal/data_imports/sources/COVERAGE_GAPS_APPENDIX.md@@ -8277,9 +8277,9 @@ Diffed against: <https://documenter.gw.postman.com/api/collections/8879268/TVmV4- [x] `PUT /api/Reports/activitydx` — bulk time-log report over a date range with client/project/user filters — the practical backfill path for logs (high). Added as `time_logs` (31-day date windows, incremental on `date`).- [x] `GET /api/Projects/{projectId}/tasklist` — project task lookup resolving the task each time log is booked against (high). Added as `project_tasks` (fan-out over `projects`, one row per task across every task list).- [x] `GET /api/Projects/{projectId}/userlist` — project-to-team-member assignment join table (medium). Added as `project_members` (fan-out over `projects`).-- [ ] `GET /api/Teams/{teamId}/teammembers` — team membership for grouping users in reporting (medium)-- [ ] `GET /api/Projects/{projectId}/overview` — project detail including budget/rate fields not on the list response (low)-- [ ] `GET /api/Reports/dashboardOptimized` — pre-aggregated dashboard totals by client/project/user (low)+- [x] `GET /api/Teams/{teamId}/teammembers` — team membership for grouping users in reporting (medium). Added as `team_members` (fan-out over the new `teams` table, read from `GET /api/Teams/getall`; Pro plan only).+- [x] `GET /api/Projects/{projectId}/overview` — project detail including budget/rate fields not on the list response (low). Added as `project_overviews` (fan-out over `projects`, one row per project).+- [ ] `GET /api/Reports/dashboardOptimized` — pre-aggregated dashboard totals by client/project/user (low). Not added: it returns totals for one caller-chosen date range and grouping, with no stable row key. The same totals can be computed from `time_logs`.Note: Doc is a Postman collection (My Hours API v1.1); parsed the raw collection JSON from documenter.gw.postman.com. Source is static (4 tables in settings.py, no dynamic discovery) and omits the time-log endpoints entirely.diff --git a/products/warehouse_sources/backend/temporal/data_imports/sources/my_hours/canonical_descriptions.py b/products/warehouse_sources/backend/temporal/data_imports/sources/my_hours/canonical_descriptions.pyindex 80951b07cce9..5c2e17714a1d 100644--- a/products/warehouse_sources/backend/temporal/data_imports/sources/my_hours/canonical_descriptions.py+++ b/products/warehouse_sources/backend/temporal/data_imports/sources/my_hours/canonical_descriptions.py@@ -139,4 +139,64 @@ CANONICAL_DESCRIPTIONS: CanonicalDescriptions = {"userIsProjectManager": "Whether the team member is a manager of this project.",},},+ "project_overviews": {+ "description": "The full detail of a project, with budget, rate and invoicing fields that the project list does not return. One row per project.",+ "docs_url": _DOCS_URL,+ "columns": {+ "id": "The unique ID of the project.",+ "projectId": "The ID of the project, the same value as `id`.",+ "name": "The project name.",+ "clientId": "The ID of the client this project belongs to.",+ "clientName": "The name of the client this project belongs to.",+ "notes": "The project notes.",+ "budgetType": "The project's budget type: 1 no budget, 2 total billable hours, 3 total billable amount.",+ "budgetValue": "The configured budget amount for the project.",+ "budgetSpent": "How much of the project's budget has been used.",+ "budgetRemaining": "How much of the project's budget is left.",+ "budgetSpentPercentage": "The share of the project's budget used, as a percentage.",+ "budgetPeriodType": "The period that the project's budget applies to.",+ "invoiceMethod": "How the project is billed: 0 not billable, 1 task rate, 2 project rate, 3 team member rate.",+ "rate": "The project's billable rate, when the project is billed by project rate.",+ "flatFee": "The project's flat fee, if set.",+ "totalTimeLogged": "Total time logged to the project.",+ "billableTimeLogged": "Billable time logged to the project.",+ "nonBillableTimeLogged": "Non-billable time logged to the project.",+ "totalAmount": "Total billable amount accrued on the project.",+ "totalExpense": "Total expenses logged to the project.",+ "totalCost": "Total labor cost accrued on the project.",+ "logCount": "The number of time logs on the project.",+ "firstLogDate": "The date of the first time log on the project.",+ "lastLogDate": "The date of the most recent time log on the project.",+ "projectUsers": "The team members assigned to the project.",+ "startDate": "When the project starts, if set.",+ "dueDate": "When the project is due, if set.",+ "dateCreated": "When the project was created.",+ "createdByUserId": "The ID of the team member who created the project.",+ "archived": "Whether the project has been archived.",+ "dateArchived": "When the project was archived, if applicable.",+ },+ },+ "teams": {+ "description": "A team: a group of team members used to manage access and group reports. Available on the Pro plan.",+ "docs_url": _DOCS_URL,+ "columns": {+ "id": "The unique ID of the team.",+ "name": "The team name.",+ "size": "The number of team members in the team.",+ "users": "The team members in the team.",+ "managers": "The managers of the team.",+ },+ },+ "team_members": {+ "description": "A team member assigned to a team. One row per team and team member.",+ "docs_url": _DOCS_URL,+ "columns": {+ "teamId": "The ID of the team.",+ "id": "The ID of the team membership.",+ "userId": "The ID of the team member.",+ "userName": "The team member's name.",+ "userCustomId": "A user-defined external identifier for the team member.",+ "userArchived": "Whether the team member has been archived.",+ },+ },}diff --git a/products/warehouse_sources/backend/temporal/data_imports/sources/my_hours/settings.py b/products/warehouse_sources/backend/temporal/data_imports/sources/my_hours/settings.pyindex 51efc9e24c14..03e6da2c2091 100644--- a/products/warehouse_sources/backend/temporal/data_imports/sources/my_hours/settings.py+++ b/products/warehouse_sources/backend/temporal/data_imports/sources/my_hours/settings.py@@ -42,6 +42,16 @@ _PROJECT_FANOUT = DependentEndpointConfig(child_response_actions=[{"status_code": 404, "action": "ignore"}],)+_TEAM_FANOUT = DependentEndpointConfig(+ parent_name="teams",+ resolve_param="team_id",+ resolve_field="id",+ include_from_parent=["id"],+ parent_field_renames={"id": "teamId"},+ # A team deleted between the team listing and its child request.+ child_response_actions=[{"status_code": 404, "action": "ignore"}],+)+# My Hours REST API list endpoints. The plain list endpoints are full refresh only: they expose no# row-level pagination and no updated-since cursor. `time_logs` reads the account-wide activity# report, which filters server-side by log date.@@ -65,10 +75,28 @@ MY_HOURS_ENDPOINTS: dict[str, MyHoursEndpointConfig] = {primary_keys=["projectId", "userId"],fanout=_PROJECT_FANOUT,),+ "project_overviews": MyHoursEndpointConfig(+ name="project_overviews",+ path="/Projects/{project_id}/overview",+ fanout=_PROJECT_FANOUT,+ ),+ # Teams are a Pro plan feature: other plans answer 403.+ "teams": MyHoursEndpointConfig(name="teams", path="/Teams/getall"),+ "team_members": MyHoursEndpointConfig(+ name="team_members",+ path="/Teams/{team_id}/teammembers",+ primary_keys=["teamId", "id"],+ fanout=_TEAM_FANOUT,+ ),}ENDPOINTS = tuple(MY_HOURS_ENDPOINTS.keys())+# Teams are a Pro plan feature, so these tables need a probe of the team listing before setup.+TEAMS_PROBE_PATH = "/Teams/getall"+TEAM_ENDPOINTS = frozenset({"teams", "team_members"})+TEAMS_UNAVAILABLE_ERROR = "Teams are only available on the My Hours Pro plan, and this API key cannot read them."+INCREMENTAL_FIELDS: dict[str, list[IncrementalField]] = {TIME_LOGS_ENDPOINT: [{diff --git a/products/warehouse_sources/backend/temporal/data_imports/sources/my_hours/source.py b/products/warehouse_sources/backend/temporal/data_imports/sources/my_hours/source.pyindex 5f52868b22eb..c84086e52a86 100644--- a/products/warehouse_sources/backend/temporal/data_imports/sources/my_hours/source.py+++ b/products/warehouse_sources/backend/temporal/data_imports/sources/my_hours/source.py@@ -29,6 +29,9 @@ from products.warehouse_sources.backend.temporal.data_imports.sources.my_hours.sENDPOINTS,INCREMENTAL_FIELDS,MY_HOURS_ENDPOINTS,+ TEAM_ENDPOINTS,+ TEAMS_PROBE_PATH,+ TEAMS_UNAVAILABLE_ERROR,TIME_LOGS_ENDPOINT,TIME_LOGS_LOOKBACK_SECONDS,)@@ -56,7 +59,7 @@ class MyHoursSource(SimpleSource[MyHoursSourceConfig]):releaseStatus=ReleaseStatus.BETA,caption="""Enter your My Hours API key to pull your time-tracking data into the PostHog Data warehouse.-API keys are available on paid plans. Create one under **Settings → Integrations → API keys** in your [My Hours account](https://app.myhours.com/). The key is tied to the user who created it, so it stops working if that user is archived or removed. The key grants read access to your clients, projects, project tasks and members, tags, users, and time logs.+API keys are available on paid plans. Create one under **Settings → Integrations → API keys** in your [My Hours account](https://app.myhours.com/). The key is tied to the user who created it, so it stops working if that user is archived or removed. The key grants read access to your clients, projects, project tasks and members, tags, users, teams, and time logs. Teams are available on the Pro plan.""",iconPath="/static/services/my_hours.png",docsUrl="https://posthog.com/docs/cdp/sources/my-hours",@@ -106,6 +109,18 @@ API keys are available on paid plans. Create one under **Settings → Integratiodefault_incremental_lookback_seconds={TIME_LOGS_ENDPOINT: TIME_LOGS_LOOKBACK_SECONDS},)+ def get_endpoint_permissions(+ self, config: MyHoursSourceConfig, team_id: int, endpoints: list[str], api_version: str | None = None+ ) -> dict[str, str | None]:+ # Report the Pro-only team tables as unavailable, rather than letting them fail every sync.+ if not TEAM_ENDPOINTS.intersection(endpoints):+ return dict.fromkeys(endpoints)++ status, _ = check_access(config.api_key, TEAMS_PROBE_PATH)+ # Only a definite refusal hides the tables; an unreachable probe leaves them selectable.+ reason = TEAMS_UNAVAILABLE_ERROR if status in (401, 403) else None+ return {name: reason if name in TEAM_ENDPOINTS else None for name in endpoints}+def validate_credentials(self,config: MyHoursSourceConfig,diff --git a/products/warehouse_sources/backend/temporal/data_imports/sources/my_hours/tests/test_my_hours.py b/products/warehouse_sources/backend/temporal/data_imports/sources/my_hours/tests/test_my_hours.pyindex e33dcae5ad73..04a04279cd6f 100644--- a/products/warehouse_sources/backend/temporal/data_imports/sources/my_hours/tests/test_my_hours.py+++ b/products/warehouse_sources/backend/temporal/data_imports/sources/my_hours/tests/test_my_hours.py@@ -26,6 +26,7 @@ from products.warehouse_sources.backend.temporal.data_imports.sources.my_hours.mfrom products.warehouse_sources.backend.temporal.data_imports.sources.my_hours.settings import (ENDPOINTS,MY_HOURS_ENDPOINTS,+ TEAMS_UNAVAILABLE_ERROR,)from products.warehouse_sources.backend.temporal.data_imports.sources.my_hours.source import MyHoursSource@@ -35,7 +36,14 @@ def _driver() -> SourceDriver:class TestSync:- @parameterized.expand([("clients", "/Clients"), ("projects", "/Projects/getAll"), ("users", "/Users/getAll")])+ @parameterized.expand(+ [+ ("clients", "/Clients"),+ ("projects", "/Projects/getAll"),+ ("users", "/Users/getAll"),+ ("teams", "/Teams/getall"),+ ]+ )def test_request_targets_the_expected_url(self, endpoint: str, path: str) -> None:result = _driver().run(endpoint, [ScriptedResponse(json=[])])@@ -92,6 +100,40 @@ class TestCheckAccess:assert {request.path for request in network.requests_log} == {"/api/Clients"}+class TestEndpointPermissions:+ def setup_method(self) -> None:+ self.source = MyHoursSource()+ self.config = MyHoursSourceConfig(api_key="mh-key")++ @parameterized.expand([("forbidden", 403), ("unauthorized", 401)])+ def test_team_tables_are_flagged_when_the_plan_refuses_teams(self, _name: str, status: int) -> None:+ with scripted_network([ScriptedResponse(status=status)]) as network:+ permissions = self.source.get_endpoint_permissions(self.config, 1, list(ENDPOINTS))++ assert permissions["teams"] == TEAMS_UNAVAILABLE_ERROR+ assert permissions["team_members"] == TEAMS_UNAVAILABLE_ERROR+ assert all(reason is None for name, reason in permissions.items() if name not in ("teams", "team_members"))+ assert [request.path for request in network.requests_log] == ["/api/Teams/getall"]++ @parameterized.expand(+ [+ ("reachable", ScriptedResponse(json=[])),+ ("server_error", ScriptedResponse(status=500)),+ ("connection_error", requests.ConnectionError("boom")),+ ]+ )+ def test_team_tables_stay_selectable_unless_refused(+ self, _name: str, answer: ScriptedResponse | BaseException+ ) -> None:+ with scripted_network(always(answer)):+ assert self.source.get_endpoint_permissions(self.config, 1, list(ENDPOINTS)) == dict.fromkeys(ENDPOINTS)++ def test_no_probe_when_no_team_table_is_requested(self) -> None:+ with scripted_network([]) as network:+ assert self.source.get_endpoint_permissions(self.config, 1, ["clients"]) == {"clients": None}+ assert network.requests_log == []++def _empty_answer(request: RecordedRequest) -> ScriptedResponse:return ScriptedResponse(json={"data": []} if request.path.endswith("/Reports/activitydx") else [])@@ -190,16 +232,60 @@ class TestProjectFanout:{"id": task_id, "projectId": 11, "taskListName": "Task list", "taskListNo": 0} for task_id in (1, 2, 3)]- def test_project_members_carry_project_id_and_skip_deleted_projects(self) -> None:+ @parameterized.expand(+ [+ (+ "project_members",+ "/Projects/getAll",+ "/Projects/11/userlist",+ [{"id": 5, "userId": 5, "userName": "A"}],+ [{"id": 5, "userId": 5, "userName": "A", "projectId": 11}],+ ),+ (+ "project_overviews",+ "/Projects/getAll",+ "/Projects/11/overview",+ {"id": 11, "name": "Website", "budgetRemaining": 9.5, "projectUsers": [{"id": 5, "name": "A"}]},+ [+ {+ "id": 11,+ "name": "Website",+ "budgetRemaining": 9.5,+ "projectUsers": [{"id": 5, "name": "A"}],+ "projectId": 11,+ }+ ],+ ),+ (+ "team_members",+ "/Teams/getall",+ "/Teams/11/teammembers",+ [{"id": 7, "userId": 5, "userName": "A", "userCustomId": None}],+ [{"id": 7, "userId": 5, "userName": "A", "userCustomId": None, "teamId": 11}],+ ),+ ]+ )+ def test_child_rows_carry_parent_id_and_skip_deleted_parents(+ self,+ endpoint: str,+ parent_path: str,+ child_path: str,+ child_body: object,+ expected_rows: list[dict[str, object]],+ ) -> None:result = _driver().run(- "project_members",+ endpoint,[- ScriptedResponse(json=[{"id": 11}, {"id": 12}]),- ScriptedResponse(json=[{"id": 5, "userId": 5, "userName": "A"}]),+ ScriptedResponse(json=[{"id": 11, "users": [], "managers": []}, {"id": 12}]),+ ScriptedResponse(json=child_body),ScriptedResponse(status=404, json={"error": "not found"}),],)assert result.raised is None- assert result.urls[1] == f"{MY_HOURS_BASE_URL}/Projects/11/userlist"- assert result.rows == [{"id": 5, "userId": 5, "userName": "A", "projectId": 11}]+ assert result.urls == [+ f"{MY_HOURS_BASE_URL}{parent_path}",+ f"{MY_HOURS_BASE_URL}{child_path}",+ f"{MY_HOURS_BASE_URL}{child_path.replace('11', '12')}",+ ]+ assert result.rows == expected_rows