Skip to content

:hedgehog: PostHog is the leading platform for building self-driving products. Our developer tools – AI observability, analytics, session replay, flags, experiments, error tracking, logs, and more – capture all the context agents need to diagnose problems, uncover opportunities, and ship fixes. Steer it all from Slack, web, desktop, or the MCP.

feat(data-warehouse): add 2 missing tables to the Hyperspell source (#119622)

Tom Owersa60e2f89a68f
diff --git a/products/warehouse_sources/backend/temporal/data_imports/sources/COVERAGE_GAPS_APPENDIX.md b/products/warehouse_sources/backend/temporal/data_imports/sources/COVERAGE_GAPS_APPENDIX.md
index f5fe97bcf127..bf9aec3e390f 100644
--- a/products/warehouse_sources/backend/temporal/data_imports/sources/COVERAGE_GAPS_APPENDIX.md
+++ b/products/warehouse_sources/backend/temporal/data_imports/sources/COVERAGE_GAPS_APPENDIX.md
@@ -6157,18 +6157,18 @@ Note: docs.huntr.co is a single-page Slate reference; parsed every https://api.h
## Hyperspell — gaps
-Today (9): `connections`, `context_documents`, `entities`, `integration_channels`, `integrations`, `memories`, `queries`, `users`, `vaults`
+Today (11): `connection_folders`, `connections`, `context_documents`, `emotional_states`, `entities`, `integration_channels`, `integrations`, `memories`, `queries`, `users`, `vaults`
Diffed against: <https://docs.hyperspell.com/llms.txt>
- [x] `GET /users` — user roster - lookup resolving the user IDs on memories, connections and queries we already sync (high). Added as `users`.
-- [ ] `GET /entities/{entity_id}/sources` — join table linking entities we sync back to the source documents they were extracted from (medium)
-- [ ] `GET /connections/{connection_id}/folders` — the folder inventory per connection, needed to see what scope each connection actually indexes (medium)
+- [ ] `GET /entities/{entity_id}/sources` — join table linking entities we sync back to the source documents they were extracted from (medium). Skipped: not in the public OpenAPI spec (v0.32.1) or the API reference, and the live API returns 404 for the route.
+- [x] `GET /connections/{connection_id}/folders` — the folder inventory per connection, needed to see what scope each connection actually indexes (medium). Added as `connection_folders` (tree walk over each user's connections whose integration supports folder selection).
- [ ] `GET /context-documents/conflicts` — detected conflicts across context documents - the quality signal for the docs we already sync (medium). Skipped: not in the public OpenAPI spec (v0.32.1) or the API reference.
- [ ] `GET /context-documents/reviews` — document review records and their suggestions, the human-in-the-loop audit trail (medium). Skipped: not in the public OpenAPI spec (v0.32.1) or the API reference.
- [x] `GET /integrations/{integration_id}/channels` — lookup of available channels per integration, resolving channel IDs on memories (medium). Added as `integration_channels` (fan-out over each user's connections whose integration supports channel selection).
-- [ ] `GET /context-documents/tree/{tree_id}/edits` — persisted user edits per document tree - edit history over synced context documents (low)
-- [ ] `GET /emotional-state/recent` — time series of stored emotional-state observations (low)
+- [ ] `GET /context-documents/tree/{tree_id}/edits` — persisted user edits per document tree - edit history over synced context documents (low). Skipped: not in the public OpenAPI spec (v0.32.1) or the API reference, so there is no documented response shape or source of tree ids.
+- [x] `GET /emotional-state/recent` — time series of stored emotional-state observations (low). Added as `emotional_states` (the API returns at most the 20 newest states per user, so the table is a rolling window).
- [ ] `GET /memories/status` — indexing progress per memory, useful for freshness/completeness checks (low)
Note: Enumerated the API reference from https://docs.hyperspell.com/llms.txt, then confirmed exact HTTP paths by fetching the individual .md pages (each embeds its Stainless OpenAPI operation line). The 7 synced tables cover the main top-level collections; the gaps are the users roster and sub-resources of things we already sync.
diff --git a/products/warehouse_sources/backend/temporal/data_imports/sources/hyperspell/canonical_descriptions.py b/products/warehouse_sources/backend/temporal/data_imports/sources/hyperspell/canonical_descriptions.py
index b6941ee9e963..e2ea86f94c5d 100644
--- a/products/warehouse_sources/backend/temporal/data_imports/sources/hyperspell/canonical_descriptions.py
+++ b/products/warehouse_sources/backend/temporal/data_imports/sources/hyperspell/canonical_descriptions.py
@@ -124,4 +124,31 @@ CANONICAL_DESCRIPTIONS: CanonicalDescriptions = {
"selected": "Whether the channel is currently selected for indexing on the connection.",
},
},
+ "connection_folders": {
+ "description": "Folders on each connection whose integration supports folder selection (for example Google Drive or Dropbox), with the explicit folder policy that controls what Hyperspell indexes.",
+ "docs_url": "https://docs.hyperspell.com/api-reference/folders/list-folders-for-a-connection",
+ "columns": {
+ "user_id": "The Hyperspell user the connection belongs to (empty when synced app-wide without a user).",
+ "connection_id": "The connection the folder was listed through.",
+ "integration_id": "The connection's integration id.",
+ "provider_folder_id": "The folder's id at the source provider.",
+ "name": "Display name of the folder.",
+ "parent_folder_id": "The provider id of the parent folder (null for root-level folders).",
+ "has_children": "Whether the folder may have subfolders.",
+ "policy": "The explicit policy set on the folder (its id and sync_mode of sync, skip, or manual), or null when the folder inherits its policy or uses the default.",
+ },
+ },
+ "emotional_states": {
+ "description": "The most recent emotional states stored for each user, extracted from conversation transcripts. The API returns at most the 20 newest states per user, so older states drop out of the table.",
+ "docs_url": "https://docs.hyperspell.com/api-reference/emotional-state/get-recent-emotional-states",
+ "columns": {
+ "user_id": "The Hyperspell user the emotional state belongs to (empty when synced app-wide without a user).",
+ "resource_id": "Identifier of the stored emotional state.",
+ "summary": "Summary of the emotional state.",
+ "extracted_at": "When the emotional state was extracted.",
+ "session_id": "The conversation session the state was extracted from, if any.",
+ "relationship_id": "The relationship the state is scoped to, if any.",
+ "summary_source": "Where the summary came from.",
+ },
+ },
}
diff --git a/products/warehouse_sources/backend/temporal/data_imports/sources/hyperspell/hyperspell.py b/products/warehouse_sources/backend/temporal/data_imports/sources/hyperspell/hyperspell.py
index 49987ea7b6ca..751544d57b8a 100644
--- a/products/warehouse_sources/backend/temporal/data_imports/sources/hyperspell/hyperspell.py
+++ b/products/warehouse_sources/backend/temporal/data_imports/sources/hyperspell/hyperspell.py
@@ -1,3 +1,4 @@
+from collections import deque
from collections.abc import Iterator
from typing import Any, Optional
from urllib.parse import quote, urlencode
@@ -26,6 +27,8 @@ DEFAULT_REGION = "us"
REQUEST_TIMEOUT_SECONDS = 60
MAX_RETRY_ATTEMPTS = 5
+# Each folders request lists the provider live, so bound the tree walk per connection.
+MAX_FOLDER_REQUESTS_PER_CONNECTION = 200
class HyperspellRetryableError(Exception):
@@ -115,9 +118,7 @@ def validate_credentials(
wait=wait_exponential_jitter(initial=1, max=30),
reraise=True,
)
-def _fetch_page(
- session: requests.Session, page_url: str, headers: dict[str, str], logger: FilteringBoundLogger
-) -> dict[str, Any]:
+def _fetch_page(session: requests.Session, page_url: str, headers: dict[str, str], logger: FilteringBoundLogger) -> Any:
response = session.get(page_url, headers=headers, timeout=REQUEST_TIMEOUT_SECONDS)
# Rate limits aren't documented; treat 429 and transient 5xx as retryable with backoff.
@@ -146,7 +147,7 @@ def _normalize_row(config: HyperspellEndpointConfig, item: dict[str, Any], user_
def _page_params(config: HyperspellEndpointConfig, cursor: str | None) -> dict[str, Any]:
- params: dict[str, Any] = {}
+ params: dict[str, Any] = dict(config.static_params)
if config.pagination == "none":
return params
if config.page_size_param:
@@ -184,7 +185,7 @@ def _get_list_rows(
url = _build_url(base_url, config.path, _page_params(config, cursor))
data = _fetch_page(session, url, headers, logger)
- items = data.get(config.data_key) or []
+ items = (data if config.data_key is None else data.get(config.data_key)) or []
next_cursor = _next_cursor(config, data, items, cursor)
if items:
@@ -199,16 +200,19 @@ def _get_list_rows(
cursor = next_cursor
-def _channel_integration_ids(
- session: requests.Session, base_url: str, api_key: str, logger: FilteringBoundLogger
+# Integration capability flags that make a connection worth fanning out over, per endpoint.
+FAN_OUT_INTEGRATION_FLAGS: dict[str, tuple[str, ...]] = {
+ "integration_channels": ("supports_channel_selection", "requires_channel_selection"),
+ "connection_folders": ("supports_folder_selection", "folder_selection_required"),
+}
+
+
+def _fan_out_integration_ids(
+ session: requests.Session, base_url: str, api_key: str, flags: tuple[str, ...], logger: FilteringBoundLogger
) -> set[str]:
url = _build_url(base_url, HYPERSPELL_ENDPOINTS["integrations"].path, {})
integrations = _fetch_page(session, url, _get_headers(api_key), logger).get("integrations") or []
- return {
- integration["id"]
- for integration in integrations
- if integration.get("supports_channel_selection") or integration.get("requires_channel_selection")
- }
+ return {integration["id"] for integration in integrations if any(integration.get(flag) for flag in flags)}
def _get_channel_rows(
@@ -258,6 +262,68 @@ def _get_channel_rows(
]
+def _get_folder_rows(
+ session: requests.Session,
+ base_url: str,
+ config: HyperspellEndpointConfig,
+ headers: dict[str, str],
+ principal: str | None,
+ folder_integration_ids: set[str],
+ logger: FilteringBoundLogger,
+ resumable_source_manager: ResumableSourceManager[HyperspellResumeConfig],
+) -> Iterator[list[dict[str, Any]]]:
+ connections_url = _build_url(base_url, HYPERSPELL_ENDPOINTS["connections"].path, {})
+ connections = _fetch_page(session, connections_url, headers, logger).get("connections") or []
+ resumable_source_manager.safe_point()
+
+ for connection in connections:
+ integration_id = connection.get("integration_id")
+ if integration_id not in folder_integration_ids:
+ continue
+
+ connection_id = connection["id"]
+ path = config.path.format(connection_id=quote(connection_id, safe=""))
+ # The endpoint lists one level per request, so walk the tree breadth-first from the root.
+ pending_parents: deque[str | None] = deque([None])
+ seen: set[str] = set()
+ requests_made = 0
+
+ while pending_parents:
+ if requests_made >= MAX_FOLDER_REQUESTS_PER_CONNECTION:
+ logger.warning(
+ f"Hyperspell: folder walk for connection {connection_id} hit the cap of "
+ f"{MAX_FOLDER_REQUESTS_PER_CONNECTION} requests, deeper folders are not synced"
+ )
+ break
+
+ parent_id = pending_parents.popleft()
+ url = _build_url(base_url, path, {"parent_id": parent_id} if parent_id else {})
+ data = _fetch_page(session, url, headers, logger)
+ requests_made += 1
+
+ rows: list[dict[str, Any]] = []
+ for folder in data.get(config.data_key) or []:
+ folder_id = folder.get("provider_folder_id")
+ # Providers such as Google Drive let one folder sit under several parents.
+ if not folder_id or folder_id in seen:
+ continue
+ seen.add(folder_id)
+ if folder.get("has_children"):
+ pending_parents.append(folder_id)
+ rows.append(
+ {
+ **_normalize_row(config, folder, principal),
+ "connection_id": connection_id,
+ "integration_id": integration_id,
+ }
+ )
+
+ if rows:
+ yield rows
+ else:
+ resumable_source_manager.safe_point()
+
+
def get_rows(
api_key: str,
region: str | None,
@@ -294,8 +360,9 @@ def get_rows(
resume_cursor = resume.cursor
logger.debug(f"Hyperspell: resuming {endpoint} from user_id={resume.user_id}, cursor={resume_cursor}")
- channel_integration_ids = (
- _channel_integration_ids(session, base_url, api_key, logger) if endpoint == "integration_channels" else set()
+ fan_out_flags = FAN_OUT_INTEGRATION_FLAGS.get(endpoint)
+ fan_out_integration_ids = (
+ _fan_out_integration_ids(session, base_url, api_key, fan_out_flags, logger) if fan_out_flags else set()
)
for index, principal in enumerate(remaining):
@@ -310,7 +377,18 @@ def get_rows(
config,
headers,
principal,
- channel_integration_ids,
+ fan_out_integration_ids,
+ logger,
+ resumable_source_manager,
+ )
+ elif endpoint == "connection_folders":
+ yield from _get_folder_rows(
+ session,
+ base_url,
+ config,
+ headers,
+ principal,
+ fan_out_integration_ids,
logger,
resumable_source_manager,
)
diff --git a/products/warehouse_sources/backend/temporal/data_imports/sources/hyperspell/settings.py b/products/warehouse_sources/backend/temporal/data_imports/sources/hyperspell/settings.py
index 07833b9d785c..730a96b9f957 100644
--- a/products/warehouse_sources/backend/temporal/data_imports/sources/hyperspell/settings.py
+++ b/products/warehouse_sources/backend/temporal/data_imports/sources/hyperspell/settings.py
@@ -1,5 +1,5 @@
from dataclasses import field
-from typing import Literal, Optional
+from typing import Any, Literal, Optional
from posthog.dataclasses import frozen
@@ -11,7 +11,8 @@ class HyperspellEndpointConfig:
name: str
path: str
# Key that wraps the list of rows in the JSON response body (e.g. {"items": [...]}).
- data_key: str
+ # None when the response body is the list itself.
+ data_key: Optional[str]
primary_keys: list[str]
# Endpoints whose data is scoped to a Hyperspell user. When the source is configured with
# user IDs, these endpoints are fetched once per user via the X-As-User header and every
@@ -23,6 +24,8 @@ class HyperspellEndpointConfig:
# Query param that controls page size ("size" or "limit", varies per endpoint).
page_size_param: Optional[str] = "size"
page_size: int = 100
+ # Query params sent on every request, independent of pagination.
+ static_params: dict[str, Any] = field(default_factory=dict)
# Stable datetime field to partition on. Never use a field that changes after creation.
partition_key: Optional[str] = None
# Nullable fields that participate in the primary key, mapped to the value that replaces
@@ -121,6 +124,29 @@ HYPERSPELL_ENDPOINTS: dict[str, HyperspellEndpointConfig] = {
user_scoped=True,
pagination="none",
),
+ # GET /connections/{connection_id}/folders?parent_id=... — one level of folders per request,
+ # walked down the tree for each of the user's connections whose integration supports
+ # folder selection. Folder ids come from the provider, so the connection is part of the key.
+ "connection_folders": HyperspellEndpointConfig(
+ name="connection_folders",
+ path="/connections/{connection_id}/folders",
+ data_key="folders",
+ primary_keys=["user_id", "connection_id", "provider_folder_id"],
+ user_scoped=True,
+ pagination="none",
+ ),
+ # GET /emotional-state/recent — a bare JSON array of the newest emotional states, newest
+ # first. The API has no pagination and caps `limit` at 20, so the table holds the 20 most
+ # recent states per user, not the full history.
+ "emotional_states": HyperspellEndpointConfig(
+ name="emotional_states",
+ path="/emotional-state/recent",
+ data_key=None,
+ primary_keys=["user_id", "resource_id"],
+ user_scoped=True,
+ pagination="none",
+ static_params={"limit": 20},
+ ),
}
ENDPOINTS = tuple(HYPERSPELL_ENDPOINTS.keys())
diff --git a/products/warehouse_sources/backend/temporal/data_imports/sources/hyperspell/tests/test_hyperspell.py b/products/warehouse_sources/backend/temporal/data_imports/sources/hyperspell/tests/test_hyperspell.py
index 342602650399..3dacd71edefd 100644
--- a/products/warehouse_sources/backend/temporal/data_imports/sources/hyperspell/tests/test_hyperspell.py
+++ b/products/warehouse_sources/backend/temporal/data_imports/sources/hyperspell/tests/test_hyperspell.py
@@ -11,6 +11,7 @@ from products.warehouse_sources.backend.temporal.data_imports.sources.common.tes
from products.warehouse_sources.backend.temporal.data_imports.sources.generated_configs.hyperspell import (
HyperspellSourceConfig,
)
+from products.warehouse_sources.backend.temporal.data_imports.sources.hyperspell import hyperspell
from products.warehouse_sources.backend.temporal.data_imports.sources.hyperspell.hyperspell import (
HyperspellResumeConfig,
validate_credentials,
@@ -22,7 +23,7 @@ def _driver(user_ids: str | None = None) -> SourceDriver:
return SourceDriver(HyperspellSource(), HyperspellSourceConfig(api_key="hs_test", region="us", user_ids=user_ids))
-def _page(body: dict[str, Any]) -> ScriptedResponse:
+def _page(body: dict[str, Any] | list[dict[str, Any]]) -> ScriptedResponse:
return ScriptedResponse(json=body)
@@ -182,6 +183,121 @@ class TestGetRows:
]
]
+ def test_connection_folders_walk_tree_of_folder_capable_connections(self) -> None:
+ result = _driver(user_ids="user-1").run(
+ "connection_folders",
+ [
+ _page(
+ {
+ "integrations": [
+ {"id": "int-drive", "supports_folder_selection": True},
+ {"id": "int-slack", "supports_folder_selection": False},
+ ]
+ }
+ ),
+ _page(
+ {
+ "connections": [
+ {"id": "conn-slack", "integration_id": "int-slack"},
+ {"id": "conn-drive", "integration_id": "int-drive"},
+ ]
+ }
+ ),
+ _page(
+ {
+ "folders": [
+ {"provider_folder_id": "F1", "name": "Projects", "has_children": True},
+ {"provider_folder_id": "F2", "name": "Archive", "has_children": False},
+ ]
+ }
+ ),
+ _page(
+ {
+ "folders": [
+ {
+ "provider_folder_id": "F3",
+ "name": "Alpha",
+ "parent_folder_id": "F1",
+ "has_children": False,
+ "policy": {"id": "pol-1", "sync_mode": "skip"},
+ },
+ # Already listed under the root; a second parent must not duplicate the row.
+ {"provider_folder_id": "F2", "name": "Archive", "has_children": False},
+ ]
+ }
+ ),
+ ],
+ )
+
+ assert result.raised is None
+ assert len(result.requests) == 4
+ assert result.requests[2].url.endswith("/connections/conn-drive/folders")
+ assert result.requests[3].url.endswith("/connections/conn-drive/folders?parent_id=F1")
+ assert result.requests[3].headers["x-as-user"] == "user-1"
+ assert [[row["provider_folder_id"] for row in batch] for batch in result.items] == [["F1", "F2"], ["F3"]]
+ assert result.items[1][0] == {
+ "provider_folder_id": "F3",
+ "name": "Alpha",
+ "parent_folder_id": "F1",
+ "has_children": False,
+ "policy": {"id": "pol-1", "sync_mode": "skip"},
+ "user_id": "user-1",
+ "connection_id": "conn-drive",
+ "integration_id": "int-drive",
+ }
+
+ def test_connection_folders_walk_stops_at_request_cap(self, monkeypatch: pytest.MonkeyPatch) -> None:
+ monkeypatch.setattr(hyperspell, "MAX_FOLDER_REQUESTS_PER_CONNECTION", 2)
+
+ result = _driver().run(
+ "connection_folders",
+ [
+ _page({"integrations": [{"id": "int-drive", "folder_selection_required": True}]}),
+ _page({"connections": [{"id": "conn-drive", "integration_id": "int-drive"}]}),
+ _page({"folders": [{"provider_folder_id": "F1", "has_children": True}]}),
+ _page({"folders": [{"provider_folder_id": "F2", "has_children": True}]}),
+ ],
+ )
+
+ assert result.raised is None
+ assert len(result.requests) == 4
+ assert [[row["provider_folder_id"] for row in batch] for batch in result.items] == [["F1"], ["F2"]]
+
+ def test_emotional_states_reads_bare_array_per_user(self) -> None:
+ result = _driver(user_ids="user-1, user-2").run(
+ "emotional_states",
+ [
+ _page(
+ [
+ {
+ "resource_id": "es-2",
+ "summary": "calm",
+ "extracted_at": "2026-01-02T00:00:00Z",
+ "session_id": None,
+ "relationship_id": None,
+ }
+ ]
+ ),
+ _page([]),
+ ],
+ )
+
+ assert result.raised is None
+ assert [request.headers["x-as-user"] for request in result.requests] == ["user-1", "user-2"]
+ assert result.params("limit") == ["20", "20"]
+ assert result.items == [
+ [
+ {
+ "resource_id": "es-2",
+ "summary": "calm",
+ "extracted_at": "2026-01-02T00:00:00Z",
+ "session_id": None,
+ "relationship_id": None,
+ "user_id": "user-1",
+ }
+ ]
+ ]
+
class TestHyperspellSource:
def test_entities_response_has_datetime_partition(self) -> None: