:hedgehog: PostHog is the leading platform for building self-driving products. Our developer tools – AI observability, analytics, session replay, flags, experiments, error tracking, logs, and more – capture all the context agents need to diagnose problems, uncover opportunities, and ship fixes. Steer it all from Slack, web, desktop, or the MCP.
Python ab-testingai-analyticsanalyticscdpdata-warehouseexperimentsfeature-flagsjavascriptproduct-analyticspythonreactsession-replaysurveystypescriptweb-analytics
feat(data-warehouse): add twelve labs knowledge store item tags table (#119534)
Tom Owers9edbc7c5f32a
diff --git a/products/warehouse_sources/backend/temporal/data_imports/sources/COVERAGE_GAPS_APPENDIX.md b/products/warehouse_sources/backend/temporal/data_imports/sources/COVERAGE_GAPS_APPENDIX.mdindex 788b643d6d52..2f36dc90433b 100644--- a/products/warehouse_sources/backend/temporal/data_imports/sources/COVERAGE_GAPS_APPENDIX.md+++ b/products/warehouse_sources/backend/temporal/data_imports/sources/COVERAGE_GAPS_APPENDIX.md@@ -12626,7 +12626,7 @@ Note: No OpenAPI spec is published; the full endpoint inventory was read from th## TwelveLabs — gaps-Today (4): `asset_transcriptions`, `indexes`, `tasks`, `videos`+Today (5): `asset_transcriptions`, `indexes`, `knowledge_store_item_tags`, `tasks`, `videos`Diffed against: <https://docs.twelvelabs.io/openapi.json>@@ -12640,6 +12640,7 @@ Diffed against: <https://docs.twelvelabs.io/openapi.json>- [x] `asset transcriptions (GET /assets/{asset_id}/transcription)` — per-asset speech transcription with status and timestamped sentences and speaker turns (medium)- [ ] `knowledge-stores, item-collections, items (GET /knowledge-stores, .../item-collections, .../items)` — the retrieval corpus objects; items resolve ids used by search (medium)- [ ] `connection imports (GET /connections/{connection_id}/imports)` — per-import job records for bulk ingestion runs (low)+- [x] `knowledge store item tags (GET /knowledge-stores/{knowledge_store_id}/items/{item_id}/tags)` — moment and range tags with custom metadata on knowledge store video items (medium)Note: Fetched the live OpenAPI (server https://api.twelvelabs.io/v1.3 — same version the PostHog source targets). The API has grown well past indexes/tasks/videos: assets, analysis tasks/batches, embeddings, entity collections and knowledge stores are all listable.diff --git a/products/warehouse_sources/backend/temporal/data_imports/sources/twelve_labs/canonical_descriptions.py b/products/warehouse_sources/backend/temporal/data_imports/sources/twelve_labs/canonical_descriptions.pyindex 54fcfc3123ba..16c30c5673e2 100644--- a/products/warehouse_sources/backend/temporal/data_imports/sources/twelve_labs/canonical_descriptions.py+++ b/products/warehouse_sources/backend/temporal/data_imports/sources/twelve_labs/canonical_descriptions.py@@ -57,4 +57,18 @@ CANONICAL_DESCRIPTIONS: CanonicalDescriptions = {"error": "Details about the failure, with a human-readable message. Present when the status is failed.",},},+ "knowledge_store_item_tags": {+ "description": "An item tag marks a moment or a range of a knowledge store video item and carries custom metadata.",+ "docs_url": "https://docs.twelvelabs.io/v1.3/api-reference/knowledge-store-items/item-tags/list",+ "columns": {+ "_id": "Unique identifier of the item tag.",+ "knowledge_store_id": "Identifier of the knowledge store.",+ "item_id": "Identifier of the knowledge store item that holds the tag.",+ "start_sec": "Start of the tag, in seconds from the start of the video.",+ "end_sec": "End of the range, in seconds from the start of the video. Null for a tag on a single moment.",+ "tag_metadata": "Custom metadata of the tag. Each value is a string, a number, a boolean, or an array of strings.",+ "created_at": "Date and time, in RFC 3339 format, the tag was created.",+ "updated_at": "Date and time, in RFC 3339 format, the tag was last updated.",+ },+ },}diff --git a/products/warehouse_sources/backend/temporal/data_imports/sources/twelve_labs/settings.py b/products/warehouse_sources/backend/temporal/data_imports/sources/twelve_labs/settings.pyindex 26ef429cc55b..8bdbe004cc6b 100644--- a/products/warehouse_sources/backend/temporal/data_imports/sources/twelve_labs/settings.py+++ b/products/warehouse_sources/backend/temporal/data_imports/sources/twelve_labs/settings.py@@ -16,7 +16,8 @@ def _datetime_field(name: str) -> IncrementalField:@dataclass(frozen=True)class TwelveLabsEndpointConfig:name: str- # `path` may contain an `{index_id}` or `{asset_id}` placeholder for fan-out endpoints.+ # `path` may contain `{index_id}`, `{asset_id}`, or `{knowledge_store_id}` + `{item_id}` placeholders+ # for fan-out endpoints.path: strincremental_fields: list[IncrementalField]# Field to partition by. Must be a STABLE creation-time field so partitions don't rewrite@@ -24,9 +25,9 @@ class TwelveLabsEndpointConfig:partition_key: Optional[str] = "created_at"primary_keys: list[str] = field(default_factory=lambda: ["_id"])should_sync_default: bool = True- # Fan out one request per parent index or asset. `path` is then a template with the parent's- # placeholder, and the parent id is injected into every row.- fan_out_over: Optional[Literal["indexes", "assets"]] = None+ # Fan out one request per parent index, asset, or knowledge store item. `path` is then a template+ # with the parent's placeholders.+ fan_out_over: Optional[Literal["indexes", "assets", "knowledge_store_items"]] = None# The child returns one object per parent instead of a paginated `data` list.single_object: bool = False# Extra static query params sent with every request to this endpoint.@@ -82,6 +83,19 @@ TWELVE_LABS_ENDPOINTS: dict[str, TwelveLabsEndpointConfig] = {single_object=True,params={"include": "sentences,utterances"},),+ # Tags mark a moment or range of a knowledge store video item with custom metadata. Items are+ # nested under knowledge stores, so this fans out over every store, then every video item. The+ # endpoint has no time filter and sorts by `start_sec`, so it ships as full refresh. Tag ids are+ # only documented per item, so the key carries both parent ids. Off by default because of the+ # per-item API cost.+ "knowledge_store_item_tags": TwelveLabsEndpointConfig(+ name="knowledge_store_item_tags",+ path="/knowledge-stores/{knowledge_store_id}/items/{item_id}/tags",+ incremental_fields=[],+ primary_keys=["knowledge_store_id", "item_id", "_id"],+ should_sync_default=False,+ fan_out_over="knowledge_store_items",+ ),}ENDPOINTS = tuple(TWELVE_LABS_ENDPOINTS.keys())diff --git a/products/warehouse_sources/backend/temporal/data_imports/sources/twelve_labs/tests/test_twelve_labs.py b/products/warehouse_sources/backend/temporal/data_imports/sources/twelve_labs/tests/test_twelve_labs.pyindex be94576c2130..1282a9ac394d 100644--- a/products/warehouse_sources/backend/temporal/data_imports/sources/twelve_labs/tests/test_twelve_labs.py+++ b/products/warehouse_sources/backend/temporal/data_imports/sources/twelve_labs/tests/test_twelve_labs.py@@ -185,6 +185,32 @@ class TestFanOut:assert result.requests[0].query["asset_types"] == ("video", "audio")assert dict(result.requests[1].query) == {"include": ("sentences,utterances",)}+ def test_item_tags_fan_out_over_each_store_and_its_video_items(self) -> None:+ # The tags path needs the store id from two levels up, so it must be carried through the+ # items level. Image items hold no tags and must not cost a request.+ tag = {"_id": "t1", "knowledge_store_id": "ks2", "item_id": "i2", "start_sec": 1.5, "end_sec": None}+ result = _run(+ "knowledge_store_item_tags",+ [+ _page([{"_id": "ks1"}, {"_id": "ks2"}], page=1, total_page=1),+ _page([{"_id": "i1", "asset_type": "image"}], page=1, total_page=1),+ _page([{"_id": "i2", "asset_type": "video"}], page=1, total_page=1),+ _page([tag], page=1, total_page=1),+ ],+ )++ assert result.rows == [tag]+ assert result.paths == [+ "/v1.3/knowledge-stores",+ "/v1.3/knowledge-stores/ks1/items",+ "/v1.3/knowledge-stores/ks2/items",+ "/v1.3/knowledge-stores/ks2/items/i2/tags",+ ]+ assert result.requests[1].param("sort_option") == "asc"+ assert result.requests[3].param("knowledge_store_id") is None+ assert result.response is not None+ assert result.response.supports_resume is False+def test_old_shape_saved_state_still_parses(self) -> None:# ResumableSourceManager._load_json does dataclass(**saved) — state saved before the# migration must still construct.diff --git a/products/warehouse_sources/backend/temporal/data_imports/sources/twelve_labs/twelve-labs.md b/products/warehouse_sources/backend/temporal/data_imports/sources/twelve_labs/twelve-labs.mdindex 0e039479531b..4110d630ec3e 100644--- a/products/warehouse_sources/backend/temporal/data_imports/sources/twelve_labs/twelve-labs.md+++ b/products/warehouse_sources/backend/temporal/data_imports/sources/twelve_labs/twelve-labs.md@@ -15,8 +15,8 @@ import AlphaRelease from "../\_snippets/alpha-release.mdx"Sync your [Twelve Labs](https://www.twelvelabs.io) video understanding library into PostHog. Thisconnector imports your indexes, the videos in each index, the video indexing tasks that track the-upload and indexing lifecycle, and the transcriptions of your video and audio assets, so you can-analyze library growth, indexing throughput, indexing failures, and spoken content alongside the rest+upload and indexing lifecycle, the transcriptions of your video and audio assets, and the tags on+your knowledge store video items, so you can analyze library growth, indexing throughput, indexing failures, and spoken content alongside the restof your data.## Prerequisites@@ -45,6 +45,10 @@ daily request quota. Enable it when you need per-video metadata.full refresh, and is disabled by default. Each row holds the transcription status plus its sentencesand speaker turns. Assets without a transcription are skipped.+`knowledge_store_item_tags` holds the tags on the video items of your knowledge stores. Each row is+one tag with its start and end time and its custom metadata. It makes one request per knowledge store+and one per video item, syncs by full refresh, and is disabled by default.+## Configuration<SourceParameters />diff --git a/products/warehouse_sources/backend/temporal/data_imports/sources/twelve_labs/twelve_labs.py b/products/warehouse_sources/backend/temporal/data_imports/sources/twelve_labs/twelve_labs.pyindex 1a85eae3d921..e158958e2ba5 100644--- a/products/warehouse_sources/backend/temporal/data_imports/sources/twelve_labs/twelve_labs.py+++ b/products/warehouse_sources/backend/temporal/data_imports/sources/twelve_labs/twelve_labs.py@@ -21,6 +21,7 @@ from products.warehouse_sources.backend.temporal.data_imports.sources.common.resAuthConfig,Endpoint,EndpointResource,+ ResponseAction,)from products.warehouse_sources.backend.temporal.data_imports.sources.common.resumable import ResumableSourceManagerfrom products.warehouse_sources.backend.temporal.data_imports.sources.common.source_helpers import validate_via_probe@@ -308,6 +309,84 @@ def _fan_out_resource(return next(resource for resource in resources if resource.name == config.name)+def _knowledge_store_item_tags_resource(+ config: TwelveLabsEndpointConfig,+ api_key: str,+ team_id: int,+ job_id: str,+) -> Resource:+ """Fan out over every knowledge store, then every video item in it, and list each item's tags.++ Tag rows already carry `knowledge_store_id` and `item_id`, so nothing is injected into them. The+ framework does not resume a two-level fan-out, so a retried run starts over and merge dedupes the+ re-pulled rows on the [knowledge_store_id, item_id, _id] key.+ """+ stores_name = "knowledge_stores_list"+ items_name = "knowledge_store_items_list"+ # Sort parents ascending on creation so page boundaries stay stable if new rows land mid-sync.+ parent_params: dict[str, Any] = {"page_limit": PAGE_LIMIT, "sort_by": "created_at", "sort_option": "asc"}+ # A store or item deleted between the parent list and the child request returns 404.+ skip_deleted_parent: list[ResponseAction] = [{"status_code": 404, "action": "ignore"}]++ def _add_store_id(row: dict[str, Any]) -> dict[str, Any]:+ row["knowledge_store_id"] = row.pop(make_parent_key_name(stores_name, "_id"))+ return row++ stores_resource: EndpointResource = {+ "name": stores_name,+ "endpoint": {+ "path": "/knowledge-stores",+ "params": parent_params,+ "paginator": _paginator(),+ "data_selector": "data",+ },+ }+ items_resource: EndpointResource = {+ "name": items_name,+ "include_from_parent": ["_id"],+ "data_map": _add_store_id,+ "endpoint": {+ "path": "/knowledge-stores/{knowledge_store_id}/items",+ "params": {+ "knowledge_store_id": {"type": "resolve", "resource": stores_name, "field": "_id"},+ **parent_params,+ },+ "paginator": _paginator(),+ "data_selector": "data",+ "response_actions": skip_deleted_parent,+ },+ }+ tags_resource: EndpointResource = {+ "name": config.name,+ "endpoint": {+ "path": config.path,+ "params": {+ "knowledge_store_id": {"type": "resolve", "resource": items_name, "field": "knowledge_store_id"},+ "item_id": {"type": "resolve", "resource": items_name, "field": "_id"},+ "page_limit": PAGE_LIMIT,+ },+ "paginator": _paginator(),+ "data_selector": "data",+ "response_actions": skip_deleted_parent,+ },+ }++ rest_config: RESTAPIConfig = {+ "client": {+ "base_url": TWELVE_LABS_BASE_URL,+ "headers": {"Accept": "application/json"},+ "auth": _auth_config(api_key),+ "allow_redirects": False,+ },+ "resources": [stores_resource, items_resource, tags_resource],+ }++ resources = {resource.name: resource for resource in rest_api_resources(rest_config, team_id, job_id, None)}+ # Only video items hold tags, so image items are dropped instead of paying a request each.+ resources[items_name].add_filter(lambda item: item.get("asset_type") == "video")+ return resources[config.name]++def twelve_labs_source(api_key: str,endpoint: str,@@ -320,7 +399,9 @@ def twelve_labs_source() -> SourceResponse:config = TWELVE_LABS_ENDPOINTS[endpoint]- if config.fan_out_over:+ if config.fan_out_over == "knowledge_store_items":+ resource = _knowledge_store_item_tags_resource(config, api_key, team_id, job_id)+ elif config.fan_out_over:resource = _fan_out_resource(config, _FAN_OUT_PARENTS[config.fan_out_over], api_key, team_id, job_id, resumable_source_manager)@@ -349,4 +430,6 @@ def twelve_labs_source(partition_format="week" if config.partition_key else None,partition_keys=[config.partition_key] if config.partition_key else None,column_hints=resource.column_hints,+ # The two-level item tags fan-out saves no checkpoint, so a shutdown cannot resume it.+ supports_resume=config.fan_out_over != "knowledge_store_items",)