Skip to content

:hedgehog: PostHog is the leading platform for building self-driving products. Our developer tools – AI observability, analytics, session replay, flags, experiments, error tracking, logs, and more – capture all the context agents need to diagnose problems, uncover opportunities, and ship fixes. Steer it all from Slack, web, desktop, or the MCP.

feat(data-warehouse): add pinecone namespaces and assistants tables (#125438)

Tom Owersc7d88f876a63
diff --git a/products/warehouse_sources/backend/temporal/data_imports/sources/COVERAGE_GAPS_APPENDIX.md b/products/warehouse_sources/backend/temporal/data_imports/sources/COVERAGE_GAPS_APPENDIX.md
index 609d3af7bc8e..c672e465b5b2 100644
--- a/products/warehouse_sources/backend/temporal/data_imports/sources/COVERAGE_GAPS_APPENDIX.md
+++ b/products/warehouse_sources/backend/temporal/data_imports/sources/COVERAGE_GAPS_APPENDIX.md
@@ -9414,14 +9414,14 @@ Note: Full vendor resource list read from the docs navigation plus each resource
## Pinecone — gaps
-Today (4): `backups`, `collections`, `indexes`, `restore_jobs`
+Today (7): `assistant_files`, `assistants`, `backups`, `collections`, `indexes`, `namespaces`, `restore_jobs`
Diffed against: <https://raw.githubusercontent.com/pinecone-io/pinecone-api/main/2026-07/db_data_2026-07.oas.yaml>
-- [ ] `Namespaces (GET /namespaces)` — measure record counts and stored data size by namespace within each index (high)
-- [ ] `Record IDs and Metadata (GET /vectors/list; GET /vectors/fetch)` — analyze stored record metadata and compare dataset composition across namespaces (medium)
-- [ ] `Assistants (GET /assistant/assistants)` — track knowledge assistants and compare their status and creation dates (medium)
-- [ ] `Assistant Files (GET /files/{assistant_name})` — analyze knowledge file inventory, processing status, and metadata by assistant (medium)
+- [x] `Namespaces (GET /namespaces)` — measure record counts and stored data size by namespace within each index (high). Added as `namespaces` (fan-out over ready serverless and BYOC `indexes`). The response carries record counts, not data size.
+- ~~`Record IDs and Metadata (GET /vectors/list; GET /vectors/fetch)` — analyze stored record metadata and compare dataset composition across namespaces (medium)~~ — skipped: it exports every record in every namespace, `fetch` always returns the vector values, and each sync would spend the customer's Pinecone read units with no server-side change filter
+- [x] `Assistants (GET /assistant/assistants)` — track knowledge assistants and compare their status and creation dates (medium). Added as `assistants`.
+- [x] `Assistant Files (GET /files/{assistant_name})` — analyze knowledge file inventory, processing status, and metadata by assistant (medium). Added as `assistant_files` (fan-out over ready `assistants`); `signed_url` is dropped.
Note: Parsed Pinecone's official 2026-07 database and assistant specifications. Namespace and record tables need requests to each index host. Record metadata requires listing IDs, then fetching their records; these list operations support serverless indexes. Assistant files use each assistant's host; excluded write operations, import plumbing, and authentication.
diff --git a/products/warehouse_sources/backend/temporal/data_imports/sources/pinecone/canonical_descriptions.py b/products/warehouse_sources/backend/temporal/data_imports/sources/pinecone/canonical_descriptions.py
index f07e71ded89f..39506685f1ea 100644
--- a/products/warehouse_sources/backend/temporal/data_imports/sources/pinecone/canonical_descriptions.py
+++ b/products/warehouse_sources/backend/temporal/data_imports/sources/pinecone/canonical_descriptions.py
@@ -56,4 +56,44 @@ CANONICAL_DESCRIPTIONS: CanonicalDescriptions = {
"percent_complete": "Completion indicator, reported as 100 for completed jobs.",
},
},
+ "namespaces": {
+ "description": "Namespaces in each ready serverless or BYOC index of the connected Pinecone project.",
+ "docs_url": "https://docs.pinecone.io/reference/api/2026-07/data-plane/listnamespaces",
+ "columns": {
+ "index_name": "Name of the index that contains the namespace.",
+ "name": "Namespace name, unique within the index.",
+ "record_count": "Total number of records in the namespace, as a decimal string.",
+ "schema": "Metadata fields that the namespace indexes for filtering.",
+ "indexed_fields": "All indexed metadata fields in the namespace.",
+ },
+ },
+ "assistants": {
+ "description": "Pinecone Assistants in the connected project.",
+ "docs_url": "https://docs.pinecone.io/reference/api/2026-07/assistant/list_assistants",
+ "columns": {
+ "name": "Assistant name, unique within the project.",
+ "instructions": "Guidance applied to all responses the assistant generates.",
+ "metadata": "Custom metadata attached to the assistant.",
+ "status": "Operational status of the assistant.",
+ "host": "Host where the assistant is deployed.",
+ "region": "Region where the assistant is deployed.",
+ "created_at": "Time the assistant was created.",
+ "updated_at": "Time the assistant was last updated.",
+ },
+ },
+ "assistant_files": {
+ "description": "Files uploaded to each ready Pinecone Assistant in the connected project.",
+ "docs_url": "https://docs.pinecone.io/reference/api/2026-07/assistant/list_files",
+ "columns": {
+ "assistant_name": "Name of the assistant that the file belongs to.",
+ "id": "Unique identifier of the file.",
+ "name": "Name of the uploaded file.",
+ "size": "File size in bytes.",
+ "metadata": "Custom metadata attached to the file.",
+ "created_on": "Time the file was uploaded.",
+ "updated_on": "Time the file was last updated.",
+ "status": "Processing state of the file.",
+ "multimodal": "Whether the file was processed as multimodal.",
+ },
+ },
}
diff --git a/products/warehouse_sources/backend/temporal/data_imports/sources/pinecone/pinecone.py b/products/warehouse_sources/backend/temporal/data_imports/sources/pinecone/pinecone.py
index da0e750b5e66..54aa00b8950d 100644
--- a/products/warehouse_sources/backend/temporal/data_imports/sources/pinecone/pinecone.py
+++ b/products/warehouse_sources/backend/temporal/data_imports/sources/pinecone/pinecone.py
@@ -1,5 +1,9 @@
-from collections.abc import Generator
-from typing import TYPE_CHECKING, cast
+from collections.abc import Generator, Iterator
+from typing import TYPE_CHECKING, Any, cast
+from urllib.parse import quote, urlsplit
+
+import structlog
+from requests.exceptions import HTTPError
from posthog.dataclasses import frozen
@@ -8,10 +12,22 @@ from products.warehouse_sources.backend.temporal.data_imports.sources.common.res
rest_api_resource,
)
from products.warehouse_sources.backend.temporal.data_imports.sources.common.rest_source.config_setup import create_auth
+from products.warehouse_sources.backend.temporal.data_imports.sources.common.rest_source.paginators import (
+ BasePaginator,
+ JSONResponseCursorPaginator,
+ SinglePagePaginator,
+)
from products.warehouse_sources.backend.temporal.data_imports.sources.common.rest_source.rest_client import RESTClient
from products.warehouse_sources.backend.temporal.data_imports.sources.common.schema import schema_for_resource
-from products.warehouse_sources.backend.temporal.data_imports.sources.common.typings import SourceResponse
-from products.warehouse_sources.backend.temporal.data_imports.sources.pinecone.settings import BASE_URL, ENDPOINTS
+from products.warehouse_sources.backend.temporal.data_imports.sources.common.typings import (
+ PartitionFormat,
+ SourceResponse,
+)
+from products.warehouse_sources.backend.temporal.data_imports.sources.pinecone.settings import (
+ BASE_URL,
+ ENDPOINTS,
+ PineconeEndpoint,
+)
if TYPE_CHECKING:
from products.warehouse_sources.backend.temporal.data_imports.sources.common.rest_source.typing import (
@@ -21,6 +37,13 @@ if TYPE_CHECKING:
)
from products.warehouse_sources.backend.temporal.data_imports.sources.common.resumable import ResumableSourceManager
+logger = structlog.get_logger(__name__)
+
+# Index and assistant hosts come from the control plane; the API key is only ever sent to Pinecone's domain.
+DATA_PLANE_HOST_SUFFIX = ".pinecone.io"
+# A file listing can carry a signed URL that grants anyone read access to the file contents.
+DROPPED_FILE_FIELDS = ("signed_url",)
+
@frozen
class PineconeResumeConfig:
@@ -47,7 +70,7 @@ def get_resource(endpoint: str) -> "EndpointResource":
"paginator": {
"type": "cursor",
"cursor_path": "pagination.next",
- "cursor_param": "paginationToken",
+ "cursor_param": settings.cursor_param,
}
if settings.paginated
else "single_page",
@@ -63,17 +86,106 @@ def get_resource(endpoint: str) -> "EndpointResource":
}
-def validate_credentials(api_key: str, api_version: str, endpoint: str) -> None:
- settings = schema_for_resource(ENDPOINTS, endpoint)
+def _client(api_key: str, api_version: str, base_url: str = BASE_URL) -> RESTClient:
config = get_client_config(api_key, api_version)
- client = RESTClient(
- base_url=config["base_url"],
+ return RESTClient(
+ base_url=base_url,
auth=create_auth(config["auth"]),
headers=config["headers"],
request_timeout=config["request_timeout"],
allowed_hosts=config["allowed_hosts"],
allow_redirects=False,
)
+
+
+def _paginator(settings: PineconeEndpoint) -> BasePaginator:
+ if settings.paginated:
+ return JSONResponseCursorPaginator(cursor_path="pagination.next", cursor_param=settings.cursor_param)
+ return SinglePagePaginator()
+
+
+def _data_plane_url(parent: str, row: dict[str, Any]) -> str | None:
+ if parent == "indexes":
+ deployment = row.get("deployment") or {}
+ status = row.get("status") or {}
+ # Pinecone lists namespaces only on serverless and BYOC indexes that are ready.
+ if deployment.get("deployment_type") == "pod" or not status.get("ready"):
+ return None
+ elif row.get("status") != "Ready":
+ return None
+ host = row.get("host")
+ if not isinstance(host, str) or not host:
+ return None
+ split = urlsplit(host if "://" in host else f"https://{host}")
+ hostname = split.hostname
+ if split.scheme != "https" or not hostname or not hostname.endswith(DATA_PLANE_HOST_SUFFIX):
+ logger.warning("pinecone_data_plane_host_skipped", parent=parent, name=row.get("name"))
+ return None
+ return f"https://{hostname}"
+
+
+def _child_row(settings: PineconeEndpoint, parent_name: str, row: dict[str, Any]) -> dict[str, Any]:
+ assert settings.parent_key is not None
+ return {
+ settings.parent_key: parent_name,
+ **{key: value for key, value in row.items() if key not in DROPPED_FILE_FIELDS},
+ }
+
+
+def fan_out_rows(
+ api_key: str,
+ api_version: str,
+ settings: PineconeEndpoint,
+ resumable_source_manager: "ResumableSourceManager[PineconeResumeConfig]",
+) -> Iterator[list[dict[str, Any]]]:
+ assert settings.parent is not None and settings.parent_key is not None
+ parent_settings = ENDPOINTS[settings.parent]
+ control_client = _client(api_key, api_version)
+ try:
+ for parents in control_client.paginate(
+ path=parent_settings.path,
+ params={"limit": 100} if parent_settings.paginated else {},
+ paginator=_paginator(parent_settings),
+ data_selector=parent_settings.data_selector,
+ data_selector_required=True,
+ ):
+ for parent in parents:
+ base_url = _data_plane_url(settings.parent, parent)
+ if base_url is None:
+ continue
+ parent_name = str(parent["name"])
+ path = settings.path.format(**{settings.parent_key: quote(parent_name, safe="")})
+ client = _client(api_key, api_version, base_url)
+ try:
+ # A data plane listing can omit an empty list, so a missing selector means zero rows.
+ for rows in client.paginate(
+ path=path,
+ params={"limit": 100},
+ paginator=_paginator(settings),
+ data_selector=settings.data_selector,
+ ):
+ if rows:
+ yield [_child_row(settings, parent_name, row) for row in rows]
+ else:
+ resumable_source_manager.safe_point()
+ except HTTPError as error:
+ # The parent was deleted after the parent listing.
+ if error.response is None or error.response.status_code != 404:
+ raise
+ resumable_source_manager.safe_point()
+ finally:
+ client.session.close()
+ resumable_source_manager.safe_point()
+ finally:
+ control_client.session.close()
+
+
+def validate_credentials(api_key: str, api_version: str, endpoint: str) -> None:
+ settings = schema_for_resource(ENDPOINTS, endpoint)
+ # A fan-out table needs a reachable parent listing; its data plane hosts are only known per parent.
+ if settings.parent is not None:
+ settings = ENDPOINTS[settings.parent]
+ client = _client(api_key, api_version)
pages = cast(
"Generator[list[object]]",
client.paginate(
@@ -99,6 +211,22 @@ def pinecone_source(
resumable_source_manager: "ResumableSourceManager[PineconeResumeConfig]",
) -> SourceResponse:
settings = schema_for_resource(ENDPOINTS, endpoint)
+ partition_count = 1 if settings.partition_mode == "md5" else None
+ partition_format: PartitionFormat | None = "month" if settings.partition_mode == "datetime" else None
+
+ if settings.parent is not None:
+ return SourceResponse(
+ name=endpoint,
+ items=lambda: fan_out_rows(api_key, api_version, settings, resumable_source_manager),
+ primary_keys=[settings.parent_key, settings.primary_key] if settings.parent_key else None,
+ partition_count=partition_count,
+ partition_keys=[settings.partition_key],
+ partition_mode=settings.partition_mode,
+ partition_format=partition_format,
+ sort_mode=None,
+ supports_resume=False,
+ )
+
config: RESTAPIConfig = {
"client": get_client_config(api_key, api_version),
"resources": [get_resource(endpoint)],
@@ -125,10 +253,10 @@ def pinecone_source(
name=endpoint,
items=lambda: resource,
primary_keys=[settings.primary_key],
- partition_count=1 if settings.partition_mode == "md5" else None,
+ partition_count=partition_count,
partition_keys=[settings.partition_key],
partition_mode=settings.partition_mode,
- partition_format="month" if settings.partition_mode == "datetime" else None,
+ partition_format=partition_format,
sort_mode=None,
supports_resume=settings.paginated,
)
diff --git a/products/warehouse_sources/backend/temporal/data_imports/sources/pinecone/settings.py b/products/warehouse_sources/backend/temporal/data_imports/sources/pinecone/settings.py
index 998d5f7cf3f9..a13b0ad2fc95 100644
--- a/products/warehouse_sources/backend/temporal/data_imports/sources/pinecone/settings.py
+++ b/products/warehouse_sources/backend/temporal/data_imports/sources/pinecone/settings.py
@@ -1,4 +1,4 @@
-from typing import TYPE_CHECKING
+from typing import TYPE_CHECKING, Literal
from posthog.dataclasses import frozen
@@ -18,6 +18,10 @@ class PineconeEndpoint:
partition_key: str
partition_mode: "PartitionMode"
paginated: bool = False
+ cursor_param: str = "paginationToken"
+ # Fan-out tables are read from each parent's own data plane host and keyed by the parent's name.
+ parent: Literal["indexes", "assistants"] | None = None
+ parent_key: str | None = None
ENDPOINTS: dict[str, PineconeEndpoint] = {
@@ -44,6 +48,36 @@ ENDPOINTS: dict[str, PineconeEndpoint] = {
partition_mode="md5",
paginated=True,
),
+ "namespaces": PineconeEndpoint(
+ path="/namespaces",
+ data_selector="namespaces",
+ primary_key="name",
+ partition_key="index_name",
+ partition_mode="md5",
+ paginated=True,
+ parent="indexes",
+ parent_key="index_name",
+ ),
+ "assistants": PineconeEndpoint(
+ path="/assistant/assistants",
+ data_selector="assistants",
+ primary_key="name",
+ partition_key="name",
+ partition_mode="md5",
+ paginated=True,
+ cursor_param="pagination_token",
+ ),
+ "assistant_files": PineconeEndpoint(
+ path="/files/{assistant_name}",
+ data_selector="files",
+ primary_key="id",
+ partition_key="assistant_name",
+ partition_mode="md5",
+ paginated=True,
+ cursor_param="pagination_token",
+ parent="assistants",
+ parent_key="assistant_name",
+ ),
}
INCREMENTAL_FIELDS: dict[str, list["IncrementalField"]] = {}
diff --git a/products/warehouse_sources/backend/temporal/data_imports/sources/pinecone/source.py b/products/warehouse_sources/backend/temporal/data_imports/sources/pinecone/source.py
index 4cffc33573e6..f1f0ade97a26 100644
--- a/products/warehouse_sources/backend/temporal/data_imports/sources/pinecone/source.py
+++ b/products/warehouse_sources/backend/temporal/data_imports/sources/pinecone/source.py
@@ -67,7 +67,7 @@ class PineconeSource(ResumableSource[PineconeSourceConfig, PineconeResumeConfig]
name=ExternalDataSourceType.PINECONE,
category=DataWarehouseSourceCategory.DATABASES,
label="Pinecone",
- caption="Import index, collection, backup, and restore job metadata from one Pinecone project. "
+ caption="Import index, namespace, collection, backup, restore job, and assistant metadata from one Pinecone project. "
"Use a project API key with permission to read the resources you select.",
docsUrl="https://posthog.com/docs/cdp/sources/pinecone",
iconPath="/static/services/pinecone.png",
diff --git a/products/warehouse_sources/backend/temporal/data_imports/sources/pinecone/tests/test_pinecone.py b/products/warehouse_sources/backend/temporal/data_imports/sources/pinecone/tests/test_pinecone.py
index 2f47b3749c0d..dd9ecbe47c15 100644
--- a/products/warehouse_sources/backend/temporal/data_imports/sources/pinecone/tests/test_pinecone.py
+++ b/products/warehouse_sources/backend/temporal/data_imports/sources/pinecone/tests/test_pinecone.py
@@ -44,6 +44,14 @@ def source_inputs(endpoint: str) -> SourceInputs:
)
+# endpoint -> (path, data selector, row key, cursor param)
+PAGINATED_ENDPOINTS = {
+ "backups": ("backups", "data", "backup_id", "paginationToken"),
+ "restore_jobs": ("restore-jobs", "data", "restore_job_id", "paginationToken"),
+ "assistants": ("assistant/assistants", "assistants", "name", "pagination_token"),
+}
+
+
def pipeline(endpoint: str) -> "SourceResponse":
source = PineconeSource()
inputs = source_inputs(endpoint)
@@ -127,6 +135,7 @@ class TestPineconeTransport(SimpleTestCase):
("restore_jobs", "absent", {}),
("restore_jobs", "null", {"pagination": None}),
("restore_jobs", "empty_token", {"pagination": {"next": ""}}),
+ ("assistants", "absent", {}),
]
)
def test_cursor_pages_checkpoint_after_yield_and_stop(
@@ -137,8 +146,7 @@ class TestPineconeTransport(SimpleTestCase):
inputs = source_inputs(endpoint)
source = PineconeSource()
manager = source.get_resumable_source_manager(inputs)
- path = endpoint.replace("_", "-")
- key = "backup_id" if endpoint == "backups" else "restore_job_id"
+ path, selector, key, cursor_param = PAGINATED_ENDPOINTS[endpoint]
with (
patch(
"products.warehouse_sources.backend.temporal.data_imports.sources.common.resumable.get_client",
@@ -149,14 +157,14 @@ class TestPineconeTransport(SimpleTestCase):
http.add(
responses.GET,
f"https://api.pinecone.io/{path}",
- json={"data": [{key: "first"}], "pagination": {"next": "next+/="}},
+ json={selector: [{key: "first"}], "pagination": {"next": "next+/="}},
)
http.add(
responses.GET,
f"https://api.pinecone.io/{path}",
- json={"data": [{key: "second"}], "pagination": {"next": "last-token"}},
+ json={selector: [{key: "second"}], "pagination": {"next": "last-token"}},
)
- http.add(responses.GET, f"https://api.pinecone.io/{path}", json={"data": [{key: "third"}], **terminal})
+ http.add(responses.GET, f"https://api.pinecone.io/{path}", json={selector: [{key: "third"}], **terminal})
response = source.source_for_pipeline(PineconeSourceConfig(api_key="fake-pinecone-key"), manager, inputs)
pages = iter(sync_items(response))
assert next(pages) == [{key: "first"}]
@@ -174,7 +182,7 @@ class TestPineconeTransport(SimpleTestCase):
manager.confirm()
assert not manager.has_staged_state()
assert redis.set.call_count == 2
- assert [parse_qs(urlsplit(call.request.url or "").query).get("paginationToken") for call in http.calls] == [
+ assert [parse_qs(urlsplit(call.request.url or "").query).get(cursor_param) for call in http.calls] == [
None,
["next+/="],
["last-token"],
@@ -204,6 +212,109 @@ class TestPineconeTransport(SimpleTestCase):
}
redis.set.assert_not_called()
+ def test_namespaces_fan_out_over_ready_data_plane_indexes(self) -> None:
+ def index(name: str, host: str, deployment_type: str = "managed", ready: bool = True) -> dict[str, object]:
+ return {
+ "name": name,
+ "host": host,
+ "deployment": {"deployment_type": deployment_type},
+ "status": {"ready": ready},
+ }
+
+ indexes = [
+ index("serverless", "serverless-abc.svc.aped-1.pinecone.io"),
+ index("byoc", "byoc-abc.svc.aped-1.pinecone.io", deployment_type="byoc"),
+ index("pod", "pod-abc.svc.us-east1-gcp.pinecone.io", deployment_type="pod"),
+ index("initializing", "initializing-abc.svc.aped-1.pinecone.io", ready=False),
+ index("off-domain", "index.example.com"),
+ ]
+ with responses.RequestsMock() as http:
+ http.add(responses.GET, "https://api.pinecone.io/indexes", json={"indexes": indexes})
+ http.add(
+ responses.GET,
+ "https://serverless-abc.svc.aped-1.pinecone.io/namespaces",
+ json={"namespaces": [{"name": "docs", "record_count": "3"}], "pagination": {"next": "ns-token"}},
+ )
+ http.add(
+ responses.GET,
+ "https://serverless-abc.svc.aped-1.pinecone.io/namespaces",
+ json={"namespaces": [{"name": "faq", "record_count": "5"}], "total_count": 2},
+ )
+ http.add(responses.GET, "https://byoc-abc.svc.aped-1.pinecone.io/namespaces", json={"total_count": 0})
+ response = pipeline("namespaces")
+ assert list(sync_items(response)) == [
+ [{"index_name": "serverless", "name": "docs", "record_count": "3"}],
+ [{"index_name": "serverless", "name": "faq", "record_count": "5"}],
+ ]
+ assert response.primary_keys == ["index_name", "name"]
+ assert [
+ (urlsplit(call.request.url or "").hostname, parse_qs(urlsplit(call.request.url or "").query))
+ for call in http.calls
+ ] == [
+ ("api.pinecone.io", {}),
+ ("serverless-abc.svc.aped-1.pinecone.io", {"limit": ["100"]}),
+ ("serverless-abc.svc.aped-1.pinecone.io", {"limit": ["100"], "paginationToken": ["ns-token"]}),
+ ("byoc-abc.svc.aped-1.pinecone.io", {"limit": ["100"]}),
+ ]
+ assert {call.request.headers["Api-Key"] for call in http.calls} == {"fake-pinecone-key"}
+
+ def test_assistant_files_fan_out_skips_deleted_assistants_and_signed_urls(self) -> None:
+ host = "https://prod-1-data.ke.pinecone.io"
+ with responses.RequestsMock() as http:
+ http.add(
+ responses.GET,
+ "https://api.pinecone.io/assistant/assistants",
+ json={
+ "assistants": [
+ {"name": "support", "status": "Ready", "host": host},
+ {"name": "deleted", "status": "Ready", "host": host},
+ ],
+ "pagination": {"next": "assistant-token"},
+ },
+ )
+ http.add(
+ responses.GET,
+ "https://api.pinecone.io/assistant/assistants",
+ json={"assistants": [{"name": "starting", "status": "Initializing", "host": host}]},
+ )
+ http.add(
+ responses.GET,
+ f"{host}/files/support",
+ json={
+ "files": [
+ {
+ "id": "file-1",
+ "name": "guide.pdf",
+ "status": "Available",
+ "signed_url": "https://storage.example.com/guide.pdf?signature=fake",
+ }
+ ],
+ "pagination": {"next": "file-token"},
+ },
+ )
+ http.add(
+ responses.GET,
+ f"{host}/files/support",
+ json={"files": [{"id": "file-2", "name": "faq.md", "status": "Processing"}]},
+ )
+ http.add(responses.GET, f"{host}/files/deleted", json={"error": {"code": "NOT_FOUND"}}, status=404)
+ response = pipeline("assistant_files")
+ assert list(sync_items(response)) == [
+ [{"assistant_name": "support", "id": "file-1", "name": "guide.pdf", "status": "Available"}],
+ [{"assistant_name": "support", "id": "file-2", "name": "faq.md", "status": "Processing"}],
+ ]
+ assert response.primary_keys == ["assistant_name", "id"]
+ assert [
+ (urlsplit(call.request.url or "").path, parse_qs(urlsplit(call.request.url or "").query))
+ for call in http.calls
+ ] == [
+ ("/assistant/assistants", {"limit": ["100"]}),
+ ("/files/support", {"limit": ["100"]}),
+ ("/files/support", {"limit": ["100"], "pagination_token": ["file-token"]}),
+ ("/files/deleted", {"limit": ["100"]}),
+ ("/assistant/assistants", {"limit": ["100"], "pagination_token": ["assistant-token"]}),
+ ]
+
@parameterized.expand(
[("indexes", "indexes"), ("collections", "collections"), ("backups", "data"), ("restore_jobs", "data")]
)
diff --git a/products/warehouse_sources/backend/temporal/data_imports/sources/pinecone/tests/test_pinecone_source.py b/products/warehouse_sources/backend/temporal/data_imports/sources/pinecone/tests/test_pinecone_source.py
index f3124947b580..b3d045dac2b7 100644
--- a/products/warehouse_sources/backend/temporal/data_imports/sources/pinecone/tests/test_pinecone_source.py
+++ b/products/warehouse_sources/backend/temporal/data_imports/sources/pinecone/tests/test_pinecone_source.py
@@ -45,6 +45,22 @@ class TestPineconeCredentials(SimpleTestCase):
assert request.url == f"https://api.pinecone.io{path}" + ("?limit=1" if schema_name else "")
assert len(http.calls) == 1
+ @parameterized.expand(
+ [
+ ("namespaces", "https://api.pinecone.io/indexes", {"indexes": []}),
+ ("assistant_files", "https://api.pinecone.io/assistant/assistants?limit=1", {"assistants": []}),
+ ]
+ )
+ def test_fan_out_table_validates_against_its_parent_listing(
+ self, schema_name: str, url: str, body: dict[str, object]
+ ) -> None:
+ with responses.RequestsMock() as http:
+ http.add(responses.GET, url.split("?")[0], json=body)
+ assert PineconeSource().validate_credentials(
+ PineconeSourceConfig(api_key="fake-pinecone-key"), team_id=1, schema_name=schema_name
+ ) == (True, None)
+ assert [call.request.url for call in http.calls] == [url]
+
def test_unknown_table_is_rejected_without_http(self) -> None:
with responses.RequestsMock() as http:
valid, error = PineconeSource().validate_credentials(