Skip to content

Graph-Native Infrastructure for Context and Accountable AI Systems

feat(ingest): add Looker ingestor (#1595)

Som Samantray9bea6abeac61
diff --git a/docs/docs.json b/docs/docs.json
index 2c5da4c3..8d0fda41 100644
--- a/docs/docs.json
+++ b/docs/docs.json
@@ -111,7 +111,8 @@
"integrations/salesforce",
"integrations/airflow",
"integrations/redshift",
- "integrations/powerbi"
+ "integrations/powerbi",
+ "integrations/looker"
]
},
{
diff --git a/docs/integrations/looker.md b/docs/integrations/looker.md
new file mode 100644
index 00000000..64fbb504
--- /dev/null
+++ b/docs/integrations/looker.md
@@ -0,0 +1,288 @@
+---
+title: "Looker Integration"
+description: "Ingest Looker Looks, Dashboards, LookML models, Folders, and Projects metadata into Semantica's KG pipeline."
+icon: "chart-line"
+---
+
+> Extract Looker metadata into Semantica with API client credentials, using the official `looker-sdk` and the validated instance endpoint.
+
+
+## Installation
+
+```bash
+# Install with Looker support
+pip install "semantica[ingest-looker]"
+
+# Or install the connector separately
+pip install looker-sdk>=24.0.0
+```
+
+`looker-sdk` is an optional dependency. A plain `pip install semantica` never pulls it in, and `import semantica.ingest` never loads it eagerly — the SDK is imported only when you first use `LookerConnector` or `LookerIngestor`. `LookerData` is importable without the SDK, so type annotations and data classes keep working in an SDK-free environment.
+
+<Note>
+This connector ingests Looker metadata only — Looks, Dashboards, LookML models (with their Explores), Folders, and Projects. Query result rows, scheduled plans, users, and groups are outside the current scope of this integration.
+</Note>
+
+
+## Basic Usage
+
+```python
+import os
+from semantica.ingest import LookerIngestor
+
+with LookerIngestor(
+ base_url=os.getenv("LOOKERSDK_BASE_URL"), # e.g. https://looker.example.com
+ client_id=os.getenv("LOOKERSDK_CLIENT_ID"),
+ client_secret=os.getenv("LOOKERSDK_CLIENT_SECRET"),
+) as ingestor:
+ looks = ingestor.ingest_looks()
+ print(f"Retrieved {looks.row_count} looks — columns: {looks.columns}")
+
+ documents = ingestor.export_as_documents(looks)
+```
+
+<Note>
+`LookerIngestor` opens one authenticated session on entry and closes it on exit, so every call inside the `with` block reuses the validated connection. Standalone calls (without `with`) open and close a transient connection per call.
+</Note>
+
+<Tip>
+Use environment variables (or a `.env` file with `python-dotenv`) to keep credentials out of source code. `LookerIngestor()` with no arguments reads from the `LOOKERSDK_*` environment variables automatically.
+</Tip>
+
+
+## Authentication
+
+<Tabs>
+ <Tab title="Client ID / Client Secret">
+ Looker API credentials are the `client_id` and `client_secret` pair that
+ Looker issues for API3 access (Admin → Users → API Keys). Pass them
+ explicitly, or let the connector read the `LOOKERSDK_CLIENT_ID` and
+ `LOOKERSDK_CLIENT_SECRET` environment variables:
+
+ ```python
+ import os
+ from semantica.ingest import LookerIngestor
+
+ ingestor = LookerIngestor(
+ base_url=os.getenv("LOOKERSDK_BASE_URL"), # must be https
+ client_id=os.getenv("LOOKERSDK_CLIENT_ID"),
+ client_secret=os.getenv("LOOKERSDK_CLIENT_SECRET"),
+ )
+ ```
+
+ Required environment variables:
+
+ ```bash
+ export LOOKERSDK_BASE_URL="https://looker.example.com"
+ export LOOKERSDK_CLIENT_ID="your-client-id"
+ export LOOKERSDK_CLIENT_SECRET="your-client-secret"
+ ```
+
+ The endpoint must use `https`. A non-`https` scheme (for example
+ `http://looker.internal`) is rejected with a `ValidationError` before any
+ client is constructed.
+ </Tab>
+ <Tab title="Existing looker.ini">
+ The connector also accepts a Looker SDK configuration file. Point
+ `config_file` at the file and, optionally, select a `section` inside it:
+
+ ```python
+ from semantica.ingest import LookerIngestor
+
+ ingestor = LookerIngestor(
+ config_file="looker.ini",
+ section="Looker",
+ )
+ ```
+
+ `config_file` defaults to `"looker.ini"`. Values already present in the
+ file are used when the matching constructor argument or environment
+ variable is absent.
+ </Tab>
+</Tabs>
+
+<Warning>
+`allow_private_ips=True` is required to connect to a private, loopback, or
+link-local endpoint, and it also relaxes the `https` requirement so a
+non-`https` scheme is accepted. Leave it off unless you are deliberately
+targeting an internal instance.
+</Warning>
+
+If an explicit `base_url` conflicts with `LOOKERSDK_BASE_URL` or the value in
+the config file, the connector raises a `ValidationError` rather than
+connecting to a host you did not validate.
+
+
+## Environment Variables
+
+All connection parameters have `LOOKERSDK_*` environment-variable fallbacks,
+resolved through the SDK's own `ApiSettings`. Explicit constructor values
+always take precedence.
+
+| Variable | Parameter | Default |
+|---|---|---|
+| `LOOKERSDK_BASE_URL` | `base_url` | — |
+| `LOOKERSDK_CLIENT_ID` | `client_id` | — |
+| `LOOKERSDK_CLIENT_SECRET` | `client_secret` | — |
+
+`config_file` and `section` are constructor parameters rather than environment
+variables:
+
+| Parameter | Description | Default |
+|---|---|---|
+| `config_file` | Path to an existing `looker.ini` file | `"looker.ini"` |
+| `section` | Section name inside `config_file` | — |
+| `allow_private_ips` | Allow private endpoints and non-`https` schemes | `False` |
+
+
+## Metadata Ingestion
+
+Each read returns a `LookerData` object with `data` (normalized metadata
+dictionaries), `row_count`, `columns`, `content_type`, `base_url`, `metadata`,
+and `ingested_at`.
+
+| Method | SDK read | `content_type` |
+|---|---|---|
+| `ingest_looks()` | `all_looks` | `"look"` |
+| `ingest_dashboards()` | `all_dashboards` | `"dashboard"` |
+| `ingest_lookml_models()` | `all_lookml_models` | `"lookml_model"` |
+| `ingest_folders()` | `all_folders` | `"folder"` |
+| `ingest_projects()` | `all_projects` | `"project"` |
+
+Reads use the SDK's `all_*` collection methods. Each method accepts optional
+keyword arguments that are forwarded to the SDK call — `transport_options` and
+`fields` for most reads, plus `limit`/`offset` for `ingest_lookml_models()`:
+
+```python
+looks = ingestor.ingest_looks(fields="id,title,description,model")
+models = ingestor.ingest_lookml_models(limit=100)
+```
+
+<Note>
+This is a metadata-only connector. Dashboards are read as `DashboardBase`
+records without tiles, so no query result rows are fetched or stored.
+</Note>
+
+### LookML models and Explores
+
+LookML models are returned with their Explores nested inside the parent model
+record. Explores carry no parent reference of their own, so the parent
+`project_name`/`name` pair is composed into each nested identifier:
+
+```python
+models = ingestor.ingest_lookml_models()
+
+for model in models.data:
+ print(f"{model['project_name']}.{model['name']}")
+ for explore in model["explores"]:
+ print(f" {explore['name']}")
+```
+
+
+## Export as Semantica Documents
+
+`export_as_documents(data)` converts a `LookerData` result into the
+`{"id", "text", "metadata"}` document shape that `GraphBuilder` consumes:
+
+```python
+documents = ingestor.export_as_documents(looks)
+
+print(f"Created {len(documents)} documents")
+# Each document:
+# {
+# "id": "looker:look:42",
+# "text": "Monthly Revenue Revenue by region orders jdoe",
+# "metadata": {
+# "source": "looker",
+# "content_type": "look",
+# "row_data": { ... allowlisted Look metadata ... },
+# "look_id": 42
+# }
+# }
+```
+
+Document `id` values are namespaced per content type so they stay unique
+across collections: `looker:look:<id>`, `looker:dashboard:<id>`,
+`looker:folder:<id>`, `looker:project:<id>`, and
+`looker:lookml_model:<project>.<name>` for models. Every document's
+`metadata["source"]` is `"looker"`, and `metadata["content_type"]` records
+which collection it came from. The full normalized record is preserved under
+`metadata["row_data"]`.
+
+LookML model documents additionally nest their Explores, each with its own
+namespaced id and parent model reference:
+
+```python
+# {
+# "id": "looker:lookml_model:ecommerce.orders",
+# "text": "Orders orders ecommerce Order Items order_items",
+# "metadata": {
+# "source": "looker",
+# "content_type": "lookml_model",
+# "row_data": { ... allowlisted LookML model metadata ... },
+# "project_name": "ecommerce",
+# "model_name": "orders",
+# "explores": [
+# {
+# "id": "looker:lookml_model:ecommerce.orders:explore:order_items",
+# "name": "order_items",
+# "label": "Order Items",
+# "group_label": "Order Items",
+# "description": None,
+# "hidden": False,
+# "parent_model": "ecommerce.orders"
+# }
+# ],
+# "explore_count": 1
+# }
+# }
+```
+
+Pass the documents directly to `GraphBuilder`:
+
+```python
+from semantica.kg import GraphBuilder
+
+builder = GraphBuilder()
+graph = builder.build(documents)
+print(f"Entities: {graph['metadata']['num_entities']}")
+```
+
+Exported documents are JSON-serializable: nested SDK model objects become
+plain dictionaries or lists, and `datetime` values become ISO 8601 strings.
+
+
+## Security
+
+The connector validates the user-supplied endpoint before any client is
+constructed, then binds the client to that exact validated `base_url` — if the
+SDK resolves a different host from the environment or config file, it fails
+closed instead of connecting.
+
+It also routes every outbound request through the same SSRF validation the
+repository's other connectors use, so the OAuth login and each metadata read
+are checked individually rather than only at construction. Connections are
+pinned to the addresses resolved during validation, so a DNS rebind between
+validation and dial cannot redirect them. Requests must use `https` unless
+`allow_private_ips` is explicitly enabled, TLS certificate verification is
+forced on (so `LOOKERSDK_VERIFY_SSL` or a `looker.ini` cannot downgrade it),
+redirects are not followed, and proxy environment variables are not trusted.
+The SDK's error-documentation lookup (a separate, unguarded `requests.get` it
+performs on a non-2xx response) is disabled so a failed request cannot open a
+second egress path.
+
+Record normalization projects every SDK object through an explicit allowlist of
+retained fields. Secret-adjacent fields such as `Project.git_password`,
+`Project.deploy_secret`, `Dashboard.password`, `Dashboard.pdt_password`, and
+`LookmlModel.device_token` are dropped, and any `user:password@` userinfo in
+`Project.git_remote_url` is scrubbed (a URL whose userinfo cannot be rewritten
+safely is redacted wholesale). Documents carry no `base_url` or credentials.
+
+
+## See Also
+
+- [Ingest Module](../reference/ingest) — Full `LookerIngestor` reference and all other ingestors.
+- [Snowflake Integration](/integrations/snowflake) — SQL data warehouse connector with similar metadata ingestion.
+- [Redshift Integration](/integrations/redshift) — Warehouse connector for tabular and query-based ingestion.
+- [Installation](../installation) — All optional dependency extras.
+- [Knowledge Graph](../reference/kg) — Build a knowledge graph from ingested Looker metadata.
diff --git a/pyproject.toml b/pyproject.toml
index b7383fe9..6024b1bd 100644
--- a/pyproject.toml
+++ b/pyproject.toml
@@ -128,6 +128,7 @@ db-arrow = ["pyarrow>=24.0.0"]
db-salesforce = ["simple-salesforce>=1.12.0"]
db-redshift = ["redshift-connector>=2.0.0"]
db-bigquery = ["google-cloud-bigquery>=3.0.0"]
+ingest-looker = ["looker-sdk>=24.0.0"]
ingest-parquet = ["pyarrow>=24.0.0"]
ingest-arrow = ["pyarrow>=24.0.0"]
ingest-sap = ["requests>=2.28.0"]
diff --git a/semantica/ingest/__init__.py b/semantica/ingest/__init__.py
index 93ad67a9..4440750a 100644
--- a/semantica/ingest/__init__.py
+++ b/semantica/ingest/__init__.py
@@ -273,6 +273,10 @@ _LAZY_EXPORTS: Dict[str, Tuple[str, str]] = {
"BigQueryIngestor": (".bigquery_ingestor", "BigQueryIngestor"),
"BigQueryData": (".bigquery_ingestor", "BigQueryData"),
"BigQueryConnector": (".bigquery_ingestor", "BigQueryConnector"),
+ # Looker ingestion
+ "LookerIngestor": (".looker_ingestor", "LookerIngestor"),
+ "LookerData": (".looker_ingestor", "LookerData"),
+ "LookerConnector": (".looker_ingestor", "LookerConnector"),
}
_OPTIONAL_DEPENDENCY_MESSAGES = {
@@ -328,6 +332,10 @@ _OPTIONAL_DEPENDENCY_MESSAGES = {
"BigQuery ingestion requires optional dependency 'google-cloud-bigquery'. "
"Install it with: pip install 'semantica[db-bigquery]'"
),
+ ".looker_ingestor": (
+ "Looker ingestion requires optional dependency 'looker-sdk'. "
+ "Install it with: pip install 'semantica[ingest-looker]'"
+ ),
}
@@ -354,6 +362,7 @@ def __getattr__(name: str) -> Any:
"lxml",
"redshift_connector",
"google",
+ "looker_sdk",
)
)
):
@@ -414,6 +423,15 @@ def __getattr__(name: str) -> Any:
if message:
raise ImportError(message)
+ if module_name == ".looker_ingestor" and name in {
+ "LookerIngestor",
+ "LookerConnector",
+ }:
+ if not getattr(module, "LOOKER_AVAILABLE", True):
+ message = _OPTIONAL_DEPENDENCY_MESSAGES.get(module_name)
+ if message:
+ raise ImportError(message)
+
value = getattr(module, attr_name)
globals()[name] = value
return value
@@ -520,6 +538,10 @@ __all__ = [
"BigQueryIngestor",
"BigQueryData",
"BigQueryConnector",
+ # Looker ingestion
+ "LookerIngestor",
+ "LookerData",
+ "LookerConnector",
# Registry and Methods
"MethodRegistry",
"method_registry",
diff --git a/semantica/ingest/looker_ingestor.py b/semantica/ingest/looker_ingestor.py
new file mode 100644
index 00000000..9d9661cc
--- /dev/null
+++ b/semantica/ingest/looker_ingestor.py
@@ -0,0 +1,1707 @@
+"""
+Looker Ingestion Module
+
+This module provides Looker metadata ingestion for the Semantica
+framework, enabling extraction of Looks, Dashboards, LookML models (with
+their Explores), Folders, and Projects via the official ``looker-sdk``
+4.0 API.
+
+Design notes
+------------
+Optional dependency
+ ``looker_sdk`` is imported inside a guard. The module always imports
+ cleanly; ``LookerData`` is usable without the SDK, and
+ :class:`LookerConnector` raises a clear ``ImportError`` naming the
+ ``ingest-looker`` extra at instantiation time.
+
+Validate the endpoint once and guard every request
+ ``looker_sdk.init40`` builds ``RequestsTransport`` internally, so the
+ repository's ``request_with_ssrf_guard`` hook cannot wrap the SDK's
+ calls. The connector resolves the effective ``base_url`` from a
+ constructed ``ApiSettings`` object, validates it, and passes that
+ *same* object to ``init40``. The binding mechanism is the
+ ``read_config`` override on that object, which the SDK consults for
+ every login. ``connect()`` then mounts a validating adapter on the
+ SDK session — the choke point every API request (including the lazy
+ OAuth login) passes through — so each outbound URL is checked with
+ ``validate_url_for_request`` before it is sent. The controls applied
+ are:
+
+ 1. ``https`` is required unless ``allow_private_ips`` is enabled;
+ 2. TLS verification is forced on, so ``LOOKERSDK_VERIFY_SSL`` or the
+ ini file cannot downgrade the validated connection;
+ 3. the SDK session's redirect cap is set to zero;
+ 4. proxy environment trust is disabled;
+ 5. every request URL is re-validated by the mounted guard adapter;
+ 6. connections are pinned to the addresses resolved during validation,
+ closing the DNS-rebinding window between validation and dial.
+
+ The guard covers the SDK's own session. The SDK's error-documentation
+ helper is a second, separate egress path (a bare ``requests.get`` with
+ no validation, timeout, redirect cap, or proxy opt-out), so
+ ``connect()`` also disables it explicitly; see
+ :meth:`LookerConnector._disable_error_doc_lookup`.
+
+ A post-construction check also compares the client's resolved
+ ``base_url`` with the validated one. The current SDK returns the same
+ ``ApiSettings`` object, so that check guards against future drift
+ rather than being a security control.
+
+Credentials are injected through the ``ApiSettings`` object rather than
+by mutating ``os.environ``, so no secret becomes process-global.
+
+Allowlist redaction
+ Every SDK record is projected through an explicit allowlist of
+ retained fields. ``Project.git_password``, ``Project.deploy_secret``,
+ ``Project.git_remote_url`` userinfo, ``Dashboard.password``,
+ ``Dashboard.pdt_password``, and ``LookmlModel.device_token`` can
+ therefore never reach :class:`LookerData`, an exported document, or a
+ log record.
+
+Main Classes:
+ - LookerConnector: Manages the ``looker_sdk`` client lifecycle
+ - LookerData: Dataclass representing ingested Looker metadata
+ - LookerIngestor: Orchestrates metadata reads and document export
+
+Optional Dependency:
+ Install via the ``ingest-looker`` extra::
+
+ pip install "semantica[ingest-looker]"
+
+Example Usage::
+
+ >>> import os
+ >>> from semantica.ingest.looker_ingestor import LookerIngestor
+ >>> with LookerIngestor(
+ ... base_url=os.getenv("LOOKERSDK_BASE_URL"),
+ ... client_id=os.getenv("LOOKERSDK_CLIENT_ID"),
+ ... client_secret=os.getenv("LOOKERSDK_CLIENT_SECRET"),
+ ... ) as ingestor:
+ ... looks = ingestor.ingest_looks()
+ ... documents = ingestor.export_as_documents(looks)
+
+Author: Semantica Contributors
+License: MIT
+"""
+
+from __future__ import annotations
+
+import enum
+import os
+from dataclasses import dataclass, field
+from datetime import date, datetime
+from typing import Any, Dict, List, Mapping, Optional, Sequence
+from urllib.parse import urlparse
+
+import requests
+
+from ..utils.exceptions import ProcessingError, ValidationError
+from ..utils.logging import get_logger
+from ..utils.progress_tracker import get_progress_tracker
+from .ssrf import (
+ _format_host_header,
+ _make_pinned_adapter,
+ _resolve_pinned_ips,
+ parse_bool,
+ validate_url_for_request,
+)
+
+# ---------------------------------------------------------------------------
+# Optional-dependency guard — mirrors the pattern in salesforce_ingestor.
+# The module always imports cleanly; the sentinel fires at instantiation
+# time inside LookerConnector.__init__ so that:
+# - ``from semantica.ingest.looker_ingestor import LookerData`` works
+# without the SDK, and
+# - ``LookerConnector(...)`` raises a clear ImportError when it is absent.
+# ---------------------------------------------------------------------------
+try:
+ import looker_sdk
+ from looker_sdk.rtl import api_settings
+
+ LOOKER_AVAILABLE = True
+except (ImportError, OSError):
+ looker_sdk = None # type: ignore[assignment]
+ api_settings = None # type: ignore[assignment]
+ LOOKER_AVAILABLE = False
+
+__all__ = [
+ "LookerData",
+ "LookerConnector",
+ "LookerIngestor",
+]
+
+_logger = get_logger("looker_ingestor")
+
+# Exact install hint required by R8.
+_MISSING_SDK_MESSAGE = (
+ "looker-sdk is required for the Looker connector. "
+ 'Install it with: pip install "semantica[ingest-looker]"'
+)
+
+# Environment prefix read by the SDK's own ApiSettings.
+_ENV_PREFIX = "LOOKERSDK"
+
+
+def _normalize_url(value: str) -> str:
+ """Return *value* with a trailing slash removed for comparison.
+
+ Args:
+ value: URL to normalize.
+
+ Returns:
+ The URL without a trailing slash.
+ """
+ return (value or "").strip().rstrip("/")
+
+
+# Returned in place of a remote URL that cannot be safely rewritten: it may
+# be ``user:password`` userinfo whose password contains a ``/``.
+_REDACTED_URL = "[redacted-looker-remote-url]"
+
+
+def _authority_is_host(authority: str) -> bool:
+ """Return True when *authority* is unambiguously a host, not userinfo.
+
+ A ``host:port`` authority is only distinguishable from
+ ``user:password`` by the port being numeric; a bracketed ``[...]``
+ literal is always an IPv6 host.
+ """
+ if authority.startswith("["):
+ return True
+ _, separator, port = authority.rpartition(":")
+ return bool(separator) and port.isdigit()
+
+
+def _scrub_url_userinfo(value: str) -> str:
+ """Strip ``user:password@`` userinfo from *value*.
+
+ ``Project.git_remote_url`` may embed credentials. Keep the host and
+ path so the metadata stays useful, but never the userinfo.
+
+ The deciding ``@`` is the one inside the authority section, not
+ necessarily the last ``@`` in the string: a path or query may contain
+ an ``@`` of its own (``https://user:[email protected]/path@file``). The
+ authority is therefore delimited first — an optional ``scheme://``,
+ then everything up to the first ``/``, ``?`` or ``#`` — and only that
+ slice is rewritten.
+
+ A value whose authority has no ``@`` but whose text contains one is
+ ambiguous: it may be ``https://user:pa/ss@host/repo.git``, where the
+ password contains the ``/`` that ends the authority. That cannot be
+ rewritten safely, so it fails closed to :data:`_REDACTED_URL` — unless
+ the authority is a plain host or a numeric ``host:port``
+ (``https://example.com/path@file``), where the ``@`` is ordinary
+ content.
+
+ Args:
+ value: Remote URL that may contain userinfo.
+
+ Returns:
+ The URL with any userinfo removed, or :data:`_REDACTED_URL` when the
+ safe rewrite is ambiguous.
+ """
+ if "@" not in value:
+ return value
+
+ scheme_index = value.find("://")
+ authority_start = 0 if scheme_index == -1 else scheme_index + 3
+ authority_end = len(value)
+ for separator in ("/", "?", "#"):
+ index = value.find(separator, authority_start)
+ if index != -1:
+ authority_end = min(authority_end, index)
+
+ authority = value[authority_start:authority_end]
+ if "@" in authority:
+ return (
+ value[:authority_start]
+ + authority.rpartition("@")[2]
+ + value[authority_end:]
+ )
+ # No '@' in the authority, yet the value has one. Ordinary content
+ # after a host is left alone; the ambiguous `user:password` shape fails
+ # closed.
+ if ":" not in authority or _authority_is_host(authority):
+ return value
+ return _REDACTED_URL
+
+
+# ---------------------------------------------------------------------------
+# LookerData
+# ---------------------------------------------------------------------------
+
+
+@dataclass
+class LookerData:
+ """Metadata records returned from a Looker API collection.
+
+ Attributes:
+ data: List of normalized metadata dictionaries.
+ row_count: Number of records in ``data``.
+ columns: Ordered list of field names present in ``data``.
+ content_type: One of ``look``, ``dashboard``, ``lookml_model``,
+ ``folder``, or ``project``.
+ base_url: Validated Looker endpoint the data was read from.
+ metadata: Arbitrary extra metadata (e.g. the SDK endpoint used).
+ ingested_at: Timestamp recorded when this object was created.
+ """
+
+ data: List[Dict[str, Any]]
+ row_count: int
+ columns: List[str]
+ content_type: str
+ base_url: Optional[str] = None
+ metadata: Dict[str, Any] = field(default_factory=dict)
+ ingested_at: datetime = field(default_factory=datetime.now)
+
+
+# ---------------------------------------------------------------------------
+# Field allowlists — explicit "retained fields" contract (R11)
+# ---------------------------------------------------------------------------
+# Only keys listed here survive normalization. Secret-adjacent SDK fields
+# (git_password, deploy_secret, password, pdt_password, device_token) are
+# deliberately absent, so they cannot reach LookerData, documents, or logs.
+
+# Values that reach the normalizer as plain mappings never pass through a
+# record allowlist, so secret-named keys are dropped explicitly as well.
+# Named arguments of LookerConnector: a config dict carrying any of these
+# must not be re-forwarded as a keyword or the call raises TypeError.
+_CONNECTOR_ARGUMENT_NAMES = frozenset(
+ {
+ "allow_private_ips",
+ "base_url",
+ "client_id",
+ "client_secret",
+ "config_file",
+ "section",
+ }
+)
+
+_SECRET_FIELD_NAMES = frozenset(
+ {
+ "deploy_secret",
+ "device_token",
+ "git_password",
+ "password",
+ "pdt_password",
+ }
+)
+
+# ``all_lookml_models()`` nests ``LookmlModelNavExplore`` records: the only
+# fields the navigation object carries are the ones below. ``view_name``,
+# ``joins``, and ``fields`` belong to ``LookmlModelExplore``, which only the
+# deferred ``lookml_model_explore`` endpoint returns and this connector never
+# calls.
+_EXPLORE_NAV_FIELDS = frozenset(
+ {
+ "description",
+ "group_label",
+ "hidden",
+ "label",
+ "name",
+ }
+)
+
+_NESTED_EXTRA_FIELDS = frozenset(
+ {
+ "certification_status",
+ "joins",
+ "condition",
+ "dependency_status",
+ "details",
+ "error_pos",
+ "field",
+ "field_error",
+ "git_branch",
+ "git_head",
+ "git_status",
+ "label",
+ "lookml_type",
+ "measure_types",
+ "message",
+ "project_id",
+ "user_attribute",
+ "value",
+ "workspace_id",
+ }
+)
+
+_RETAINED_FIELDS: Dict[str, frozenset] = {
+ "look": frozenset(
+ {
+ "can",
+ "certification_metadata",
+ "content_favorite_id",
+ "content_metadata_id",
+ "created_at",
+ "deleted",
+ "deleted_at",
+ "deleter_id",
+ "description",
+ "embed_url",
+ "excel_file_url",
+ "favorite_count",
+ "folder",
+ "folder_id",
+ "google_spreadsheet_formula",
+ "id",
+ "image_embed_url",
+ "is_owner_disabled",
+ "is_run_on_load",
+ "last_accessed_at",
+ "last_updater_id",
+ "last_viewed_at",
+ "model",
+ "public",
+ "public_slug",
+ "public_url",
+ "query_id",
+ "short_url",
+ "title",
+ "updated_at",
+ "usage_count",
+ "user_id",
+ "user_name",
+ "view_count",
+ }
+ ),
+ "dashboard": frozenset(
+ {
+ "alert_sync_with_dashboard_filter_enabled",
+ "appearance",
+ "background_color",
+ "can",
+ "certification_metadata",
+ "chat_enabled",
+ "content_favorite_id",
+ "content_metadata_id",
+ "created_at",
+ "crossfilter_enabled",
+ "deleted",
+ "deleted_at",
+ "deleter_id",
+ "description",
+ "download_settings",
+ "edit_uri",
+ "enable_viz_full_screen",
+ "favorite_count",
+ "filters_bar_collapsed",
+ "filters_location_top",
+ "folder",
+ "folder_id",
+ "hidden",
+ "id",
+ "is_owner_disabled",
+ "last_accessed_at",
+ "last_updater_id",
+ "last_updater_name",
+ "last_viewed_at",
+ "load_configuration",
+ "lookml_link_id",
+ "model",
+ "preferred_viewer",
+ "preserve_desktop_layout",
+ "query_timezone",
+ "readonly",
+ "refresh_interval",
+ "refresh_interval_to_i",
+ "show_filters_bar",
+ "show_title",
+ "slug",
+ "text_tile_text_color",
+ "tile_background_color",
+ "tile_text_color",
+ "title",
+ "title_color",
+ "updated_at",
+ "url",
+ "usage_count",
+ "user_id",
+ "user_name",
+ "view_count",
+ }
+ ),
+ # ``device_token`` is intentionally absent (R11).
+ "lookml_model": frozenset(
+ {
+ "allowed_db_connection_names",
+ "can",
+ "explores",
+ "has_content",
+ "label",
+ "name",
+ "project_name",
+ "unlimited_db_connections",
+ }
+ ),
+ "folder": frozenset(
+ {
+ "can",
+ "child_count",
+ "content_metadata_id",
+ "created_at",
+ "creator_id",
+ "dashboards",
+ "external_id",
+ "id",
+ "is_embed",
+ "is_embed_shared_root",
+ "is_embed_users_root",
+ "is_personal",
+ "is_personal_descendant",
+ "is_shared_root",
+ "is_users_root",
+ "looks",
+ "name",
+ "parent_id",
+ }
+ ),
+ # ``git_password``, ``deploy_secret``, and the password user-attribute
+ # fields are intentionally absent (R11). ``git_remote_url`` is kept
+ # but has its userinfo scrubbed during normalization.
+ "project": frozenset(
+ {
+ "allow_warnings",
+ "can",
+ "dependency_status",
+ "git_application_server_http_port",
+ "git_application_server_http_scheme",
+ "git_production_branch_name",
+ "git_remote_url",
+ "git_release_mgmt_enabled",
+ "git_service_name",
+ "git_username",
+ "git_username_user_attribute",
+ "has_production_counterpart",
+ "id",
+ "is_example",
+ "is_git_dev_locked",
+ "name",
+ "pull_request_mode",
+ "unset_deploy_secret",
+ "use_git_cookie_auth",
+ "uses_git",
+ "validation_required",
+ }
+ ),
+}
+
+# Nested SDK models (Explores, folders, and so on) are filtered against the
+# union of every top-level allowlist plus the nested-only field sets.
+_NESTED_RETAINED_FIELDS = frozenset(
+ set().union(
+ *(_RETAINED_FIELDS.values()),
+ _EXPLORE_NAV_FIELDS,
+ _NESTED_EXTRA_FIELDS,
+ )
+)
+
+# Human-readable field selection used to compose a document's ``text``.
+_TEXT_FIELDS: Dict[str, Sequence[str]] = {
+ "look": ("title", "description", "model", "user_name"),
+ "dashboard": ("title", "description", "model", "user_name"),
+ "lookml_model": ("label", "name", "project_name"),
+ "folder": ("name",),
+ "project": ("name", "git_username"),
+}
+
+# Flattened identifier exposed in document metadata, per content type.
+_ID_METADATA_KEYS: Dict[str, str] = {
+ "look": "look_id",
+ "dashboard": "dashboard_id",
+ "folder": "folder_id",
+ "project": "project_id",
+}
+
+
+# ---------------------------------------------------------------------------
+# SSRF guard adapter — mounted on the SDK session by connect()
+# ---------------------------------------------------------------------------
+
+
+class _SSRFGuardAdapter(requests.adapters.HTTPAdapter):
+ """Validate every request URL, then send it through the pinned delegate.
+
+ ``looker_sdk.init40`` builds its own transport inside the SDK, so the
+ repository's ``request_with_ssrf_guard`` helper cannot wrap the SDK's
+ calls. Mounting this adapter on the SDK session puts the same
+ per-request ``validate_url_for_request`` check on the single choke
+ point every request — the OAuth login and all five metadata reads —
+ passes through.
+
+ When addresses were resolved for pinning, ``send`` delegates to the
+ repository's pinned adapter so the connection still targets those
+ addresses and the hostname still supplies TLS identity. The
+ ``_semantica_pinned`` marker is exposed in that case.
+ """
+
+ def __init__(
+ self,
+ *,
+ allow_private_ips: bool,
+ pinned_adapter: Optional[requests.adapters.HTTPAdapter] = None,
+ ) -> None:
+ super().__init__()
+ self._allow_private_ips = allow_private_ips
+ self._pinned_adapter = pinned_adapter
+
+ @property
+ def _semantica_pinned(self) -> bool:
+ """Whether a pinning delegate is mounted beneath this adapter.
+
+ Derived from the delegate itself rather than cached, so the marker
+ read by :func:`~semantica.ingest.ssrf._apply_connection_pin` cannot
+ drift from the state it describes.
+ """
+ return self._pinned_adapter is not None
+
+ def send(self, request: Any, **kwargs: Any) -> Any:
+ """Validate *request* and send it through the pinned delegate."""
+ validate_url_for_request(request.url, allow_private_ips=self._allow_private_ips)
+ if self._pinned_adapter is not None:
+ return self._pinned_adapter.send(request, **kwargs)
+ return super().send(request, **kwargs)
+
+ def close(self) -> None:
+ """Close the pinned delegate, then this adapter."""
+ if self._pinned_adapter is not None:
+ self._pinned_adapter.close()
+ super().close()
+
+
+# ---------------------------------------------------------------------------
+# LookerConnector
+# ---------------------------------------------------------------------------
+
+
+class LookerConnector:
+ """Manages the ``looker_sdk`` client lifecycle.
+
+ Responsibilities:
+
+ * Resolves credentials from constructor arguments, falling back to
+ the ``LOOKERSDK_BASE_URL``, ``LOOKERSDK_CLIENT_ID``, and
+ ``LOOKERSDK_CLIENT_SECRET`` environment variables (and, via the
+ SDK's own ``ApiSettings``, a ``looker.ini`` file).
+ * Resolves the effective endpoint once, validates the scheme and the
+ SSRF contract, and binds the client to that exact value (KTD2/KTD4).
+ * Disables redirect following and proxy environment trust on the SDK
+ session and mounts a per-request SSRF guard that also pins the
+ connection; never mutates ``os.environ``.
+ * Exposes :meth:`connect`, :meth:`disconnect`, and
+ :meth:`test_connection`, plus the context-manager protocol.
+
+ Args:
+ base_url: Looker instance URL. Defaults to
+ ``LOOKERSDK_BASE_URL``.
+ client_id: Looker API client id. Defaults to
+ ``LOOKERSDK_CLIENT_ID``.
+ client_secret: Looker API client secret. Defaults to
+ ``LOOKERSDK_CLIENT_SECRET``.
+ config_file: Path to a ``looker.ini`` file. Defaults to
+ ``"looker.ini"``.
+ section: Section name inside *config_file*.
+ allow_private_ips: Opt into private/loopback/link-local endpoints
+ and non-``https`` schemes. Defaults to ``False``.
+ config: Optional extra configuration dict. ``allow_private_ips``
+ is read from here when present, mirroring the SAP connector.
+ **kwargs: Additional keyword arguments merged into ``config``.
+
+ Raises:
+ ImportError: When ``looker-sdk`` is not installed.
+ ValidationError: When credentials are missing, the endpoint fails
+ validation, or the environment conflicts with the explicit
+ ``base_url``.
+ """
+
+ def __init__(
+ self,
+ base_url: Optional[str] = None,
+ client_id: Optional[str] = None,
+ client_secret: Optional[str] = None,
+ config_file: Optional[str] = None,
+ section: Optional[str] = None,
+ allow_private_ips: bool = False,
+ config: Optional[Dict[str, Any]] = None,
+ **kwargs: Any,
+ ) -> None:
+ if not LOOKER_AVAILABLE:
+ raise ImportError(_MISSING_SDK_MESSAGE)
+
+ self.logger = _logger
+
+ self.config: Dict[str, Any] = dict(config or {})
+ self.config.update(kwargs)
+ self.allow_private_ips = parse_bool(
+ self.config.pop("allow_private_ips", allow_private_ips), default=False
+ )
+
+ # Credentials: explicit arguments win over environment variables.
+ self._explicit_base_url: Optional[str] = base_url
+ self.client_id: Optional[str] = client_id or os.getenv("LOOKERSDK_CLIENT_ID")
+ self._client_secret: Optional[str] = client_secret or os.getenv(
+ "LOOKERSDK_CLIENT_SECRET"
+ )
+ self.config_file: str = config_file or "looker.ini"
+ self.section: Optional[str] = section
+
+ self._client: Optional[Any] = None
+ # Endpoint validated at construction time, reused so the SSRF host
+ # check (which resolves DNS) runs at most once per connector.
+ self._validated_base_url: Optional[str] = None
+
+ # Fail fast on an explicit endpoint before any SDK object exists:
+ # a conflicting environment value must never silently win (AE2),
+ # and a bad scheme or blocked host must never reach the SDK.
+ self._validate_explicit_endpoint()
+
+ settings = self._build_settings()
+ raw_config = self._raw_read_config(settings)
+
+ # KTD4/AE2 — the SDK's ApiSettings lets the environment (and the
+ # ini file) override the constructor. Refuse to proceed rather
+ # than silently connecting somewhere the caller did not validate.
+ configured_base_url = str(raw_config.get("base_url") or "")
+ if self._explicit_base_url and configured_base_url:
+ if _normalize_url(configured_base_url) != _normalize_url(
+ self._explicit_base_url
+ ):
+ raise ValidationError(
+ "Looker base_url from the environment or config file "
+ f"({_scrub_url_userinfo(configured_base_url)!r}) conflicts "
+ "with the explicit 'base_url' "
+ f"({_scrub_url_userinfo(self._explicit_base_url)!r}); "
+ "refusing to connect rather than binding the client to an "
+ "unvalidated host."
+ )
+
+ effective_base_url = (
+ self._explicit_base_url
+ or configured_base_url
+ or str(getattr(settings, "base_url", "") or "")
+ )
+ effective_client_id = self.client_id or str(raw_config.get("client_id") or "")
+ effective_client_secret = self._client_secret or str(
+ raw_config.get("client_secret") or ""
+ )
+
+ self.base_url: str = effective_base_url
+ self.client_id = effective_client_id
+ self._client_secret = effective_client_secret
+
+ # ``looker_sdk`` re-reads settings for every login, so the merged
+ # values must live on ``read_config`` — assigning attributes alone
+ # would not reach the OAuth2 exchange.
+ settings.base_url = effective_base_url
+ # `verify_ssl` comes from LOOKERSDK_VERIFY_SSL / the ini file and would
+ # otherwise let an environment detail silently turn the enforced https
+ # requirement into an unverified connection carrying the client secret.
+ if not self.allow_private_ips:
+ settings.verify_ssl = True
+ base_read_config = settings.read_config
+
+ def _effective_read_config() -> Dict[str, Any]:
+ data: Dict[str, Any] = dict(base_read_config())
+ if effective_base_url:
+ data["base_url"] = effective_base_url
+ if effective_client_id:
+ data["client_id"] = effective_client_id
+ if effective_client_secret:
+ data["client_secret"] = effective_client_secret
+ if not self.allow_private_ips:
+ data["verify_ssl"] = True
+ return data
+
+ settings.read_config = _effective_read_config # type: ignore[method-assign]
+ self._settings = settings
+
+ self._validate_auth()
+ # An explicit endpoint was already validated above and always wins
+ # the resolution order, so it is never validated twice.
+ if self._validated_base_url is None:
+ self._validated_base_url = self._validate_endpoint(effective_base_url)
+
+ self.logger.debug(
+ "Looker connector initialised (base_url=%s, allow_private_ips=%s)",
+ self._validated_base_url,
+ self.allow_private_ips,
+ )
+
+ # ------------------------------------------------------------------
+ # Internal helpers
+ # ------------------------------------------------------------------
+
+ def _validate_explicit_endpoint(self) -> None:
+ """Validate an explicitly supplied ``base_url`` before SDK use.
+
+ Runs before any ``ApiSettings``/``init40`` work so a conflicting
+ ``LOOKERSDK_BASE_URL`` (AE2) or an unsafe explicit endpoint fails
+ immediately and without touching the SDK.
+
+ Raises:
+ ValidationError: When the environment conflicts with the
+ explicit value, or the explicit value fails scheme/SSRF
+ validation.
+ """
+ if not self._explicit_base_url:
+ return
+
+ env_base_url = os.getenv("LOOKERSDK_BASE_URL")
+ if env_base_url and _normalize_url(env_base_url) != _normalize_url(
+ self._explicit_base_url
+ ):
+ raise ValidationError(
+ "LOOKERSDK_BASE_URL "
+ f"({_scrub_url_userinfo(env_base_url)!r}) conflicts with the "
+ f"explicit 'base_url' "
+ f"({_scrub_url_userinfo(self._explicit_base_url)!r}); "
+ "refusing to connect rather than binding the client to an "
+ "unvalidated host."
+ )
+
+ self._validated_base_url = self._validate_endpoint(self._explicit_base_url)
+
+ def _build_settings(self) -> Any:
+ """Construct the SDK ``ApiSettings`` used to bind the client.
+
+ Returns:
+ A configured ``looker_sdk.rtl.api_settings.ApiSettings``.
+
+ Raises:
+ ValidationError: If an explicitly named config file is missing.
+ ProcessingError: If settings cannot be constructed.
+ """
+ try:
+ return api_settings.ApiSettings(
+ filename=self.config_file,
+ section=self.section,
+ env_prefix=_ENV_PREFIX,
+ )
+ except FileNotFoundError as exc:
+ raise ValidationError(
+ f"Looker config file not found: {self.config_file!r}."
+ ) from exc
+ except ValidationError:
+ raise
+ except Exception as exc: # noqa: BLE001 - translated with type only
+ self.logger.error(
+ "Failed to load Looker SDK settings: %s", type(exc).__name__
+ )
+ raise ProcessingError(
+ f"Failed to load Looker SDK settings: {type(exc).__name__}"
+ ) from None
+
+ @staticmethod
+ def _raw_read_config(settings: Any) -> Dict[str, Any]:
+ """Read the SDK's resolved config without applying our overrides."""
+ try:
+ data = settings.read_config()
+ except Exception as exc: # noqa: BLE001 - config read is best-effort here
+ # Without the config we cannot detect an endpoint conflict, so
+ # surface the cause instead of failing later as "missing
+ # credentials".
+ _logger.debug("Could not read Looker SDK config: %s", type(exc).__name__)
+ return {}
+ if isinstance(data, Mapping):
+ return dict(data)
+ return {}
+
+ def _validate_auth(self) -> None:
+ """Raise ``ValidationError`` when required credentials are absent.
+
+ Validates before any network call so failures are immediate and
+ never leak partial state.
+
+ Raises:
+ ValidationError: When ``base_url``, ``client_id``, or
+ ``client_secret`` is missing.
+ """
+ if not self.base_url:
+ raise ValidationError(
+ "Looker base_url is required. Provide via 'base_url' or the "
+ "LOOKERSDK_BASE_URL environment variable."
+ )
+ if not self.client_id:
+ raise ValidationError(
+ "Looker client_id is required. Provide via 'client_id' or the "
+ "LOOKERSDK_CLIENT_ID environment variable."
+ )
+ if not self._client_secret:
+ raise ValidationError(
+ "Looker client_secret is required. Provide via "
+ "'client_secret' or the LOOKERSDK_CLIENT_SECRET environment "
+ "variable."
+ )
+
+ def _validate_endpoint(self, base_url: str) -> str:
+ """Validate scheme and SSRF constraints for *base_url*.
+
+ Args:
+ base_url: Effective endpoint resolved from kwargs, environment,
+ or the ini file.
+
+ Returns:
+ The unchanged, validated endpoint.
+
+ Raises:
+ ValidationError: When the scheme is not ``https`` and
+ ``allow_private_ips`` is off, or when the host is blocked.
+ """
+ parsed = urlparse(base_url)
+ scheme = (parsed.scheme or "").lower()
+ if scheme != "https" and not self.allow_private_ips:
+ raise ValidationError(
+ "Looker base_url must use https unless allow_private_ips is "
+ f"enabled. Got scheme {parsed.scheme!r}."
+ )
+ if parsed.username or parsed.password:
+ raise ValidationError(
+ "Looker base_url must not embed userinfo credentials; pass "
+ "them as client_id/client_secret instead."
+ )
+ validate_url_for_request(base_url, allow_private_ips=self.allow_private_ips)
+ return base_url
+
+ @staticmethod
+ def _client_base_url(client: Any) -> Optional[str]:
+ """Return the ``base_url`` the constructed client is bound to."""
+ settings = getattr(getattr(client, "auth", None), "settings", None)
+ value = getattr(settings, "base_url", None)
+ return value if isinstance(value, str) else None
+
+ def _apply_transport_controls(self, client: Any) -> None:
+ """Apply the compensating controls to the SDK client's transport.
+
+ Disables redirect following and proxy trust on the SDK session,
+ forces TLS verification, pins connections to the validated
+ addresses, and disables the SDK's separate error-documentation
+ fetch.
+
+ Args:
+ client: A constructed ``Looker40SDK`` instance.
+
+ Raises:
+ ProcessingError: If the client exposes no transport session,
+ because the compensating controls could not be applied.
+ """
+ session = self._transport_session(client)
+ if session is None:
+ raise ProcessingError(
+ "Looker SDK client exposes no transport session; refusing to "
+ "use a client without redirect and proxy controls."
+ )
+ session.max_redirects = 0
+ session.trust_env = False
+ if session.verify is False:
+ raise ProcessingError(
+ "Looker SDK session has TLS verification disabled; refusing to "
+ "send credentials over an unverified connection."
+ )
+ session.verify = True
+ self._pin_session(session)
+ self._disable_error_doc_lookup()
+
+ @staticmethod
+ def _disable_error_doc_lookup() -> None:
+ """Stop the SDK's error helper from making its own network calls.
+
+ On a non-2xx API response the SDK builds an ``ErrorDocHelper`` and
+ looks up its error codes at ``ERROR_CODES_URL`` through a bare
+ module-level ``requests.get`` — a fresh session with
+ ``trust_env=True``, no timeout, no redirect cap, no SSRF validation,
+ and no pin. That is a second egress path beside the guarded SDK
+ session, so it is disabled here.
+
+ The override is deliberately installed on the SDK *class* rather
+ than the instance: the helper is constructed deep inside the SDK
+ with no injection point, and a per-call class override leaves the
+ unguarded, untimed, proxy-honouring request reachable. Overriding
+ ``ErrorDocHelper.get_index`` to return ``None`` makes ``lookup()``
+ take its ``KeyError`` branch and skip the second fetch entirely.
+ SDK internals are not a stable contract, so every access is
+ defensive; when it cannot be applied the fact is logged at debug.
+ """
+ try:
+ import looker_sdk.error as looker_error
+ except Exception as exc: # noqa: BLE001 - SDK internals may move
+ _logger.debug(
+ "Could not import looker_sdk.error to disable error-doc lookup: %s",
+ type(exc).__name__,
+ )
+ return
+
+ helper = getattr(looker_error, "ErrorDocHelper", None)
+ if helper is None or not callable(getattr(helper, "get_index", None)):
+ _logger.debug(
+ "looker_sdk.error exposes no ErrorDocHelper.get_index; "
+ "error-doc lookup could not be disabled."
+ )
+ return
+
+ def _no_error_doc_index(self: Any, url: Any = None) -> None:
+ """Return without fetching; the SDK uses its inline fallback."""
+ return None
+
+ try:
+ helper.get_index = _no_error_doc_index # type: ignore[method-assign]
+ except Exception as exc: # noqa: BLE001 - SDK internals may move
+ _logger.debug(
+ "Could not disable Looker error-doc lookup: %s",
+ type(exc).__name__,
+ )
+
+ def _pin_session(self, session: Any) -> None:
+ """Guard every request and pin connections to validated addresses.
+
+ The endpoint is validated once at construction, but the SDK resolves
+ the hostname again at request time, which reopens a DNS-rebinding
+ window and lets any later request (including the lazy OAuth login)
+ reach an address that was never validated. The mounted adapter
+ re-validates every request URL by calling
+ ``validate_url_for_request``; when addresses were resolved it then
+ delegates to a pinning adapter that connects only to the addresses
+ validated here, while the hostname still supplies TLS identity.
+
+ Raises:
+ ValidationError: If the endpoint no longer resolves to a
+ permitted address.
+ """
+ parsed = urlparse(self._validated_base_url)
+ pinned_ips = _resolve_pinned_ips(
+ self._validated_base_url, allow_private_ips=self.allow_private_ips
+ )
+ pinned_adapter: Optional[requests.adapters.HTTPAdapter] = None
+ if pinned_ips:
+ pinned_adapter = _make_pinned_adapter(pinned_ips, parsed.hostname or "")
+ pinned_adapter._semantica_pinned = True
+
+ # The guard is what gets mounted for both schemes; pinning rides
+ # beneath it so the validated addresses are still dialed.
+ adapter = _SSRFGuardAdapter(
+ allow_private_ips=self.allow_private_ips,
+ pinned_adapter=pinned_adapter,
+ )
+ session.mount("http://", adapter)
+ session.mount("https://", adapter)
+
+ if not pinned_ips:
+ return
+ session.headers["Host"] = _format_host_header(
+ parsed.hostname or "", parsed.port, parsed.scheme
+ )
+
+ @staticmethod
+ def _transport_session(client: Any) -> Optional[Any]:
+ """Return the SDK transport's HTTP session, if it exposes one."""
+ return getattr(getattr(client, "transport", None), "session", None)
+
+ def _close_client(self, client: Any) -> None:
+ """Best-effort close of a client that must not be used."""
+ session = self._transport_session(client)
+ if session is None:
+ return
+ try:
+ session.close()
+ except Exception: # noqa: BLE001 - cleanup must never raise
+ pass
+
+ # ------------------------------------------------------------------
+ # Public properties
+ # ------------------------------------------------------------------
+
+ @property
+ def client(self) -> Optional[Any]:
+ """The active ``Looker40SDK`` client, or ``None``."""
+ return self._client
+
+ @property
+ def connected(self) -> bool:
+ """Whether a client is currently connected."""
+ return self._client is not None
+
+ # ------------------------------------------------------------------
+ # Lifecycle
+ # ------------------------------------------------------------------
+
+ def connect(self) -> Any:
+ """Create and return a ``looker_sdk`` client.
+
+ If a client is already connected, returns it immediately without
+ constructing a new one.
+
+ Returns:
+ The ``Looker40SDK`` client bound to the validated endpoint.
+
+ Raises:
+ ValidationError: If the constructed client is bound to a
+ different endpoint than the validated one.
+ ProcessingError: If client construction fails for any reason.
+ Credentials are never included in the raised message.
+ """
+ if self._client is not None:
+ return self._client
+
+ try:
+ client = looker_sdk.init40(config_settings=self._settings)
+ except ValidationError:
+ raise
+ except Exception as exc: # noqa: BLE001 - translated with type only
+ self.logger.error(
+ "Failed to initialise Looker client: %s", type(exc).__name__
+ )
+ raise ProcessingError(
+ f"Failed to initialise Looker client: {type(exc).__name__}"
+ ) from None
+
+ resolved = self._client_base_url(client)
+ if _normalize_url(resolved or "") != _normalize_url(self._validated_base_url):
+ self._close_client(client)
+ self.logger.error(
+ "Looker client bound to an unexpected base_url; failing closed."
+ )
+ raise ValidationError(
+ "Looker client is bound to a different base_url than the "
+ "validated endpoint; refusing to use it."
+ )
+
+ try:
+ self._apply_transport_controls(client)
+ except Exception:
+ self._close_client(client)
+ raise
+
+ self._client = client
+ self.logger.info("Connected to Looker: base_url=%s", self._validated_base_url)
+ return self._client
+
+ def disconnect(self) -> None:
+ """Close the client session and release the reference.
+
+ Safe to call when already disconnected — subsequent calls are
+ no-ops, and cleanup failures are swallowed.
+ """
+ client, self._client = self._client, None
+ if client is None:
+ return
+ self._close_client(client)
+ self.logger.debug("Disconnected from Looker.")
+
+ def test_connection(self) -> bool:
+ """Verify connectivity with a cheap authenticated read.
+
+ Uses ``client.me()`` (a single ``GET /user``) to confirm the
+ session is live. The client is disconnected only when this call
+ opened it.
+
+ Returns:
+ ``True`` if the probe succeeds, ``False`` for any failure.
+ """
+ opened_here = self._client is None
+ try:
+ client = self.connect()
+ client.me()
+ return True
+ except Exception as exc: # noqa: BLE001 - bool result contract
+ self.logger.debug("Looker connection test failed: %s", type(exc).__name__)
+ return False
+ finally:
+ if opened_here:
+ self.disconnect()
+
+ def __enter__(self) -> "LookerConnector":
+ """Open the Looker client on context entry."""
+ self.connect()
+ return self
+
+ def __exit__(
+ self,
+ exc_type: Any,
+ exc_val: Any,
+ exc_tb: Any,
+ ) -> None:
+ """Close the Looker client on context exit."""
+ self.close()
+
+ def close(self) -> None:
+ """Disconnect from Looker and release the client."""
+ self.disconnect()
+
+
+# ---------------------------------------------------------------------------
+# LookerIngestor
+# ---------------------------------------------------------------------------
+
+
+class LookerIngestor:
+ """Looker metadata ingestor for the Semantica framework.
+
+ Wraps :class:`LookerConnector` and exposes the five metadata reads,
+ the deep normalizer with allowlist redaction, and the document
+ projection consumed by :class:`~semantica.kg.graph_builder.GraphBuilder`.
+
+ Args:
+ base_url: Looker instance URL; see :class:`LookerConnector`.
+ client_id: Looker API client id.
+ client_secret: Looker API client secret.
+ config_file: Path to a ``looker.ini`` file.
+ section: Section inside *config_file*.
+ allow_private_ips: Opt into private endpoints and non-``https``
+ schemes. Defaults to ``False``. When ``config`` (or ``**kwargs``)
+ carries ``allow_private_ips``, that value wins and is parsed with
+ :func:`~semantica.ingest.ssrf.parse_bool`, mirroring
+ :class:`LookerConnector`.
+ connector: An existing :class:`LookerConnector` to reuse.
+ config: Optional extra configuration dict forwarded to the
+ connector. A value here for any named connector argument that was
+ not supplied explicitly (``base_url``, ``client_id``,
+ ``client_secret``, ``config_file``, ``section``) is resolved from
+ the config, and ``allow_private_ips`` follows the config-wins
+ precedence described above.
+ **kwargs: Additional keyword arguments merged into ``config``.
+
+ Raises:
+ ImportError: When ``looker-sdk`` is not installed and no connector
+ is supplied.
+ ValidationError: When required credentials are incomplete.
+
+ Example::
+
+ from semantica.ingest.looker_ingestor import LookerIngestor
+
+ with LookerIngestor(
+ base_url="https://looker.example.com",
+ client_id="...",
+ client_secret="...",
+ ) as ingestor:
+ data = ingestor.ingest_looks()
+ documents = ingestor.export_as_documents(data)
+ """
+
+ def __init__(
+ self,
+ base_url: Optional[str] = None,
+ client_id: Optional[str] = None,
+ client_secret: Optional[str] = None,
+ config_file: Optional[str] = None,
+ section: Optional[str] = None,
+ allow_private_ips: bool = False,
+ connector: Optional[LookerConnector] = None,
+ config: Optional[Dict[str, Any]] = None,
+ **kwargs: Any,
+ ) -> None:
+ self.logger = _logger
+
+ self.config: Dict[str, Any] = dict(config or {})
+ self.config.update(kwargs)
+
+ # KTD3 — config wins when it carries the key, matching
+ # LookerConnector, and the value is parsed the same way so a string
+ # such as "0" cannot silently enable private access. The key stays
+ # excluded from the forwarded kwargs so the connector call cannot
+ # collide with the named argument.
+ resolved_allow_private_ips = parse_bool(
+ self.config.get("allow_private_ips", allow_private_ips),
+ default=False,
+ )
+
+ # The remaining named connector arguments are also excluded from the
+ # forwarded kwargs, so a config-supplied value must be resolved
+ # explicitly (an explicitly supplied argument always wins).
+ resolved_base_url = base_url or self.config.get("base_url")
+ resolved_client_id = client_id or self.config.get("client_id")
+ resolved_client_secret = client_secret or self.config.get("client_secret")
+ resolved_config_file = config_file or self.config.get("config_file")
+ resolved_section = section or self.config.get("section")
+
+ self.connector: LookerConnector = connector or LookerConnector(
+ base_url=resolved_base_url,
+ client_id=resolved_client_id,
+ client_secret=resolved_client_secret,
+ config_file=resolved_config_file,
+ section=resolved_section,
+ allow_private_ips=resolved_allow_private_ips,
+ **{
+ key: value
+ for key, value in self.config.items()
+ if key not in _CONNECTOR_ARGUMENT_NAMES
+ },
+ )
+
+ # Progress tracker — consistent with all other ingestors.
+ self.progress_tracker = get_progress_tracker()
+ if not self.progress_tracker.enabled:
+ self.progress_tracker.enabled = True
+
+ self.logger.debug("Looker ingestor initialised.")
+
+ # ------------------------------------------------------------------
+ # Context-manager support
+ # ------------------------------------------------------------------
+
+ def __enter__(self) -> "LookerIngestor":
+ """Open the Looker connection on context entry."""
+ self.connector.connect()
+ return self
+
+ def __exit__(
+ self,
+ exc_type: Any,
+ exc_val: Any,
+ exc_tb: Any,
+ ) -> None:
+ """Close the Looker connection on context exit."""
+ self.close()
+
+ def close(self) -> None:
+ """Disconnect from Looker and release the client."""
+ self.connector.disconnect()
+
+ # ------------------------------------------------------------------
+ # Metadata reads
+ # ------------------------------------------------------------------
+
+ def ingest_looks(self, **options: Any) -> LookerData:
+ """Fetch Look metadata via ``all_looks()``.
+
+ Args:
+ **options: Optional SDK ``transport_options`` / ``fields``.
+
+ Returns:
+ :class:`LookerData` with ``content_type == "look"``.
+
+ Raises:
+ ProcessingError: If the SDK read fails.
+ """
+ return self._ingest("all_looks", "look", **options)
+
+ def ingest_dashboards(self, **options: Any) -> LookerData:
+ """Fetch Dashboard metadata via ``all_dashboards()``.
+
+ The SDK returns ``DashboardBase`` objects without tiles, which
+ enforces the connector's metadata-only boundary.
+
+ Args:
+ **options: Optional SDK ``transport_options`` / ``fields``.
+
+ Returns:
+ :class:`LookerData` with ``content_type == "dashboard"``.
+
+ Raises:
+ ProcessingError: If the SDK read fails.
+ """
+ return self._ingest("all_dashboards", "dashboard", **options)
+
+ def ingest_lookml_models(self, **options: Any) -> LookerData:
+ """Fetch LookML model metadata via ``all_lookml_models()``.
+
+ Explores are nested inside their parent model record.
+
+ Args:
+ **options: Optional SDK ``limit`` / ``offset`` /
+ ``transport_options``.
+
+ Returns:
+ :class:`LookerData` with ``content_type == "lookml_model"``.
+
+ Raises:
+ ProcessingError: If the SDK read fails.
+ """
+ return self._ingest("all_lookml_models", "lookml_model", **options)
+
+ def ingest_folders(self, **options: Any) -> LookerData:
+ """Fetch Folder metadata via ``all_folders()``.
+
+ Args:
+ **options: Optional SDK ``transport_options`` / ``fields``.
+
+ Returns:
+ :class:`LookerData` with ``content_type == "folder"``.
+
+ Raises:
+ ProcessingError: If the SDK read fails.
+ """
+ return self._ingest("all_folders", "folder", **options)
+
+ def ingest_projects(self, **options: Any) -> LookerData:
+ """Fetch Project metadata via ``all_projects()``.
+
+ Secret-adjacent project fields are removed by the normalizer and
+ ``git_remote_url`` userinfo is scrubbed.
+
+ Args:
+ **options: Optional SDK ``transport_options`` / ``fields``.
+
+ Returns:
+ :class:`LookerData` with ``content_type == "project"``.
+
+ Raises:
+ ProcessingError: If the SDK read fails.
+ """
+ return self._ingest("all_projects", "project", **options)
+
+ # ------------------------------------------------------------------
+ # Internal read pipeline
+ # ------------------------------------------------------------------
+
+ def _ingest(self, sdk_method: str, content_type: str, **options: Any) -> LookerData:
+ """Run one SDK collection read and normalize its records.
+
+ The connection is opened transiently unless the caller already
+ connected the connector (for example via the context manager).
+
+ Args:
+ sdk_method: Name of the ``looker_sdk`` collection method.
+ content_type: Content type recorded on the result.
+ **options: Options forwarded to the SDK method.
+
+ Returns:
+ A populated :class:`LookerData`.
+
+ Raises:
+ ProcessingError: If the SDK read fails.
+ """
+ already_connected = bool(getattr(self.connector, "connected", False))
+ tracking_id = self.progress_tracker.start_tracking(
+ file=f"looker:{content_type}",
+ module="ingest",
+ submodule="LookerIngestor",
+ message=f"Reading Looker {content_type} metadata",
+ )
+
+ try:
+ client = self.connector.connect()
+ try:
+ method = getattr(client, sdk_method)
+ raw = method(**options) if options else method()
+ except ValidationError:
+ raise
+ except Exception as exc: # noqa: BLE001 - type name only
+ self.logger.error(
+ "Looker %s read failed: %s", sdk_method, type(exc).__name__
+ )
+ raise ProcessingError(
+ f"Looker {sdk_method} failed: {type(exc).__name__}"
+ ) from None
+
+ records = self._to_records(raw or [], content_type)
+ data = LookerData(
+ data=records,
+ row_count=len(records),
+ columns=self._derive_columns(records),
+ content_type=content_type,
+ base_url=getattr(self.connector, "base_url", None),
+ metadata={"content_type": content_type, "endpoint": sdk_method},
+ )
+ self.progress_tracker.stop_tracking(
+ tracking_id,
+ status="completed",
+ message=f"Read {data.row_count} {content_type} record(s)",
+ )
+ self.logger.debug(
+ "ingest_%s: normalized %d record(s)",
+ content_type,
+ data.row_count,
+ )
+ return data
+ except (ValidationError, ProcessingError):
+ self.progress_tracker.stop_tracking(
+ tracking_id,
+ status="failed",
+ message=f"Failed to read Looker {content_type} metadata",
+ )
+ raise
+ except Exception as exc: # noqa: BLE001 - finalized then re-raised
+ self.progress_tracker.stop_tracking(
+ tracking_id, status="failed", message=type(exc).__name__
+ )
+ self.logger.error(
+ "Failed to ingest Looker %s: %s", content_type, type(exc).__name__
+ )
+ raise
+ finally:
+ if not already_connected:
+ self.connector.disconnect()
+
+ @staticmethod
+ def _derive_columns(records: List[Dict[str, Any]]) -> List[str]:
+ """Return the ordered union of keys present across *records*."""
+ return list(dict.fromkeys(key for record in records for key in record))
+
+ # ------------------------------------------------------------------
+ # Deep normalizer
+ # ------------------------------------------------------------------
+
+ def _to_records(
+ self, raw: Sequence[Any], content_type: str
+ ) -> List[Dict[str, Any]]:
+ """Normalize SDK records into plain, JSON-serializable dicts.
+
+ Args:
+ raw: Sequence of SDK ``Model`` objects or mappings.
+ content_type: One of the allowlisted content types.
+
+ Returns:
+ A list of normalized dictionaries.
+
+ Raises:
+ ValidationError: If *content_type* has no allowlist.
+ ProcessingError: If a record is not a mapping-like object.
+ """
+ if content_type not in _RETAINED_FIELDS:
+ raise ValidationError(f"Unsupported Looker content type: {content_type!r}.")
+ allowed = _RETAINED_FIELDS[content_type]
+ return [self._normalize_record(record, allowed) for record in raw]
+
+ def _normalize_record(self, record: Any, allowed: frozenset) -> Dict[str, Any]:
+ """Project a single record through the retained-fields allowlist."""
+ data = self._record_mapping(record)
+ return {
+ key: self._normalize_field(key, value)
+ for key, value in data.items()
+ if key in allowed
+ }
+
+ def _normalize_field(self, key: str, value: Any) -> Any:
+ """Normalize one field, scrubbing any credential-bearing URL.
+
+ ``Project.git_remote_url`` may embed ``user:password@`` userinfo;
+ the field is useful metadata, the userinfo is not.
+ """
+ if key == "git_remote_url" and isinstance(value, str):
+ value = _scrub_url_userinfo(value)
+ return self._normalize_value(value)
+
+ def _normalize_value(self, value: Any) -> Any:
+ """Recursively convert *value* into JSON-serializable primitives.
+
+ SDK ``Model`` objects become filtered dictionaries, sequences
+ become lists, and dates/datetimes become ISO 8601 strings.
+
+ The recursion assumes an acyclic value graph: a Looker JSON
+ response is a tree, so there is no depth or cycle guard here. A
+ hand-built cyclic object would exhaust the interpreter stack and
+ raise ``RecursionError``.
+ """
+ if value is None or isinstance(value, (str, int, float, bool)):
+ return value
+ if isinstance(value, enum.Enum):
+ return value.value
+ if isinstance(value, date):
+ return value.isoformat()
+ if isinstance(value, Mapping):
+ # Route through _normalize_field so credential-bearing URLs are
+ # scrubbed, and drop secret-named keys, at every depth.
+ return {
+ key: self._normalize_field(key, item)
+ for key, item in value.items()
+ if key not in _SECRET_FIELD_NAMES
+ }
+ if isinstance(value, (list, tuple, set, frozenset)):
+ return [self._normalize_value(item) for item in value]
+ if self._is_model_like(value):
+ return self._normalize_record(value, _NESTED_RETAINED_FIELDS)
+ return str(value)
+
+ @staticmethod
+ def _is_model_like(value: Any) -> bool:
+ """Return True for SDK ``Model``-like objects (keys/items/iter)."""
+ return callable(getattr(value, "keys", None)) and callable(
+ getattr(value, "items", None)
+ )
+
+ @staticmethod
+ def _record_mapping(record: Any) -> Dict[str, Any]:
+ """Return a mapping view of *record*.
+
+ ``dict(record)`` is the SDK primitive, but nested values stay
+ ``Model`` objects, so callers recurse through
+ :meth:`_normalize_value`.
+
+ Raises:
+ ProcessingError: If *record* is not mapping-like. A record the
+ SDK returned in an unexpected shape is a data problem, not
+ misconfiguration, so it is chained from the conversion
+ failure when one occurred.
+ """
+ if isinstance(record, Mapping):
+ return dict(record)
+ last_error: Optional[BaseException] = None
+ items = getattr(record, "items", None)
+ if callable(items):
+ try:
+ return dict(items())
+ except Exception as exc: # noqa: BLE001 - fall through to keys()
+ last_error = exc
+ keys = getattr(record, "keys", None)
+ if callable(keys):
+ try:
+ return {str(key): record[key] for key in keys()}
+ except Exception as exc: # noqa: BLE001 - fall through to __dict__
+ last_error = exc
+ data = getattr(record, "__dict__", None)
+ if isinstance(data, dict):
+ return dict(data)
+ raise ProcessingError(
+ f"Unsupported Looker record type: {type(record).__name__}"
+ ) from last_error
+
+ # ------------------------------------------------------------------
+ # Document export
+ # ------------------------------------------------------------------
+
+ def export_as_documents(self, data: LookerData) -> List[Dict[str, Any]]:
+ """Convert :class:`LookerData` to Semantica documents.
+
+ Produces the ``{"id", "text", "metadata"}`` shape consumed by
+ :class:`~semantica.kg.graph_builder.GraphBuilder`. Document ids
+ are namespaced per content type so they stay unique across
+ collections, and LookML Explores are nested inside their parent
+ model document.
+
+ Metadata always carries ``source`` (``"looker"``) and
+ ``content_type``; it never carries ``base_url`` or credentials.
+
+ Args:
+ data: A :class:`LookerData` returned by one of the
+ ``ingest_*`` methods.
+
+ Returns:
+ A list of document dictionaries.
+
+ Raises:
+ ValidationError: If ``data.content_type`` is unsupported, or a
+ row is not a mapping. A row the SDK returned in an
+ unexpected shape cannot be projected, so the failure names
+ the offending type instead of leaking an ``AttributeError``.
+ """
+ if data.content_type not in _RETAINED_FIELDS:
+ raise ValidationError(
+ f"Unsupported Looker content type: {data.content_type!r}."
+ )
+
+ documents: List[Dict[str, Any]] = []
+ for index, row in enumerate(data.data):
+ if not isinstance(row, Mapping):
+ raise ValidationError(
+ "Looker document export requires mapping records; got "
+ f"{type(row).__name__} at index {index}."
+ )
+ document_id = self._document_id(row, data.content_type, index)
+ documents.append(
+ {
+ "id": document_id,
+ "text": self._document_text(row, data.content_type),
+ "metadata": self._document_metadata(
+ row, data.content_type, document_id
+ ),
+ }
+ )
+
+ self.logger.debug(
+ "export_as_documents: exported %d document(s)", len(documents)
+ )
+ return documents
+
+ @staticmethod
+ def _document_id(row: Dict[str, Any], content_type: str, index: int) -> str:
+ """Compose the namespaced document id for one record."""
+ if content_type == "lookml_model":
+ project = row.get("project_name") or "unknown"
+ name = row.get("name") or index
+ return f"looker:lookml_model:{project}.{name}"
+ identifier = row.get("id")
+ if identifier is None or identifier == "":
+ identifier = index
+ return f"looker:{content_type}:{identifier}"
+
+ def _document_metadata(
+ self, row: Dict[str, Any], content_type: str, document_id: str
+ ) -> Dict[str, Any]:
+ """Build the metadata block for one document."""
+ metadata: Dict[str, Any] = {
+ "source": "looker",
+ "content_type": content_type,
+ "row_data": row,
+ }
+
+ identifier_key = _ID_METADATA_KEYS.get(content_type)
+ if identifier_key is not None:
+ metadata[identifier_key] = row.get("id")
+
+ if content_type == "lookml_model":
+ project = row.get("project_name") or "unknown"
+ name = row.get("name") or "unknown"
+ metadata["project_name"] = row.get("project_name")
+ metadata["model_name"] = row.get("name")
+ metadata["explores"] = self._nest_explores(
+ row.get("explores"), project, name, document_id
+ )
+ metadata["explore_count"] = len(metadata["explores"])
+
+ return metadata
+
+ @staticmethod
+ def _nest_explores(
+ explores: Any,
+ project: str,
+ model: str,
+ document_id: str,
+ ) -> List[Dict[str, Any]]:
+ """Nest Explore summaries inside their parent model document.
+
+ Only the ``LookmlModelNavExplore`` fields exist for an
+ ``all_lookml_models()`` read; ``view_name``/``joins``/``fields``
+ belong to the deferred ``lookml_model_explore`` endpoint. Explores
+ carry no parent reference of their own, so the parent project/model
+ name is composed into every nested id.
+ """
+ nested: List[Dict[str, Any]] = []
+ if not isinstance(explores, (list, tuple)):
+ return nested
+ for explore in explores:
+ if not isinstance(explore, Mapping):
+ continue
+ explore_name = explore.get("name") or "unknown"
+ nested.append(
+ {
+ "id": f"{document_id}:explore:{explore_name}",
+ "name": explore_name,
+ "label": explore.get("label"),
+ "group_label": explore.get("group_label"),
+ "description": explore.get("description"),
+ "hidden": explore.get("hidden"),
+ "parent_model": f"{project}.{model}",
+ }
+ )
+ return nested
+
+ def _document_text(self, row: Dict[str, Any], content_type: str) -> str:
+ """Compose the searchable text for one document."""
+ parts: List[str] = []
+
+ for field_name in _TEXT_FIELDS.get(content_type, ()):
+ value = row.get(field_name)
+ if isinstance(value, str) and value:
+ parts.append(value)
+
+ folder = row.get("folder")
+ if isinstance(folder, Mapping):
+ folder_name = folder.get("name")
+ if isinstance(folder_name, str) and folder_name:
+ parts.append(folder_name)
+
+ if content_type == "lookml_model":
+ explores = row.get("explores")
+ # A malformed record (e.g. ``explores: 5``) must degrade to no
+ # explore text instead of raising ``TypeError``.
+ if not isinstance(explores, (list, tuple)):
+ return " ".join(parts)
+ for explore in explores:
+ if not isinstance(explore, Mapping):
+ continue
+ for field_name in ("label", "name", "description", "group_label"):
+ value = explore.get(field_name)
+ if isinstance(value, str) and value:
+ parts.append(value)
+
+ return " ".join(parts)
diff --git a/semantica/ingest/ssrf.py b/semantica/ingest/ssrf.py
index f97659ea..8fedcfba 100644
--- a/semantica/ingest/ssrf.py
+++ b/semantica/ingest/ssrf.py
@@ -37,6 +37,29 @@ _STRIP_BODY_ON_REDIRECT = frozenset({301, 302, 303})
_DEFAULT_PORTS = {"http": 80, "https": 443}
+def _format_host_header(hostname: str, port: Optional[int], scheme: str) -> str:
+ """Render the HTTP ``Host`` authority for *hostname*:*port*.
+
+ ``urlparse`` strips the brackets from an IPv6 literal, but a valid HTTP
+ authority requires them (RFC 3986 section 3.2.2), so the literal is
+ re-bracketed here. The port is omitted when it is absent or already the
+ scheme's default (``_DEFAULT_PORTS.get(scheme, 80)``).
+
+ Args:
+ hostname: Host without brackets (as returned by ``urlparse``).
+ port: Explicit port, or ``None`` when the URL carries none.
+ scheme: URL scheme used to look up the default port.
+
+ Returns:
+ The value for the ``Host`` header.
+ """
+ host = hostname
+ if ":" in host and not host.startswith("["):
+ host = f"[{host}]"
+ default_port = _DEFAULT_PORTS.get(scheme, 80)
+ return host if port in (None, default_port) else f"{host}:{port}"
+
+
def _should_strip_auth(old_url: str, new_url: str) -> bool:
"""Decide whether credentials must not follow a redirect.
@@ -422,12 +445,8 @@ def _apply_connection_pin(
parsed = urlparse(url)
if pinned_ips:
- port = parsed.port
- default_port = _DEFAULT_PORTS.get(parsed.scheme, 80)
- host_header = (
- parsed.hostname
- if port in (None, default_port)
- else f"{parsed.hostname}:{port}"
+ host_header = _format_host_header(
+ parsed.hostname or "", parsed.port, parsed.scheme
)
adapter = _make_pinned_adapter(pinned_ips, parsed.hostname or "")
adapter._semantica_pinned = True
diff --git a/tests/ingest/test_optional_imports.py b/tests/ingest/test_optional_imports.py
index 76da6cc4..c3446de5 100644
--- a/tests/ingest/test_optional_imports.py
+++ b/tests/ingest/test_optional_imports.py
@@ -212,6 +212,7 @@ from semantica.ingest import (
GitAnalyzer,
XMLIngestionData,
SalesforceData,
+ LookerData,
)
print(
CodeExtractor.__name__,
@@ -220,15 +221,16 @@ print(
GitAnalyzer.__name__,
XMLIngestionData.__name__,
SalesforceData.__name__,
+ LookerData.__name__,
)
""",
- ("git", "lxml", "simple_salesforce"),
+ ("git", "lxml", "simple_salesforce", "looker_sdk"),
)
assert result.returncode == 0, result.stderr
assert (
- "CodeExtractor CodeFile CommitInfo GitAnalyzer XMLIngestionData SalesforceData"
- in result.stdout
+ "CodeExtractor CodeFile CommitInfo GitAnalyzer XMLIngestionData "
+ "SalesforceData LookerData" in result.stdout
)
@@ -347,3 +349,78 @@ else:
assert result.returncode == 0, result.stderr
assert "ImportError" in result.stdout
assert "db-redshift" in result.stdout
+
+
+def test_looker_package_imports_without_sdk() -> None:
+ """``import semantica.ingest`` must not eagerly pull in looker_sdk."""
+ result = _run_python_with_blocked_modules(
+ """
+from semantica.ingest import FileIngestor, ingest_file
+print(FileIngestor.__name__, callable(ingest_file))
+""",
+ ("looker_sdk",),
+ )
+
+ assert result.returncode == 0, result.stderr
+ assert "FileIngestor True" in result.stdout
+
+
+def test_looker_data_importable_without_sdk() -> None:
+ """``LookerData`` is a plain dataclass — no SDK needed to import it."""
+ result = _run_python_with_blocked_modules(
+ """
+from semantica.ingest import LookerData
+print(LookerData.__name__)
+""",
+ ("looker_sdk",),
+ )
+
+ assert result.returncode == 0, result.stderr
+ assert "LookerData" in result.stdout
+
+
+def test_looker_ingestor_probe_fails_without_sdk() -> None:
+ """``from semantica.ingest import LookerIngestor`` must raise
+ ``ImportError`` when ``looker_sdk`` is absent."""
+ result = _run_python_with_blocked_modules(
+ """
+try:
+ from semantica.ingest import LookerIngestor
+ has_looker = True
+except ImportError:
+ has_looker = False
+
+assert not has_looker, (
+ "Expected LookerIngestor import to fail without looker-sdk"
+)
+print("LookerIngestor probe passed")
+""",
+ ("looker_sdk",),
+ )
+
+ assert result.returncode == 0, result.stderr
+ assert "LookerIngestor probe passed" in result.stdout
+
+
+def test_looker_connector_reports_missing_dep_with_install_hint() -> None:
+ """Constructing ``LookerConnector`` without the SDK must raise
+ ``ImportError`` with a message that names the ``ingest-looker`` extra."""
+ result = _run_python_with_blocked_modules(
+ """
+from semantica.ingest.looker_ingestor import LookerConnector
+
+try:
+ LookerConnector()
+except ImportError as exc:
+ print(type(exc).__name__, exc)
+else:
+ raise SystemExit(
+ "expected LookerConnector() to fail without looker-sdk"
+ )
+""",
+ ("looker_sdk",),
+ )
+
+ assert result.returncode == 0, result.stderr
+ assert "ImportError" in result.stdout
+ assert "ingest-looker" in result.stdout
diff --git a/tests/ingest/test_ssrf_protection.py b/tests/ingest/test_ssrf_protection.py
index 8d6b9116..9d8c7f6b 100644
--- a/tests/ingest/test_ssrf_protection.py
+++ b/tests/ingest/test_ssrf_protection.py
@@ -3,10 +3,13 @@
from unittest.mock import MagicMock, patch
import pytest
+import requests
import urllib3.connectionpool as pool
from semantica.ingest.api_ingestor import RESTIngestor
from semantica.ingest.ssrf import (
+ _apply_connection_pin,
+ _format_host_header,
parse_bool,
request_with_ssrf_guard,
validate_url_for_request,
@@ -15,6 +18,55 @@ from semantica.ingest.web_ingestor import SitemapCrawler, WebIngestor
from semantica.utils.exceptions import ValidationError
+class TestFormatHostHeader:
+ """The shared authority formatter brackets IPv6 and elides default ports."""
+
+ def test_ipv6_literal_without_explicit_port(self):
+ assert _format_host_header("2001:db8::1", None, "https") == "[2001:db8::1]"
+ assert _format_host_header("2001:db8::1", 443, "https") == "[2001:db8::1]"
+
+ def test_ipv6_literal_with_explicit_non_default_port(self):
+ assert _format_host_header("2001:db8::1", 8443, "https") == "[2001:db8::1]:8443"
+ assert _format_host_header("2001:db8::1", 8080, "http") == "[2001:db8::1]:8080"
+
+ def test_ipv4_literal_is_unchanged(self):
+ assert _format_host_header("93.184.216.34", None, "https") == "93.184.216.34"
+ assert _format_host_header("93.184.216.34", 443, "https") == "93.184.216.34"
+ assert (
+ _format_host_header("93.184.216.34", 8443, "https") == "93.184.216.34:8443"
+ )
+
+ def test_unknown_scheme_falls_back_to_port_80(self):
+ assert _format_host_header("example.com", 80, "ftp") == "example.com"
+ assert _format_host_header("example.com", 8080, "ftp") == "example.com:8080"
+
+
+class TestApplyConnectionPinHostHeader:
+ """The shared guarded path emits a valid authority for IPv6 targets."""
+
+ @pytest.mark.parametrize(
+ ("url", "expected_host"),
+ [
+ ("https://[2001:db8::1]/x", "[2001:db8::1]"),
+ ("https://[2001:db8::1]:8443/x", "[2001:db8::1]:8443"),
+ ],
+ )
+ def test_ipv6_authority_is_bracketed(self, url, expected_host):
+ session = requests.Session()
+
+ _apply_connection_pin(
+ session,
+ url,
+ ["2001:db8::1"],
+ requests.adapters.HTTPAdapter(),
+ requests.adapters.HTTPAdapter(),
+ False,
+ None,
+ )
+
+ assert session.headers["Host"] == expected_host
+
+
class TestParseBool:
def test_bool_passthrough(self):
assert parse_bool(True) is True
diff --git a/tests/test_looker_ingestor.py b/tests/test_looker_ingestor.py
new file mode 100644
index 00000000..61ffa327
--- /dev/null
+++ b/tests/test_looker_ingestor.py
@@ -0,0 +1,2070 @@
+"""
+Unit tests for LookerConnector and LookerIngestor.
+
+All ``looker_sdk`` calls are mocked — no live Looker instance is contacted
+and no network I/O occurs.
+
+Test structure mirrors ``tests/test_redshift_ingestor.py`` and
+``tests/test_salesforce_ingestor.py``:
+
+ - an autouse fixture injects a ``looker_sdk`` stub into ``sys.modules``
+ when the real SDK is absent, so tests that reference the symbol can
+ still run; the module-level optional-dependency guard has already run
+ at import time, so this fixture cannot change ``LOOKER_AVAILABLE``;
+ - an autouse fixture clears ``LOOKERSDK_*`` variables so a developer's
+ live configuration can never leak into a test;
+ - tests that need the real ``ApiSettings``/``init40`` implementation are
+ skipped when ``looker-sdk`` is not installed; the import guard, the
+ normalizer, the redaction contract, and the document projection are
+ covered without it.
+
+SSRF assertions use literal hosts (``localhost``, ``127.0.0.1``) only.
+The DNS-stubbing autouse fixture lives in ``tests/ingest/conftest.py``,
+not at the repository root, so no test here may require DNS resolution.
+"""
+
+from __future__ import annotations
+
+import json
+import os
+import unittest
+from datetime import datetime, timezone
+from unittest.mock import MagicMock, Mock, call, patch
+
+import pytest
+import requests
+from requests.adapters import BaseAdapter
+from requests.exceptions import TooManyRedirects
+
+# ---------------------------------------------------------------------------
+# Optional SDK detection
+# ---------------------------------------------------------------------------
+try:
+ import looker_sdk as _looker_sdk # noqa: F401
+
+ LOOKER_SDK_AVAILABLE = True
+except ImportError:
+ LOOKER_SDK_AVAILABLE = False
+
+requires_looker_sdk = pytest.mark.skipif(
+ not LOOKER_SDK_AVAILABLE,
+ reason="looker-sdk is required for this test",
+)
+
+# A literal public IP is used instead of a hostname: the SSRF validator
+# performs real DNS lookups, and the DNS-stubbing fixture lives only in
+# tests/ingest/conftest.py, not at the repository root.
+PUBLIC_BASE_URL = "https://93.184.216.34"
+PRIVATE_BASE_URL = "http://127.0.0.1:19999"
+CLIENT_ID = "test-client-id"
+CLIENT_SECRET = "test-client-secret"
+ACCESS_TOKEN = "test-access-token-do-not-log"
+
+# Literal secret values that must never survive normalization or logging.
+SECRET_FIELD_VALUES = {
+ "git_password": "git-password-do-not-log",
+ "deploy_secret": "deploy-secret-do-not-log",
+ "password": "dashboard-password-do-not-log",
+ "pdt_password": "pdt-password-do-not-log",
+ "device_token": ACCESS_TOKEN,
+}
+
+# Logger names whose records must never carry a credential or token.
+CONNECTOR_LOGGER = "semantica.looker_ingestor"
+TRANSPORT_LOGGER = "looker_sdk.rtl.requests_transport"
+
+_LOOKER_ENV_KEYS = (
+ "LOOKERSDK_BASE_URL",
+ "LOOKERSDK_CLIENT_ID",
+ "LOOKERSDK_CLIENT_SECRET",
+ "LOOKERSDK_VERIFY_SSL",
+ "LOOKERSDK_TIMEOUT",
+)
+
+
+# ---------------------------------------------------------------------------
+# autouse fixtures
+# ---------------------------------------------------------------------------
+
+
[email protected](autouse=True)
+def _mock_looker_sdk_if_needed():
+ """Expose a minimal ``looker_sdk`` stub when the SDK is not installed.
+
+ The stub is installed in ``sys.modules`` for the duration of each test,
+ but ``semantica.ingest.looker_ingestor`` has already evaluated its
+ module-level import guard by then, so this fixture does **not** exercise
+ that guard — it only keeps ``looker_sdk`` importable for tests that
+ reference the module object directly.
+ """
+ if not LOOKER_SDK_AVAILABLE:
+ stub = MagicMock()
+ stub.error.SDKError = type("SDKError", (Exception,), {})
+
+ with patch.dict("sys.modules", {"looker_sdk": stub}):
+ yield
+ else:
+ yield
+
+
[email protected](autouse=True)
+def _clean_looker_env(monkeypatch):
+ """Remove inherited ``LOOKERSDK_*`` variables before every test."""
+ for key in _LOOKER_ENV_KEYS:
+ monkeypatch.delenv(key, raising=False)
+
+
+# ---------------------------------------------------------------------------
+# Helpers
+# ---------------------------------------------------------------------------
+
+
+def _make_mock_client(base_url: str = PUBLIC_BASE_URL) -> MagicMock:
+ """Return a mock that behaves like a ``Looker40SDK`` instance.
+
+ ``connect()`` asserts that the client's resolved ``base_url`` matches
+ the validated endpoint, and applies the redirect/proxy transport
+ controls to ``client.transport.session``, so the mock must expose
+ both paths with realistic starting values.
+ """
+ client = MagicMock()
+ client.auth.settings.base_url = base_url
+ client.transport.session.max_redirects = 5
+ client.transport.session.trust_env = True
+ return client
+
+
+class _FakeConnector:
+ """Minimal connector double for ingestor tests (no SDK required)."""
+
+ def __init__(self, client, base_url: str = PUBLIC_BASE_URL):
+ self.client = client
+ self.base_url = base_url
+ self.connected = False
+ self.connect_calls = 0
+ self.disconnect_calls = 0
+
+ def connect(self):
+ self.connect_calls += 1
+ self.connected = True
+ return self.client
+
+ def disconnect(self):
+ self.disconnect_calls += 1
+ self.connected = False
+
+
+def _make_ingestor(client) -> "object":
+ """Return a LookerIngestor wired to a fake connector."""
+ from semantica.ingest.looker_ingestor import LookerIngestor
+
+ return LookerIngestor(connector=_FakeConnector(client))
+
+
+# ---------------------------------------------------------------------------
+# TestLookerConnectorDataContract — LookerData is SDK-independent
+# ---------------------------------------------------------------------------
+
+
+class TestLookerConnectorDataContract:
+ """``LookerData`` must import and construct without ``looker_sdk``."""
+
+ def test_importable_without_sdk(self):
+ from semantica.ingest.looker_ingestor import LookerData
+
+ with patch("semantica.ingest.looker_ingestor.LOOKER_AVAILABLE", False):
+ data = LookerData(data=[], row_count=0, columns=[], content_type="look")
+
+ assert data.data == []
+ assert data.row_count == 0
+ assert data.columns == []
+ assert data.content_type == "look"
+ assert data.base_url is None
+ assert data.metadata == {}
+ assert isinstance(data.ingested_at, datetime)
+
+ def test_full_field_contract(self):
+ from semantica.ingest.looker_ingestor import LookerData
+
+ rows = [{"id": 1, "title": "Orders"}]
+ data = LookerData(
+ data=rows,
+ row_count=1,
+ columns=["id", "title"],
+ content_type="look",
+ base_url=PUBLIC_BASE_URL,
+ metadata={"content_type": "look"},
+ )
+
+ assert data.data == rows
+ assert data.row_count == 1
+ assert data.base_url == PUBLIC_BASE_URL
+ assert data.metadata == {"content_type": "look"}
+
+ def test_module_all_and_logger_are_module_level(self):
+ from semantica.ingest import looker_ingestor as mod
+
+ assert mod.__all__ == [
+ "LookerData",
+ "LookerConnector",
+ "LookerIngestor",
+ ]
+ assert mod._logger is not None
+
+
+# ---------------------------------------------------------------------------
+# TestLookerConnectorImportGuard
+# ---------------------------------------------------------------------------
+
+
+class TestLookerConnectorImportGuard:
+ """R8 — the optional-dependency guard and its install hint."""
+
+ def test_missing_sdk_raises_import_error_with_exact_hint(self):
+ from semantica.ingest.looker_ingestor import LookerConnector
+
+ with patch("semantica.ingest.looker_ingestor.LOOKER_AVAILABLE", False):
+ with pytest.raises(ImportError) as exc_info:
+ LookerConnector(
+ base_url=PUBLIC_BASE_URL,
+ client_id=CLIENT_ID,
+ client_secret=CLIENT_SECRET,
+ )
+
+ message = str(exc_info.value)
+ assert 'pip install "semantica[ingest-looker]"' in message
+ assert CLIENT_SECRET not in message
+
+ def test_missing_sdk_error_does_not_leak_secret(self):
+ from semantica.ingest.looker_ingestor import LookerConnector
+
+ with patch("semantica.ingest.looker_ingestor.LOOKER_AVAILABLE", False):
+ with pytest.raises(ImportError) as exc_info:
+ LookerConnector(client_secret="super-sensitive")
+
+ assert "super-sensitive" not in str(exc_info.value)
+
+
+# ---------------------------------------------------------------------------
+# TestLookerConnectorInit
+# ---------------------------------------------------------------------------
+
+
+@requires_looker_sdk
+class TestLookerConnectorInit:
+ """Credential resolution, defaults, and pre-flight validation."""
+
+ def test_explicit_credentials_stored(self):
+ from semantica.ingest.looker_ingestor import LookerConnector
+
+ conn = LookerConnector(
+ base_url=PUBLIC_BASE_URL,
+ client_id=CLIENT_ID,
+ client_secret=CLIENT_SECRET,
+ )
+
+ assert conn.base_url == PUBLIC_BASE_URL
+ assert conn.client_id == CLIENT_ID
+ assert conn._client_secret == CLIENT_SECRET
+ assert conn.connected is False
+ assert conn.client is None
+ assert conn.allow_private_ips is False
+
+ def test_credentials_default_to_environment(self):
+ from semantica.ingest.looker_ingestor import LookerConnector
+
+ env = {
+ "LOOKERSDK_BASE_URL": PUBLIC_BASE_URL,
+ "LOOKERSDK_CLIENT_ID": "env-client",
+ "LOOKERSDK_CLIENT_SECRET": "env-secret",
+ }
+ with patch.dict(os.environ, env):
+ conn = LookerConnector()
+
+ assert conn.base_url == PUBLIC_BASE_URL
+ assert conn.client_id == "env-client"
+ assert conn._client_secret == "env-secret"
+
+ def test_explicit_args_override_env_credentials(self):
+ from semantica.ingest.looker_ingestor import LookerConnector
+
+ env = {
+ "LOOKERSDK_CLIENT_ID": "env-client",
+ "LOOKERSDK_CLIENT_SECRET": "env-secret",
+ }
+ with patch.dict(os.environ, env):
+ conn = LookerConnector(
+ base_url=PUBLIC_BASE_URL,
+ client_id=CLIENT_ID,
+ client_secret=CLIENT_SECRET,
+ )
+
+ assert conn.client_id == CLIENT_ID
+ assert conn._client_secret == CLIENT_SECRET
+
+ def test_missing_base_url_raises_validation_error(self):
+ from semantica.ingest.looker_ingestor import LookerConnector
+ from semantica.utils.exceptions import ValidationError
+
+ with pytest.raises(ValidationError):
+ LookerConnector(client_id=CLIENT_ID, client_secret=CLIENT_SECRET)
+
+ def test_missing_client_id_raises_validation_error(self):
+ from semantica.ingest.looker_ingestor import LookerConnector
+ from semantica.utils.exceptions import ValidationError
+
+ with pytest.raises(ValidationError):
+ LookerConnector(base_url=PUBLIC_BASE_URL, client_secret=CLIENT_SECRET)
+
+ def test_missing_client_secret_raises_validation_error(self):
+ from semantica.ingest.looker_ingestor import LookerConnector
+ from semantica.utils.exceptions import ValidationError
+
+ with pytest.raises(ValidationError):
+ LookerConnector(base_url=PUBLIC_BASE_URL, client_id=CLIENT_ID)
+
+ def test_validation_error_does_not_contain_secret(self):
+ from semantica.ingest.looker_ingestor import LookerConnector
+ from semantica.utils.exceptions import ValidationError
+
+ with pytest.raises(ValidationError) as exc_info:
+ LookerConnector(client_id=CLIENT_ID, client_secret="s3cr3t-value")
+
+ assert "s3cr3t-value" not in str(exc_info.value)
+
+ def test_api_settings_built_with_config_file_section_and_env_prefix(self):
+ from semantica.ingest.looker_ingestor import LookerConnector
+
+ mock_settings = MagicMock()
+ mock_settings.base_url = PUBLIC_BASE_URL
+ mock_settings.client_id = CLIENT_ID
+ mock_settings.client_secret = CLIENT_SECRET
+
+ with patch(
+ "semantica.ingest.looker_ingestor.api_settings.ApiSettings",
+ return_value=mock_settings,
+ ) as mock_api_settings:
+ LookerConnector(
+ base_url=PUBLIC_BASE_URL,
+ client_id=CLIENT_ID,
+ client_secret=CLIENT_SECRET,
+ config_file="custom.ini",
+ section="looker",
+ )
+
+ mock_api_settings.assert_called_once_with(
+ filename="custom.ini", section="looker", env_prefix="LOOKERSDK"
+ )
+
+ def test_does_not_mutate_os_environ(self):
+ from semantica.ingest.looker_ingestor import LookerConnector
+
+ before = dict(os.environ)
+ LookerConnector(
+ base_url=PUBLIC_BASE_URL,
+ client_id=CLIENT_ID,
+ client_secret=CLIENT_SECRET,
+ )
+ assert dict(os.environ) == before
+
+
+# ---------------------------------------------------------------------------
+# TestLookerConnectorFailClosed
+# ---------------------------------------------------------------------------
+
+
+class TestLookerConnectorFailClosed:
+ """AE1, AE2 — unsafe endpoints fail before any SDK object is built.
+
+ These cases exercise only the pre-flight checks, so they run whether
+ or not ``looker-sdk`` is installed; the fixture patches
+ ``LOOKER_AVAILABLE`` so the optional-dependency guard does not
+ short-circuit first.
+ """
+
+ @pytest.fixture(autouse=True)
+ def _force_available(self):
+ with patch("semantica.ingest.looker_ingestor.LOOKER_AVAILABLE", True):
+ yield
+
+ def test_http_base_url_rejected_by_default(self):
+ from semantica.ingest.looker_ingestor import LookerConnector
+ from semantica.utils.exceptions import ValidationError
+
+ with pytest.raises(ValidationError) as exc_info:
+ LookerConnector(
+ base_url="http://looker.internal",
+ client_id=CLIENT_ID,
+ client_secret=CLIENT_SECRET,
+ )
+
+ assert "https" in str(exc_info.value)
+
+ def test_http_rejected_before_any_client_is_built(self):
+ from semantica.ingest.looker_ingestor import LookerConnector
+ from semantica.utils.exceptions import ValidationError
+
+ with patch("semantica.ingest.looker_ingestor.looker_sdk") as mock_sdk:
+ with pytest.raises(ValidationError):
+ LookerConnector(
+ base_url="http://looker.internal",
+ client_id=CLIENT_ID,
+ client_secret=CLIENT_SECRET,
+ )
+
+ mock_sdk.init40.assert_not_called()
+
+ def test_localhost_rejected_by_default(self):
+ from semantica.ingest.looker_ingestor import LookerConnector
+ from semantica.utils.exceptions import ValidationError
+
+ with pytest.raises(ValidationError):
+ LookerConnector(
+ base_url="https://localhost",
+ client_id=CLIENT_ID,
+ client_secret=CLIENT_SECRET,
+ )
+
+ def test_loopback_ip_rejected_by_default(self):
+ from semantica.ingest.looker_ingestor import LookerConnector
+ from semantica.utils.exceptions import ValidationError
+
+ with pytest.raises(ValidationError):
+ LookerConnector(
+ base_url="https://127.0.0.1",
+ client_id=CLIENT_ID,
+ client_secret=CLIENT_SECRET,
+ )
+
+ @pytest.mark.parametrize("value", ["false", "0", "no", "off"])
+ def test_falsey_strings_do_not_enable_allow_private_ips(self, value):
+ from semantica.ingest.looker_ingestor import LookerConnector
+ from semantica.utils.exceptions import ValidationError
+
+ with pytest.raises(ValidationError):
+ LookerConnector(
+ base_url=PRIVATE_BASE_URL,
+ client_id=CLIENT_ID,
+ client_secret=CLIENT_SECRET,
+ config={"allow_private_ips": value},
+ )
+
+ def test_config_dict_overrides_kwarg_allow_private_ips(self):
+ from semantica.ingest.looker_ingestor import LookerConnector
+ from semantica.utils.exceptions import ValidationError
+
+ with pytest.raises(ValidationError):
+ LookerConnector(
+ base_url=PRIVATE_BASE_URL,
+ client_id=CLIENT_ID,
+ client_secret=CLIENT_SECRET,
+ allow_private_ips=True,
+ config={"allow_private_ips": "0"},
+ )
+
+ def test_env_base_url_conflicting_with_constructor_fails_closed(self):
+ """AE2 — the environment must not silently redirect the client."""
+ from semantica.ingest.looker_ingestor import LookerConnector
+ from semantica.utils.exceptions import ValidationError
+
+ with patch.dict(os.environ, {"LOOKERSDK_BASE_URL": "https://10.0.0.5"}):
+ with pytest.raises(ValidationError):
+ LookerConnector(
+ base_url=PUBLIC_BASE_URL,
+ client_id=CLIENT_ID,
+ client_secret=CLIENT_SECRET,
+ )
+
+ def test_env_base_url_conflict_message_does_not_leak_secret(self):
+ from semantica.ingest.looker_ingestor import LookerConnector
+ from semantica.utils.exceptions import ValidationError
+
+ with patch.dict(os.environ, {"LOOKERSDK_BASE_URL": "https://10.0.0.5"}):
+ with pytest.raises(ValidationError) as exc_info:
+ LookerConnector(
+ base_url=PUBLIC_BASE_URL,
+ client_id=CLIENT_ID,
+ client_secret="do-not-leak",
+ )
+
+ assert "do-not-leak" not in str(exc_info.value)
+
+
+# ---------------------------------------------------------------------------
+# TestLookerConnectorEndpointValidation
+# ---------------------------------------------------------------------------
+
+
+@requires_looker_sdk
+class TestLookerConnectorEndpointValidation:
+ """R10 — validate-once binding and the allow_private_ips opt-in."""
+
+ def test_http_allowed_with_allow_private_ips_true(self):
+ from semantica.ingest.looker_ingestor import LookerConnector
+
+ conn = LookerConnector(
+ base_url=PRIVATE_BASE_URL,
+ client_id=CLIENT_ID,
+ client_secret=CLIENT_SECRET,
+ allow_private_ips=True,
+ )
+
+ assert conn.allow_private_ips is True
+ assert conn.base_url == PRIVATE_BASE_URL
+
+ def test_config_dict_true_string_enables_allow_private_ips(self):
+ from semantica.ingest.looker_ingestor import LookerConnector
+
+ conn = LookerConnector(
+ base_url=PRIVATE_BASE_URL,
+ client_id=CLIENT_ID,
+ client_secret=CLIENT_SECRET,
+ config={"allow_private_ips": "true"},
+ )
+
+ assert conn.allow_private_ips is True
+
+ def test_env_private_base_url_rejected_without_allow_private_ips(self):
+ """A private endpoint supplied only by the environment fails closed."""
+ from semantica.ingest.looker_ingestor import LookerConnector
+ from semantica.utils.exceptions import ValidationError
+
+ env = {
+ "LOOKERSDK_BASE_URL": "https://10.0.0.5",
+ "LOOKERSDK_CLIENT_ID": CLIENT_ID,
+ "LOOKERSDK_CLIENT_SECRET": CLIENT_SECRET,
+ }
+ with patch.dict(os.environ, env):
+ with pytest.raises(ValidationError):
+ LookerConnector()
+
+ def test_settings_read_config_carries_explicit_credentials(self):
+ """KTD4 — credentials ride the settings object, not os.environ."""
+ from semantica.ingest.looker_ingestor import LookerConnector
+
+ conn = LookerConnector(
+ base_url=PUBLIC_BASE_URL,
+ client_id=CLIENT_ID,
+ client_secret=CLIENT_SECRET,
+ )
+ resolved = conn._settings.read_config()
+
+ assert resolved["base_url"] == PUBLIC_BASE_URL
+ assert resolved["client_id"] == CLIENT_ID
+ assert resolved["client_secret"] == CLIENT_SECRET
+ assert os.getenv("LOOKERSDK_CLIENT_SECRET") is None
+
+
+# ---------------------------------------------------------------------------
+# TestLookerConnectorLifecycle
+# ---------------------------------------------------------------------------
+
+
+@requires_looker_sdk
+class TestLookerConnectorLifecycle:
+ """connect()/disconnect()/context-manager and the transport controls."""
+
+ def _connected_connector(self, client):
+ from semantica.ingest.looker_ingestor import LookerConnector
+
+ with patch(
+ "semantica.ingest.looker_ingestor.looker_sdk.init40",
+ return_value=client,
+ ) as mock_init:
+ conn = LookerConnector(
+ base_url=PUBLIC_BASE_URL,
+ client_id=CLIENT_ID,
+ client_secret=CLIENT_SECRET,
+ )
+ result = conn.connect()
+ return conn, result, mock_init
+
+ def test_connect_passes_same_settings_object(self):
+ client = _make_mock_client()
+
+ with patch(
+ "semantica.ingest.looker_ingestor.looker_sdk.init40",
+ return_value=client,
+ ) as mock_init:
+ from semantica.ingest.looker_ingestor import LookerConnector
+
+ conn = LookerConnector(
+ base_url=PUBLIC_BASE_URL,
+ client_id=CLIENT_ID,
+ client_secret=CLIENT_SECRET,
+ )
+ conn.connect()
+
+ assert mock_init.call_args[1]["config_settings"] is conn._settings
+ assert conn._settings.base_url == PUBLIC_BASE_URL
+
+ def test_connect_returns_existing_client(self):
+ client = _make_mock_client()
+
+ with patch(
+ "semantica.ingest.looker_ingestor.looker_sdk.init40",
+ return_value=client,
+ ) as mock_init:
+ from semantica.ingest.looker_ingestor import LookerConnector
+
+ conn = LookerConnector(
+ base_url=PUBLIC_BASE_URL,
+ client_id=CLIENT_ID,
+ client_secret=CLIENT_SECRET,
+ )
+ first = conn.connect()
+ second = conn.connect()
+
+ assert first is client
+ assert second is client
+ mock_init.assert_called_once()
+
+ def test_connect_applies_transport_controls(self):
+ client = _make_mock_client()
+
+ with patch(
+ "semantica.ingest.looker_ingestor.looker_sdk.init40",
+ return_value=client,
+ ):
+ from semantica.ingest.looker_ingestor import LookerConnector
+
+ conn = LookerConnector(
+ base_url=PUBLIC_BASE_URL,
+ client_id=CLIENT_ID,
+ client_secret=CLIENT_SECRET,
+ )
+ conn.connect()
+
+ assert client.transport.session.max_redirects == 0
+ assert client.transport.session.trust_env is False
+
+ def test_connect_fails_closed_when_client_base_url_differs(self):
+ from semantica.ingest.looker_ingestor import LookerConnector
+ from semantica.utils.exceptions import ValidationError
+
+ client = _make_mock_client(base_url="https://elsewhere.example.com")
+
+ with patch(
+ "semantica.ingest.looker_ingestor.looker_sdk.init40",
+ return_value=client,
+ ):
+ conn = LookerConnector(
+ base_url=PUBLIC_BASE_URL,
+ client_id=CLIENT_ID,
+ client_secret=CLIENT_SECRET,
+ )
+ with pytest.raises(ValidationError):
+ conn.connect()
+
+ assert conn.connected is False
+ assert conn.client is None
+
+ def test_connect_failure_raises_processing_error_without_secret(self):
+ from semantica.ingest.looker_ingestor import LookerConnector
+ from semantica.utils.exceptions import ProcessingError
+
+ error = _looker_sdk.error.SDKError(f"auth failed client_secret={CLIENT_SECRET}")
+
+ with patch(
+ "semantica.ingest.looker_ingestor.looker_sdk.init40",
+ side_effect=error,
+ ):
+ conn = LookerConnector(
+ base_url=PUBLIC_BASE_URL,
+ client_id=CLIENT_ID,
+ client_secret=CLIENT_SECRET,
+ )
+ with pytest.raises(ProcessingError) as exc_info:
+ conn.connect()
+
+ message = str(exc_info.value)
+ assert CLIENT_SECRET not in message
+ assert "SDKError" in message
+
+ def test_disconnect_clears_client_and_is_idempotent(self):
+ client = _make_mock_client()
+ conn, _, _ = self._connected_connector(client)
+
+ conn.disconnect()
+ assert conn.connected is False
+ assert conn.client is None
+
+ conn.disconnect()
+ assert conn.connected is False
+
+ def test_context_manager_connects_and_disconnects(self):
+ client = _make_mock_client()
+
+ with patch(
+ "semantica.ingest.looker_ingestor.looker_sdk.init40",
+ return_value=client,
+ ):
+ from semantica.ingest.looker_ingestor import LookerConnector
+
+ with LookerConnector(
+ base_url=PUBLIC_BASE_URL,
+ client_id=CLIENT_ID,
+ client_secret=CLIENT_SECRET,
+ ) as conn:
+ assert conn.connected is True
+
+ assert conn.connected is False
+
+ def test_close_delegates_to_disconnect(self):
+ client = _make_mock_client()
+ conn, _, _ = self._connected_connector(client)
+
+ conn.close()
+
+ assert conn.connected is False
+
+ def test_real_init40_client_is_bound_and_constrained(self):
+ """End-to-end (offline) proof that the validated value is dialed.
+
+ Real ``looker_sdk.init40`` constructs the client and its
+ ``requests.Session`` without any network I/O, which lets this test
+ assert the binding and the KTD2 transport controls for real rather
+ than against a mock.
+ """
+ from semantica.ingest.looker_ingestor import LookerConnector
+
+ conn = LookerConnector(
+ base_url=PUBLIC_BASE_URL,
+ client_id=CLIENT_ID,
+ client_secret=CLIENT_SECRET,
+ )
+
+ client = conn.connect()
+
+ assert conn.connected is True
+ assert client.auth.settings is conn._settings
+ assert client.auth.settings.base_url == PUBLIC_BASE_URL
+ assert client.transport.session.max_redirects == 0
+ assert client.transport.session.trust_env is False
+
+ resolved = client.auth.settings.read_config()
+ assert resolved["base_url"] == PUBLIC_BASE_URL
+ assert resolved["client_id"] == CLIENT_ID
+ assert resolved["client_secret"] == CLIENT_SECRET
+
+ conn.disconnect()
+ assert conn.connected is False
+
+
+# ---------------------------------------------------------------------------
+# TestLookerConnectorTestConnection
+# ---------------------------------------------------------------------------
+
+
+@requires_looker_sdk
+class TestLookerConnectorTestConnection:
+ """test_connection() — bool result, safe logging, lifecycle discipline."""
+
+ def _connector(self):
+ from semantica.ingest.looker_ingestor import LookerConnector
+
+ conn = LookerConnector(
+ base_url=PUBLIC_BASE_URL,
+ client_id=CLIENT_ID,
+ client_secret=CLIENT_SECRET,
+ )
+ conn.logger = Mock()
+ return conn
+
+ def test_returns_true_on_success(self):
+ client = _make_mock_client()
+ client.me.return_value = object()
+ conn = self._connector()
+
+ with patch(
+ "semantica.ingest.looker_ingestor.looker_sdk.init40",
+ return_value=client,
+ ):
+ assert conn.test_connection() is True
+
+ client.me.assert_called_once()
+
+ def test_opens_and_closes_transient_client(self):
+ client = _make_mock_client()
+ client.me.return_value = object()
+ conn = self._connector()
+
+ with patch(
+ "semantica.ingest.looker_ingestor.looker_sdk.init40",
+ return_value=client,
+ ):
+ conn.test_connection()
+
+ assert conn.connected is False
+
+ def test_returns_false_and_logs_only_exception_type(self):
+ client = _make_mock_client()
+ client.me.side_effect = _looker_sdk.error.SDKError(
+ f"access_token=leaked client_secret={CLIENT_SECRET}"
+ )
+ conn = self._connector()
+
+ with patch(
+ "semantica.ingest.looker_ingestor.looker_sdk.init40",
+ return_value=client,
+ ):
+ assert conn.test_connection() is False
+
+ rendered = " ".join(
+ str(argument)
+ for call in conn.logger.debug.call_args_list
+ for argument in call.args
+ )
+ assert CLIENT_SECRET not in rendered
+ assert "leaked" not in rendered
+ assert "SDKError" in rendered
+
+ def test_does_not_disconnect_pre_existing_client(self):
+ client = _make_mock_client()
+ client.me.return_value = object()
+ conn = self._connector()
+
+ with patch(
+ "semantica.ingest.looker_ingestor.looker_sdk.init40",
+ return_value=client,
+ ):
+ conn.connect()
+ assert conn.connected is True
+ with patch.object(conn, "disconnect") as mock_disconnect:
+ assert conn.test_connection() is True
+
+ mock_disconnect.assert_not_called()
+ assert conn.connected is True
+
+
+# ---------------------------------------------------------------------------
+# TestLookerIngestorReads
+# ---------------------------------------------------------------------------
+
+
+class TestLookerIngestorReads:
+ """U2 — the five metadata reads and the connection lifecycle."""
+
+ @pytest.mark.parametrize(
+ ("method_name", "sdk_method", "content_type", "rows", "columns"),
+ [
+ (
+ "ingest_looks",
+ "all_looks",
+ "look",
+ [{"id": 1, "title": "one"}, {"id": 2, "title": "two"}],
+ ["id", "title"],
+ ),
+ (
+ "ingest_dashboards",
+ "all_dashboards",
+ "dashboard",
+ [{"id": 1, "title": "one"}, {"id": 2, "title": "two"}],
+ ["id", "title"],
+ ),
+ (
+ "ingest_lookml_models",
+ "all_lookml_models",
+ "lookml_model",
+ [{"name": "one"}, {"name": "two"}],
+ ["name"],
+ ),
+ (
+ "ingest_folders",
+ "all_folders",
+ "folder",
+ [{"id": 1, "name": "one"}, {"id": 2, "name": "two"}],
+ ["id", "name"],
+ ),
+ (
+ "ingest_projects",
+ "all_projects",
+ "project",
+ [{"id": "a", "name": "one"}, {"id": "b", "name": "two"}],
+ ["id", "name"],
+ ),
+ ],
+ )
+ def test_read_returns_rows_and_row_count(
+ self, method_name, sdk_method, content_type, rows, columns
+ ):
+ client = MagicMock()
+ getattr(client, sdk_method).return_value = rows
+ ingestor = _make_ingestor(client)
+
+ data = getattr(ingestor, method_name)()
+
+ assert data.row_count == 2
+ assert len(data.data) == 2
+ assert data.content_type == content_type
+ assert data.base_url == PUBLIC_BASE_URL
+ assert data.columns == columns
+
+ def test_read_calls_sdk_method(self):
+ client = MagicMock()
+ client.all_looks.return_value = []
+ ingestor = _make_ingestor(client)
+
+ ingestor.ingest_looks()
+
+ client.all_looks.assert_called_once()
+
+ def test_empty_result_set_returns_empty_list(self):
+ client = MagicMock()
+ client.all_looks.return_value = []
+ ingestor = _make_ingestor(client)
+
+ data = ingestor.ingest_looks()
+
+ assert data.data == []
+ assert data.row_count == 0
+ assert ingestor.export_as_documents(data) == []
+
+ def test_none_result_set_is_treated_as_empty(self):
+ client = MagicMock()
+ client.all_looks.return_value = None
+ ingestor = _make_ingestor(client)
+
+ data = ingestor.ingest_looks()
+
+ assert data.data == []
+ assert data.row_count == 0
+
+ def test_transient_connection_is_closed(self):
+ client = MagicMock()
+ client.all_looks.return_value = []
+ fake = _FakeConnector(client)
+ from semantica.ingest.looker_ingestor import LookerIngestor
+
+ ingestor = LookerIngestor(connector=fake)
+ ingestor.ingest_looks()
+
+ assert fake.connect_calls == 1
+ assert fake.disconnect_calls == 1
+ assert fake.connected is False
+
+ def test_pre_existing_connection_is_reused(self):
+ client = MagicMock()
+ client.all_looks.return_value = []
+ fake = _FakeConnector(client)
+ fake.connected = True
+ from semantica.ingest.looker_ingestor import LookerIngestor
+
+ ingestor = LookerIngestor(connector=fake)
+ ingestor.ingest_looks()
+
+ assert fake.disconnect_calls == 0
+ assert fake.connected is True
+
+ @requires_looker_sdk
+ def test_sdk_error_translated_to_processing_error(self):
+ from semantica.utils.exceptions import ProcessingError
+
+ client = MagicMock()
+ client.all_looks.side_effect = _looker_sdk.error.SDKError(
+ f"token=leaked secret={CLIENT_SECRET}"
+ )
+ fake = _FakeConnector(client)
+ from semantica.ingest.looker_ingestor import LookerIngestor
+
+ ingestor = LookerIngestor(connector=fake)
+ ingestor.logger = Mock()
+
+ with pytest.raises(ProcessingError) as exc_info:
+ ingestor.ingest_looks()
+
+ message = str(exc_info.value)
+ assert CLIENT_SECRET not in message
+ assert "SDKError" in message
+ assert fake.disconnect_calls == 1
+
+ def test_ingestor_context_manager_uses_connector(self):
+ client = MagicMock()
+ client.all_looks.return_value = []
+ fake = _FakeConnector(client)
+ from semantica.ingest.looker_ingestor import LookerIngestor
+
+ with LookerIngestor(connector=fake) as ingestor:
+ assert fake.connected is True
+ ingestor.ingest_looks()
+
+ assert fake.connected is False
+
+
+# ---------------------------------------------------------------------------
+# TestLookerIngestorNormalization
+# ---------------------------------------------------------------------------
+
+
+class TestLookerIngestorNormalization:
+ """U2 — deep normalization and allowlist redaction."""
+
+ def test_plain_dict_records_pass_through_allowed_fields(self):
+ client = MagicMock()
+ client.all_looks.return_value = [
+ {"id": 7, "title": "Revenue", "unexpected": "drop-me"}
+ ]
+ ingestor = _make_ingestor(client)
+
+ data = ingestor.ingest_looks()
+
+ assert data.data == [{"id": 7, "title": "Revenue"}]
+
+ def test_unknown_content_type_raises_validation_error(self):
+ from semantica.ingest.looker_ingestor import LookerData
+ from semantica.utils.exceptions import ValidationError
+
+ ingestor = _make_ingestor(MagicMock())
+ data = LookerData(
+ data=[{"id": 1}], row_count=1, columns=[], content_type="tile"
+ )
+
+ with pytest.raises(ValidationError):
+ ingestor.export_as_documents(data)
+
+ def test_datetime_values_become_iso_8601_strings(self):
+ stamp = datetime(2024, 1, 2, 3, 4, 5, tzinfo=timezone.utc)
+ client = MagicMock()
+ client.all_looks.return_value = [{"id": 1, "title": "T", "created_at": stamp}]
+ ingestor = _make_ingestor(client)
+
+ data = ingestor.ingest_looks()
+
+ assert data.data[0]["created_at"] == stamp.isoformat()
+ json.dumps(data.data)
+
+ def test_nested_models_and_sequences_are_normalized(self):
+ class _Nested:
+ def __init__(self, **kwargs):
+ self._data = kwargs
+
+ def keys(self):
+ return self._data.keys()
+
+ def items(self):
+ return self._data.items()
+
+ client = MagicMock()
+ client.all_lookml_models.return_value = [
+ {
+ "name": "model",
+ "project_name": "proj",
+ "explores": [
+ _Nested(name="orders", label="Orders", group_label="Commerce")
+ ],
+ }
+ ]
+ ingestor = _make_ingestor(client)
+
+ data = ingestor.ingest_lookml_models()
+ record = data.data[0]
+
+ assert record["explores"][0] == {
+ "name": "orders",
+ "label": "Orders",
+ "group_label": "Commerce",
+ }
+ json.dumps(record)
+
+ def test_secret_fields_absent_from_plain_dict_records(self):
+ client = MagicMock()
+ client.all_dashboards.return_value = [
+ {
+ "id": 1,
+ "title": "Ops",
+ "password": "DASH_PASSWORD",
+ "pdt_password": "PDT_PASSWORD",
+ }
+ ]
+ client.all_lookml_models.return_value = [
+ {
+ "name": "model",
+ "project_name": "proj",
+ "device_token": "DEVICE_TOKEN",
+ }
+ ]
+ ingestor = _make_ingestor(client)
+
+ dashboards = ingestor.ingest_dashboards()
+ models = ingestor.ingest_lookml_models()
+
+ rendered = json.dumps([dashboards.data, models.data])
+ for secret in ("DASH_PASSWORD", "PDT_PASSWORD", "DEVICE_TOKEN"):
+ assert secret not in rendered
+
+ def test_nested_secret_fields_are_redacted(self):
+ class _NestedProject:
+ def __init__(self, **kwargs):
+ self._data = kwargs
+
+ def keys(self):
+ return self._data.keys()
+
+ def items(self):
+ return self._data.items()
+
+ client = MagicMock()
+ client.all_folders.return_value = [
+ {
+ "id": 1,
+ "name": "Shared",
+ "looks": [
+ _NestedProject(
+ id=9,
+ title="Nested look",
+ password="NESTED_PASSWORD",
+ )
+ ],
+ }
+ ]
+ ingestor = _make_ingestor(client)
+
+ data = ingestor.ingest_folders()
+
+ rendered = json.dumps(data.data)
+ assert "NESTED_PASSWORD" not in rendered
+ assert "password" not in rendered
+
+ @requires_looker_sdk
+ def test_project_secrets_absent_from_data_and_documents(self):
+ from looker_sdk.sdk.api40 import models as sdk_models
+
+ project = sdk_models.Project(
+ id="shop",
+ name="shop",
+ git_remote_url="https://gituser:[email protected]/shop.git",
+ git_username="gituser",
+ git_password="GIT_PASSWORD",
+ deploy_secret="DEPLOY_SECRET",
+ )
+ client = MagicMock()
+ client.all_projects.return_value = [project]
+ ingestor = _make_ingestor(client)
+
+ data = ingestor.ingest_projects()
+ documents = ingestor.export_as_documents(data)
+
+ payload = json.dumps({"data": data.data, "documents": documents})
+ for secret in (
+ "GIT_PASSWORD",
+ "DEPLOY_SECRET",
+ "gitpass",
+ "gituser:gitpass",
+ ):
+ assert secret not in payload
+
+ record = data.data[0]
+ assert record["name"] == "shop"
+ assert record["git_remote_url"] == "https://git.example.com/shop.git"
+ assert "git_username" in record
+ assert "git_password" not in record
+ assert "deploy_secret" not in record
+
+ @requires_looker_sdk
+ def test_lookml_model_explores_are_nested_with_parent_identity(self):
+ from looker_sdk.sdk.api40 import models as sdk_models
+
+ explore = sdk_models.LookmlModelNavExplore(
+ name="orders",
+ label="Orders",
+ description="Order facts",
+ hidden=False,
+ group_label="Commerce",
+ )
+ model = sdk_models.LookmlModel(
+ name="ecommerce",
+ project_name="shop",
+ label="Ecommerce",
+ explores=[explore],
+ )
+ client = MagicMock()
+ client.all_lookml_models.return_value = [model]
+ ingestor = _make_ingestor(client)
+
+ data = ingestor.ingest_lookml_models()
+ record = data.data[0]
+
+ assert record["name"] == "ecommerce"
+ assert record["project_name"] == "shop"
+ # ``all_lookml_models()`` yields ``LookmlModelNavExplore`` records: only
+ # the navigation fields exist, and ``group_label`` must survive.
+ assert record["explores"][0] == {
+ "name": "orders",
+ "description": "Order facts",
+ "label": "Orders",
+ "hidden": False,
+ "group_label": "Commerce",
+ }
+ assert "device_token" not in record
+ json.dumps(record)
+
+
+# ---------------------------------------------------------------------------
+# TestLookerIngestorExport
+# ---------------------------------------------------------------------------
+
+
+class TestLookerIngestorExport:
+ """U2/R3/R4 — document projection, ids, metadata, and JSON safety."""
+
+ def _sample_rows_by_content_type(self):
+ return {
+ "look": [{"id": 1, "title": "Orders look", "description": "desc"}],
+ "dashboard": [{"id": 2, "title": "Ops dashboard"}],
+ "lookml_model": [
+ {
+ "name": "ecommerce",
+ "project_name": "shop",
+ "label": "Ecommerce",
+ "explores": [
+ {
+ "name": "orders",
+ "label": "Orders",
+ "description": "Order facts",
+ "hidden": False,
+ "group_label": "Commerce",
+ }
+ ],
+ }
+ ],
+ "folder": [{"id": 3, "name": "Shared"}],
+ "project": [{"id": "shop", "name": "shop"}],
+ }
+
+ def test_document_shape_and_metadata(self):
+ client = MagicMock()
+ client.all_looks.return_value = [
+ {"id": 1, "title": "Orders look", "description": "desc"}
+ ]
+ ingestor = _make_ingestor(client)
+
+ data = ingestor.ingest_looks()
+ documents = ingestor.export_as_documents(data)
+
+ assert len(documents) == 1
+ document = documents[0]
+ assert set(document) == {"id", "text", "metadata"}
+ assert document["id"] == "looker:look:1"
+ assert "Orders look" in document["text"]
+ assert document["metadata"]["source"] == "looker"
+ assert document["metadata"]["content_type"] == "look"
+ assert "base_url" not in document["metadata"]
+ assert "base_url" not in json.dumps(document["metadata"])
+
+ def test_metadata_never_contains_credentials(self):
+ client = MagicMock()
+ client.all_projects.return_value = [
+ {
+ "id": "shop",
+ "name": "shop",
+ "client_id": "cid",
+ "client_secret": "csecret",
+ "secret": "nope",
+ }
+ ]
+ ingestor = _make_ingestor(client)
+
+ data = ingestor.ingest_projects()
+ documents = ingestor.export_as_documents(data)
+
+ rendered = json.dumps(documents)
+ assert "csecret" not in rendered
+ assert "client_secret" not in rendered
+
+ def test_document_ids_are_unique_across_content_types(self):
+ from semantica.ingest.looker_ingestor import LookerData
+
+ ingestor = _make_ingestor(MagicMock())
+ documents = []
+ for content_type, rows in self._sample_rows_by_content_type().items():
+ data = LookerData(
+ data=rows,
+ row_count=len(rows),
+ columns=[],
+ content_type=content_type,
+ )
+ documents.extend(ingestor.export_as_documents(data))
+
+ ids = [document["id"] for document in documents]
+ assert ids == [
+ "looker:look:1",
+ "looker:dashboard:2",
+ "looker:lookml_model:shop.ecommerce",
+ "looker:folder:3",
+ "looker:project:shop",
+ ]
+ assert len(ids) == len(set(ids))
+
+ def test_lookml_model_document_nests_explores(self):
+ from semantica.ingest.looker_ingestor import LookerData
+
+ ingestor = _make_ingestor(MagicMock())
+ rows = self._sample_rows_by_content_type()["lookml_model"]
+ data = LookerData(
+ data=rows,
+ row_count=len(rows),
+ columns=[],
+ content_type="lookml_model",
+ )
+
+ documents = ingestor.export_as_documents(data)
+ metadata = documents[0]["metadata"]
+
+ assert documents[0]["id"] == "looker:lookml_model:shop.ecommerce"
+ assert metadata["project_name"] == "shop"
+ assert metadata["model_name"] == "ecommerce"
+ assert metadata["explores"][0]["id"] == (
+ "looker:lookml_model:shop.ecommerce:explore:orders"
+ )
+ assert metadata["explores"][0]["name"] == "orders"
+ assert metadata["explores"][0]["label"] == "Orders"
+ assert metadata["explores"][0]["description"] == "Order facts"
+ assert metadata["explores"][0]["hidden"] is False
+ assert metadata["explores"][0]["group_label"] == "Commerce"
+ assert metadata["explores"][0]["parent_model"] == "shop.ecommerce"
+ assert "view_name" not in metadata["explores"][0]
+
+ def test_full_export_is_json_serializable(self):
+ from semantica.ingest.looker_ingestor import LookerData
+
+ ingestor = _make_ingestor(MagicMock())
+ documents = []
+ for content_type, rows in self._sample_rows_by_content_type().items():
+ data = LookerData(
+ data=rows,
+ row_count=len(rows),
+ columns=[],
+ content_type=content_type,
+ )
+ documents.extend(ingestor.export_as_documents(data))
+
+ json.dumps(documents)
+
+ @requires_looker_sdk
+ def test_export_with_sdk_models_and_datetimes_is_json_serializable(self):
+ from looker_sdk.sdk.api40 import models as sdk_models
+
+ look = sdk_models.Look(
+ id=11,
+ title="Timed look",
+ created_at=datetime(2024, 5, 6, 7, 8, 9, tzinfo=timezone.utc),
+ )
+ model = sdk_models.LookmlModel(
+ name="ecommerce",
+ project_name="shop",
+ explores=[
+ sdk_models.LookmlModelNavExplore(
+ name="orders",
+ label="Orders",
+ description="Order facts",
+ hidden=False,
+ group_label="Commerce",
+ )
+ ],
+ )
+ client = MagicMock()
+ client.all_looks.return_value = [look]
+ client.all_lookml_models.return_value = [model]
+ ingestor = _make_ingestor(client)
+
+ looks_data = ingestor.ingest_looks()
+ models_data = ingestor.ingest_lookml_models()
+
+ json.dumps(looks_data.data)
+ json.dumps(models_data.data)
+ json.dumps(ingestor.export_as_documents(looks_data))
+ json.dumps(ingestor.export_as_documents(models_data))
+
+
+# ---------------------------------------------------------------------------
+# TestLookerConnectorSecretLogging — R11/R13
+# ---------------------------------------------------------------------------
+
+
+@requires_looker_sdk
+class TestLookerConnectorSecretLogging(unittest.TestCase):
+ """No credential, PAT, or secret-adjacent field may reach a log record.
+
+ Records are captured with :meth:`unittest.TestCase.assertLogs` from the
+ connector logger and from the SDK's own transport logger across a
+ mocked ``connect()``, the ``test_connection()`` probe, and all five
+ metadata reads.
+ """
+
+ @staticmethod
+ def _secret_rows():
+ """Rows carrying every secret-adjacent SDK field."""
+ return [
+ {
+ "id": 1,
+ "title": "Orders",
+ "project_name": "shop",
+ "name": "ecommerce",
+ "git_remote_url": (
+ f"https://user:{CLIENT_SECRET}@git.example.com/shop.git"
+ ),
+ **SECRET_FIELD_VALUES,
+ }
+ ]
+
+ def test_secret_and_access_token_never_reach_log_records(self):
+ from semantica.ingest.looker_ingestor import LookerConnector, LookerIngestor
+
+ client = _make_mock_client()
+ rows = self._secret_rows()
+ for sdk_method in (
+ "all_looks",
+ "all_dashboards",
+ "all_lookml_models",
+ "all_folders",
+ "all_projects",
+ ):
+ getattr(client, sdk_method).return_value = rows
+
+ # The probe fails with an SDK error whose text embeds both a token
+ # and the client secret: only the exception type may be logged.
+ client.me.side_effect = _looker_sdk.error.SDKError(
+ f"auth failed token={ACCESS_TOKEN} secret={CLIENT_SECRET}"
+ )
+
+ with patch(
+ "semantica.ingest.looker_ingestor.looker_sdk.init40",
+ return_value=client,
+ ):
+ connector = LookerConnector(
+ base_url=PUBLIC_BASE_URL,
+ client_id=CLIENT_ID,
+ client_secret=CLIENT_SECRET,
+ )
+
+ with self.assertLogs(CONNECTOR_LOGGER, level="DEBUG") as connector_logs:
+ connector.connect()
+ assert connector.test_connection() is False
+
+ ingestor = LookerIngestor(connector=connector)
+ for method_name in (
+ "ingest_looks",
+ "ingest_dashboards",
+ "ingest_lookml_models",
+ "ingest_folders",
+ "ingest_projects",
+ ):
+ data = getattr(ingestor, method_name)()
+ ingestor.export_as_documents(data)
+
+ captured = "\n".join(connector_logs.output)
+ assert connector_logs.output, "expected connector log records"
+
+ for secret in (CLIENT_SECRET, ACCESS_TOKEN, *SECRET_FIELD_VALUES.values()):
+ assert secret not in captured
+
+
+# ---------------------------------------------------------------------------
+# TestLookerConnectorTransportHardening — KTD2 compensating controls
+# ---------------------------------------------------------------------------
+
+
+@requires_looker_sdk
+class TestLookerConnectorTransportHardening:
+ """The SDK session must not follow redirects or trust proxy env vars."""
+
+ def _connect_with_real_session(self, session, base_url: str = PUBLIC_BASE_URL):
+ """Connect a LookerConnector whose client carries *session*."""
+ from semantica.ingest.looker_ingestor import LookerConnector
+
+ client = _make_mock_client(base_url=base_url)
+ client.transport.session = session
+
+ with patch(
+ "semantica.ingest.looker_ingestor.looker_sdk.init40",
+ return_value=client,
+ ):
+ connector = LookerConnector(
+ base_url=base_url,
+ client_id=CLIENT_ID,
+ client_secret=CLIENT_SECRET,
+ )
+ connector.connect()
+ return connector
+
+ def test_redirect_response_is_not_followed(self):
+ target = f"{PUBLIC_BASE_URL}/api/4.0/looks"
+ seen = []
+
+ class _RedirectAdapter(BaseAdapter):
+ def send(self, request, **kwargs):
+ seen.append(request.url)
+ response = requests.Response()
+ response.status_code = 302
+ response.headers["Location"] = "https://evil.example.com/steal"
+ response.url = request.url
+ response.request = request
+ response.raw = None
+ return response
+
+ def close(self):
+ pass
+
+ session = requests.Session()
+ assert session.max_redirects != 0 # default is a real redirect cap
+
+ self._connect_with_real_session(session)
+ session.mount("https://", _RedirectAdapter())
+
+ assert session.max_redirects == 0
+ with pytest.raises(TooManyRedirects):
+ session.get(target)
+
+ # The redirect target was never requested: only the original hop.
+ assert seen == [target]
+
+ def test_proxy_environment_variable_is_not_honoured(self, monkeypatch):
+ monkeypatch.setenv("HTTP_PROXY", "http://127.0.0.1:9")
+ monkeypatch.setenv("HTTPS_PROXY", "http://127.0.0.1:9")
+
+ session = requests.Session()
+ assert session.trust_env is True # default honours the environment
+
+ self._connect_with_real_session(session)
+
+ assert session.trust_env is False
+ settings = session.merge_environment_settings(
+ f"{PUBLIC_BASE_URL}/api/4.0/looks", {}, None, None, None
+ )
+ assert settings["proxies"] == {}
+
+ def test_connections_are_pinned_to_the_validated_address(self):
+ session = requests.Session()
+
+ self._connect_with_real_session(session)
+
+ adapter = session.adapters.get("https://")
+ assert getattr(adapter, "_semantica_pinned", False) is True
+
+ def test_metadata_read_is_routed_through_the_ssrf_validator(self):
+ """AE1 — every SDK request must pass the repository SSRF validator.
+
+ The SDK's ``all_looks`` is wired to issue a real request through the
+ session that ``connect()`` hardens. With the validator patched to
+ raise, the read must fail *before* the pinned transport is reached.
+ """
+ from semantica.ingest.looker_ingestor import LookerIngestor
+ from semantica.utils.exceptions import ValidationError
+
+ target = f"{PUBLIC_BASE_URL}/api/4.0/looks"
+ session = requests.Session()
+ client = _make_mock_client()
+ client.transport.session = session
+
+ class _RecordingAdapter(BaseAdapter):
+ """Pinned-transport stand-in that never dials the network."""
+
+ def __init__(self):
+ self.seen = []
+
+ def send(self, request, **kwargs):
+ self.seen.append(request.url)
+ response = requests.Response()
+ response.status_code = 200
+ response.url = request.url
+ response.request = request
+ response._content = b"[]"
+ return response
+
+ def close(self):
+ pass
+
+ delegate = _RecordingAdapter()
+
+ def _all_looks(**_kwargs):
+ session.get(target)
+ return []
+
+ client.all_looks.side_effect = _all_looks
+
+ with (
+ patch(
+ "semantica.ingest.looker_ingestor.looker_sdk.init40",
+ return_value=client,
+ ),
+ patch(
+ "semantica.ingest.looker_ingestor._make_pinned_adapter",
+ return_value=delegate,
+ ),
+ ):
+ ingestor = LookerIngestor(
+ base_url=PUBLIC_BASE_URL,
+ client_id=CLIENT_ID,
+ client_secret=CLIENT_SECRET,
+ )
+
+ with patch(
+ "semantica.ingest.looker_ingestor.validate_url_for_request",
+ side_effect=ValidationError("blocked by the test double"),
+ ):
+ with pytest.raises(ValidationError):
+ ingestor.ingest_looks()
+
+ # The validator raised before the pinned adapter was reached.
+ assert delegate.seen == []
+
+ @pytest.mark.parametrize(
+ ("base_url", "expected_host"),
+ [
+ ("https://[2001:db8::1]", "[2001:db8::1]"),
+ ("https://[2001:db8::1]:8443", "[2001:db8::1]:8443"),
+ ],
+ )
+ def test_ipv6_host_header_is_bracketed(self, base_url, expected_host):
+ """AE4 — the pinned session's Host header is a valid authority."""
+ session = requests.Session()
+
+ with (
+ patch("semantica.ingest.looker_ingestor.validate_url_for_request"),
+ patch(
+ "semantica.ingest.looker_ingestor._resolve_pinned_ips",
+ return_value=["2001:db8::1"],
+ ),
+ ):
+ self._connect_with_real_session(session, base_url=base_url)
+
+ assert session.headers["Host"] == expected_host
+
+
+# ---------------------------------------------------------------------------
+# TestLookerHardeningRegressions — defects found in code review
+# ---------------------------------------------------------------------------
+
+
+class TestLookerHardeningRegressions:
+ """Regressions for review findings: redaction and config forwarding."""
+
+ def test_userinfo_is_scrubbed_without_a_scheme(self):
+ """Credentials in a scheme-less remote must not survive."""
+ from semantica.ingest.looker_ingestor import _scrub_url_userinfo
+
+ assert (
+ _scrub_url_userinfo("oauth2:[email protected]/org/r.git")
+ == "gitlab.example.com/org/r.git"
+ )
+ assert (
+ _scrub_url_userinfo("[email protected]:org/repo.git")
+ == "github.com:org/repo.git"
+ )
+ assert _scrub_url_userinfo("https://user:[email protected]/r.git") == (
+ "https://example.com/r.git"
+ )
+
+ def test_userinfo_in_path_is_not_treated_as_credentials(self):
+ from semantica.ingest.looker_ingestor import _scrub_url_userinfo
+
+ assert (
+ _scrub_url_userinfo("https://example.com/path@file")
+ == "https://example.com/path@file"
+ )
+
+ def test_userinfo_is_scrubbed_when_an_at_sign_follows_the_authority(self):
+ """The deciding '@' must be the one inside the authority section."""
+ from semantica.ingest.looker_ingestor import _scrub_url_userinfo
+
+ assert _scrub_url_userinfo("https://user:[email protected]/path@file") == (
+ "https://example.com/path@file"
+ )
+ assert (
+ _scrub_url_userinfo(
+ "https://oauth2:[email protected]/org/repo.git"
+ )
+ == "https://gitlab.example.com/org/[email protected]"
+ )
+
+ def test_ipv6_authority_userinfo_is_scrubbed(self):
+ from semantica.ingest.looker_ingestor import _scrub_url_userinfo
+
+ assert (
+ _scrub_url_userinfo("https://user:pw@[::1]:8080/x")
+ == "https://[::1]:8080/x"
+ )
+
+ def test_ambiguous_password_containing_a_slash_fails_closed(self):
+ """A password containing '/' defeats the authority split."""
+ from semantica.ingest.looker_ingestor import (
+ _REDACTED_URL,
+ _scrub_url_userinfo,
+ )
+
+ assert _scrub_url_userinfo("https://user:pa/ss@host/repo.git") == (
+ _REDACTED_URL
+ )
+ # A later '@' in the path must not rescue the earlier userinfo.
+ assert (
+ _scrub_url_userinfo(
+ "https://alice:[email protected]/groups/[email protected]/repo.git"
+ )
+ == "https://git.example.com/groups/[email protected]/repo.git"
+ )
+ assert (
+ _scrub_url_userinfo("https://user:pw@host/repo.git?x=@y")
+ == "https://host/repo.git?x=@y"
+ )
+
+ def test_host_port_authority_is_not_treated_as_userinfo(self):
+ from semantica.ingest.looker_ingestor import _scrub_url_userinfo
+
+ assert (
+ _scrub_url_userinfo("https://example.com:8443/path@file")
+ == "https://example.com:8443/path@file"
+ )
+ assert (
+ _scrub_url_userinfo("https://[::1]:8080/path@file")
+ == "https://[::1]:8080/path@file"
+ )
+
+ def test_guard_adapter_pinned_marker_is_derived_and_read_only(self):
+ """``_semantica_pinned`` is derived from the delegate, not cached."""
+ from semantica.ingest.looker_ingestor import _SSRFGuardAdapter
+
+ adapter = _SSRFGuardAdapter(allow_private_ips=False)
+
+ assert adapter._semantica_pinned is False
+ with pytest.raises(AttributeError):
+ adapter._semantica_pinned = True
+
+ @requires_looker_sdk
+ def test_ingestor_config_dict_does_not_collide_with_named_arguments(self):
+ """A config dict naming a connector parameter must not raise."""
+ from semantica.ingest.looker_ingestor import LookerConnector, LookerIngestor
+
+ ingestor = LookerIngestor(
+ base_url=PUBLIC_BASE_URL,
+ client_id=CLIENT_ID,
+ client_secret=CLIENT_SECRET,
+ config={"allow_private_ips": True},
+ )
+
+ assert isinstance(ingestor.connector, LookerConnector)
+
+ @requires_looker_sdk
+ def test_env_conflict_message_does_not_leak_userinfo(self):
+ from semantica.ingest.looker_ingestor import LookerConnector
+ from semantica.utils.exceptions import ValidationError
+
+ env_url = "https://envuser:[email protected]"
+ with patch.dict(os.environ, {"LOOKERSDK_BASE_URL": env_url}):
+ with pytest.raises(ValidationError) as exc_info:
+ LookerConnector(
+ base_url=PUBLIC_BASE_URL,
+ client_id=CLIENT_ID,
+ client_secret=CLIENT_SECRET,
+ )
+
+ message = str(exc_info.value)
+ assert "ENVSECRET" not in message
+ assert "envuser" not in message
+
+ @requires_looker_sdk
+ def test_ini_conflict_message_does_not_leak_userinfo(self):
+ from semantica.ingest.looker_ingestor import LookerConnector
+ from semantica.utils.exceptions import ValidationError
+
+ raw_config = {"base_url": "https://iniuser:[email protected]"}
+
+ with patch.object(LookerConnector, "_raw_read_config", return_value=raw_config):
+ with pytest.raises(ValidationError) as exc_info:
+ LookerConnector(
+ base_url=PUBLIC_BASE_URL,
+ client_id=CLIENT_ID,
+ client_secret=CLIENT_SECRET,
+ )
+
+ message = str(exc_info.value)
+ assert "INISECRET" not in message
+ assert "iniuser" not in message
+
+
+# ---------------------------------------------------------------------------
+# TestLookerIngestorConfigForwarding — config keys reach the connector
+# ---------------------------------------------------------------------------
+
+
+class TestLookerIngestorConfigForwarding:
+ """A ``config`` value for a named connector argument must reach it.
+
+ ``_CONNECTOR_ARGUMENT_NAMES`` keeps those keys out of the forwarded
+ ``**kwargs`` (so the call cannot collide), which means the ingestor must
+ resolve each one explicitly instead of silently dropping it.
+ """
+
+ @pytest.mark.parametrize(
+ ("key", "value"),
+ [
+ ("base_url", PUBLIC_BASE_URL),
+ ("client_id", "config-client-id"),
+ ("client_secret", "config-client-secret"),
+ ("config_file", "custom.ini"),
+ ("section", "production"),
+ ],
+ )
+ def test_config_value_reaches_the_connector(self, key, value):
+ from semantica.ingest.looker_ingestor import LookerIngestor
+
+ with patch(
+ "semantica.ingest.looker_ingestor.LookerConnector"
+ ) as mock_connector:
+ LookerIngestor(config={key: value})
+
+ assert mock_connector.call_args.kwargs[key] == value
+
+ def test_named_argument_wins_over_config_value(self):
+ from semantica.ingest.looker_ingestor import LookerIngestor
+
+ with patch(
+ "semantica.ingest.looker_ingestor.LookerConnector"
+ ) as mock_connector:
+ LookerIngestor(
+ base_url=PUBLIC_BASE_URL,
+ config={"base_url": "https://config.example.com"},
+ )
+
+ assert mock_connector.call_args.kwargs["base_url"] == PUBLIC_BASE_URL
+
+
+# ---------------------------------------------------------------------------
+# TestLookerIngestorMalformedRecords — SDK data problems fail safely
+# ---------------------------------------------------------------------------
+
+
+class TestLookerIngestorMalformedRecords:
+ """Malformed SDK records must use the documented exception types."""
+
+ def test_unsupported_record_type_raises_processing_error(self):
+ from semantica.utils.exceptions import ProcessingError
+
+ client = MagicMock()
+ client.all_looks.return_value = [42]
+ ingestor = _make_ingestor(client)
+
+ with pytest.raises(ProcessingError) as exc_info:
+ ingestor.ingest_looks()
+
+ assert "Unsupported Looker record type: int" in str(exc_info.value)
+
+ def test_record_conversion_failure_is_chained(self):
+ from semantica.utils.exceptions import ProcessingError
+
+ class _Unconvertible:
+ __slots__ = ()
+
+ def items(self):
+ raise RuntimeError("boom")
+
+ def keys(self):
+ raise RuntimeError("boom")
+
+ client = MagicMock()
+ client.all_looks.return_value = [_Unconvertible()]
+ ingestor = _make_ingestor(client)
+
+ with pytest.raises(ProcessingError) as exc_info:
+ ingestor.ingest_looks()
+
+ assert isinstance(exc_info.value.__cause__, RuntimeError)
+ assert "Unsupported Looker record type: _Unconvertible" in str(exc_info.value)
+
+ def test_non_mapping_row_raises_documented_validation_error(self):
+ from semantica.ingest.looker_ingestor import LookerData
+ from semantica.utils.exceptions import ValidationError
+
+ ingestor = _make_ingestor(MagicMock())
+ data = LookerData(data=["oops"], row_count=1, columns=[], content_type="look")
+
+ with pytest.raises(ValidationError) as exc_info:
+ ingestor.export_as_documents(data)
+
+ assert "str" in str(exc_info.value)
+
+ def test_non_iterable_explores_degrades_safely(self):
+ from semantica.ingest.looker_ingestor import LookerData
+
+ ingestor = _make_ingestor(MagicMock())
+ data = LookerData(
+ data=[{"name": "model", "project_name": "proj", "explores": 5}],
+ row_count=1,
+ columns=[],
+ content_type="lookml_model",
+ )
+
+ documents = ingestor.export_as_documents(data)
+
+ metadata = documents[0]["metadata"]
+ assert metadata["explores"] == []
+ assert metadata["explore_count"] == 0
+
+
+# ---------------------------------------------------------------------------
+# TestLookerIngestorOptionForwarding — **options reach the SDK method
+# ---------------------------------------------------------------------------
+
+
+class TestLookerIngestorOptionForwarding:
+ """Each read's documented ``**options`` must reach the SDK call."""
+
+ _READS = [
+ ("ingest_looks", "all_looks"),
+ ("ingest_dashboards", "all_dashboards"),
+ ("ingest_lookml_models", "all_lookml_models"),
+ ("ingest_folders", "all_folders"),
+ ("ingest_projects", "all_projects"),
+ ]
+
+ @pytest.mark.parametrize(("method_name", "sdk_method"), _READS)
+ def test_options_are_forwarded_to_the_sdk_method(self, method_name, sdk_method):
+ client = MagicMock()
+ getattr(client, sdk_method).return_value = []
+ ingestor = _make_ingestor(client)
+
+ getattr(ingestor, method_name)(fields="id,title", limit=5, offset=10)
+
+ assert getattr(client, sdk_method).call_args == call(
+ fields="id,title", limit=5, offset=10
+ )
+
+ @pytest.mark.parametrize(("method_name", "sdk_method"), _READS)
+ def test_no_options_sends_no_keyword_arguments(self, method_name, sdk_method):
+ client = MagicMock()
+ getattr(client, sdk_method).return_value = []
+ ingestor = _make_ingestor(client)
+
+ getattr(ingestor, method_name)()
+
+ assert getattr(client, sdk_method).call_args == call()
+
+
+# ---------------------------------------------------------------------------
+# TestLookerIngestorConfigOptIn — AE2/AE3 config allow_private_ips resolution
+# ---------------------------------------------------------------------------
+
+
+@requires_looker_sdk
+class TestLookerIngestorConfigOptIn:
+ """``config={"allow_private_ips": True}`` must reach the connector."""
+
+ @pytest.mark.parametrize("kwargs", [{}, {"allow_private_ips": False}])
+ def test_config_opt_in_wins_when_named_argument_is_left_at_default(self, kwargs):
+ """AE2/AE3 — the config entry wins while the named arg is unset."""
+ from semantica.ingest.looker_ingestor import LookerIngestor
+
+ ingestor = LookerIngestor(
+ base_url=PRIVATE_BASE_URL,
+ client_id=CLIENT_ID,
+ client_secret=CLIENT_SECRET,
+ config={"allow_private_ips": True},
+ **kwargs,
+ )
+
+ assert ingestor.connector.allow_private_ips is True
+
+ def test_config_value_wins_over_a_non_default_named_argument(self):
+ """The connector's config-wins precedence, parsed with parse_bool."""
+ from semantica.ingest.looker_ingestor import LookerIngestor
+
+ ingestor = LookerIngestor(
+ base_url=PUBLIC_BASE_URL,
+ client_id=CLIENT_ID,
+ client_secret=CLIENT_SECRET,
+ allow_private_ips=True,
+ config={"allow_private_ips": "0"},
+ )
+
+ assert ingestor.connector.allow_private_ips is False
+
+
+# ---------------------------------------------------------------------------
+# TestLookerConnectorErrorDocEgress — the SDK's error-doc fetch is neutralised
+# ---------------------------------------------------------------------------
+
+
+@requires_looker_sdk
+class TestLookerConnectorErrorDocEgress:
+ """The SDK's error-documentation lookup must not make its own requests.
+
+ ``looker_sdk.error.ErrorDocHelper.get_index`` performs a bare
+ ``requests.get`` (fresh session, ``trust_env=True``, no timeout, no
+ redirect cap, no SSRF validation) whenever the API returns a non-2xx
+ response. ``connect()`` disables it, so a failing API call still raises
+ the normal SDK error without opening a second, unguarded egress path.
+ """
+
+ def _connected_connector(self):
+ from semantica.ingest.looker_ingestor import LookerConnector
+
+ client = _make_mock_client()
+ with patch(
+ "semantica.ingest.looker_ingestor.looker_sdk.init40",
+ return_value=client,
+ ):
+ connector = LookerConnector(
+ base_url=PUBLIC_BASE_URL,
+ client_id=CLIENT_ID,
+ client_secret=CLIENT_SECRET,
+ )
+ connector.connect()
+ return connector
+
+ def test_error_doc_lookup_makes_no_network_call(self):
+ import looker_sdk.error as sdk_error
+
+ self._connected_connector()
+
+ helper = sdk_error.ErrorDocHelper()
+ documentation_url = "https://docs.looker.com/r/err/4.0/404/notfound"
+ with patch.object(
+ sdk_error.requests, "get", side_effect=AssertionError("network egress")
+ ) as mock_get:
+ error_doc_url, error_doc = helper.parse_and_lookup(documentation_url)
+
+ mock_get.assert_not_called()
+ # No index was fetched, so no per-code document URL was resolved
+ # either: the helper falls back to its base URL and the inline
+ # "no documentation" message.
+ assert error_doc_url == helper.ERROR_CODES_URL
+ assert "No documentation found" in error_doc
+
+ def test_non_2xx_response_raises_the_normal_sdk_error(self):
+ import looker_sdk.error as sdk_error
+ from looker_sdk.rtl import api_methods
+
+ self._connected_connector()
+
+ documentation_url = "https://docs.looker.com/r/err/4.0/404/notfound"
+
+ class _Auth:
+ class settings: # noqa: N801 - mirrors the SDK settings shape
+ base_url = PUBLIC_BASE_URL
+
+ class _Response:
+ ok = False
+ encoding = "utf-8"
+ value = b'{"message": "not found"}'
+
+ def _deserialize(*, data, structure):
+ return sdk_error.SDKError(
+ message="not found", documentation_url=documentation_url
+ )
+
+ methods = api_methods.APIMethods(
+ auth=_Auth(),
+ deserialize=_deserialize,
+ serialize=MagicMock(),
+ transport=MagicMock(),
+ api_version="4.0",
+ )
+
+ with patch.object(
+ sdk_error.requests, "get", side_effect=AssertionError("network egress")
+ ) as mock_get:
+ with pytest.raises(sdk_error.SDKError) as exc_info:
+ methods._return(_Response(), structure=sdk_error.SDKError)
+
+ mock_get.assert_not_called()
+ assert "No documentation found" in exc_info.value.error_doc
diff --git a/uv.lock b/uv.lock
index 34e8460f..e5763913 100644
--- a/uv.lock
+++ b/uv.lock
@@ -725,6 +725,20 @@ wheels = [
{ url = "https://files.pythonhosted.org/packages/9e/96/d32b941a501ab566a16358d68b6eb4e4acc373fab3c3c4d7d9e649f7b4bb/catalogue-2.0.10-py3-none-any.whl", hash = "sha256:58c2de0020aa90f4a2da7dfad161bf7b3b054c86a5f09fcedc0b2b740c109a9f", size = 17325, upload-time = "2023-09-25T06:29:23.337Z" },
]
+[[package]]
+name = "cattrs"
+version = "26.2.0"
+source = { registry = "https://pypi.org/simple" }
+dependencies = [
+ { name = "attrs" },
+ { name = "exceptiongroup", marker = "python_full_version < '3.11'" },
+ { name = "typing-extensions" },
+]
+sdist = { url = "https://files.pythonhosted.org/packages/d6/b2/42f4524e5479b090040b5fd8bb316dd8c65a079bb6492494ce2079dc91be/cattrs-26.2.0.tar.gz", hash = "sha256:3cf49f69df8326bcf17a3cb3d3d3ec4a856858fe3a7473746c9044c317d3ba55", size = 524993, upload-time = "2026-09-06T20:31:58.055Z" }
+wheels = [
+ { url = "https://files.pythonhosted.org/packages/f5/4d/6b8460462012a52075ba97932197f8c141be71b8e5f7e918af08ee73424b/cattrs-26.2.0-py3-none-any.whl", hash = "sha256:c680f17cd1df3a1c038ad1db8d90a9c595568cc54c4e8098e7ceb8d7b9bd826b", size = 74830, upload-time = "2026-09-06T20:31:56.327Z" },
+]
+
[[package]]
name = "celery"
version = "5.6.3"
@@ -4043,6 +4057,21 @@ wheels = [
{ url = "https://files.pythonhosted.org/packages/0c/29/0348de65b8cc732daa3e33e67806420b2ae89bdce2b04af740289c5c6c8c/loguru-0.7.3-py3-none-any.whl", hash = "sha256:31a33c10c8e1e10422bfd431aeb5d351c7cf7fa671e3c4df004162264b28220c", size = 61595, upload-time = "2024-12-06T11:20:54.538Z" },
]
+[[package]]
+name = "looker-sdk"
+version = "26.12.0"
+source = { registry = "https://pypi.org/simple" }
+dependencies = [
+ { name = "attrs" },
+ { name = "cattrs" },
+ { name = "requests" },
+ { name = "typing-extensions" },
+]
+sdist = { url = "https://files.pythonhosted.org/packages/7c/b2/2e2aba48b687a0237bf1d84d0e3ea99c433232b5ae662369ec8439793987/looker_sdk-26.12.0.tar.gz", hash = "sha256:4dd327927b541e322de8bee1820cf1ce212c1883dc51b8bc098efe12c314e3f5", size = 240874, upload-time = "2026-07-13T20:37:32.284Z" }
+wheels = [
+ { url = "https://files.pythonhosted.org/packages/ea/a4/94b4c13e8ddc705b0607287f4e778a1cb54aab00d330350dd85dab3310a6/looker_sdk-26.12.0-py3-none-any.whl", hash = "sha256:c765f95c01ceaf3c162a779c4794dfaeb8696b4e1e16a81fb8bb1f253ce74b57", size = 262578, upload-time = "2026-07-13T20:37:30.791Z" },
+]
+
[[package]]
name = "lxml"
version = "6.1.3"
@@ -8344,6 +8373,9 @@ ingest-arrow = [
ingest-git = [
{ name = "gitpython" },
]
+ingest-looker = [
+ { name = "looker-sdk" },
+]
ingest-parquet = [
{ name = "pyarrow" },
]
@@ -8542,6 +8574,7 @@ requires-dist = [
{ name = "librosa", marker = "extra == 'media'", specifier = ">=0.9.0" },
{ name = "litellm", marker = "extra == 'llm-litellm'", specifier = ">=1.83.9" },
{ name = "loguru", specifier = ">=0.7.3" },
+ { name = "looker-sdk", marker = "extra == 'ingest-looker'", specifier = ">=24.0.0" },
{ name = "lxml", marker = "extra == 'documents'", specifier = ">=6.1.1" },
{ name = "matplotlib", marker = "extra == 'viz'", specifier = ">=3.9.4" },
{ name = "neo4j", marker = "extra == 'graph-amazon-neptune'", specifier = ">=5.0.0" },
@@ -8625,7 +8658,7 @@ requires-dist = [
{ name = "weaviate-client", marker = "extra == 'vectorstore-weaviate'", specifier = ">=4.0.0" },
{ name = "websockets", marker = "extra == 'explorer'", specifier = ">=15.0.1" },
]
-provides-extras = ["llm-openai", "llm-groq", "llm-gemini", "llm-anthropic", "llm-ollama", "llm-deepseek", "llm-novita", "llm-litellm", "llm-instructor", "llm-all", "documents", "parse-docling", "parse-pdf", "shacl", "db-snowflake", "db-databricks", "db-arrow", "db-salesforce", "db-redshift", "db-bigquery", "ingest-parquet", "ingest-arrow", "ingest-sap", "ingest-airflow", "ingest-git", "db-all", "models-huggingface", "embeddings-local", "nlp-spacy", "graph-neo4j", "graph-falkordb", "graph-amazon-neptune", "graph-apache-age", "graph-embeddings", "graph-all", "tripletstore-oxigraph", "vectorstore-faiss", "vectorstore-qdrant", "vectorstore-weaviate", "vectorstore-pinecone", "vectorstore-milvus", "vectorstore-pgvector", "vectorstore-sqlite", "vectorstore-all", "infra", "cloud", "monitoring", "viz", "media", "gpu", "agno", "langchain", "google-adk", "open-ontologies", "watch", "split-tiktoken", "split-community", "split-topic", "split-all", "explorer", "explorer-lite", "all"]
+provides-extras = ["llm-openai", "llm-groq", "llm-gemini", "llm-anthropic", "llm-ollama", "llm-deepseek", "llm-novita", "llm-litellm", "llm-instructor", "llm-all", "documents", "parse-docling", "parse-pdf", "shacl", "db-snowflake", "db-databricks", "db-arrow", "db-salesforce", "db-redshift", "db-bigquery", "ingest-looker", "ingest-parquet", "ingest-arrow", "ingest-sap", "ingest-airflow", "ingest-git", "db-all", "models-huggingface", "embeddings-local", "nlp-spacy", "graph-neo4j", "graph-falkordb", "graph-amazon-neptune", "graph-apache-age", "graph-embeddings", "graph-all", "tripletstore-oxigraph", "vectorstore-faiss", "vectorstore-qdrant", "vectorstore-weaviate", "vectorstore-pinecone", "vectorstore-milvus", "vectorstore-pgvector", "vectorstore-sqlite", "vectorstore-all", "infra", "cloud", "monitoring", "viz", "media", "gpu", "agno", "langchain", "google-adk", "open-ontologies", "watch", "split-tiktoken", "split-community", "split-topic", "split-all", "explorer", "explorer-lite", "all"]
[package.metadata.requires-dev]
dev = [