Graph-Native Infrastructure for Context and Accountable AI Systems
Python agent-memoryaiai-governanceai-infrastructureartificial-intelligencecontext-engineeringcontext-graphsdata-engineeringdecision-intelligencedeveloper-toolsexplainable-aigenerative-aigraph-ragknowledge-graphllmontologyprovenancepythonreasoningsemantic-search
feat(ingest): add Looker ingestor (#1595)
Som Samantray9bea6abeac61
diff --git a/docs/docs.json b/docs/docs.jsonindex 2c5da4c3..8d0fda41 100644--- a/docs/docs.json+++ b/docs/docs.json@@ -111,7 +111,8 @@"integrations/salesforce","integrations/airflow","integrations/redshift",- "integrations/powerbi"+ "integrations/powerbi",+ "integrations/looker"]},{diff --git a/docs/integrations/looker.md b/docs/integrations/looker.mdnew file mode 100644index 00000000..64fbb504--- /dev/null+++ b/docs/integrations/looker.md@@ -0,0 +1,288 @@+---+title: "Looker Integration"+description: "Ingest Looker Looks, Dashboards, LookML models, Folders, and Projects metadata into Semantica's KG pipeline."+icon: "chart-line"+---++> Extract Looker metadata into Semantica with API client credentials, using the official `looker-sdk` and the validated instance endpoint.+++## Installation++```bash+# Install with Looker support+pip install "semantica[ingest-looker]"++# Or install the connector separately+pip install looker-sdk>=24.0.0+```++`looker-sdk` is an optional dependency. A plain `pip install semantica` never pulls it in, and `import semantica.ingest` never loads it eagerly — the SDK is imported only when you first use `LookerConnector` or `LookerIngestor`. `LookerData` is importable without the SDK, so type annotations and data classes keep working in an SDK-free environment.++<Note>+This connector ingests Looker metadata only — Looks, Dashboards, LookML models (with their Explores), Folders, and Projects. Query result rows, scheduled plans, users, and groups are outside the current scope of this integration.+</Note>+++## Basic Usage++```python+import os+from semantica.ingest import LookerIngestor++with LookerIngestor(+ base_url=os.getenv("LOOKERSDK_BASE_URL"), # e.g. https://looker.example.com+ client_id=os.getenv("LOOKERSDK_CLIENT_ID"),+ client_secret=os.getenv("LOOKERSDK_CLIENT_SECRET"),+) as ingestor:+ looks = ingestor.ingest_looks()+ print(f"Retrieved {looks.row_count} looks — columns: {looks.columns}")++ documents = ingestor.export_as_documents(looks)+```++<Note>+`LookerIngestor` opens one authenticated session on entry and closes it on exit, so every call inside the `with` block reuses the validated connection. Standalone calls (without `with`) open and close a transient connection per call.+</Note>++<Tip>+Use environment variables (or a `.env` file with `python-dotenv`) to keep credentials out of source code. `LookerIngestor()` with no arguments reads from the `LOOKERSDK_*` environment variables automatically.+</Tip>+++## Authentication++<Tabs>+ <Tab title="Client ID / Client Secret">+ Looker API credentials are the `client_id` and `client_secret` pair that+ Looker issues for API3 access (Admin → Users → API Keys). Pass them+ explicitly, or let the connector read the `LOOKERSDK_CLIENT_ID` and+ `LOOKERSDK_CLIENT_SECRET` environment variables:++ ```python+ import os+ from semantica.ingest import LookerIngestor++ ingestor = LookerIngestor(+ base_url=os.getenv("LOOKERSDK_BASE_URL"), # must be https+ client_id=os.getenv("LOOKERSDK_CLIENT_ID"),+ client_secret=os.getenv("LOOKERSDK_CLIENT_SECRET"),+ )+ ```++ Required environment variables:++ ```bash+ export LOOKERSDK_BASE_URL="https://looker.example.com"+ export LOOKERSDK_CLIENT_ID="your-client-id"+ export LOOKERSDK_CLIENT_SECRET="your-client-secret"+ ```++ The endpoint must use `https`. A non-`https` scheme (for example+ `http://looker.internal`) is rejected with a `ValidationError` before any+ client is constructed.+ </Tab>+ <Tab title="Existing looker.ini">+ The connector also accepts a Looker SDK configuration file. Point+ `config_file` at the file and, optionally, select a `section` inside it:++ ```python+ from semantica.ingest import LookerIngestor++ ingestor = LookerIngestor(+ config_file="looker.ini",+ section="Looker",+ )+ ```++ `config_file` defaults to `"looker.ini"`. Values already present in the+ file are used when the matching constructor argument or environment+ variable is absent.+ </Tab>+</Tabs>++<Warning>+`allow_private_ips=True` is required to connect to a private, loopback, or+link-local endpoint, and it also relaxes the `https` requirement so a+non-`https` scheme is accepted. Leave it off unless you are deliberately+targeting an internal instance.+</Warning>++If an explicit `base_url` conflicts with `LOOKERSDK_BASE_URL` or the value in+the config file, the connector raises a `ValidationError` rather than+connecting to a host you did not validate.+++## Environment Variables++All connection parameters have `LOOKERSDK_*` environment-variable fallbacks,+resolved through the SDK's own `ApiSettings`. Explicit constructor values+always take precedence.++| Variable | Parameter | Default |+|---|---|---|+| `LOOKERSDK_BASE_URL` | `base_url` | — |+| `LOOKERSDK_CLIENT_ID` | `client_id` | — |+| `LOOKERSDK_CLIENT_SECRET` | `client_secret` | — |++`config_file` and `section` are constructor parameters rather than environment+variables:++| Parameter | Description | Default |+|---|---|---|+| `config_file` | Path to an existing `looker.ini` file | `"looker.ini"` |+| `section` | Section name inside `config_file` | — |+| `allow_private_ips` | Allow private endpoints and non-`https` schemes | `False` |+++## Metadata Ingestion++Each read returns a `LookerData` object with `data` (normalized metadata+dictionaries), `row_count`, `columns`, `content_type`, `base_url`, `metadata`,+and `ingested_at`.++| Method | SDK read | `content_type` |+|---|---|---|+| `ingest_looks()` | `all_looks` | `"look"` |+| `ingest_dashboards()` | `all_dashboards` | `"dashboard"` |+| `ingest_lookml_models()` | `all_lookml_models` | `"lookml_model"` |+| `ingest_folders()` | `all_folders` | `"folder"` |+| `ingest_projects()` | `all_projects` | `"project"` |++Reads use the SDK's `all_*` collection methods. Each method accepts optional+keyword arguments that are forwarded to the SDK call — `transport_options` and+`fields` for most reads, plus `limit`/`offset` for `ingest_lookml_models()`:++```python+looks = ingestor.ingest_looks(fields="id,title,description,model")+models = ingestor.ingest_lookml_models(limit=100)+```++<Note>+This is a metadata-only connector. Dashboards are read as `DashboardBase`+records without tiles, so no query result rows are fetched or stored.+</Note>++### LookML models and Explores++LookML models are returned with their Explores nested inside the parent model+record. Explores carry no parent reference of their own, so the parent+`project_name`/`name` pair is composed into each nested identifier:++```python+models = ingestor.ingest_lookml_models()++for model in models.data:+ print(f"{model['project_name']}.{model['name']}")+ for explore in model["explores"]:+ print(f" {explore['name']}")+```+++## Export as Semantica Documents++`export_as_documents(data)` converts a `LookerData` result into the+`{"id", "text", "metadata"}` document shape that `GraphBuilder` consumes:++```python+documents = ingestor.export_as_documents(looks)++print(f"Created {len(documents)} documents")+# Each document:+# {+# "id": "looker:look:42",+# "text": "Monthly Revenue Revenue by region orders jdoe",+# "metadata": {+# "source": "looker",+# "content_type": "look",+# "row_data": { ... allowlisted Look metadata ... },+# "look_id": 42+# }+# }+```++Document `id` values are namespaced per content type so they stay unique+across collections: `looker:look:<id>`, `looker:dashboard:<id>`,+`looker:folder:<id>`, `looker:project:<id>`, and+`looker:lookml_model:<project>.<name>` for models. Every document's+`metadata["source"]` is `"looker"`, and `metadata["content_type"]` records+which collection it came from. The full normalized record is preserved under+`metadata["row_data"]`.++LookML model documents additionally nest their Explores, each with its own+namespaced id and parent model reference:++```python+# {+# "id": "looker:lookml_model:ecommerce.orders",+# "text": "Orders orders ecommerce Order Items order_items",+# "metadata": {+# "source": "looker",+# "content_type": "lookml_model",+# "row_data": { ... allowlisted LookML model metadata ... },+# "project_name": "ecommerce",+# "model_name": "orders",+# "explores": [+# {+# "id": "looker:lookml_model:ecommerce.orders:explore:order_items",+# "name": "order_items",+# "label": "Order Items",+# "group_label": "Order Items",+# "description": None,+# "hidden": False,+# "parent_model": "ecommerce.orders"+# }+# ],+# "explore_count": 1+# }+# }+```++Pass the documents directly to `GraphBuilder`:++```python+from semantica.kg import GraphBuilder++builder = GraphBuilder()+graph = builder.build(documents)+print(f"Entities: {graph['metadata']['num_entities']}")+```++Exported documents are JSON-serializable: nested SDK model objects become+plain dictionaries or lists, and `datetime` values become ISO 8601 strings.+++## Security++The connector validates the user-supplied endpoint before any client is+constructed, then binds the client to that exact validated `base_url` — if the+SDK resolves a different host from the environment or config file, it fails+closed instead of connecting.++It also routes every outbound request through the same SSRF validation the+repository's other connectors use, so the OAuth login and each metadata read+are checked individually rather than only at construction. Connections are+pinned to the addresses resolved during validation, so a DNS rebind between+validation and dial cannot redirect them. Requests must use `https` unless+`allow_private_ips` is explicitly enabled, TLS certificate verification is+forced on (so `LOOKERSDK_VERIFY_SSL` or a `looker.ini` cannot downgrade it),+redirects are not followed, and proxy environment variables are not trusted.+The SDK's error-documentation lookup (a separate, unguarded `requests.get` it+performs on a non-2xx response) is disabled so a failed request cannot open a+second egress path.++Record normalization projects every SDK object through an explicit allowlist of+retained fields. Secret-adjacent fields such as `Project.git_password`,+`Project.deploy_secret`, `Dashboard.password`, `Dashboard.pdt_password`, and+`LookmlModel.device_token` are dropped, and any `user:password@` userinfo in+`Project.git_remote_url` is scrubbed (a URL whose userinfo cannot be rewritten+safely is redacted wholesale). Documents carry no `base_url` or credentials.+++## See Also++- [Ingest Module](../reference/ingest) — Full `LookerIngestor` reference and all other ingestors.+- [Snowflake Integration](/integrations/snowflake) — SQL data warehouse connector with similar metadata ingestion.+- [Redshift Integration](/integrations/redshift) — Warehouse connector for tabular and query-based ingestion.+- [Installation](../installation) — All optional dependency extras.+- [Knowledge Graph](../reference/kg) — Build a knowledge graph from ingested Looker metadata.diff --git a/pyproject.toml b/pyproject.tomlindex b7383fe9..6024b1bd 100644--- a/pyproject.toml+++ b/pyproject.toml@@ -128,6 +128,7 @@ db-arrow = ["pyarrow>=24.0.0"]db-salesforce = ["simple-salesforce>=1.12.0"]db-redshift = ["redshift-connector>=2.0.0"]db-bigquery = ["google-cloud-bigquery>=3.0.0"]+ingest-looker = ["looker-sdk>=24.0.0"]ingest-parquet = ["pyarrow>=24.0.0"]ingest-arrow = ["pyarrow>=24.0.0"]ingest-sap = ["requests>=2.28.0"]diff --git a/semantica/ingest/__init__.py b/semantica/ingest/__init__.pyindex 93ad67a9..4440750a 100644--- a/semantica/ingest/__init__.py+++ b/semantica/ingest/__init__.py@@ -273,6 +273,10 @@ _LAZY_EXPORTS: Dict[str, Tuple[str, str]] = {"BigQueryIngestor": (".bigquery_ingestor", "BigQueryIngestor"),"BigQueryData": (".bigquery_ingestor", "BigQueryData"),"BigQueryConnector": (".bigquery_ingestor", "BigQueryConnector"),+ # Looker ingestion+ "LookerIngestor": (".looker_ingestor", "LookerIngestor"),+ "LookerData": (".looker_ingestor", "LookerData"),+ "LookerConnector": (".looker_ingestor", "LookerConnector"),}_OPTIONAL_DEPENDENCY_MESSAGES = {@@ -328,6 +332,10 @@ _OPTIONAL_DEPENDENCY_MESSAGES = {"BigQuery ingestion requires optional dependency 'google-cloud-bigquery'. ""Install it with: pip install 'semantica[db-bigquery]'"),+ ".looker_ingestor": (+ "Looker ingestion requires optional dependency 'looker-sdk'. "+ "Install it with: pip install 'semantica[ingest-looker]'"+ ),}@@ -354,6 +362,7 @@ def __getattr__(name: str) -> Any:"lxml","redshift_connector","google",+ "looker_sdk",))):@@ -414,6 +423,15 @@ def __getattr__(name: str) -> Any:if message:raise ImportError(message)+ if module_name == ".looker_ingestor" and name in {+ "LookerIngestor",+ "LookerConnector",+ }:+ if not getattr(module, "LOOKER_AVAILABLE", True):+ message = _OPTIONAL_DEPENDENCY_MESSAGES.get(module_name)+ if message:+ raise ImportError(message)+value = getattr(module, attr_name)globals()[name] = valuereturn value@@ -520,6 +538,10 @@ __all__ = ["BigQueryIngestor","BigQueryData","BigQueryConnector",+ # Looker ingestion+ "LookerIngestor",+ "LookerData",+ "LookerConnector",# Registry and Methods"MethodRegistry","method_registry",diff --git a/semantica/ingest/looker_ingestor.py b/semantica/ingest/looker_ingestor.pynew file mode 100644index 00000000..9d9661cc--- /dev/null+++ b/semantica/ingest/looker_ingestor.py@@ -0,0 +1,1707 @@+"""+Looker Ingestion Module++This module provides Looker metadata ingestion for the Semantica+framework, enabling extraction of Looks, Dashboards, LookML models (with+their Explores), Folders, and Projects via the official ``looker-sdk``+4.0 API.++Design notes+------------+Optional dependency+ ``looker_sdk`` is imported inside a guard. The module always imports+ cleanly; ``LookerData`` is usable without the SDK, and+ :class:`LookerConnector` raises a clear ``ImportError`` naming the+ ``ingest-looker`` extra at instantiation time.++Validate the endpoint once and guard every request+ ``looker_sdk.init40`` builds ``RequestsTransport`` internally, so the+ repository's ``request_with_ssrf_guard`` hook cannot wrap the SDK's+ calls. The connector resolves the effective ``base_url`` from a+ constructed ``ApiSettings`` object, validates it, and passes that+ *same* object to ``init40``. The binding mechanism is the+ ``read_config`` override on that object, which the SDK consults for+ every login. ``connect()`` then mounts a validating adapter on the+ SDK session — the choke point every API request (including the lazy+ OAuth login) passes through — so each outbound URL is checked with+ ``validate_url_for_request`` before it is sent. The controls applied+ are:++ 1. ``https`` is required unless ``allow_private_ips`` is enabled;+ 2. TLS verification is forced on, so ``LOOKERSDK_VERIFY_SSL`` or the+ ini file cannot downgrade the validated connection;+ 3. the SDK session's redirect cap is set to zero;+ 4. proxy environment trust is disabled;+ 5. every request URL is re-validated by the mounted guard adapter;+ 6. connections are pinned to the addresses resolved during validation,+ closing the DNS-rebinding window between validation and dial.++ The guard covers the SDK's own session. The SDK's error-documentation+ helper is a second, separate egress path (a bare ``requests.get`` with+ no validation, timeout, redirect cap, or proxy opt-out), so+ ``connect()`` also disables it explicitly; see+ :meth:`LookerConnector._disable_error_doc_lookup`.++ A post-construction check also compares the client's resolved+ ``base_url`` with the validated one. The current SDK returns the same+ ``ApiSettings`` object, so that check guards against future drift+ rather than being a security control.++Credentials are injected through the ``ApiSettings`` object rather than+by mutating ``os.environ``, so no secret becomes process-global.++Allowlist redaction+ Every SDK record is projected through an explicit allowlist of+ retained fields. ``Project.git_password``, ``Project.deploy_secret``,+ ``Project.git_remote_url`` userinfo, ``Dashboard.password``,+ ``Dashboard.pdt_password``, and ``LookmlModel.device_token`` can+ therefore never reach :class:`LookerData`, an exported document, or a+ log record.++Main Classes:+ - LookerConnector: Manages the ``looker_sdk`` client lifecycle+ - LookerData: Dataclass representing ingested Looker metadata+ - LookerIngestor: Orchestrates metadata reads and document export++Optional Dependency:+ Install via the ``ingest-looker`` extra::++ pip install "semantica[ingest-looker]"++Example Usage::++ >>> import os+ >>> from semantica.ingest.looker_ingestor import LookerIngestor+ >>> with LookerIngestor(+ ... base_url=os.getenv("LOOKERSDK_BASE_URL"),+ ... client_id=os.getenv("LOOKERSDK_CLIENT_ID"),+ ... client_secret=os.getenv("LOOKERSDK_CLIENT_SECRET"),+ ... ) as ingestor:+ ... looks = ingestor.ingest_looks()+ ... documents = ingestor.export_as_documents(looks)++Author: Semantica Contributors+License: MIT+"""++from __future__ import annotations++import enum+import os+from dataclasses import dataclass, field+from datetime import date, datetime+from typing import Any, Dict, List, Mapping, Optional, Sequence+from urllib.parse import urlparse++import requests++from ..utils.exceptions import ProcessingError, ValidationError+from ..utils.logging import get_logger+from ..utils.progress_tracker import get_progress_tracker+from .ssrf import (+ _format_host_header,+ _make_pinned_adapter,+ _resolve_pinned_ips,+ parse_bool,+ validate_url_for_request,+)++# ---------------------------------------------------------------------------+# Optional-dependency guard — mirrors the pattern in salesforce_ingestor.+# The module always imports cleanly; the sentinel fires at instantiation+# time inside LookerConnector.__init__ so that:+# - ``from semantica.ingest.looker_ingestor import LookerData`` works+# without the SDK, and+# - ``LookerConnector(...)`` raises a clear ImportError when it is absent.+# ---------------------------------------------------------------------------+try:+ import looker_sdk+ from looker_sdk.rtl import api_settings++ LOOKER_AVAILABLE = True+except (ImportError, OSError):+ looker_sdk = None # type: ignore[assignment]+ api_settings = None # type: ignore[assignment]+ LOOKER_AVAILABLE = False++__all__ = [+ "LookerData",+ "LookerConnector",+ "LookerIngestor",+]++_logger = get_logger("looker_ingestor")++# Exact install hint required by R8.+_MISSING_SDK_MESSAGE = (+ "looker-sdk is required for the Looker connector. "+ 'Install it with: pip install "semantica[ingest-looker]"'+)++# Environment prefix read by the SDK's own ApiSettings.+_ENV_PREFIX = "LOOKERSDK"+++def _normalize_url(value: str) -> str:+ """Return *value* with a trailing slash removed for comparison.++ Args:+ value: URL to normalize.++ Returns:+ The URL without a trailing slash.+ """+ return (value or "").strip().rstrip("/")+++# Returned in place of a remote URL that cannot be safely rewritten: it may+# be ``user:password`` userinfo whose password contains a ``/``.+_REDACTED_URL = "[redacted-looker-remote-url]"+++def _authority_is_host(authority: str) -> bool:+ """Return True when *authority* is unambiguously a host, not userinfo.++ A ``host:port`` authority is only distinguishable from+ ``user:password`` by the port being numeric; a bracketed ``[...]``+ literal is always an IPv6 host.+ """+ if authority.startswith("["):+ return True+ _, separator, port = authority.rpartition(":")+ return bool(separator) and port.isdigit()+++def _scrub_url_userinfo(value: str) -> str:+ """Strip ``user:password@`` userinfo from *value*.++ ``Project.git_remote_url`` may embed credentials. Keep the host and+ path so the metadata stays useful, but never the userinfo.++ The deciding ``@`` is the one inside the authority section, not+ necessarily the last ``@`` in the string: a path or query may contain+ an ``@`` of its own (``https://user:[email protected]/path@file``). The+ authority is therefore delimited first — an optional ``scheme://``,+ then everything up to the first ``/``, ``?`` or ``#`` — and only that+ slice is rewritten.++ A value whose authority has no ``@`` but whose text contains one is+ ambiguous: it may be ``https://user:pa/ss@host/repo.git``, where the+ password contains the ``/`` that ends the authority. That cannot be+ rewritten safely, so it fails closed to :data:`_REDACTED_URL` — unless+ the authority is a plain host or a numeric ``host:port``+ (``https://example.com/path@file``), where the ``@`` is ordinary+ content.++ Args:+ value: Remote URL that may contain userinfo.++ Returns:+ The URL with any userinfo removed, or :data:`_REDACTED_URL` when the+ safe rewrite is ambiguous.+ """+ if "@" not in value:+ return value++ scheme_index = value.find("://")+ authority_start = 0 if scheme_index == -1 else scheme_index + 3+ authority_end = len(value)+ for separator in ("/", "?", "#"):+ index = value.find(separator, authority_start)+ if index != -1:+ authority_end = min(authority_end, index)++ authority = value[authority_start:authority_end]+ if "@" in authority:+ return (+ value[:authority_start]+ + authority.rpartition("@")[2]+ + value[authority_end:]+ )+ # No '@' in the authority, yet the value has one. Ordinary content+ # after a host is left alone; the ambiguous `user:password` shape fails+ # closed.+ if ":" not in authority or _authority_is_host(authority):+ return value+ return _REDACTED_URL+++# ---------------------------------------------------------------------------+# LookerData+# ---------------------------------------------------------------------------+++@dataclass+class LookerData:+ """Metadata records returned from a Looker API collection.++ Attributes:+ data: List of normalized metadata dictionaries.+ row_count: Number of records in ``data``.+ columns: Ordered list of field names present in ``data``.+ content_type: One of ``look``, ``dashboard``, ``lookml_model``,+ ``folder``, or ``project``.+ base_url: Validated Looker endpoint the data was read from.+ metadata: Arbitrary extra metadata (e.g. the SDK endpoint used).+ ingested_at: Timestamp recorded when this object was created.+ """++ data: List[Dict[str, Any]]+ row_count: int+ columns: List[str]+ content_type: str+ base_url: Optional[str] = None+ metadata: Dict[str, Any] = field(default_factory=dict)+ ingested_at: datetime = field(default_factory=datetime.now)+++# ---------------------------------------------------------------------------+# Field allowlists — explicit "retained fields" contract (R11)+# ---------------------------------------------------------------------------+# Only keys listed here survive normalization. Secret-adjacent SDK fields+# (git_password, deploy_secret, password, pdt_password, device_token) are+# deliberately absent, so they cannot reach LookerData, documents, or logs.++# Values that reach the normalizer as plain mappings never pass through a+# record allowlist, so secret-named keys are dropped explicitly as well.+# Named arguments of LookerConnector: a config dict carrying any of these+# must not be re-forwarded as a keyword or the call raises TypeError.+_CONNECTOR_ARGUMENT_NAMES = frozenset(+ {+ "allow_private_ips",+ "base_url",+ "client_id",+ "client_secret",+ "config_file",+ "section",+ }+)++_SECRET_FIELD_NAMES = frozenset(+ {+ "deploy_secret",+ "device_token",+ "git_password",+ "password",+ "pdt_password",+ }+)++# ``all_lookml_models()`` nests ``LookmlModelNavExplore`` records: the only+# fields the navigation object carries are the ones below. ``view_name``,+# ``joins``, and ``fields`` belong to ``LookmlModelExplore``, which only the+# deferred ``lookml_model_explore`` endpoint returns and this connector never+# calls.+_EXPLORE_NAV_FIELDS = frozenset(+ {+ "description",+ "group_label",+ "hidden",+ "label",+ "name",+ }+)++_NESTED_EXTRA_FIELDS = frozenset(+ {+ "certification_status",+ "joins",+ "condition",+ "dependency_status",+ "details",+ "error_pos",+ "field",+ "field_error",+ "git_branch",+ "git_head",+ "git_status",+ "label",+ "lookml_type",+ "measure_types",+ "message",+ "project_id",+ "user_attribute",+ "value",+ "workspace_id",+ }+)++_RETAINED_FIELDS: Dict[str, frozenset] = {+ "look": frozenset(+ {+ "can",+ "certification_metadata",+ "content_favorite_id",+ "content_metadata_id",+ "created_at",+ "deleted",+ "deleted_at",+ "deleter_id",+ "description",+ "embed_url",+ "excel_file_url",+ "favorite_count",+ "folder",+ "folder_id",+ "google_spreadsheet_formula",+ "id",+ "image_embed_url",+ "is_owner_disabled",+ "is_run_on_load",+ "last_accessed_at",+ "last_updater_id",+ "last_viewed_at",+ "model",+ "public",+ "public_slug",+ "public_url",+ "query_id",+ "short_url",+ "title",+ "updated_at",+ "usage_count",+ "user_id",+ "user_name",+ "view_count",+ }+ ),+ "dashboard": frozenset(+ {+ "alert_sync_with_dashboard_filter_enabled",+ "appearance",+ "background_color",+ "can",+ "certification_metadata",+ "chat_enabled",+ "content_favorite_id",+ "content_metadata_id",+ "created_at",+ "crossfilter_enabled",+ "deleted",+ "deleted_at",+ "deleter_id",+ "description",+ "download_settings",+ "edit_uri",+ "enable_viz_full_screen",+ "favorite_count",+ "filters_bar_collapsed",+ "filters_location_top",+ "folder",+ "folder_id",+ "hidden",+ "id",+ "is_owner_disabled",+ "last_accessed_at",+ "last_updater_id",+ "last_updater_name",+ "last_viewed_at",+ "load_configuration",+ "lookml_link_id",+ "model",+ "preferred_viewer",+ "preserve_desktop_layout",+ "query_timezone",+ "readonly",+ "refresh_interval",+ "refresh_interval_to_i",+ "show_filters_bar",+ "show_title",+ "slug",+ "text_tile_text_color",+ "tile_background_color",+ "tile_text_color",+ "title",+ "title_color",+ "updated_at",+ "url",+ "usage_count",+ "user_id",+ "user_name",+ "view_count",+ }+ ),+ # ``device_token`` is intentionally absent (R11).+ "lookml_model": frozenset(+ {+ "allowed_db_connection_names",+ "can",+ "explores",+ "has_content",+ "label",+ "name",+ "project_name",+ "unlimited_db_connections",+ }+ ),+ "folder": frozenset(+ {+ "can",+ "child_count",+ "content_metadata_id",+ "created_at",+ "creator_id",+ "dashboards",+ "external_id",+ "id",+ "is_embed",+ "is_embed_shared_root",+ "is_embed_users_root",+ "is_personal",+ "is_personal_descendant",+ "is_shared_root",+ "is_users_root",+ "looks",+ "name",+ "parent_id",+ }+ ),+ # ``git_password``, ``deploy_secret``, and the password user-attribute+ # fields are intentionally absent (R11). ``git_remote_url`` is kept+ # but has its userinfo scrubbed during normalization.+ "project": frozenset(+ {+ "allow_warnings",+ "can",+ "dependency_status",+ "git_application_server_http_port",+ "git_application_server_http_scheme",+ "git_production_branch_name",+ "git_remote_url",+ "git_release_mgmt_enabled",+ "git_service_name",+ "git_username",+ "git_username_user_attribute",+ "has_production_counterpart",+ "id",+ "is_example",+ "is_git_dev_locked",+ "name",+ "pull_request_mode",+ "unset_deploy_secret",+ "use_git_cookie_auth",+ "uses_git",+ "validation_required",+ }+ ),+}++# Nested SDK models (Explores, folders, and so on) are filtered against the+# union of every top-level allowlist plus the nested-only field sets.+_NESTED_RETAINED_FIELDS = frozenset(+ set().union(+ *(_RETAINED_FIELDS.values()),+ _EXPLORE_NAV_FIELDS,+ _NESTED_EXTRA_FIELDS,+ )+)++# Human-readable field selection used to compose a document's ``text``.+_TEXT_FIELDS: Dict[str, Sequence[str]] = {+ "look": ("title", "description", "model", "user_name"),+ "dashboard": ("title", "description", "model", "user_name"),+ "lookml_model": ("label", "name", "project_name"),+ "folder": ("name",),+ "project": ("name", "git_username"),+}++# Flattened identifier exposed in document metadata, per content type.+_ID_METADATA_KEYS: Dict[str, str] = {+ "look": "look_id",+ "dashboard": "dashboard_id",+ "folder": "folder_id",+ "project": "project_id",+}+++# ---------------------------------------------------------------------------+# SSRF guard adapter — mounted on the SDK session by connect()+# ---------------------------------------------------------------------------+++class _SSRFGuardAdapter(requests.adapters.HTTPAdapter):+ """Validate every request URL, then send it through the pinned delegate.++ ``looker_sdk.init40`` builds its own transport inside the SDK, so the+ repository's ``request_with_ssrf_guard`` helper cannot wrap the SDK's+ calls. Mounting this adapter on the SDK session puts the same+ per-request ``validate_url_for_request`` check on the single choke+ point every request — the OAuth login and all five metadata reads —+ passes through.++ When addresses were resolved for pinning, ``send`` delegates to the+ repository's pinned adapter so the connection still targets those+ addresses and the hostname still supplies TLS identity. The+ ``_semantica_pinned`` marker is exposed in that case.+ """++ def __init__(+ self,+ *,+ allow_private_ips: bool,+ pinned_adapter: Optional[requests.adapters.HTTPAdapter] = None,+ ) -> None:+ super().__init__()+ self._allow_private_ips = allow_private_ips+ self._pinned_adapter = pinned_adapter++ @property+ def _semantica_pinned(self) -> bool:+ """Whether a pinning delegate is mounted beneath this adapter.++ Derived from the delegate itself rather than cached, so the marker+ read by :func:`~semantica.ingest.ssrf._apply_connection_pin` cannot+ drift from the state it describes.+ """+ return self._pinned_adapter is not None++ def send(self, request: Any, **kwargs: Any) -> Any:+ """Validate *request* and send it through the pinned delegate."""+ validate_url_for_request(request.url, allow_private_ips=self._allow_private_ips)+ if self._pinned_adapter is not None:+ return self._pinned_adapter.send(request, **kwargs)+ return super().send(request, **kwargs)++ def close(self) -> None:+ """Close the pinned delegate, then this adapter."""+ if self._pinned_adapter is not None:+ self._pinned_adapter.close()+ super().close()+++# ---------------------------------------------------------------------------+# LookerConnector+# ---------------------------------------------------------------------------+++class LookerConnector:+ """Manages the ``looker_sdk`` client lifecycle.++ Responsibilities:++ * Resolves credentials from constructor arguments, falling back to+ the ``LOOKERSDK_BASE_URL``, ``LOOKERSDK_CLIENT_ID``, and+ ``LOOKERSDK_CLIENT_SECRET`` environment variables (and, via the+ SDK's own ``ApiSettings``, a ``looker.ini`` file).+ * Resolves the effective endpoint once, validates the scheme and the+ SSRF contract, and binds the client to that exact value (KTD2/KTD4).+ * Disables redirect following and proxy environment trust on the SDK+ session and mounts a per-request SSRF guard that also pins the+ connection; never mutates ``os.environ``.+ * Exposes :meth:`connect`, :meth:`disconnect`, and+ :meth:`test_connection`, plus the context-manager protocol.++ Args:+ base_url: Looker instance URL. Defaults to+ ``LOOKERSDK_BASE_URL``.+ client_id: Looker API client id. Defaults to+ ``LOOKERSDK_CLIENT_ID``.+ client_secret: Looker API client secret. Defaults to+ ``LOOKERSDK_CLIENT_SECRET``.+ config_file: Path to a ``looker.ini`` file. Defaults to+ ``"looker.ini"``.+ section: Section name inside *config_file*.+ allow_private_ips: Opt into private/loopback/link-local endpoints+ and non-``https`` schemes. Defaults to ``False``.+ config: Optional extra configuration dict. ``allow_private_ips``+ is read from here when present, mirroring the SAP connector.+ **kwargs: Additional keyword arguments merged into ``config``.++ Raises:+ ImportError: When ``looker-sdk`` is not installed.+ ValidationError: When credentials are missing, the endpoint fails+ validation, or the environment conflicts with the explicit+ ``base_url``.+ """++ def __init__(+ self,+ base_url: Optional[str] = None,+ client_id: Optional[str] = None,+ client_secret: Optional[str] = None,+ config_file: Optional[str] = None,+ section: Optional[str] = None,+ allow_private_ips: bool = False,+ config: Optional[Dict[str, Any]] = None,+ **kwargs: Any,+ ) -> None:+ if not LOOKER_AVAILABLE:+ raise ImportError(_MISSING_SDK_MESSAGE)++ self.logger = _logger++ self.config: Dict[str, Any] = dict(config or {})+ self.config.update(kwargs)+ self.allow_private_ips = parse_bool(+ self.config.pop("allow_private_ips", allow_private_ips), default=False+ )++ # Credentials: explicit arguments win over environment variables.+ self._explicit_base_url: Optional[str] = base_url+ self.client_id: Optional[str] = client_id or os.getenv("LOOKERSDK_CLIENT_ID")+ self._client_secret: Optional[str] = client_secret or os.getenv(+ "LOOKERSDK_CLIENT_SECRET"+ )+ self.config_file: str = config_file or "looker.ini"+ self.section: Optional[str] = section++ self._client: Optional[Any] = None+ # Endpoint validated at construction time, reused so the SSRF host+ # check (which resolves DNS) runs at most once per connector.+ self._validated_base_url: Optional[str] = None++ # Fail fast on an explicit endpoint before any SDK object exists:+ # a conflicting environment value must never silently win (AE2),+ # and a bad scheme or blocked host must never reach the SDK.+ self._validate_explicit_endpoint()++ settings = self._build_settings()+ raw_config = self._raw_read_config(settings)++ # KTD4/AE2 — the SDK's ApiSettings lets the environment (and the+ # ini file) override the constructor. Refuse to proceed rather+ # than silently connecting somewhere the caller did not validate.+ configured_base_url = str(raw_config.get("base_url") or "")+ if self._explicit_base_url and configured_base_url:+ if _normalize_url(configured_base_url) != _normalize_url(+ self._explicit_base_url+ ):+ raise ValidationError(+ "Looker base_url from the environment or config file "+ f"({_scrub_url_userinfo(configured_base_url)!r}) conflicts "+ "with the explicit 'base_url' "+ f"({_scrub_url_userinfo(self._explicit_base_url)!r}); "+ "refusing to connect rather than binding the client to an "+ "unvalidated host."+ )++ effective_base_url = (+ self._explicit_base_url+ or configured_base_url+ or str(getattr(settings, "base_url", "") or "")+ )+ effective_client_id = self.client_id or str(raw_config.get("client_id") or "")+ effective_client_secret = self._client_secret or str(+ raw_config.get("client_secret") or ""+ )++ self.base_url: str = effective_base_url+ self.client_id = effective_client_id+ self._client_secret = effective_client_secret++ # ``looker_sdk`` re-reads settings for every login, so the merged+ # values must live on ``read_config`` — assigning attributes alone+ # would not reach the OAuth2 exchange.+ settings.base_url = effective_base_url+ # `verify_ssl` comes from LOOKERSDK_VERIFY_SSL / the ini file and would+ # otherwise let an environment detail silently turn the enforced https+ # requirement into an unverified connection carrying the client secret.+ if not self.allow_private_ips:+ settings.verify_ssl = True+ base_read_config = settings.read_config++ def _effective_read_config() -> Dict[str, Any]:+ data: Dict[str, Any] = dict(base_read_config())+ if effective_base_url:+ data["base_url"] = effective_base_url+ if effective_client_id:+ data["client_id"] = effective_client_id+ if effective_client_secret:+ data["client_secret"] = effective_client_secret+ if not self.allow_private_ips:+ data["verify_ssl"] = True+ return data++ settings.read_config = _effective_read_config # type: ignore[method-assign]+ self._settings = settings++ self._validate_auth()+ # An explicit endpoint was already validated above and always wins+ # the resolution order, so it is never validated twice.+ if self._validated_base_url is None:+ self._validated_base_url = self._validate_endpoint(effective_base_url)++ self.logger.debug(+ "Looker connector initialised (base_url=%s, allow_private_ips=%s)",+ self._validated_base_url,+ self.allow_private_ips,+ )++ # ------------------------------------------------------------------+ # Internal helpers+ # ------------------------------------------------------------------++ def _validate_explicit_endpoint(self) -> None:+ """Validate an explicitly supplied ``base_url`` before SDK use.++ Runs before any ``ApiSettings``/``init40`` work so a conflicting+ ``LOOKERSDK_BASE_URL`` (AE2) or an unsafe explicit endpoint fails+ immediately and without touching the SDK.++ Raises:+ ValidationError: When the environment conflicts with the+ explicit value, or the explicit value fails scheme/SSRF+ validation.+ """+ if not self._explicit_base_url:+ return++ env_base_url = os.getenv("LOOKERSDK_BASE_URL")+ if env_base_url and _normalize_url(env_base_url) != _normalize_url(+ self._explicit_base_url+ ):+ raise ValidationError(+ "LOOKERSDK_BASE_URL "+ f"({_scrub_url_userinfo(env_base_url)!r}) conflicts with the "+ f"explicit 'base_url' "+ f"({_scrub_url_userinfo(self._explicit_base_url)!r}); "+ "refusing to connect rather than binding the client to an "+ "unvalidated host."+ )++ self._validated_base_url = self._validate_endpoint(self._explicit_base_url)++ def _build_settings(self) -> Any:+ """Construct the SDK ``ApiSettings`` used to bind the client.++ Returns:+ A configured ``looker_sdk.rtl.api_settings.ApiSettings``.++ Raises:+ ValidationError: If an explicitly named config file is missing.+ ProcessingError: If settings cannot be constructed.+ """+ try:+ return api_settings.ApiSettings(+ filename=self.config_file,+ section=self.section,+ env_prefix=_ENV_PREFIX,+ )+ except FileNotFoundError as exc:+ raise ValidationError(+ f"Looker config file not found: {self.config_file!r}."+ ) from exc+ except ValidationError:+ raise+ except Exception as exc: # noqa: BLE001 - translated with type only+ self.logger.error(+ "Failed to load Looker SDK settings: %s", type(exc).__name__+ )+ raise ProcessingError(+ f"Failed to load Looker SDK settings: {type(exc).__name__}"+ ) from None++ @staticmethod+ def _raw_read_config(settings: Any) -> Dict[str, Any]:+ """Read the SDK's resolved config without applying our overrides."""+ try:+ data = settings.read_config()+ except Exception as exc: # noqa: BLE001 - config read is best-effort here+ # Without the config we cannot detect an endpoint conflict, so+ # surface the cause instead of failing later as "missing+ # credentials".+ _logger.debug("Could not read Looker SDK config: %s", type(exc).__name__)+ return {}+ if isinstance(data, Mapping):+ return dict(data)+ return {}++ def _validate_auth(self) -> None:+ """Raise ``ValidationError`` when required credentials are absent.++ Validates before any network call so failures are immediate and+ never leak partial state.++ Raises:+ ValidationError: When ``base_url``, ``client_id``, or+ ``client_secret`` is missing.+ """+ if not self.base_url:+ raise ValidationError(+ "Looker base_url is required. Provide via 'base_url' or the "+ "LOOKERSDK_BASE_URL environment variable."+ )+ if not self.client_id:+ raise ValidationError(+ "Looker client_id is required. Provide via 'client_id' or the "+ "LOOKERSDK_CLIENT_ID environment variable."+ )+ if not self._client_secret:+ raise ValidationError(+ "Looker client_secret is required. Provide via "+ "'client_secret' or the LOOKERSDK_CLIENT_SECRET environment "+ "variable."+ )++ def _validate_endpoint(self, base_url: str) -> str:+ """Validate scheme and SSRF constraints for *base_url*.++ Args:+ base_url: Effective endpoint resolved from kwargs, environment,+ or the ini file.++ Returns:+ The unchanged, validated endpoint.++ Raises:+ ValidationError: When the scheme is not ``https`` and+ ``allow_private_ips`` is off, or when the host is blocked.+ """+ parsed = urlparse(base_url)+ scheme = (parsed.scheme or "").lower()+ if scheme != "https" and not self.allow_private_ips:+ raise ValidationError(+ "Looker base_url must use https unless allow_private_ips is "+ f"enabled. Got scheme {parsed.scheme!r}."+ )+ if parsed.username or parsed.password:+ raise ValidationError(+ "Looker base_url must not embed userinfo credentials; pass "+ "them as client_id/client_secret instead."+ )+ validate_url_for_request(base_url, allow_private_ips=self.allow_private_ips)+ return base_url++ @staticmethod+ def _client_base_url(client: Any) -> Optional[str]:+ """Return the ``base_url`` the constructed client is bound to."""+ settings = getattr(getattr(client, "auth", None), "settings", None)+ value = getattr(settings, "base_url", None)+ return value if isinstance(value, str) else None++ def _apply_transport_controls(self, client: Any) -> None:+ """Apply the compensating controls to the SDK client's transport.++ Disables redirect following and proxy trust on the SDK session,+ forces TLS verification, pins connections to the validated+ addresses, and disables the SDK's separate error-documentation+ fetch.++ Args:+ client: A constructed ``Looker40SDK`` instance.++ Raises:+ ProcessingError: If the client exposes no transport session,+ because the compensating controls could not be applied.+ """+ session = self._transport_session(client)+ if session is None:+ raise ProcessingError(+ "Looker SDK client exposes no transport session; refusing to "+ "use a client without redirect and proxy controls."+ )+ session.max_redirects = 0+ session.trust_env = False+ if session.verify is False:+ raise ProcessingError(+ "Looker SDK session has TLS verification disabled; refusing to "+ "send credentials over an unverified connection."+ )+ session.verify = True+ self._pin_session(session)+ self._disable_error_doc_lookup()++ @staticmethod+ def _disable_error_doc_lookup() -> None:+ """Stop the SDK's error helper from making its own network calls.++ On a non-2xx API response the SDK builds an ``ErrorDocHelper`` and+ looks up its error codes at ``ERROR_CODES_URL`` through a bare+ module-level ``requests.get`` — a fresh session with+ ``trust_env=True``, no timeout, no redirect cap, no SSRF validation,+ and no pin. That is a second egress path beside the guarded SDK+ session, so it is disabled here.++ The override is deliberately installed on the SDK *class* rather+ than the instance: the helper is constructed deep inside the SDK+ with no injection point, and a per-call class override leaves the+ unguarded, untimed, proxy-honouring request reachable. Overriding+ ``ErrorDocHelper.get_index`` to return ``None`` makes ``lookup()``+ take its ``KeyError`` branch and skip the second fetch entirely.+ SDK internals are not a stable contract, so every access is+ defensive; when it cannot be applied the fact is logged at debug.+ """+ try:+ import looker_sdk.error as looker_error+ except Exception as exc: # noqa: BLE001 - SDK internals may move+ _logger.debug(+ "Could not import looker_sdk.error to disable error-doc lookup: %s",+ type(exc).__name__,+ )+ return++ helper = getattr(looker_error, "ErrorDocHelper", None)+ if helper is None or not callable(getattr(helper, "get_index", None)):+ _logger.debug(+ "looker_sdk.error exposes no ErrorDocHelper.get_index; "+ "error-doc lookup could not be disabled."+ )+ return++ def _no_error_doc_index(self: Any, url: Any = None) -> None:+ """Return without fetching; the SDK uses its inline fallback."""+ return None++ try:+ helper.get_index = _no_error_doc_index # type: ignore[method-assign]+ except Exception as exc: # noqa: BLE001 - SDK internals may move+ _logger.debug(+ "Could not disable Looker error-doc lookup: %s",+ type(exc).__name__,+ )++ def _pin_session(self, session: Any) -> None:+ """Guard every request and pin connections to validated addresses.++ The endpoint is validated once at construction, but the SDK resolves+ the hostname again at request time, which reopens a DNS-rebinding+ window and lets any later request (including the lazy OAuth login)+ reach an address that was never validated. The mounted adapter+ re-validates every request URL by calling+ ``validate_url_for_request``; when addresses were resolved it then+ delegates to a pinning adapter that connects only to the addresses+ validated here, while the hostname still supplies TLS identity.++ Raises:+ ValidationError: If the endpoint no longer resolves to a+ permitted address.+ """+ parsed = urlparse(self._validated_base_url)+ pinned_ips = _resolve_pinned_ips(+ self._validated_base_url, allow_private_ips=self.allow_private_ips+ )+ pinned_adapter: Optional[requests.adapters.HTTPAdapter] = None+ if pinned_ips:+ pinned_adapter = _make_pinned_adapter(pinned_ips, parsed.hostname or "")+ pinned_adapter._semantica_pinned = True++ # The guard is what gets mounted for both schemes; pinning rides+ # beneath it so the validated addresses are still dialed.+ adapter = _SSRFGuardAdapter(+ allow_private_ips=self.allow_private_ips,+ pinned_adapter=pinned_adapter,+ )+ session.mount("http://", adapter)+ session.mount("https://", adapter)++ if not pinned_ips:+ return+ session.headers["Host"] = _format_host_header(+ parsed.hostname or "", parsed.port, parsed.scheme+ )++ @staticmethod+ def _transport_session(client: Any) -> Optional[Any]:+ """Return the SDK transport's HTTP session, if it exposes one."""+ return getattr(getattr(client, "transport", None), "session", None)++ def _close_client(self, client: Any) -> None:+ """Best-effort close of a client that must not be used."""+ session = self._transport_session(client)+ if session is None:+ return+ try:+ session.close()+ except Exception: # noqa: BLE001 - cleanup must never raise+ pass++ # ------------------------------------------------------------------+ # Public properties+ # ------------------------------------------------------------------++ @property+ def client(self) -> Optional[Any]:+ """The active ``Looker40SDK`` client, or ``None``."""+ return self._client++ @property+ def connected(self) -> bool:+ """Whether a client is currently connected."""+ return self._client is not None++ # ------------------------------------------------------------------+ # Lifecycle+ # ------------------------------------------------------------------++ def connect(self) -> Any:+ """Create and return a ``looker_sdk`` client.++ If a client is already connected, returns it immediately without+ constructing a new one.++ Returns:+ The ``Looker40SDK`` client bound to the validated endpoint.++ Raises:+ ValidationError: If the constructed client is bound to a+ different endpoint than the validated one.+ ProcessingError: If client construction fails for any reason.+ Credentials are never included in the raised message.+ """+ if self._client is not None:+ return self._client++ try:+ client = looker_sdk.init40(config_settings=self._settings)+ except ValidationError:+ raise+ except Exception as exc: # noqa: BLE001 - translated with type only+ self.logger.error(+ "Failed to initialise Looker client: %s", type(exc).__name__+ )+ raise ProcessingError(+ f"Failed to initialise Looker client: {type(exc).__name__}"+ ) from None++ resolved = self._client_base_url(client)+ if _normalize_url(resolved or "") != _normalize_url(self._validated_base_url):+ self._close_client(client)+ self.logger.error(+ "Looker client bound to an unexpected base_url; failing closed."+ )+ raise ValidationError(+ "Looker client is bound to a different base_url than the "+ "validated endpoint; refusing to use it."+ )++ try:+ self._apply_transport_controls(client)+ except Exception:+ self._close_client(client)+ raise++ self._client = client+ self.logger.info("Connected to Looker: base_url=%s", self._validated_base_url)+ return self._client++ def disconnect(self) -> None:+ """Close the client session and release the reference.++ Safe to call when already disconnected — subsequent calls are+ no-ops, and cleanup failures are swallowed.+ """+ client, self._client = self._client, None+ if client is None:+ return+ self._close_client(client)+ self.logger.debug("Disconnected from Looker.")++ def test_connection(self) -> bool:+ """Verify connectivity with a cheap authenticated read.++ Uses ``client.me()`` (a single ``GET /user``) to confirm the+ session is live. The client is disconnected only when this call+ opened it.++ Returns:+ ``True`` if the probe succeeds, ``False`` for any failure.+ """+ opened_here = self._client is None+ try:+ client = self.connect()+ client.me()+ return True+ except Exception as exc: # noqa: BLE001 - bool result contract+ self.logger.debug("Looker connection test failed: %s", type(exc).__name__)+ return False+ finally:+ if opened_here:+ self.disconnect()++ def __enter__(self) -> "LookerConnector":+ """Open the Looker client on context entry."""+ self.connect()+ return self++ def __exit__(+ self,+ exc_type: Any,+ exc_val: Any,+ exc_tb: Any,+ ) -> None:+ """Close the Looker client on context exit."""+ self.close()++ def close(self) -> None:+ """Disconnect from Looker and release the client."""+ self.disconnect()+++# ---------------------------------------------------------------------------+# LookerIngestor+# ---------------------------------------------------------------------------+++class LookerIngestor:+ """Looker metadata ingestor for the Semantica framework.++ Wraps :class:`LookerConnector` and exposes the five metadata reads,+ the deep normalizer with allowlist redaction, and the document+ projection consumed by :class:`~semantica.kg.graph_builder.GraphBuilder`.++ Args:+ base_url: Looker instance URL; see :class:`LookerConnector`.+ client_id: Looker API client id.+ client_secret: Looker API client secret.+ config_file: Path to a ``looker.ini`` file.+ section: Section inside *config_file*.+ allow_private_ips: Opt into private endpoints and non-``https``+ schemes. Defaults to ``False``. When ``config`` (or ``**kwargs``)+ carries ``allow_private_ips``, that value wins and is parsed with+ :func:`~semantica.ingest.ssrf.parse_bool`, mirroring+ :class:`LookerConnector`.+ connector: An existing :class:`LookerConnector` to reuse.+ config: Optional extra configuration dict forwarded to the+ connector. A value here for any named connector argument that was+ not supplied explicitly (``base_url``, ``client_id``,+ ``client_secret``, ``config_file``, ``section``) is resolved from+ the config, and ``allow_private_ips`` follows the config-wins+ precedence described above.+ **kwargs: Additional keyword arguments merged into ``config``.++ Raises:+ ImportError: When ``looker-sdk`` is not installed and no connector+ is supplied.+ ValidationError: When required credentials are incomplete.++ Example::++ from semantica.ingest.looker_ingestor import LookerIngestor++ with LookerIngestor(+ base_url="https://looker.example.com",+ client_id="...",+ client_secret="...",+ ) as ingestor:+ data = ingestor.ingest_looks()+ documents = ingestor.export_as_documents(data)+ """++ def __init__(+ self,+ base_url: Optional[str] = None,+ client_id: Optional[str] = None,+ client_secret: Optional[str] = None,+ config_file: Optional[str] = None,+ section: Optional[str] = None,+ allow_private_ips: bool = False,+ connector: Optional[LookerConnector] = None,+ config: Optional[Dict[str, Any]] = None,+ **kwargs: Any,+ ) -> None:+ self.logger = _logger++ self.config: Dict[str, Any] = dict(config or {})+ self.config.update(kwargs)++ # KTD3 — config wins when it carries the key, matching+ # LookerConnector, and the value is parsed the same way so a string+ # such as "0" cannot silently enable private access. The key stays+ # excluded from the forwarded kwargs so the connector call cannot+ # collide with the named argument.+ resolved_allow_private_ips = parse_bool(+ self.config.get("allow_private_ips", allow_private_ips),+ default=False,+ )++ # The remaining named connector arguments are also excluded from the+ # forwarded kwargs, so a config-supplied value must be resolved+ # explicitly (an explicitly supplied argument always wins).+ resolved_base_url = base_url or self.config.get("base_url")+ resolved_client_id = client_id or self.config.get("client_id")+ resolved_client_secret = client_secret or self.config.get("client_secret")+ resolved_config_file = config_file or self.config.get("config_file")+ resolved_section = section or self.config.get("section")++ self.connector: LookerConnector = connector or LookerConnector(+ base_url=resolved_base_url,+ client_id=resolved_client_id,+ client_secret=resolved_client_secret,+ config_file=resolved_config_file,+ section=resolved_section,+ allow_private_ips=resolved_allow_private_ips,+ **{+ key: value+ for key, value in self.config.items()+ if key not in _CONNECTOR_ARGUMENT_NAMES+ },+ )++ # Progress tracker — consistent with all other ingestors.+ self.progress_tracker = get_progress_tracker()+ if not self.progress_tracker.enabled:+ self.progress_tracker.enabled = True++ self.logger.debug("Looker ingestor initialised.")++ # ------------------------------------------------------------------+ # Context-manager support+ # ------------------------------------------------------------------++ def __enter__(self) -> "LookerIngestor":+ """Open the Looker connection on context entry."""+ self.connector.connect()+ return self++ def __exit__(+ self,+ exc_type: Any,+ exc_val: Any,+ exc_tb: Any,+ ) -> None:+ """Close the Looker connection on context exit."""+ self.close()++ def close(self) -> None:+ """Disconnect from Looker and release the client."""+ self.connector.disconnect()++ # ------------------------------------------------------------------+ # Metadata reads+ # ------------------------------------------------------------------++ def ingest_looks(self, **options: Any) -> LookerData:+ """Fetch Look metadata via ``all_looks()``.++ Args:+ **options: Optional SDK ``transport_options`` / ``fields``.++ Returns:+ :class:`LookerData` with ``content_type == "look"``.++ Raises:+ ProcessingError: If the SDK read fails.+ """+ return self._ingest("all_looks", "look", **options)++ def ingest_dashboards(self, **options: Any) -> LookerData:+ """Fetch Dashboard metadata via ``all_dashboards()``.++ The SDK returns ``DashboardBase`` objects without tiles, which+ enforces the connector's metadata-only boundary.++ Args:+ **options: Optional SDK ``transport_options`` / ``fields``.++ Returns:+ :class:`LookerData` with ``content_type == "dashboard"``.++ Raises:+ ProcessingError: If the SDK read fails.+ """+ return self._ingest("all_dashboards", "dashboard", **options)++ def ingest_lookml_models(self, **options: Any) -> LookerData:+ """Fetch LookML model metadata via ``all_lookml_models()``.++ Explores are nested inside their parent model record.++ Args:+ **options: Optional SDK ``limit`` / ``offset`` /+ ``transport_options``.++ Returns:+ :class:`LookerData` with ``content_type == "lookml_model"``.++ Raises:+ ProcessingError: If the SDK read fails.+ """+ return self._ingest("all_lookml_models", "lookml_model", **options)++ def ingest_folders(self, **options: Any) -> LookerData:+ """Fetch Folder metadata via ``all_folders()``.++ Args:+ **options: Optional SDK ``transport_options`` / ``fields``.++ Returns:+ :class:`LookerData` with ``content_type == "folder"``.++ Raises:+ ProcessingError: If the SDK read fails.+ """+ return self._ingest("all_folders", "folder", **options)++ def ingest_projects(self, **options: Any) -> LookerData:+ """Fetch Project metadata via ``all_projects()``.++ Secret-adjacent project fields are removed by the normalizer and+ ``git_remote_url`` userinfo is scrubbed.++ Args:+ **options: Optional SDK ``transport_options`` / ``fields``.++ Returns:+ :class:`LookerData` with ``content_type == "project"``.++ Raises:+ ProcessingError: If the SDK read fails.+ """+ return self._ingest("all_projects", "project", **options)++ # ------------------------------------------------------------------+ # Internal read pipeline+ # ------------------------------------------------------------------++ def _ingest(self, sdk_method: str, content_type: str, **options: Any) -> LookerData:+ """Run one SDK collection read and normalize its records.++ The connection is opened transiently unless the caller already+ connected the connector (for example via the context manager).++ Args:+ sdk_method: Name of the ``looker_sdk`` collection method.+ content_type: Content type recorded on the result.+ **options: Options forwarded to the SDK method.++ Returns:+ A populated :class:`LookerData`.++ Raises:+ ProcessingError: If the SDK read fails.+ """+ already_connected = bool(getattr(self.connector, "connected", False))+ tracking_id = self.progress_tracker.start_tracking(+ file=f"looker:{content_type}",+ module="ingest",+ submodule="LookerIngestor",+ message=f"Reading Looker {content_type} metadata",+ )++ try:+ client = self.connector.connect()+ try:+ method = getattr(client, sdk_method)+ raw = method(**options) if options else method()+ except ValidationError:+ raise+ except Exception as exc: # noqa: BLE001 - type name only+ self.logger.error(+ "Looker %s read failed: %s", sdk_method, type(exc).__name__+ )+ raise ProcessingError(+ f"Looker {sdk_method} failed: {type(exc).__name__}"+ ) from None++ records = self._to_records(raw or [], content_type)+ data = LookerData(+ data=records,+ row_count=len(records),+ columns=self._derive_columns(records),+ content_type=content_type,+ base_url=getattr(self.connector, "base_url", None),+ metadata={"content_type": content_type, "endpoint": sdk_method},+ )+ self.progress_tracker.stop_tracking(+ tracking_id,+ status="completed",+ message=f"Read {data.row_count} {content_type} record(s)",+ )+ self.logger.debug(+ "ingest_%s: normalized %d record(s)",+ content_type,+ data.row_count,+ )+ return data+ except (ValidationError, ProcessingError):+ self.progress_tracker.stop_tracking(+ tracking_id,+ status="failed",+ message=f"Failed to read Looker {content_type} metadata",+ )+ raise+ except Exception as exc: # noqa: BLE001 - finalized then re-raised+ self.progress_tracker.stop_tracking(+ tracking_id, status="failed", message=type(exc).__name__+ )+ self.logger.error(+ "Failed to ingest Looker %s: %s", content_type, type(exc).__name__+ )+ raise+ finally:+ if not already_connected:+ self.connector.disconnect()++ @staticmethod+ def _derive_columns(records: List[Dict[str, Any]]) -> List[str]:+ """Return the ordered union of keys present across *records*."""+ return list(dict.fromkeys(key for record in records for key in record))++ # ------------------------------------------------------------------+ # Deep normalizer+ # ------------------------------------------------------------------++ def _to_records(+ self, raw: Sequence[Any], content_type: str+ ) -> List[Dict[str, Any]]:+ """Normalize SDK records into plain, JSON-serializable dicts.++ Args:+ raw: Sequence of SDK ``Model`` objects or mappings.+ content_type: One of the allowlisted content types.++ Returns:+ A list of normalized dictionaries.++ Raises:+ ValidationError: If *content_type* has no allowlist.+ ProcessingError: If a record is not a mapping-like object.+ """+ if content_type not in _RETAINED_FIELDS:+ raise ValidationError(f"Unsupported Looker content type: {content_type!r}.")+ allowed = _RETAINED_FIELDS[content_type]+ return [self._normalize_record(record, allowed) for record in raw]++ def _normalize_record(self, record: Any, allowed: frozenset) -> Dict[str, Any]:+ """Project a single record through the retained-fields allowlist."""+ data = self._record_mapping(record)+ return {+ key: self._normalize_field(key, value)+ for key, value in data.items()+ if key in allowed+ }++ def _normalize_field(self, key: str, value: Any) -> Any:+ """Normalize one field, scrubbing any credential-bearing URL.++ ``Project.git_remote_url`` may embed ``user:password@`` userinfo;+ the field is useful metadata, the userinfo is not.+ """+ if key == "git_remote_url" and isinstance(value, str):+ value = _scrub_url_userinfo(value)+ return self._normalize_value(value)++ def _normalize_value(self, value: Any) -> Any:+ """Recursively convert *value* into JSON-serializable primitives.++ SDK ``Model`` objects become filtered dictionaries, sequences+ become lists, and dates/datetimes become ISO 8601 strings.++ The recursion assumes an acyclic value graph: a Looker JSON+ response is a tree, so there is no depth or cycle guard here. A+ hand-built cyclic object would exhaust the interpreter stack and+ raise ``RecursionError``.+ """+ if value is None or isinstance(value, (str, int, float, bool)):+ return value+ if isinstance(value, enum.Enum):+ return value.value+ if isinstance(value, date):+ return value.isoformat()+ if isinstance(value, Mapping):+ # Route through _normalize_field so credential-bearing URLs are+ # scrubbed, and drop secret-named keys, at every depth.+ return {+ key: self._normalize_field(key, item)+ for key, item in value.items()+ if key not in _SECRET_FIELD_NAMES+ }+ if isinstance(value, (list, tuple, set, frozenset)):+ return [self._normalize_value(item) for item in value]+ if self._is_model_like(value):+ return self._normalize_record(value, _NESTED_RETAINED_FIELDS)+ return str(value)++ @staticmethod+ def _is_model_like(value: Any) -> bool:+ """Return True for SDK ``Model``-like objects (keys/items/iter)."""+ return callable(getattr(value, "keys", None)) and callable(+ getattr(value, "items", None)+ )++ @staticmethod+ def _record_mapping(record: Any) -> Dict[str, Any]:+ """Return a mapping view of *record*.++ ``dict(record)`` is the SDK primitive, but nested values stay+ ``Model`` objects, so callers recurse through+ :meth:`_normalize_value`.++ Raises:+ ProcessingError: If *record* is not mapping-like. A record the+ SDK returned in an unexpected shape is a data problem, not+ misconfiguration, so it is chained from the conversion+ failure when one occurred.+ """+ if isinstance(record, Mapping):+ return dict(record)+ last_error: Optional[BaseException] = None+ items = getattr(record, "items", None)+ if callable(items):+ try:+ return dict(items())+ except Exception as exc: # noqa: BLE001 - fall through to keys()+ last_error = exc+ keys = getattr(record, "keys", None)+ if callable(keys):+ try:+ return {str(key): record[key] for key in keys()}+ except Exception as exc: # noqa: BLE001 - fall through to __dict__+ last_error = exc+ data = getattr(record, "__dict__", None)+ if isinstance(data, dict):+ return dict(data)+ raise ProcessingError(+ f"Unsupported Looker record type: {type(record).__name__}"+ ) from last_error++ # ------------------------------------------------------------------+ # Document export+ # ------------------------------------------------------------------++ def export_as_documents(self, data: LookerData) -> List[Dict[str, Any]]:+ """Convert :class:`LookerData` to Semantica documents.++ Produces the ``{"id", "text", "metadata"}`` shape consumed by+ :class:`~semantica.kg.graph_builder.GraphBuilder`. Document ids+ are namespaced per content type so they stay unique across+ collections, and LookML Explores are nested inside their parent+ model document.++ Metadata always carries ``source`` (``"looker"``) and+ ``content_type``; it never carries ``base_url`` or credentials.++ Args:+ data: A :class:`LookerData` returned by one of the+ ``ingest_*`` methods.++ Returns:+ A list of document dictionaries.++ Raises:+ ValidationError: If ``data.content_type`` is unsupported, or a+ row is not a mapping. A row the SDK returned in an+ unexpected shape cannot be projected, so the failure names+ the offending type instead of leaking an ``AttributeError``.+ """+ if data.content_type not in _RETAINED_FIELDS:+ raise ValidationError(+ f"Unsupported Looker content type: {data.content_type!r}."+ )++ documents: List[Dict[str, Any]] = []+ for index, row in enumerate(data.data):+ if not isinstance(row, Mapping):+ raise ValidationError(+ "Looker document export requires mapping records; got "+ f"{type(row).__name__} at index {index}."+ )+ document_id = self._document_id(row, data.content_type, index)+ documents.append(+ {+ "id": document_id,+ "text": self._document_text(row, data.content_type),+ "metadata": self._document_metadata(+ row, data.content_type, document_id+ ),+ }+ )++ self.logger.debug(+ "export_as_documents: exported %d document(s)", len(documents)+ )+ return documents++ @staticmethod+ def _document_id(row: Dict[str, Any], content_type: str, index: int) -> str:+ """Compose the namespaced document id for one record."""+ if content_type == "lookml_model":+ project = row.get("project_name") or "unknown"+ name = row.get("name") or index+ return f"looker:lookml_model:{project}.{name}"+ identifier = row.get("id")+ if identifier is None or identifier == "":+ identifier = index+ return f"looker:{content_type}:{identifier}"++ def _document_metadata(+ self, row: Dict[str, Any], content_type: str, document_id: str+ ) -> Dict[str, Any]:+ """Build the metadata block for one document."""+ metadata: Dict[str, Any] = {+ "source": "looker",+ "content_type": content_type,+ "row_data": row,+ }++ identifier_key = _ID_METADATA_KEYS.get(content_type)+ if identifier_key is not None:+ metadata[identifier_key] = row.get("id")++ if content_type == "lookml_model":+ project = row.get("project_name") or "unknown"+ name = row.get("name") or "unknown"+ metadata["project_name"] = row.get("project_name")+ metadata["model_name"] = row.get("name")+ metadata["explores"] = self._nest_explores(+ row.get("explores"), project, name, document_id+ )+ metadata["explore_count"] = len(metadata["explores"])++ return metadata++ @staticmethod+ def _nest_explores(+ explores: Any,+ project: str,+ model: str,+ document_id: str,+ ) -> List[Dict[str, Any]]:+ """Nest Explore summaries inside their parent model document.++ Only the ``LookmlModelNavExplore`` fields exist for an+ ``all_lookml_models()`` read; ``view_name``/``joins``/``fields``+ belong to the deferred ``lookml_model_explore`` endpoint. Explores+ carry no parent reference of their own, so the parent project/model+ name is composed into every nested id.+ """+ nested: List[Dict[str, Any]] = []+ if not isinstance(explores, (list, tuple)):+ return nested+ for explore in explores:+ if not isinstance(explore, Mapping):+ continue+ explore_name = explore.get("name") or "unknown"+ nested.append(+ {+ "id": f"{document_id}:explore:{explore_name}",+ "name": explore_name,+ "label": explore.get("label"),+ "group_label": explore.get("group_label"),+ "description": explore.get("description"),+ "hidden": explore.get("hidden"),+ "parent_model": f"{project}.{model}",+ }+ )+ return nested++ def _document_text(self, row: Dict[str, Any], content_type: str) -> str:+ """Compose the searchable text for one document."""+ parts: List[str] = []++ for field_name in _TEXT_FIELDS.get(content_type, ()):+ value = row.get(field_name)+ if isinstance(value, str) and value:+ parts.append(value)++ folder = row.get("folder")+ if isinstance(folder, Mapping):+ folder_name = folder.get("name")+ if isinstance(folder_name, str) and folder_name:+ parts.append(folder_name)++ if content_type == "lookml_model":+ explores = row.get("explores")+ # A malformed record (e.g. ``explores: 5``) must degrade to no+ # explore text instead of raising ``TypeError``.+ if not isinstance(explores, (list, tuple)):+ return " ".join(parts)+ for explore in explores:+ if not isinstance(explore, Mapping):+ continue+ for field_name in ("label", "name", "description", "group_label"):+ value = explore.get(field_name)+ if isinstance(value, str) and value:+ parts.append(value)++ return " ".join(parts)diff --git a/semantica/ingest/ssrf.py b/semantica/ingest/ssrf.pyindex f97659ea..8fedcfba 100644--- a/semantica/ingest/ssrf.py+++ b/semantica/ingest/ssrf.py@@ -37,6 +37,29 @@ _STRIP_BODY_ON_REDIRECT = frozenset({301, 302, 303})_DEFAULT_PORTS = {"http": 80, "https": 443}+def _format_host_header(hostname: str, port: Optional[int], scheme: str) -> str:+ """Render the HTTP ``Host`` authority for *hostname*:*port*.++ ``urlparse`` strips the brackets from an IPv6 literal, but a valid HTTP+ authority requires them (RFC 3986 section 3.2.2), so the literal is+ re-bracketed here. The port is omitted when it is absent or already the+ scheme's default (``_DEFAULT_PORTS.get(scheme, 80)``).++ Args:+ hostname: Host without brackets (as returned by ``urlparse``).+ port: Explicit port, or ``None`` when the URL carries none.+ scheme: URL scheme used to look up the default port.++ Returns:+ The value for the ``Host`` header.+ """+ host = hostname+ if ":" in host and not host.startswith("["):+ host = f"[{host}]"+ default_port = _DEFAULT_PORTS.get(scheme, 80)+ return host if port in (None, default_port) else f"{host}:{port}"++def _should_strip_auth(old_url: str, new_url: str) -> bool:"""Decide whether credentials must not follow a redirect.@@ -422,12 +445,8 @@ def _apply_connection_pin(parsed = urlparse(url)if pinned_ips:- port = parsed.port- default_port = _DEFAULT_PORTS.get(parsed.scheme, 80)- host_header = (- parsed.hostname- if port in (None, default_port)- else f"{parsed.hostname}:{port}"+ host_header = _format_host_header(+ parsed.hostname or "", parsed.port, parsed.scheme)adapter = _make_pinned_adapter(pinned_ips, parsed.hostname or "")adapter._semantica_pinned = Truediff --git a/tests/ingest/test_optional_imports.py b/tests/ingest/test_optional_imports.pyindex 76da6cc4..c3446de5 100644--- a/tests/ingest/test_optional_imports.py+++ b/tests/ingest/test_optional_imports.py@@ -212,6 +212,7 @@ from semantica.ingest import (GitAnalyzer,XMLIngestionData,SalesforceData,+ LookerData,)print(CodeExtractor.__name__,@@ -220,15 +221,16 @@ print(GitAnalyzer.__name__,XMLIngestionData.__name__,SalesforceData.__name__,+ LookerData.__name__,)""",- ("git", "lxml", "simple_salesforce"),+ ("git", "lxml", "simple_salesforce", "looker_sdk"),)assert result.returncode == 0, result.stderrassert (- "CodeExtractor CodeFile CommitInfo GitAnalyzer XMLIngestionData SalesforceData"- in result.stdout+ "CodeExtractor CodeFile CommitInfo GitAnalyzer XMLIngestionData "+ "SalesforceData LookerData" in result.stdout)@@ -347,3 +349,78 @@ else:assert result.returncode == 0, result.stderrassert "ImportError" in result.stdoutassert "db-redshift" in result.stdout+++def test_looker_package_imports_without_sdk() -> None:+ """``import semantica.ingest`` must not eagerly pull in looker_sdk."""+ result = _run_python_with_blocked_modules(+ """+from semantica.ingest import FileIngestor, ingest_file+print(FileIngestor.__name__, callable(ingest_file))+""",+ ("looker_sdk",),+ )++ assert result.returncode == 0, result.stderr+ assert "FileIngestor True" in result.stdout+++def test_looker_data_importable_without_sdk() -> None:+ """``LookerData`` is a plain dataclass — no SDK needed to import it."""+ result = _run_python_with_blocked_modules(+ """+from semantica.ingest import LookerData+print(LookerData.__name__)+""",+ ("looker_sdk",),+ )++ assert result.returncode == 0, result.stderr+ assert "LookerData" in result.stdout+++def test_looker_ingestor_probe_fails_without_sdk() -> None:+ """``from semantica.ingest import LookerIngestor`` must raise+ ``ImportError`` when ``looker_sdk`` is absent."""+ result = _run_python_with_blocked_modules(+ """+try:+ from semantica.ingest import LookerIngestor+ has_looker = True+except ImportError:+ has_looker = False++assert not has_looker, (+ "Expected LookerIngestor import to fail without looker-sdk"+)+print("LookerIngestor probe passed")+""",+ ("looker_sdk",),+ )++ assert result.returncode == 0, result.stderr+ assert "LookerIngestor probe passed" in result.stdout+++def test_looker_connector_reports_missing_dep_with_install_hint() -> None:+ """Constructing ``LookerConnector`` without the SDK must raise+ ``ImportError`` with a message that names the ``ingest-looker`` extra."""+ result = _run_python_with_blocked_modules(+ """+from semantica.ingest.looker_ingestor import LookerConnector++try:+ LookerConnector()+except ImportError as exc:+ print(type(exc).__name__, exc)+else:+ raise SystemExit(+ "expected LookerConnector() to fail without looker-sdk"+ )+""",+ ("looker_sdk",),+ )++ assert result.returncode == 0, result.stderr+ assert "ImportError" in result.stdout+ assert "ingest-looker" in result.stdoutdiff --git a/tests/ingest/test_ssrf_protection.py b/tests/ingest/test_ssrf_protection.pyindex 8d6b9116..9d8c7f6b 100644--- a/tests/ingest/test_ssrf_protection.py+++ b/tests/ingest/test_ssrf_protection.py@@ -3,10 +3,13 @@from unittest.mock import MagicMock, patchimport pytest+import requestsimport urllib3.connectionpool as poolfrom semantica.ingest.api_ingestor import RESTIngestorfrom semantica.ingest.ssrf import (+ _apply_connection_pin,+ _format_host_header,parse_bool,request_with_ssrf_guard,validate_url_for_request,@@ -15,6 +18,55 @@ from semantica.ingest.web_ingestor import SitemapCrawler, WebIngestorfrom semantica.utils.exceptions import ValidationError+class TestFormatHostHeader:+ """The shared authority formatter brackets IPv6 and elides default ports."""++ def test_ipv6_literal_without_explicit_port(self):+ assert _format_host_header("2001:db8::1", None, "https") == "[2001:db8::1]"+ assert _format_host_header("2001:db8::1", 443, "https") == "[2001:db8::1]"++ def test_ipv6_literal_with_explicit_non_default_port(self):+ assert _format_host_header("2001:db8::1", 8443, "https") == "[2001:db8::1]:8443"+ assert _format_host_header("2001:db8::1", 8080, "http") == "[2001:db8::1]:8080"++ def test_ipv4_literal_is_unchanged(self):+ assert _format_host_header("93.184.216.34", None, "https") == "93.184.216.34"+ assert _format_host_header("93.184.216.34", 443, "https") == "93.184.216.34"+ assert (+ _format_host_header("93.184.216.34", 8443, "https") == "93.184.216.34:8443"+ )++ def test_unknown_scheme_falls_back_to_port_80(self):+ assert _format_host_header("example.com", 80, "ftp") == "example.com"+ assert _format_host_header("example.com", 8080, "ftp") == "example.com:8080"+++class TestApplyConnectionPinHostHeader:+ """The shared guarded path emits a valid authority for IPv6 targets."""++ @pytest.mark.parametrize(+ ("url", "expected_host"),+ [+ ("https://[2001:db8::1]/x", "[2001:db8::1]"),+ ("https://[2001:db8::1]:8443/x", "[2001:db8::1]:8443"),+ ],+ )+ def test_ipv6_authority_is_bracketed(self, url, expected_host):+ session = requests.Session()++ _apply_connection_pin(+ session,+ url,+ ["2001:db8::1"],+ requests.adapters.HTTPAdapter(),+ requests.adapters.HTTPAdapter(),+ False,+ None,+ )++ assert session.headers["Host"] == expected_host++class TestParseBool:def test_bool_passthrough(self):assert parse_bool(True) is Truediff --git a/tests/test_looker_ingestor.py b/tests/test_looker_ingestor.pynew file mode 100644index 00000000..61ffa327--- /dev/null+++ b/tests/test_looker_ingestor.py@@ -0,0 +1,2070 @@+"""+Unit tests for LookerConnector and LookerIngestor.++All ``looker_sdk`` calls are mocked — no live Looker instance is contacted+and no network I/O occurs.++Test structure mirrors ``tests/test_redshift_ingestor.py`` and+``tests/test_salesforce_ingestor.py``:++ - an autouse fixture injects a ``looker_sdk`` stub into ``sys.modules``+ when the real SDK is absent, so tests that reference the symbol can+ still run; the module-level optional-dependency guard has already run+ at import time, so this fixture cannot change ``LOOKER_AVAILABLE``;+ - an autouse fixture clears ``LOOKERSDK_*`` variables so a developer's+ live configuration can never leak into a test;+ - tests that need the real ``ApiSettings``/``init40`` implementation are+ skipped when ``looker-sdk`` is not installed; the import guard, the+ normalizer, the redaction contract, and the document projection are+ covered without it.++SSRF assertions use literal hosts (``localhost``, ``127.0.0.1``) only.+The DNS-stubbing autouse fixture lives in ``tests/ingest/conftest.py``,+not at the repository root, so no test here may require DNS resolution.+"""++from __future__ import annotations++import json+import os+import unittest+from datetime import datetime, timezone+from unittest.mock import MagicMock, Mock, call, patch++import pytest+import requests+from requests.adapters import BaseAdapter+from requests.exceptions import TooManyRedirects++# ---------------------------------------------------------------------------+# Optional SDK detection+# ---------------------------------------------------------------------------+try:+ import looker_sdk as _looker_sdk # noqa: F401++ LOOKER_SDK_AVAILABLE = True+except ImportError:+ LOOKER_SDK_AVAILABLE = False++requires_looker_sdk = pytest.mark.skipif(+ not LOOKER_SDK_AVAILABLE,+ reason="looker-sdk is required for this test",+)++# A literal public IP is used instead of a hostname: the SSRF validator+# performs real DNS lookups, and the DNS-stubbing fixture lives only in+# tests/ingest/conftest.py, not at the repository root.+PUBLIC_BASE_URL = "https://93.184.216.34"+PRIVATE_BASE_URL = "http://127.0.0.1:19999"+CLIENT_ID = "test-client-id"+CLIENT_SECRET = "test-client-secret"+ACCESS_TOKEN = "test-access-token-do-not-log"++# Literal secret values that must never survive normalization or logging.+SECRET_FIELD_VALUES = {+ "git_password": "git-password-do-not-log",+ "deploy_secret": "deploy-secret-do-not-log",+ "password": "dashboard-password-do-not-log",+ "pdt_password": "pdt-password-do-not-log",+ "device_token": ACCESS_TOKEN,+}++# Logger names whose records must never carry a credential or token.+CONNECTOR_LOGGER = "semantica.looker_ingestor"+TRANSPORT_LOGGER = "looker_sdk.rtl.requests_transport"++_LOOKER_ENV_KEYS = (+ "LOOKERSDK_BASE_URL",+ "LOOKERSDK_CLIENT_ID",+ "LOOKERSDK_CLIENT_SECRET",+ "LOOKERSDK_VERIFY_SSL",+ "LOOKERSDK_TIMEOUT",+)+++# ---------------------------------------------------------------------------+# autouse fixtures+# ---------------------------------------------------------------------------++[email protected](autouse=True)+def _mock_looker_sdk_if_needed():+ """Expose a minimal ``looker_sdk`` stub when the SDK is not installed.++ The stub is installed in ``sys.modules`` for the duration of each test,+ but ``semantica.ingest.looker_ingestor`` has already evaluated its+ module-level import guard by then, so this fixture does **not** exercise+ that guard — it only keeps ``looker_sdk`` importable for tests that+ reference the module object directly.+ """+ if not LOOKER_SDK_AVAILABLE:+ stub = MagicMock()+ stub.error.SDKError = type("SDKError", (Exception,), {})++ with patch.dict("sys.modules", {"looker_sdk": stub}):+ yield+ else:+ yield++[email protected](autouse=True)+def _clean_looker_env(monkeypatch):+ """Remove inherited ``LOOKERSDK_*`` variables before every test."""+ for key in _LOOKER_ENV_KEYS:+ monkeypatch.delenv(key, raising=False)+++# ---------------------------------------------------------------------------+# Helpers+# ---------------------------------------------------------------------------+++def _make_mock_client(base_url: str = PUBLIC_BASE_URL) -> MagicMock:+ """Return a mock that behaves like a ``Looker40SDK`` instance.++ ``connect()`` asserts that the client's resolved ``base_url`` matches+ the validated endpoint, and applies the redirect/proxy transport+ controls to ``client.transport.session``, so the mock must expose+ both paths with realistic starting values.+ """+ client = MagicMock()+ client.auth.settings.base_url = base_url+ client.transport.session.max_redirects = 5+ client.transport.session.trust_env = True+ return client+++class _FakeConnector:+ """Minimal connector double for ingestor tests (no SDK required)."""++ def __init__(self, client, base_url: str = PUBLIC_BASE_URL):+ self.client = client+ self.base_url = base_url+ self.connected = False+ self.connect_calls = 0+ self.disconnect_calls = 0++ def connect(self):+ self.connect_calls += 1+ self.connected = True+ return self.client++ def disconnect(self):+ self.disconnect_calls += 1+ self.connected = False+++def _make_ingestor(client) -> "object":+ """Return a LookerIngestor wired to a fake connector."""+ from semantica.ingest.looker_ingestor import LookerIngestor++ return LookerIngestor(connector=_FakeConnector(client))+++# ---------------------------------------------------------------------------+# TestLookerConnectorDataContract — LookerData is SDK-independent+# ---------------------------------------------------------------------------+++class TestLookerConnectorDataContract:+ """``LookerData`` must import and construct without ``looker_sdk``."""++ def test_importable_without_sdk(self):+ from semantica.ingest.looker_ingestor import LookerData++ with patch("semantica.ingest.looker_ingestor.LOOKER_AVAILABLE", False):+ data = LookerData(data=[], row_count=0, columns=[], content_type="look")++ assert data.data == []+ assert data.row_count == 0+ assert data.columns == []+ assert data.content_type == "look"+ assert data.base_url is None+ assert data.metadata == {}+ assert isinstance(data.ingested_at, datetime)++ def test_full_field_contract(self):+ from semantica.ingest.looker_ingestor import LookerData++ rows = [{"id": 1, "title": "Orders"}]+ data = LookerData(+ data=rows,+ row_count=1,+ columns=["id", "title"],+ content_type="look",+ base_url=PUBLIC_BASE_URL,+ metadata={"content_type": "look"},+ )++ assert data.data == rows+ assert data.row_count == 1+ assert data.base_url == PUBLIC_BASE_URL+ assert data.metadata == {"content_type": "look"}++ def test_module_all_and_logger_are_module_level(self):+ from semantica.ingest import looker_ingestor as mod++ assert mod.__all__ == [+ "LookerData",+ "LookerConnector",+ "LookerIngestor",+ ]+ assert mod._logger is not None+++# ---------------------------------------------------------------------------+# TestLookerConnectorImportGuard+# ---------------------------------------------------------------------------+++class TestLookerConnectorImportGuard:+ """R8 — the optional-dependency guard and its install hint."""++ def test_missing_sdk_raises_import_error_with_exact_hint(self):+ from semantica.ingest.looker_ingestor import LookerConnector++ with patch("semantica.ingest.looker_ingestor.LOOKER_AVAILABLE", False):+ with pytest.raises(ImportError) as exc_info:+ LookerConnector(+ base_url=PUBLIC_BASE_URL,+ client_id=CLIENT_ID,+ client_secret=CLIENT_SECRET,+ )++ message = str(exc_info.value)+ assert 'pip install "semantica[ingest-looker]"' in message+ assert CLIENT_SECRET not in message++ def test_missing_sdk_error_does_not_leak_secret(self):+ from semantica.ingest.looker_ingestor import LookerConnector++ with patch("semantica.ingest.looker_ingestor.LOOKER_AVAILABLE", False):+ with pytest.raises(ImportError) as exc_info:+ LookerConnector(client_secret="super-sensitive")++ assert "super-sensitive" not in str(exc_info.value)+++# ---------------------------------------------------------------------------+# TestLookerConnectorInit+# ---------------------------------------------------------------------------+++@requires_looker_sdk+class TestLookerConnectorInit:+ """Credential resolution, defaults, and pre-flight validation."""++ def test_explicit_credentials_stored(self):+ from semantica.ingest.looker_ingestor import LookerConnector++ conn = LookerConnector(+ base_url=PUBLIC_BASE_URL,+ client_id=CLIENT_ID,+ client_secret=CLIENT_SECRET,+ )++ assert conn.base_url == PUBLIC_BASE_URL+ assert conn.client_id == CLIENT_ID+ assert conn._client_secret == CLIENT_SECRET+ assert conn.connected is False+ assert conn.client is None+ assert conn.allow_private_ips is False++ def test_credentials_default_to_environment(self):+ from semantica.ingest.looker_ingestor import LookerConnector++ env = {+ "LOOKERSDK_BASE_URL": PUBLIC_BASE_URL,+ "LOOKERSDK_CLIENT_ID": "env-client",+ "LOOKERSDK_CLIENT_SECRET": "env-secret",+ }+ with patch.dict(os.environ, env):+ conn = LookerConnector()++ assert conn.base_url == PUBLIC_BASE_URL+ assert conn.client_id == "env-client"+ assert conn._client_secret == "env-secret"++ def test_explicit_args_override_env_credentials(self):+ from semantica.ingest.looker_ingestor import LookerConnector++ env = {+ "LOOKERSDK_CLIENT_ID": "env-client",+ "LOOKERSDK_CLIENT_SECRET": "env-secret",+ }+ with patch.dict(os.environ, env):+ conn = LookerConnector(+ base_url=PUBLIC_BASE_URL,+ client_id=CLIENT_ID,+ client_secret=CLIENT_SECRET,+ )++ assert conn.client_id == CLIENT_ID+ assert conn._client_secret == CLIENT_SECRET++ def test_missing_base_url_raises_validation_error(self):+ from semantica.ingest.looker_ingestor import LookerConnector+ from semantica.utils.exceptions import ValidationError++ with pytest.raises(ValidationError):+ LookerConnector(client_id=CLIENT_ID, client_secret=CLIENT_SECRET)++ def test_missing_client_id_raises_validation_error(self):+ from semantica.ingest.looker_ingestor import LookerConnector+ from semantica.utils.exceptions import ValidationError++ with pytest.raises(ValidationError):+ LookerConnector(base_url=PUBLIC_BASE_URL, client_secret=CLIENT_SECRET)++ def test_missing_client_secret_raises_validation_error(self):+ from semantica.ingest.looker_ingestor import LookerConnector+ from semantica.utils.exceptions import ValidationError++ with pytest.raises(ValidationError):+ LookerConnector(base_url=PUBLIC_BASE_URL, client_id=CLIENT_ID)++ def test_validation_error_does_not_contain_secret(self):+ from semantica.ingest.looker_ingestor import LookerConnector+ from semantica.utils.exceptions import ValidationError++ with pytest.raises(ValidationError) as exc_info:+ LookerConnector(client_id=CLIENT_ID, client_secret="s3cr3t-value")++ assert "s3cr3t-value" not in str(exc_info.value)++ def test_api_settings_built_with_config_file_section_and_env_prefix(self):+ from semantica.ingest.looker_ingestor import LookerConnector++ mock_settings = MagicMock()+ mock_settings.base_url = PUBLIC_BASE_URL+ mock_settings.client_id = CLIENT_ID+ mock_settings.client_secret = CLIENT_SECRET++ with patch(+ "semantica.ingest.looker_ingestor.api_settings.ApiSettings",+ return_value=mock_settings,+ ) as mock_api_settings:+ LookerConnector(+ base_url=PUBLIC_BASE_URL,+ client_id=CLIENT_ID,+ client_secret=CLIENT_SECRET,+ config_file="custom.ini",+ section="looker",+ )++ mock_api_settings.assert_called_once_with(+ filename="custom.ini", section="looker", env_prefix="LOOKERSDK"+ )++ def test_does_not_mutate_os_environ(self):+ from semantica.ingest.looker_ingestor import LookerConnector++ before = dict(os.environ)+ LookerConnector(+ base_url=PUBLIC_BASE_URL,+ client_id=CLIENT_ID,+ client_secret=CLIENT_SECRET,+ )+ assert dict(os.environ) == before+++# ---------------------------------------------------------------------------+# TestLookerConnectorFailClosed+# ---------------------------------------------------------------------------+++class TestLookerConnectorFailClosed:+ """AE1, AE2 — unsafe endpoints fail before any SDK object is built.++ These cases exercise only the pre-flight checks, so they run whether+ or not ``looker-sdk`` is installed; the fixture patches+ ``LOOKER_AVAILABLE`` so the optional-dependency guard does not+ short-circuit first.+ """++ @pytest.fixture(autouse=True)+ def _force_available(self):+ with patch("semantica.ingest.looker_ingestor.LOOKER_AVAILABLE", True):+ yield++ def test_http_base_url_rejected_by_default(self):+ from semantica.ingest.looker_ingestor import LookerConnector+ from semantica.utils.exceptions import ValidationError++ with pytest.raises(ValidationError) as exc_info:+ LookerConnector(+ base_url="http://looker.internal",+ client_id=CLIENT_ID,+ client_secret=CLIENT_SECRET,+ )++ assert "https" in str(exc_info.value)++ def test_http_rejected_before_any_client_is_built(self):+ from semantica.ingest.looker_ingestor import LookerConnector+ from semantica.utils.exceptions import ValidationError++ with patch("semantica.ingest.looker_ingestor.looker_sdk") as mock_sdk:+ with pytest.raises(ValidationError):+ LookerConnector(+ base_url="http://looker.internal",+ client_id=CLIENT_ID,+ client_secret=CLIENT_SECRET,+ )++ mock_sdk.init40.assert_not_called()++ def test_localhost_rejected_by_default(self):+ from semantica.ingest.looker_ingestor import LookerConnector+ from semantica.utils.exceptions import ValidationError++ with pytest.raises(ValidationError):+ LookerConnector(+ base_url="https://localhost",+ client_id=CLIENT_ID,+ client_secret=CLIENT_SECRET,+ )++ def test_loopback_ip_rejected_by_default(self):+ from semantica.ingest.looker_ingestor import LookerConnector+ from semantica.utils.exceptions import ValidationError++ with pytest.raises(ValidationError):+ LookerConnector(+ base_url="https://127.0.0.1",+ client_id=CLIENT_ID,+ client_secret=CLIENT_SECRET,+ )++ @pytest.mark.parametrize("value", ["false", "0", "no", "off"])+ def test_falsey_strings_do_not_enable_allow_private_ips(self, value):+ from semantica.ingest.looker_ingestor import LookerConnector+ from semantica.utils.exceptions import ValidationError++ with pytest.raises(ValidationError):+ LookerConnector(+ base_url=PRIVATE_BASE_URL,+ client_id=CLIENT_ID,+ client_secret=CLIENT_SECRET,+ config={"allow_private_ips": value},+ )++ def test_config_dict_overrides_kwarg_allow_private_ips(self):+ from semantica.ingest.looker_ingestor import LookerConnector+ from semantica.utils.exceptions import ValidationError++ with pytest.raises(ValidationError):+ LookerConnector(+ base_url=PRIVATE_BASE_URL,+ client_id=CLIENT_ID,+ client_secret=CLIENT_SECRET,+ allow_private_ips=True,+ config={"allow_private_ips": "0"},+ )++ def test_env_base_url_conflicting_with_constructor_fails_closed(self):+ """AE2 — the environment must not silently redirect the client."""+ from semantica.ingest.looker_ingestor import LookerConnector+ from semantica.utils.exceptions import ValidationError++ with patch.dict(os.environ, {"LOOKERSDK_BASE_URL": "https://10.0.0.5"}):+ with pytest.raises(ValidationError):+ LookerConnector(+ base_url=PUBLIC_BASE_URL,+ client_id=CLIENT_ID,+ client_secret=CLIENT_SECRET,+ )++ def test_env_base_url_conflict_message_does_not_leak_secret(self):+ from semantica.ingest.looker_ingestor import LookerConnector+ from semantica.utils.exceptions import ValidationError++ with patch.dict(os.environ, {"LOOKERSDK_BASE_URL": "https://10.0.0.5"}):+ with pytest.raises(ValidationError) as exc_info:+ LookerConnector(+ base_url=PUBLIC_BASE_URL,+ client_id=CLIENT_ID,+ client_secret="do-not-leak",+ )++ assert "do-not-leak" not in str(exc_info.value)+++# ---------------------------------------------------------------------------+# TestLookerConnectorEndpointValidation+# ---------------------------------------------------------------------------+++@requires_looker_sdk+class TestLookerConnectorEndpointValidation:+ """R10 — validate-once binding and the allow_private_ips opt-in."""++ def test_http_allowed_with_allow_private_ips_true(self):+ from semantica.ingest.looker_ingestor import LookerConnector++ conn = LookerConnector(+ base_url=PRIVATE_BASE_URL,+ client_id=CLIENT_ID,+ client_secret=CLIENT_SECRET,+ allow_private_ips=True,+ )++ assert conn.allow_private_ips is True+ assert conn.base_url == PRIVATE_BASE_URL++ def test_config_dict_true_string_enables_allow_private_ips(self):+ from semantica.ingest.looker_ingestor import LookerConnector++ conn = LookerConnector(+ base_url=PRIVATE_BASE_URL,+ client_id=CLIENT_ID,+ client_secret=CLIENT_SECRET,+ config={"allow_private_ips": "true"},+ )++ assert conn.allow_private_ips is True++ def test_env_private_base_url_rejected_without_allow_private_ips(self):+ """A private endpoint supplied only by the environment fails closed."""+ from semantica.ingest.looker_ingestor import LookerConnector+ from semantica.utils.exceptions import ValidationError++ env = {+ "LOOKERSDK_BASE_URL": "https://10.0.0.5",+ "LOOKERSDK_CLIENT_ID": CLIENT_ID,+ "LOOKERSDK_CLIENT_SECRET": CLIENT_SECRET,+ }+ with patch.dict(os.environ, env):+ with pytest.raises(ValidationError):+ LookerConnector()++ def test_settings_read_config_carries_explicit_credentials(self):+ """KTD4 — credentials ride the settings object, not os.environ."""+ from semantica.ingest.looker_ingestor import LookerConnector++ conn = LookerConnector(+ base_url=PUBLIC_BASE_URL,+ client_id=CLIENT_ID,+ client_secret=CLIENT_SECRET,+ )+ resolved = conn._settings.read_config()++ assert resolved["base_url"] == PUBLIC_BASE_URL+ assert resolved["client_id"] == CLIENT_ID+ assert resolved["client_secret"] == CLIENT_SECRET+ assert os.getenv("LOOKERSDK_CLIENT_SECRET") is None+++# ---------------------------------------------------------------------------+# TestLookerConnectorLifecycle+# ---------------------------------------------------------------------------+++@requires_looker_sdk+class TestLookerConnectorLifecycle:+ """connect()/disconnect()/context-manager and the transport controls."""++ def _connected_connector(self, client):+ from semantica.ingest.looker_ingestor import LookerConnector++ with patch(+ "semantica.ingest.looker_ingestor.looker_sdk.init40",+ return_value=client,+ ) as mock_init:+ conn = LookerConnector(+ base_url=PUBLIC_BASE_URL,+ client_id=CLIENT_ID,+ client_secret=CLIENT_SECRET,+ )+ result = conn.connect()+ return conn, result, mock_init++ def test_connect_passes_same_settings_object(self):+ client = _make_mock_client()++ with patch(+ "semantica.ingest.looker_ingestor.looker_sdk.init40",+ return_value=client,+ ) as mock_init:+ from semantica.ingest.looker_ingestor import LookerConnector++ conn = LookerConnector(+ base_url=PUBLIC_BASE_URL,+ client_id=CLIENT_ID,+ client_secret=CLIENT_SECRET,+ )+ conn.connect()++ assert mock_init.call_args[1]["config_settings"] is conn._settings+ assert conn._settings.base_url == PUBLIC_BASE_URL++ def test_connect_returns_existing_client(self):+ client = _make_mock_client()++ with patch(+ "semantica.ingest.looker_ingestor.looker_sdk.init40",+ return_value=client,+ ) as mock_init:+ from semantica.ingest.looker_ingestor import LookerConnector++ conn = LookerConnector(+ base_url=PUBLIC_BASE_URL,+ client_id=CLIENT_ID,+ client_secret=CLIENT_SECRET,+ )+ first = conn.connect()+ second = conn.connect()++ assert first is client+ assert second is client+ mock_init.assert_called_once()++ def test_connect_applies_transport_controls(self):+ client = _make_mock_client()++ with patch(+ "semantica.ingest.looker_ingestor.looker_sdk.init40",+ return_value=client,+ ):+ from semantica.ingest.looker_ingestor import LookerConnector++ conn = LookerConnector(+ base_url=PUBLIC_BASE_URL,+ client_id=CLIENT_ID,+ client_secret=CLIENT_SECRET,+ )+ conn.connect()++ assert client.transport.session.max_redirects == 0+ assert client.transport.session.trust_env is False++ def test_connect_fails_closed_when_client_base_url_differs(self):+ from semantica.ingest.looker_ingestor import LookerConnector+ from semantica.utils.exceptions import ValidationError++ client = _make_mock_client(base_url="https://elsewhere.example.com")++ with patch(+ "semantica.ingest.looker_ingestor.looker_sdk.init40",+ return_value=client,+ ):+ conn = LookerConnector(+ base_url=PUBLIC_BASE_URL,+ client_id=CLIENT_ID,+ client_secret=CLIENT_SECRET,+ )+ with pytest.raises(ValidationError):+ conn.connect()++ assert conn.connected is False+ assert conn.client is None++ def test_connect_failure_raises_processing_error_without_secret(self):+ from semantica.ingest.looker_ingestor import LookerConnector+ from semantica.utils.exceptions import ProcessingError++ error = _looker_sdk.error.SDKError(f"auth failed client_secret={CLIENT_SECRET}")++ with patch(+ "semantica.ingest.looker_ingestor.looker_sdk.init40",+ side_effect=error,+ ):+ conn = LookerConnector(+ base_url=PUBLIC_BASE_URL,+ client_id=CLIENT_ID,+ client_secret=CLIENT_SECRET,+ )+ with pytest.raises(ProcessingError) as exc_info:+ conn.connect()++ message = str(exc_info.value)+ assert CLIENT_SECRET not in message+ assert "SDKError" in message++ def test_disconnect_clears_client_and_is_idempotent(self):+ client = _make_mock_client()+ conn, _, _ = self._connected_connector(client)++ conn.disconnect()+ assert conn.connected is False+ assert conn.client is None++ conn.disconnect()+ assert conn.connected is False++ def test_context_manager_connects_and_disconnects(self):+ client = _make_mock_client()++ with patch(+ "semantica.ingest.looker_ingestor.looker_sdk.init40",+ return_value=client,+ ):+ from semantica.ingest.looker_ingestor import LookerConnector++ with LookerConnector(+ base_url=PUBLIC_BASE_URL,+ client_id=CLIENT_ID,+ client_secret=CLIENT_SECRET,+ ) as conn:+ assert conn.connected is True++ assert conn.connected is False++ def test_close_delegates_to_disconnect(self):+ client = _make_mock_client()+ conn, _, _ = self._connected_connector(client)++ conn.close()++ assert conn.connected is False++ def test_real_init40_client_is_bound_and_constrained(self):+ """End-to-end (offline) proof that the validated value is dialed.++ Real ``looker_sdk.init40`` constructs the client and its+ ``requests.Session`` without any network I/O, which lets this test+ assert the binding and the KTD2 transport controls for real rather+ than against a mock.+ """+ from semantica.ingest.looker_ingestor import LookerConnector++ conn = LookerConnector(+ base_url=PUBLIC_BASE_URL,+ client_id=CLIENT_ID,+ client_secret=CLIENT_SECRET,+ )++ client = conn.connect()++ assert conn.connected is True+ assert client.auth.settings is conn._settings+ assert client.auth.settings.base_url == PUBLIC_BASE_URL+ assert client.transport.session.max_redirects == 0+ assert client.transport.session.trust_env is False++ resolved = client.auth.settings.read_config()+ assert resolved["base_url"] == PUBLIC_BASE_URL+ assert resolved["client_id"] == CLIENT_ID+ assert resolved["client_secret"] == CLIENT_SECRET++ conn.disconnect()+ assert conn.connected is False+++# ---------------------------------------------------------------------------+# TestLookerConnectorTestConnection+# ---------------------------------------------------------------------------+++@requires_looker_sdk+class TestLookerConnectorTestConnection:+ """test_connection() — bool result, safe logging, lifecycle discipline."""++ def _connector(self):+ from semantica.ingest.looker_ingestor import LookerConnector++ conn = LookerConnector(+ base_url=PUBLIC_BASE_URL,+ client_id=CLIENT_ID,+ client_secret=CLIENT_SECRET,+ )+ conn.logger = Mock()+ return conn++ def test_returns_true_on_success(self):+ client = _make_mock_client()+ client.me.return_value = object()+ conn = self._connector()++ with patch(+ "semantica.ingest.looker_ingestor.looker_sdk.init40",+ return_value=client,+ ):+ assert conn.test_connection() is True++ client.me.assert_called_once()++ def test_opens_and_closes_transient_client(self):+ client = _make_mock_client()+ client.me.return_value = object()+ conn = self._connector()++ with patch(+ "semantica.ingest.looker_ingestor.looker_sdk.init40",+ return_value=client,+ ):+ conn.test_connection()++ assert conn.connected is False++ def test_returns_false_and_logs_only_exception_type(self):+ client = _make_mock_client()+ client.me.side_effect = _looker_sdk.error.SDKError(+ f"access_token=leaked client_secret={CLIENT_SECRET}"+ )+ conn = self._connector()++ with patch(+ "semantica.ingest.looker_ingestor.looker_sdk.init40",+ return_value=client,+ ):+ assert conn.test_connection() is False++ rendered = " ".join(+ str(argument)+ for call in conn.logger.debug.call_args_list+ for argument in call.args+ )+ assert CLIENT_SECRET not in rendered+ assert "leaked" not in rendered+ assert "SDKError" in rendered++ def test_does_not_disconnect_pre_existing_client(self):+ client = _make_mock_client()+ client.me.return_value = object()+ conn = self._connector()++ with patch(+ "semantica.ingest.looker_ingestor.looker_sdk.init40",+ return_value=client,+ ):+ conn.connect()+ assert conn.connected is True+ with patch.object(conn, "disconnect") as mock_disconnect:+ assert conn.test_connection() is True++ mock_disconnect.assert_not_called()+ assert conn.connected is True+++# ---------------------------------------------------------------------------+# TestLookerIngestorReads+# ---------------------------------------------------------------------------+++class TestLookerIngestorReads:+ """U2 — the five metadata reads and the connection lifecycle."""++ @pytest.mark.parametrize(+ ("method_name", "sdk_method", "content_type", "rows", "columns"),+ [+ (+ "ingest_looks",+ "all_looks",+ "look",+ [{"id": 1, "title": "one"}, {"id": 2, "title": "two"}],+ ["id", "title"],+ ),+ (+ "ingest_dashboards",+ "all_dashboards",+ "dashboard",+ [{"id": 1, "title": "one"}, {"id": 2, "title": "two"}],+ ["id", "title"],+ ),+ (+ "ingest_lookml_models",+ "all_lookml_models",+ "lookml_model",+ [{"name": "one"}, {"name": "two"}],+ ["name"],+ ),+ (+ "ingest_folders",+ "all_folders",+ "folder",+ [{"id": 1, "name": "one"}, {"id": 2, "name": "two"}],+ ["id", "name"],+ ),+ (+ "ingest_projects",+ "all_projects",+ "project",+ [{"id": "a", "name": "one"}, {"id": "b", "name": "two"}],+ ["id", "name"],+ ),+ ],+ )+ def test_read_returns_rows_and_row_count(+ self, method_name, sdk_method, content_type, rows, columns+ ):+ client = MagicMock()+ getattr(client, sdk_method).return_value = rows+ ingestor = _make_ingestor(client)++ data = getattr(ingestor, method_name)()++ assert data.row_count == 2+ assert len(data.data) == 2+ assert data.content_type == content_type+ assert data.base_url == PUBLIC_BASE_URL+ assert data.columns == columns++ def test_read_calls_sdk_method(self):+ client = MagicMock()+ client.all_looks.return_value = []+ ingestor = _make_ingestor(client)++ ingestor.ingest_looks()++ client.all_looks.assert_called_once()++ def test_empty_result_set_returns_empty_list(self):+ client = MagicMock()+ client.all_looks.return_value = []+ ingestor = _make_ingestor(client)++ data = ingestor.ingest_looks()++ assert data.data == []+ assert data.row_count == 0+ assert ingestor.export_as_documents(data) == []++ def test_none_result_set_is_treated_as_empty(self):+ client = MagicMock()+ client.all_looks.return_value = None+ ingestor = _make_ingestor(client)++ data = ingestor.ingest_looks()++ assert data.data == []+ assert data.row_count == 0++ def test_transient_connection_is_closed(self):+ client = MagicMock()+ client.all_looks.return_value = []+ fake = _FakeConnector(client)+ from semantica.ingest.looker_ingestor import LookerIngestor++ ingestor = LookerIngestor(connector=fake)+ ingestor.ingest_looks()++ assert fake.connect_calls == 1+ assert fake.disconnect_calls == 1+ assert fake.connected is False++ def test_pre_existing_connection_is_reused(self):+ client = MagicMock()+ client.all_looks.return_value = []+ fake = _FakeConnector(client)+ fake.connected = True+ from semantica.ingest.looker_ingestor import LookerIngestor++ ingestor = LookerIngestor(connector=fake)+ ingestor.ingest_looks()++ assert fake.disconnect_calls == 0+ assert fake.connected is True++ @requires_looker_sdk+ def test_sdk_error_translated_to_processing_error(self):+ from semantica.utils.exceptions import ProcessingError++ client = MagicMock()+ client.all_looks.side_effect = _looker_sdk.error.SDKError(+ f"token=leaked secret={CLIENT_SECRET}"+ )+ fake = _FakeConnector(client)+ from semantica.ingest.looker_ingestor import LookerIngestor++ ingestor = LookerIngestor(connector=fake)+ ingestor.logger = Mock()++ with pytest.raises(ProcessingError) as exc_info:+ ingestor.ingest_looks()++ message = str(exc_info.value)+ assert CLIENT_SECRET not in message+ assert "SDKError" in message+ assert fake.disconnect_calls == 1++ def test_ingestor_context_manager_uses_connector(self):+ client = MagicMock()+ client.all_looks.return_value = []+ fake = _FakeConnector(client)+ from semantica.ingest.looker_ingestor import LookerIngestor++ with LookerIngestor(connector=fake) as ingestor:+ assert fake.connected is True+ ingestor.ingest_looks()++ assert fake.connected is False+++# ---------------------------------------------------------------------------+# TestLookerIngestorNormalization+# ---------------------------------------------------------------------------+++class TestLookerIngestorNormalization:+ """U2 — deep normalization and allowlist redaction."""++ def test_plain_dict_records_pass_through_allowed_fields(self):+ client = MagicMock()+ client.all_looks.return_value = [+ {"id": 7, "title": "Revenue", "unexpected": "drop-me"}+ ]+ ingestor = _make_ingestor(client)++ data = ingestor.ingest_looks()++ assert data.data == [{"id": 7, "title": "Revenue"}]++ def test_unknown_content_type_raises_validation_error(self):+ from semantica.ingest.looker_ingestor import LookerData+ from semantica.utils.exceptions import ValidationError++ ingestor = _make_ingestor(MagicMock())+ data = LookerData(+ data=[{"id": 1}], row_count=1, columns=[], content_type="tile"+ )++ with pytest.raises(ValidationError):+ ingestor.export_as_documents(data)++ def test_datetime_values_become_iso_8601_strings(self):+ stamp = datetime(2024, 1, 2, 3, 4, 5, tzinfo=timezone.utc)+ client = MagicMock()+ client.all_looks.return_value = [{"id": 1, "title": "T", "created_at": stamp}]+ ingestor = _make_ingestor(client)++ data = ingestor.ingest_looks()++ assert data.data[0]["created_at"] == stamp.isoformat()+ json.dumps(data.data)++ def test_nested_models_and_sequences_are_normalized(self):+ class _Nested:+ def __init__(self, **kwargs):+ self._data = kwargs++ def keys(self):+ return self._data.keys()++ def items(self):+ return self._data.items()++ client = MagicMock()+ client.all_lookml_models.return_value = [+ {+ "name": "model",+ "project_name": "proj",+ "explores": [+ _Nested(name="orders", label="Orders", group_label="Commerce")+ ],+ }+ ]+ ingestor = _make_ingestor(client)++ data = ingestor.ingest_lookml_models()+ record = data.data[0]++ assert record["explores"][0] == {+ "name": "orders",+ "label": "Orders",+ "group_label": "Commerce",+ }+ json.dumps(record)++ def test_secret_fields_absent_from_plain_dict_records(self):+ client = MagicMock()+ client.all_dashboards.return_value = [+ {+ "id": 1,+ "title": "Ops",+ "password": "DASH_PASSWORD",+ "pdt_password": "PDT_PASSWORD",+ }+ ]+ client.all_lookml_models.return_value = [+ {+ "name": "model",+ "project_name": "proj",+ "device_token": "DEVICE_TOKEN",+ }+ ]+ ingestor = _make_ingestor(client)++ dashboards = ingestor.ingest_dashboards()+ models = ingestor.ingest_lookml_models()++ rendered = json.dumps([dashboards.data, models.data])+ for secret in ("DASH_PASSWORD", "PDT_PASSWORD", "DEVICE_TOKEN"):+ assert secret not in rendered++ def test_nested_secret_fields_are_redacted(self):+ class _NestedProject:+ def __init__(self, **kwargs):+ self._data = kwargs++ def keys(self):+ return self._data.keys()++ def items(self):+ return self._data.items()++ client = MagicMock()+ client.all_folders.return_value = [+ {+ "id": 1,+ "name": "Shared",+ "looks": [+ _NestedProject(+ id=9,+ title="Nested look",+ password="NESTED_PASSWORD",+ )+ ],+ }+ ]+ ingestor = _make_ingestor(client)++ data = ingestor.ingest_folders()++ rendered = json.dumps(data.data)+ assert "NESTED_PASSWORD" not in rendered+ assert "password" not in rendered++ @requires_looker_sdk+ def test_project_secrets_absent_from_data_and_documents(self):+ from looker_sdk.sdk.api40 import models as sdk_models++ project = sdk_models.Project(+ id="shop",+ name="shop",+ git_remote_url="https://gituser:[email protected]/shop.git",+ git_username="gituser",+ git_password="GIT_PASSWORD",+ deploy_secret="DEPLOY_SECRET",+ )+ client = MagicMock()+ client.all_projects.return_value = [project]+ ingestor = _make_ingestor(client)++ data = ingestor.ingest_projects()+ documents = ingestor.export_as_documents(data)++ payload = json.dumps({"data": data.data, "documents": documents})+ for secret in (+ "GIT_PASSWORD",+ "DEPLOY_SECRET",+ "gitpass",+ "gituser:gitpass",+ ):+ assert secret not in payload++ record = data.data[0]+ assert record["name"] == "shop"+ assert record["git_remote_url"] == "https://git.example.com/shop.git"+ assert "git_username" in record+ assert "git_password" not in record+ assert "deploy_secret" not in record++ @requires_looker_sdk+ def test_lookml_model_explores_are_nested_with_parent_identity(self):+ from looker_sdk.sdk.api40 import models as sdk_models++ explore = sdk_models.LookmlModelNavExplore(+ name="orders",+ label="Orders",+ description="Order facts",+ hidden=False,+ group_label="Commerce",+ )+ model = sdk_models.LookmlModel(+ name="ecommerce",+ project_name="shop",+ label="Ecommerce",+ explores=[explore],+ )+ client = MagicMock()+ client.all_lookml_models.return_value = [model]+ ingestor = _make_ingestor(client)++ data = ingestor.ingest_lookml_models()+ record = data.data[0]++ assert record["name"] == "ecommerce"+ assert record["project_name"] == "shop"+ # ``all_lookml_models()`` yields ``LookmlModelNavExplore`` records: only+ # the navigation fields exist, and ``group_label`` must survive.+ assert record["explores"][0] == {+ "name": "orders",+ "description": "Order facts",+ "label": "Orders",+ "hidden": False,+ "group_label": "Commerce",+ }+ assert "device_token" not in record+ json.dumps(record)+++# ---------------------------------------------------------------------------+# TestLookerIngestorExport+# ---------------------------------------------------------------------------+++class TestLookerIngestorExport:+ """U2/R3/R4 — document projection, ids, metadata, and JSON safety."""++ def _sample_rows_by_content_type(self):+ return {+ "look": [{"id": 1, "title": "Orders look", "description": "desc"}],+ "dashboard": [{"id": 2, "title": "Ops dashboard"}],+ "lookml_model": [+ {+ "name": "ecommerce",+ "project_name": "shop",+ "label": "Ecommerce",+ "explores": [+ {+ "name": "orders",+ "label": "Orders",+ "description": "Order facts",+ "hidden": False,+ "group_label": "Commerce",+ }+ ],+ }+ ],+ "folder": [{"id": 3, "name": "Shared"}],+ "project": [{"id": "shop", "name": "shop"}],+ }++ def test_document_shape_and_metadata(self):+ client = MagicMock()+ client.all_looks.return_value = [+ {"id": 1, "title": "Orders look", "description": "desc"}+ ]+ ingestor = _make_ingestor(client)++ data = ingestor.ingest_looks()+ documents = ingestor.export_as_documents(data)++ assert len(documents) == 1+ document = documents[0]+ assert set(document) == {"id", "text", "metadata"}+ assert document["id"] == "looker:look:1"+ assert "Orders look" in document["text"]+ assert document["metadata"]["source"] == "looker"+ assert document["metadata"]["content_type"] == "look"+ assert "base_url" not in document["metadata"]+ assert "base_url" not in json.dumps(document["metadata"])++ def test_metadata_never_contains_credentials(self):+ client = MagicMock()+ client.all_projects.return_value = [+ {+ "id": "shop",+ "name": "shop",+ "client_id": "cid",+ "client_secret": "csecret",+ "secret": "nope",+ }+ ]+ ingestor = _make_ingestor(client)++ data = ingestor.ingest_projects()+ documents = ingestor.export_as_documents(data)++ rendered = json.dumps(documents)+ assert "csecret" not in rendered+ assert "client_secret" not in rendered++ def test_document_ids_are_unique_across_content_types(self):+ from semantica.ingest.looker_ingestor import LookerData++ ingestor = _make_ingestor(MagicMock())+ documents = []+ for content_type, rows in self._sample_rows_by_content_type().items():+ data = LookerData(+ data=rows,+ row_count=len(rows),+ columns=[],+ content_type=content_type,+ )+ documents.extend(ingestor.export_as_documents(data))++ ids = [document["id"] for document in documents]+ assert ids == [+ "looker:look:1",+ "looker:dashboard:2",+ "looker:lookml_model:shop.ecommerce",+ "looker:folder:3",+ "looker:project:shop",+ ]+ assert len(ids) == len(set(ids))++ def test_lookml_model_document_nests_explores(self):+ from semantica.ingest.looker_ingestor import LookerData++ ingestor = _make_ingestor(MagicMock())+ rows = self._sample_rows_by_content_type()["lookml_model"]+ data = LookerData(+ data=rows,+ row_count=len(rows),+ columns=[],+ content_type="lookml_model",+ )++ documents = ingestor.export_as_documents(data)+ metadata = documents[0]["metadata"]++ assert documents[0]["id"] == "looker:lookml_model:shop.ecommerce"+ assert metadata["project_name"] == "shop"+ assert metadata["model_name"] == "ecommerce"+ assert metadata["explores"][0]["id"] == (+ "looker:lookml_model:shop.ecommerce:explore:orders"+ )+ assert metadata["explores"][0]["name"] == "orders"+ assert metadata["explores"][0]["label"] == "Orders"+ assert metadata["explores"][0]["description"] == "Order facts"+ assert metadata["explores"][0]["hidden"] is False+ assert metadata["explores"][0]["group_label"] == "Commerce"+ assert metadata["explores"][0]["parent_model"] == "shop.ecommerce"+ assert "view_name" not in metadata["explores"][0]++ def test_full_export_is_json_serializable(self):+ from semantica.ingest.looker_ingestor import LookerData++ ingestor = _make_ingestor(MagicMock())+ documents = []+ for content_type, rows in self._sample_rows_by_content_type().items():+ data = LookerData(+ data=rows,+ row_count=len(rows),+ columns=[],+ content_type=content_type,+ )+ documents.extend(ingestor.export_as_documents(data))++ json.dumps(documents)++ @requires_looker_sdk+ def test_export_with_sdk_models_and_datetimes_is_json_serializable(self):+ from looker_sdk.sdk.api40 import models as sdk_models++ look = sdk_models.Look(+ id=11,+ title="Timed look",+ created_at=datetime(2024, 5, 6, 7, 8, 9, tzinfo=timezone.utc),+ )+ model = sdk_models.LookmlModel(+ name="ecommerce",+ project_name="shop",+ explores=[+ sdk_models.LookmlModelNavExplore(+ name="orders",+ label="Orders",+ description="Order facts",+ hidden=False,+ group_label="Commerce",+ )+ ],+ )+ client = MagicMock()+ client.all_looks.return_value = [look]+ client.all_lookml_models.return_value = [model]+ ingestor = _make_ingestor(client)++ looks_data = ingestor.ingest_looks()+ models_data = ingestor.ingest_lookml_models()++ json.dumps(looks_data.data)+ json.dumps(models_data.data)+ json.dumps(ingestor.export_as_documents(looks_data))+ json.dumps(ingestor.export_as_documents(models_data))+++# ---------------------------------------------------------------------------+# TestLookerConnectorSecretLogging — R11/R13+# ---------------------------------------------------------------------------+++@requires_looker_sdk+class TestLookerConnectorSecretLogging(unittest.TestCase):+ """No credential, PAT, or secret-adjacent field may reach a log record.++ Records are captured with :meth:`unittest.TestCase.assertLogs` from the+ connector logger and from the SDK's own transport logger across a+ mocked ``connect()``, the ``test_connection()`` probe, and all five+ metadata reads.+ """++ @staticmethod+ def _secret_rows():+ """Rows carrying every secret-adjacent SDK field."""+ return [+ {+ "id": 1,+ "title": "Orders",+ "project_name": "shop",+ "name": "ecommerce",+ "git_remote_url": (+ f"https://user:{CLIENT_SECRET}@git.example.com/shop.git"+ ),+ **SECRET_FIELD_VALUES,+ }+ ]++ def test_secret_and_access_token_never_reach_log_records(self):+ from semantica.ingest.looker_ingestor import LookerConnector, LookerIngestor++ client = _make_mock_client()+ rows = self._secret_rows()+ for sdk_method in (+ "all_looks",+ "all_dashboards",+ "all_lookml_models",+ "all_folders",+ "all_projects",+ ):+ getattr(client, sdk_method).return_value = rows++ # The probe fails with an SDK error whose text embeds both a token+ # and the client secret: only the exception type may be logged.+ client.me.side_effect = _looker_sdk.error.SDKError(+ f"auth failed token={ACCESS_TOKEN} secret={CLIENT_SECRET}"+ )++ with patch(+ "semantica.ingest.looker_ingestor.looker_sdk.init40",+ return_value=client,+ ):+ connector = LookerConnector(+ base_url=PUBLIC_BASE_URL,+ client_id=CLIENT_ID,+ client_secret=CLIENT_SECRET,+ )++ with self.assertLogs(CONNECTOR_LOGGER, level="DEBUG") as connector_logs:+ connector.connect()+ assert connector.test_connection() is False++ ingestor = LookerIngestor(connector=connector)+ for method_name in (+ "ingest_looks",+ "ingest_dashboards",+ "ingest_lookml_models",+ "ingest_folders",+ "ingest_projects",+ ):+ data = getattr(ingestor, method_name)()+ ingestor.export_as_documents(data)++ captured = "\n".join(connector_logs.output)+ assert connector_logs.output, "expected connector log records"++ for secret in (CLIENT_SECRET, ACCESS_TOKEN, *SECRET_FIELD_VALUES.values()):+ assert secret not in captured+++# ---------------------------------------------------------------------------+# TestLookerConnectorTransportHardening — KTD2 compensating controls+# ---------------------------------------------------------------------------+++@requires_looker_sdk+class TestLookerConnectorTransportHardening:+ """The SDK session must not follow redirects or trust proxy env vars."""++ def _connect_with_real_session(self, session, base_url: str = PUBLIC_BASE_URL):+ """Connect a LookerConnector whose client carries *session*."""+ from semantica.ingest.looker_ingestor import LookerConnector++ client = _make_mock_client(base_url=base_url)+ client.transport.session = session++ with patch(+ "semantica.ingest.looker_ingestor.looker_sdk.init40",+ return_value=client,+ ):+ connector = LookerConnector(+ base_url=base_url,+ client_id=CLIENT_ID,+ client_secret=CLIENT_SECRET,+ )+ connector.connect()+ return connector++ def test_redirect_response_is_not_followed(self):+ target = f"{PUBLIC_BASE_URL}/api/4.0/looks"+ seen = []++ class _RedirectAdapter(BaseAdapter):+ def send(self, request, **kwargs):+ seen.append(request.url)+ response = requests.Response()+ response.status_code = 302+ response.headers["Location"] = "https://evil.example.com/steal"+ response.url = request.url+ response.request = request+ response.raw = None+ return response++ def close(self):+ pass++ session = requests.Session()+ assert session.max_redirects != 0 # default is a real redirect cap++ self._connect_with_real_session(session)+ session.mount("https://", _RedirectAdapter())++ assert session.max_redirects == 0+ with pytest.raises(TooManyRedirects):+ session.get(target)++ # The redirect target was never requested: only the original hop.+ assert seen == [target]++ def test_proxy_environment_variable_is_not_honoured(self, monkeypatch):+ monkeypatch.setenv("HTTP_PROXY", "http://127.0.0.1:9")+ monkeypatch.setenv("HTTPS_PROXY", "http://127.0.0.1:9")++ session = requests.Session()+ assert session.trust_env is True # default honours the environment++ self._connect_with_real_session(session)++ assert session.trust_env is False+ settings = session.merge_environment_settings(+ f"{PUBLIC_BASE_URL}/api/4.0/looks", {}, None, None, None+ )+ assert settings["proxies"] == {}++ def test_connections_are_pinned_to_the_validated_address(self):+ session = requests.Session()++ self._connect_with_real_session(session)++ adapter = session.adapters.get("https://")+ assert getattr(adapter, "_semantica_pinned", False) is True++ def test_metadata_read_is_routed_through_the_ssrf_validator(self):+ """AE1 — every SDK request must pass the repository SSRF validator.++ The SDK's ``all_looks`` is wired to issue a real request through the+ session that ``connect()`` hardens. With the validator patched to+ raise, the read must fail *before* the pinned transport is reached.+ """+ from semantica.ingest.looker_ingestor import LookerIngestor+ from semantica.utils.exceptions import ValidationError++ target = f"{PUBLIC_BASE_URL}/api/4.0/looks"+ session = requests.Session()+ client = _make_mock_client()+ client.transport.session = session++ class _RecordingAdapter(BaseAdapter):+ """Pinned-transport stand-in that never dials the network."""++ def __init__(self):+ self.seen = []++ def send(self, request, **kwargs):+ self.seen.append(request.url)+ response = requests.Response()+ response.status_code = 200+ response.url = request.url+ response.request = request+ response._content = b"[]"+ return response++ def close(self):+ pass++ delegate = _RecordingAdapter()++ def _all_looks(**_kwargs):+ session.get(target)+ return []++ client.all_looks.side_effect = _all_looks++ with (+ patch(+ "semantica.ingest.looker_ingestor.looker_sdk.init40",+ return_value=client,+ ),+ patch(+ "semantica.ingest.looker_ingestor._make_pinned_adapter",+ return_value=delegate,+ ),+ ):+ ingestor = LookerIngestor(+ base_url=PUBLIC_BASE_URL,+ client_id=CLIENT_ID,+ client_secret=CLIENT_SECRET,+ )++ with patch(+ "semantica.ingest.looker_ingestor.validate_url_for_request",+ side_effect=ValidationError("blocked by the test double"),+ ):+ with pytest.raises(ValidationError):+ ingestor.ingest_looks()++ # The validator raised before the pinned adapter was reached.+ assert delegate.seen == []++ @pytest.mark.parametrize(+ ("base_url", "expected_host"),+ [+ ("https://[2001:db8::1]", "[2001:db8::1]"),+ ("https://[2001:db8::1]:8443", "[2001:db8::1]:8443"),+ ],+ )+ def test_ipv6_host_header_is_bracketed(self, base_url, expected_host):+ """AE4 — the pinned session's Host header is a valid authority."""+ session = requests.Session()++ with (+ patch("semantica.ingest.looker_ingestor.validate_url_for_request"),+ patch(+ "semantica.ingest.looker_ingestor._resolve_pinned_ips",+ return_value=["2001:db8::1"],+ ),+ ):+ self._connect_with_real_session(session, base_url=base_url)++ assert session.headers["Host"] == expected_host+++# ---------------------------------------------------------------------------+# TestLookerHardeningRegressions — defects found in code review+# ---------------------------------------------------------------------------+++class TestLookerHardeningRegressions:+ """Regressions for review findings: redaction and config forwarding."""++ def test_userinfo_is_scrubbed_without_a_scheme(self):+ """Credentials in a scheme-less remote must not survive."""+ from semantica.ingest.looker_ingestor import _scrub_url_userinfo++ assert (+ _scrub_url_userinfo("oauth2:[email protected]/org/r.git")+ == "gitlab.example.com/org/r.git"+ )+ assert (+ _scrub_url_userinfo("[email protected]:org/repo.git")+ == "github.com:org/repo.git"+ )+ assert _scrub_url_userinfo("https://user:[email protected]/r.git") == (+ "https://example.com/r.git"+ )++ def test_userinfo_in_path_is_not_treated_as_credentials(self):+ from semantica.ingest.looker_ingestor import _scrub_url_userinfo++ assert (+ _scrub_url_userinfo("https://example.com/path@file")+ == "https://example.com/path@file"+ )++ def test_userinfo_is_scrubbed_when_an_at_sign_follows_the_authority(self):+ """The deciding '@' must be the one inside the authority section."""+ from semantica.ingest.looker_ingestor import _scrub_url_userinfo++ assert _scrub_url_userinfo("https://user:[email protected]/path@file") == (+ "https://example.com/path@file"+ )+ assert (+ _scrub_url_userinfo(+ "https://oauth2:[email protected]/org/repo.git"+ )+ == "https://gitlab.example.com/org/[email protected]"+ )++ def test_ipv6_authority_userinfo_is_scrubbed(self):+ from semantica.ingest.looker_ingestor import _scrub_url_userinfo++ assert (+ _scrub_url_userinfo("https://user:pw@[::1]:8080/x")+ == "https://[::1]:8080/x"+ )++ def test_ambiguous_password_containing_a_slash_fails_closed(self):+ """A password containing '/' defeats the authority split."""+ from semantica.ingest.looker_ingestor import (+ _REDACTED_URL,+ _scrub_url_userinfo,+ )++ assert _scrub_url_userinfo("https://user:pa/ss@host/repo.git") == (+ _REDACTED_URL+ )+ # A later '@' in the path must not rescue the earlier userinfo.+ assert (+ _scrub_url_userinfo(+ "https://alice:[email protected]/groups/[email protected]/repo.git"+ )+ == "https://git.example.com/groups/[email protected]/repo.git"+ )+ assert (+ _scrub_url_userinfo("https://user:pw@host/repo.git?x=@y")+ == "https://host/repo.git?x=@y"+ )++ def test_host_port_authority_is_not_treated_as_userinfo(self):+ from semantica.ingest.looker_ingestor import _scrub_url_userinfo++ assert (+ _scrub_url_userinfo("https://example.com:8443/path@file")+ == "https://example.com:8443/path@file"+ )+ assert (+ _scrub_url_userinfo("https://[::1]:8080/path@file")+ == "https://[::1]:8080/path@file"+ )++ def test_guard_adapter_pinned_marker_is_derived_and_read_only(self):+ """``_semantica_pinned`` is derived from the delegate, not cached."""+ from semantica.ingest.looker_ingestor import _SSRFGuardAdapter++ adapter = _SSRFGuardAdapter(allow_private_ips=False)++ assert adapter._semantica_pinned is False+ with pytest.raises(AttributeError):+ adapter._semantica_pinned = True++ @requires_looker_sdk+ def test_ingestor_config_dict_does_not_collide_with_named_arguments(self):+ """A config dict naming a connector parameter must not raise."""+ from semantica.ingest.looker_ingestor import LookerConnector, LookerIngestor++ ingestor = LookerIngestor(+ base_url=PUBLIC_BASE_URL,+ client_id=CLIENT_ID,+ client_secret=CLIENT_SECRET,+ config={"allow_private_ips": True},+ )++ assert isinstance(ingestor.connector, LookerConnector)++ @requires_looker_sdk+ def test_env_conflict_message_does_not_leak_userinfo(self):+ from semantica.ingest.looker_ingestor import LookerConnector+ from semantica.utils.exceptions import ValidationError++ env_url = "https://envuser:[email protected]"+ with patch.dict(os.environ, {"LOOKERSDK_BASE_URL": env_url}):+ with pytest.raises(ValidationError) as exc_info:+ LookerConnector(+ base_url=PUBLIC_BASE_URL,+ client_id=CLIENT_ID,+ client_secret=CLIENT_SECRET,+ )++ message = str(exc_info.value)+ assert "ENVSECRET" not in message+ assert "envuser" not in message++ @requires_looker_sdk+ def test_ini_conflict_message_does_not_leak_userinfo(self):+ from semantica.ingest.looker_ingestor import LookerConnector+ from semantica.utils.exceptions import ValidationError++ raw_config = {"base_url": "https://iniuser:[email protected]"}++ with patch.object(LookerConnector, "_raw_read_config", return_value=raw_config):+ with pytest.raises(ValidationError) as exc_info:+ LookerConnector(+ base_url=PUBLIC_BASE_URL,+ client_id=CLIENT_ID,+ client_secret=CLIENT_SECRET,+ )++ message = str(exc_info.value)+ assert "INISECRET" not in message+ assert "iniuser" not in message+++# ---------------------------------------------------------------------------+# TestLookerIngestorConfigForwarding — config keys reach the connector+# ---------------------------------------------------------------------------+++class TestLookerIngestorConfigForwarding:+ """A ``config`` value for a named connector argument must reach it.++ ``_CONNECTOR_ARGUMENT_NAMES`` keeps those keys out of the forwarded+ ``**kwargs`` (so the call cannot collide), which means the ingestor must+ resolve each one explicitly instead of silently dropping it.+ """++ @pytest.mark.parametrize(+ ("key", "value"),+ [+ ("base_url", PUBLIC_BASE_URL),+ ("client_id", "config-client-id"),+ ("client_secret", "config-client-secret"),+ ("config_file", "custom.ini"),+ ("section", "production"),+ ],+ )+ def test_config_value_reaches_the_connector(self, key, value):+ from semantica.ingest.looker_ingestor import LookerIngestor++ with patch(+ "semantica.ingest.looker_ingestor.LookerConnector"+ ) as mock_connector:+ LookerIngestor(config={key: value})++ assert mock_connector.call_args.kwargs[key] == value++ def test_named_argument_wins_over_config_value(self):+ from semantica.ingest.looker_ingestor import LookerIngestor++ with patch(+ "semantica.ingest.looker_ingestor.LookerConnector"+ ) as mock_connector:+ LookerIngestor(+ base_url=PUBLIC_BASE_URL,+ config={"base_url": "https://config.example.com"},+ )++ assert mock_connector.call_args.kwargs["base_url"] == PUBLIC_BASE_URL+++# ---------------------------------------------------------------------------+# TestLookerIngestorMalformedRecords — SDK data problems fail safely+# ---------------------------------------------------------------------------+++class TestLookerIngestorMalformedRecords:+ """Malformed SDK records must use the documented exception types."""++ def test_unsupported_record_type_raises_processing_error(self):+ from semantica.utils.exceptions import ProcessingError++ client = MagicMock()+ client.all_looks.return_value = [42]+ ingestor = _make_ingestor(client)++ with pytest.raises(ProcessingError) as exc_info:+ ingestor.ingest_looks()++ assert "Unsupported Looker record type: int" in str(exc_info.value)++ def test_record_conversion_failure_is_chained(self):+ from semantica.utils.exceptions import ProcessingError++ class _Unconvertible:+ __slots__ = ()++ def items(self):+ raise RuntimeError("boom")++ def keys(self):+ raise RuntimeError("boom")++ client = MagicMock()+ client.all_looks.return_value = [_Unconvertible()]+ ingestor = _make_ingestor(client)++ with pytest.raises(ProcessingError) as exc_info:+ ingestor.ingest_looks()++ assert isinstance(exc_info.value.__cause__, RuntimeError)+ assert "Unsupported Looker record type: _Unconvertible" in str(exc_info.value)++ def test_non_mapping_row_raises_documented_validation_error(self):+ from semantica.ingest.looker_ingestor import LookerData+ from semantica.utils.exceptions import ValidationError++ ingestor = _make_ingestor(MagicMock())+ data = LookerData(data=["oops"], row_count=1, columns=[], content_type="look")++ with pytest.raises(ValidationError) as exc_info:+ ingestor.export_as_documents(data)++ assert "str" in str(exc_info.value)++ def test_non_iterable_explores_degrades_safely(self):+ from semantica.ingest.looker_ingestor import LookerData++ ingestor = _make_ingestor(MagicMock())+ data = LookerData(+ data=[{"name": "model", "project_name": "proj", "explores": 5}],+ row_count=1,+ columns=[],+ content_type="lookml_model",+ )++ documents = ingestor.export_as_documents(data)++ metadata = documents[0]["metadata"]+ assert metadata["explores"] == []+ assert metadata["explore_count"] == 0+++# ---------------------------------------------------------------------------+# TestLookerIngestorOptionForwarding — **options reach the SDK method+# ---------------------------------------------------------------------------+++class TestLookerIngestorOptionForwarding:+ """Each read's documented ``**options`` must reach the SDK call."""++ _READS = [+ ("ingest_looks", "all_looks"),+ ("ingest_dashboards", "all_dashboards"),+ ("ingest_lookml_models", "all_lookml_models"),+ ("ingest_folders", "all_folders"),+ ("ingest_projects", "all_projects"),+ ]++ @pytest.mark.parametrize(("method_name", "sdk_method"), _READS)+ def test_options_are_forwarded_to_the_sdk_method(self, method_name, sdk_method):+ client = MagicMock()+ getattr(client, sdk_method).return_value = []+ ingestor = _make_ingestor(client)++ getattr(ingestor, method_name)(fields="id,title", limit=5, offset=10)++ assert getattr(client, sdk_method).call_args == call(+ fields="id,title", limit=5, offset=10+ )++ @pytest.mark.parametrize(("method_name", "sdk_method"), _READS)+ def test_no_options_sends_no_keyword_arguments(self, method_name, sdk_method):+ client = MagicMock()+ getattr(client, sdk_method).return_value = []+ ingestor = _make_ingestor(client)++ getattr(ingestor, method_name)()++ assert getattr(client, sdk_method).call_args == call()+++# ---------------------------------------------------------------------------+# TestLookerIngestorConfigOptIn — AE2/AE3 config allow_private_ips resolution+# ---------------------------------------------------------------------------+++@requires_looker_sdk+class TestLookerIngestorConfigOptIn:+ """``config={"allow_private_ips": True}`` must reach the connector."""++ @pytest.mark.parametrize("kwargs", [{}, {"allow_private_ips": False}])+ def test_config_opt_in_wins_when_named_argument_is_left_at_default(self, kwargs):+ """AE2/AE3 — the config entry wins while the named arg is unset."""+ from semantica.ingest.looker_ingestor import LookerIngestor++ ingestor = LookerIngestor(+ base_url=PRIVATE_BASE_URL,+ client_id=CLIENT_ID,+ client_secret=CLIENT_SECRET,+ config={"allow_private_ips": True},+ **kwargs,+ )++ assert ingestor.connector.allow_private_ips is True++ def test_config_value_wins_over_a_non_default_named_argument(self):+ """The connector's config-wins precedence, parsed with parse_bool."""+ from semantica.ingest.looker_ingestor import LookerIngestor++ ingestor = LookerIngestor(+ base_url=PUBLIC_BASE_URL,+ client_id=CLIENT_ID,+ client_secret=CLIENT_SECRET,+ allow_private_ips=True,+ config={"allow_private_ips": "0"},+ )++ assert ingestor.connector.allow_private_ips is False+++# ---------------------------------------------------------------------------+# TestLookerConnectorErrorDocEgress — the SDK's error-doc fetch is neutralised+# ---------------------------------------------------------------------------+++@requires_looker_sdk+class TestLookerConnectorErrorDocEgress:+ """The SDK's error-documentation lookup must not make its own requests.++ ``looker_sdk.error.ErrorDocHelper.get_index`` performs a bare+ ``requests.get`` (fresh session, ``trust_env=True``, no timeout, no+ redirect cap, no SSRF validation) whenever the API returns a non-2xx+ response. ``connect()`` disables it, so a failing API call still raises+ the normal SDK error without opening a second, unguarded egress path.+ """++ def _connected_connector(self):+ from semantica.ingest.looker_ingestor import LookerConnector++ client = _make_mock_client()+ with patch(+ "semantica.ingest.looker_ingestor.looker_sdk.init40",+ return_value=client,+ ):+ connector = LookerConnector(+ base_url=PUBLIC_BASE_URL,+ client_id=CLIENT_ID,+ client_secret=CLIENT_SECRET,+ )+ connector.connect()+ return connector++ def test_error_doc_lookup_makes_no_network_call(self):+ import looker_sdk.error as sdk_error++ self._connected_connector()++ helper = sdk_error.ErrorDocHelper()+ documentation_url = "https://docs.looker.com/r/err/4.0/404/notfound"+ with patch.object(+ sdk_error.requests, "get", side_effect=AssertionError("network egress")+ ) as mock_get:+ error_doc_url, error_doc = helper.parse_and_lookup(documentation_url)++ mock_get.assert_not_called()+ # No index was fetched, so no per-code document URL was resolved+ # either: the helper falls back to its base URL and the inline+ # "no documentation" message.+ assert error_doc_url == helper.ERROR_CODES_URL+ assert "No documentation found" in error_doc++ def test_non_2xx_response_raises_the_normal_sdk_error(self):+ import looker_sdk.error as sdk_error+ from looker_sdk.rtl import api_methods++ self._connected_connector()++ documentation_url = "https://docs.looker.com/r/err/4.0/404/notfound"++ class _Auth:+ class settings: # noqa: N801 - mirrors the SDK settings shape+ base_url = PUBLIC_BASE_URL++ class _Response:+ ok = False+ encoding = "utf-8"+ value = b'{"message": "not found"}'++ def _deserialize(*, data, structure):+ return sdk_error.SDKError(+ message="not found", documentation_url=documentation_url+ )++ methods = api_methods.APIMethods(+ auth=_Auth(),+ deserialize=_deserialize,+ serialize=MagicMock(),+ transport=MagicMock(),+ api_version="4.0",+ )++ with patch.object(+ sdk_error.requests, "get", side_effect=AssertionError("network egress")+ ) as mock_get:+ with pytest.raises(sdk_error.SDKError) as exc_info:+ methods._return(_Response(), structure=sdk_error.SDKError)++ mock_get.assert_not_called()+ assert "No documentation found" in exc_info.value.error_docdiff --git a/uv.lock b/uv.lockindex 34e8460f..e5763913 100644--- a/uv.lock+++ b/uv.lock@@ -725,6 +725,20 @@ wheels = [{ url = "https://files.pythonhosted.org/packages/9e/96/d32b941a501ab566a16358d68b6eb4e4acc373fab3c3c4d7d9e649f7b4bb/catalogue-2.0.10-py3-none-any.whl", hash = "sha256:58c2de0020aa90f4a2da7dfad161bf7b3b054c86a5f09fcedc0b2b740c109a9f", size = 17325, upload-time = "2023-09-25T06:29:23.337Z" },]+[[package]]+name = "cattrs"+version = "26.2.0"+source = { registry = "https://pypi.org/simple" }+dependencies = [+ { name = "attrs" },+ { name = "exceptiongroup", marker = "python_full_version < '3.11'" },+ { name = "typing-extensions" },+]+sdist = { url = "https://files.pythonhosted.org/packages/d6/b2/42f4524e5479b090040b5fd8bb316dd8c65a079bb6492494ce2079dc91be/cattrs-26.2.0.tar.gz", hash = "sha256:3cf49f69df8326bcf17a3cb3d3d3ec4a856858fe3a7473746c9044c317d3ba55", size = 524993, upload-time = "2026-09-06T20:31:58.055Z" }+wheels = [+ { url = "https://files.pythonhosted.org/packages/f5/4d/6b8460462012a52075ba97932197f8c141be71b8e5f7e918af08ee73424b/cattrs-26.2.0-py3-none-any.whl", hash = "sha256:c680f17cd1df3a1c038ad1db8d90a9c595568cc54c4e8098e7ceb8d7b9bd826b", size = 74830, upload-time = "2026-09-06T20:31:56.327Z" },+]+[[package]]name = "celery"version = "5.6.3"@@ -4043,6 +4057,21 @@ wheels = [{ url = "https://files.pythonhosted.org/packages/0c/29/0348de65b8cc732daa3e33e67806420b2ae89bdce2b04af740289c5c6c8c/loguru-0.7.3-py3-none-any.whl", hash = "sha256:31a33c10c8e1e10422bfd431aeb5d351c7cf7fa671e3c4df004162264b28220c", size = 61595, upload-time = "2024-12-06T11:20:54.538Z" },]+[[package]]+name = "looker-sdk"+version = "26.12.0"+source = { registry = "https://pypi.org/simple" }+dependencies = [+ { name = "attrs" },+ { name = "cattrs" },+ { name = "requests" },+ { name = "typing-extensions" },+]+sdist = { url = "https://files.pythonhosted.org/packages/7c/b2/2e2aba48b687a0237bf1d84d0e3ea99c433232b5ae662369ec8439793987/looker_sdk-26.12.0.tar.gz", hash = "sha256:4dd327927b541e322de8bee1820cf1ce212c1883dc51b8bc098efe12c314e3f5", size = 240874, upload-time = "2026-07-13T20:37:32.284Z" }+wheels = [+ { url = "https://files.pythonhosted.org/packages/ea/a4/94b4c13e8ddc705b0607287f4e778a1cb54aab00d330350dd85dab3310a6/looker_sdk-26.12.0-py3-none-any.whl", hash = "sha256:c765f95c01ceaf3c162a779c4794dfaeb8696b4e1e16a81fb8bb1f253ce74b57", size = 262578, upload-time = "2026-07-13T20:37:30.791Z" },+]+[[package]]name = "lxml"version = "6.1.3"@@ -8344,6 +8373,9 @@ ingest-arrow = [ingest-git = [{ name = "gitpython" },]+ingest-looker = [+ { name = "looker-sdk" },+]ingest-parquet = [{ name = "pyarrow" },]@@ -8542,6 +8574,7 @@ requires-dist = [{ name = "librosa", marker = "extra == 'media'", specifier = ">=0.9.0" },{ name = "litellm", marker = "extra == 'llm-litellm'", specifier = ">=1.83.9" },{ name = "loguru", specifier = ">=0.7.3" },+ { name = "looker-sdk", marker = "extra == 'ingest-looker'", specifier = ">=24.0.0" },{ name = "lxml", marker = "extra == 'documents'", specifier = ">=6.1.1" },{ name = "matplotlib", marker = "extra == 'viz'", specifier = ">=3.9.4" },{ name = "neo4j", marker = "extra == 'graph-amazon-neptune'", specifier = ">=5.0.0" },@@ -8625,7 +8658,7 @@ requires-dist = [{ name = "weaviate-client", marker = "extra == 'vectorstore-weaviate'", specifier = ">=4.0.0" },{ name = "websockets", marker = "extra == 'explorer'", specifier = ">=15.0.1" },]-provides-extras = ["llm-openai", "llm-groq", "llm-gemini", "llm-anthropic", "llm-ollama", "llm-deepseek", "llm-novita", "llm-litellm", "llm-instructor", "llm-all", "documents", "parse-docling", "parse-pdf", "shacl", "db-snowflake", "db-databricks", "db-arrow", "db-salesforce", "db-redshift", "db-bigquery", "ingest-parquet", "ingest-arrow", "ingest-sap", "ingest-airflow", "ingest-git", "db-all", "models-huggingface", "embeddings-local", "nlp-spacy", "graph-neo4j", "graph-falkordb", "graph-amazon-neptune", "graph-apache-age", "graph-embeddings", "graph-all", "tripletstore-oxigraph", "vectorstore-faiss", "vectorstore-qdrant", "vectorstore-weaviate", "vectorstore-pinecone", "vectorstore-milvus", "vectorstore-pgvector", "vectorstore-sqlite", "vectorstore-all", "infra", "cloud", "monitoring", "viz", "media", "gpu", "agno", "langchain", "google-adk", "open-ontologies", "watch", "split-tiktoken", "split-community", "split-topic", "split-all", "explorer", "explorer-lite", "all"]+provides-extras = ["llm-openai", "llm-groq", "llm-gemini", "llm-anthropic", "llm-ollama", "llm-deepseek", "llm-novita", "llm-litellm", "llm-instructor", "llm-all", "documents", "parse-docling", "parse-pdf", "shacl", "db-snowflake", "db-databricks", "db-arrow", "db-salesforce", "db-redshift", "db-bigquery", "ingest-looker", "ingest-parquet", "ingest-arrow", "ingest-sap", "ingest-airflow", "ingest-git", "db-all", "models-huggingface", "embeddings-local", "nlp-spacy", "graph-neo4j", "graph-falkordb", "graph-amazon-neptune", "graph-apache-age", "graph-embeddings", "graph-all", "tripletstore-oxigraph", "vectorstore-faiss", "vectorstore-qdrant", "vectorstore-weaviate", "vectorstore-pinecone", "vectorstore-milvus", "vectorstore-pgvector", "vectorstore-sqlite", "vectorstore-all", "infra", "cloud", "monitoring", "viz", "media", "gpu", "agno", "langchain", "google-adk", "open-ontologies", "watch", "split-tiktoken", "split-community", "split-topic", "split-all", "explorer", "explorer-lite", "all"][package.metadata.requires-dev]dev = [