Skip to content

LLM-Driven Extraction of Unstructured Data — Built for API Deployments & ETL Pipeline Workflows

UN-4402 [FEAT] Global search API (#2343)

Ritwik Ga0fdc1a3c006
diff --git a/backend/backend/settings/base.py b/backend/backend/settings/base.py
index 375cc73ce..4ad2ed50a 100644
--- a/backend/backend/settings/base.py
+++ b/backend/backend/settings/base.py
@@ -456,6 +456,7 @@ SHARED_APPS = (
"dashboard_metrics",
"platform_api",
"global_api_deployment_key",
+ "global_search",
)
TENANT_APPS = []
diff --git a/backend/backend/urls_v2.py b/backend/backend/urls_v2.py
index 4b9636a35..0fef98ea8 100644
--- a/backend/backend/urls_v2.py
+++ b/backend/backend/urls_v2.py
@@ -67,6 +67,7 @@ urlpatterns = [
path("metrics/", include("dashboard_metrics.urls")),
path("platform-api/", include("platform_api.urls")),
path("global-api-deployment/", include("global_api_deployment_key.urls")),
+ path("", include("global_search.urls")),
]
# Organization-scoped MCP server — OFF by default.
diff --git a/backend/global_search/README.md b/backend/global_search/README.md
new file mode 100644
index 000000000..fe6d4bbfe
--- /dev/null
+++ b/backend/global_search/README.md
@@ -0,0 +1,156 @@
+# Global search
+
+One endpoint that finds anything the caller can open, grouped by type. This
+file is the published contract for consumers: P2-T10, the top-bar search
+panel, and PB-T18, the portal search, which shares the envelope. The
+response JSON Schema is
+[`contract/search-response.schema.json`](contract/search-response.schema.json);
+tests validate real responses against it and check the type table's order,
+type, label and route.key columns against the registered providers.
+
+## Endpoint
+
+```
+GET /api/v1/unstract/{org_id}/search/?q=<text>&types=<csv>&limit=<n>
+```
+
+Authentication uses the session auth stack shared by the other tenant
+endpoints. Service accounts get the same visibility rules their `for_user()`
+gives them in each list. The endpoint is additive: no existing endpoint
+changes.
+
+## Parameters
+
+| param | default | rules |
+|---|---|---|
+| `q` | `""` | Trimmed. After trimming, **fewer than 2 characters returns 200 with `groups: []`**. More than 100 characters returns 400. Matches name and description case-insensitively; `%` and `_` match literally. |
+| `types` | all | Comma-separated type keys. Unknown or unavailable keys are **ignored**, so one panel can send cloud keys to an OSS backend. |
+| `limit` | `5` | Results per type, 1 to 20. A non-integer or out-of-range value returns 400. |
+
+## Response
+
+```json
+{
+ "query": "invoice",
+ "groups": [
+ {
+ "type": "prompt_studio_project",
+ "label": "Prompt Studio projects",
+ "has_more": true,
+ "results": [
+ {
+ "type": "prompt_studio_project",
+ "id": "6f1c2a4e-1d1b-4c8e-9a51-0c7b2f1e9d10",
+ "name": "Invoice parser",
+ "context": "[email protected]",
+ "match": "name_prefix",
+ "route": {
+ "key": "prompt_studio_project.detail",
+ "params": {"id": "6f1c2a4e-1d1b-4c8e-9a51-0c7b2f1e9d10"}
+ }
+ }
+ ]
+ }
+ ],
+ "failed_types": []
+}
+```
+
+- Groups are returned in the `order` of the table below. **Empty groups are
+ omitted.**
+- `has_more` is true when the type has more matches than `limit`. There is
+ no total count; the "More" link opens the type's own list.
+- `match` names the tier that ranked the result. Results are sorted by tier
+ (`name_prefix`, `name`, `description_prefix`, `description`), then by most
+ recently modified, then by id.
+- `context` is a single display-only line, or `null`. Do not parse it.
+- `route.key` is a route-registry name, not a URL. `route.params` values are
+ strings. It is one of the keys the table below lists for the result's
+ type; adapters have two (see below the table). A client that does not
+ know a key should render the result without a link, never guess a
+ screen.
+- `type` is an open string. A client should render an unknown type with its
+ `label`, or skip it.
+- `failed_types` lists types whose search failed; their groups are missing
+ even if they have matches, so a client can say "some results could not be
+ loaded". It is empty on a normal response.
+- `route.params` always includes `id`.
+
+Errors use the standard error envelope:
+
+- `400 {"type": "validation_error", "errors": [{"code", "detail", "attr"}]}`
+ for bad parameters.
+- `503` with code `search_unavailable` when every searched type failed with
+ a database error a retry may clear (unreachable, timed out, locked), or
+ when a failure left the connection unusable (even if other types had
+ already answered). Safe to retry; never show it as "no results". A slow or
+ locked query on a healthy connection otherwise only drops its own type
+ into `failed_types` while other types still answer.
+- `500` with code `search_failed` when every searched type failed and not
+ all with such database errors (a bug). Retrying will not help.
+- 401 and 403 behave as on every tenant endpoint.
+
+## Types
+
+| order | type | label | route.key | route.params | context |
+|---|---|---|---|---|---|
+| 10 | `prompt_studio_project` | Prompt Studio projects | `prompt_studio_project.detail` | `id` | owner e-mail |
+| 30 | `workflow` | Workflows | `workflow.detail` | `id` | owner e-mail |
+| 40 | `api_deployment` | API deployments | `api_deployment.detail` | `id` | parent workflow name |
+| 50 | `etl_pipeline` | ETL pipelines | `etl_pipeline.detail` | `id` | parent workflow name |
+| 60 | `task_pipeline` | Task pipelines | `task_pipeline.detail` | `id` | parent workflow name |
+| 80 | `adapter` | Adapters | `adapter.detail` / `adapter.info` | `id`, `adapter_type` | adapter type · provider |
+| 90 | `connector` | Connectors | `connector.detail` | `id` | provider |
+
+An adapter opens on `adapter.detail` only for a caller who may edit it
+(owner or co-owner, org admin, service account), the same rule as the
+adapter `retrieve` endpoint. Everyone else gets `adapter.info`, with the
+same params, a read-only view backed by `adapter/info/<id>/`, which allows
+every caller search sends there.
+
+Cloud and on-prem add `agentic_project` (20) and `review_queue` (70). Those
+apps register these types themselves, and they are documented in the cloud
+repo.
+
+### What is matched and who sees it
+
+Each type starts from the same `for_user()` queryset its list view uses.
+That covers owners, co-owners, direct viewers, group shares, org shares,
+admins and service accounts. Search rules:
+
+- Names matched: `tool_name`, `workflow_name`, `display_name` and
+ `api_name`, `pipeline_name`, `adapter_name`, `connector_name`.
+- Descriptions matched: Prompt Studio projects, workflows, API deployments
+ and adapters. Pipelines and connectors have no description.
+- Pipelines: ETL and Task only. DEFAULT and APP pipelines are excluded.
+- Adapters: platform-provided (frictionless) adapters are never returned.
+ The adapter list shows them so any member can delete one, but nobody can
+ open their detail view. No Op adapters are never returned either.
+- Owner e-mail is not matched. The workflow, Prompt Studio, adapter and
+ connector list `search` parameters do match it, so the panel and those
+ filtered lists can differ slightly.
+
+## Response-time budget
+
+**p95 ≤ 300 ms server time for all types at `limit=5`**, on an organisation
+of the largest current size. The fixture used is 2,000 rows per type, 200
+users and 20 groups; it will be updated when real largest-org counts are
+available.
+
+CI does not gate on latency, because runners are shared. `tests/test_budget.py`
+gates on query count instead: a request may issue at most three SQL queries
+per type in total (one ranked search per type, plus the organisation point
+lookups the existing `for_user()` makes) plus a small fixed overhead, and
+the count must not change between `limit=1` and `limit=20`, so an N+1 query
+pattern fails the test. The measured p95 is printed on every run and is
+asserted only with `GLOBAL_SEARCH_ASSERT_P95=1`, against
+`GLOBAL_SEARCH_CEILING_MS` (default 300). No trigram index is used: every
+query is filtered by organisation first.
+
+## Adding a type
+
+Subclass `global_search.providers.SearchProvider`, or use
+`ModelSearchProvider` for a single queryset. It must apply the type's own
+list visibility rule. Register it with
+`global_search.registry.register_provider()` from the owning app's
+`AppConfig.ready()`, add a row to the table above, and add visibility tests.
diff --git a/backend/global_search/__init__.py b/backend/global_search/__init__.py
new file mode 100644
index 000000000..e69de29bb
diff --git a/backend/global_search/apps.py b/backend/global_search/apps.py
new file mode 100644
index 000000000..bad24c2ec
--- /dev/null
+++ b/backend/global_search/apps.py
@@ -0,0 +1,12 @@
+from django.apps import AppConfig
+
+
+class GlobalSearchConfig(AppConfig):
+ name = "global_search"
+
+ def ready(self) -> None:
+ from global_search.oss_providers import OSS_PROVIDERS
+ from global_search.registry import register_provider
+
+ for provider_cls in OSS_PROVIDERS:
+ register_provider(provider_cls())
diff --git a/backend/global_search/constants.py b/backend/global_search/constants.py
new file mode 100644
index 000000000..ffec01c2a
--- /dev/null
+++ b/backend/global_search/constants.py
@@ -0,0 +1,45 @@
+"""Contract constants for the global search endpoint: parameter bounds,
+match tiers and the OSS type keys. Cloud-only type keys are deliberately
+absent: cloud apps define and register their own.
+"""
+
+from enum import IntEnum
+
+
+class SearchParams:
+ QUERY = "q"
+ TYPES = "types"
+ LIMIT = "limit"
+
+ # A shorter query answers an empty result, not an error.
+ MIN_QUERY_LENGTH = 2
+ MAX_QUERY_LENGTH = 100
+ DEFAULT_LIMIT = 5
+ MAX_LIMIT = 20
+
+
+class MatchTier(IntEnum):
+ """Ranking tiers, best first. The value is the sort key; the label is
+ what the response's ``match`` field carries.
+ """
+
+ NAME_PREFIX = 0
+ NAME = 1
+ DESCRIPTION_PREFIX = 2
+ DESCRIPTION = 3
+
+ @property
+ def label(self) -> str:
+ return self.name.lower()
+
+
+class OSSSearchType:
+ """Type keys contributed by OSS."""
+
+ PROMPT_STUDIO_PROJECT = "prompt_studio_project"
+ WORKFLOW = "workflow"
+ API_DEPLOYMENT = "api_deployment"
+ ETL_PIPELINE = "etl_pipeline"
+ TASK_PIPELINE = "task_pipeline"
+ ADAPTER = "adapter"
+ CONNECTOR = "connector"
diff --git a/backend/global_search/contract/search-response.schema.json b/backend/global_search/contract/search-response.schema.json
new file mode 100644
index 000000000..cc5180ed2
--- /dev/null
+++ b/backend/global_search/contract/search-response.schema.json
@@ -0,0 +1,120 @@
+{
+ "$schema": "https://json-schema.org/draft/2020-12/schema",
+ "$id": "unstract/global-search/response/v1",
+ "type": "object",
+ "required": [
+ "query",
+ "groups",
+ "failed_types"
+ ],
+ "additionalProperties": false,
+ "properties": {
+ "query": {
+ "type": "string"
+ },
+ "groups": {
+ "type": "array",
+ "items": {
+ "$ref": "#/$defs/group"
+ }
+ },
+ "failed_types": {
+ "type": "array",
+ "items": {
+ "type": "string"
+ }
+ }
+ },
+ "$defs": {
+ "group": {
+ "type": "object",
+ "required": [
+ "type",
+ "label",
+ "has_more",
+ "results"
+ ],
+ "additionalProperties": false,
+ "properties": {
+ "type": {
+ "type": "string",
+ "pattern": "^[a-z][a-z0-9_]*$"
+ },
+ "label": {
+ "type": "string"
+ },
+ "has_more": {
+ "type": "boolean"
+ },
+ "results": {
+ "type": "array",
+ "minItems": 1,
+ "maxItems": 20,
+ "items": {
+ "$ref": "#/$defs/result"
+ }
+ }
+ }
+ },
+ "result": {
+ "type": "object",
+ "required": [
+ "type",
+ "id",
+ "name",
+ "context",
+ "match",
+ "route"
+ ],
+ "additionalProperties": false,
+ "properties": {
+ "type": {
+ "type": "string"
+ },
+ "id": {
+ "type": "string"
+ },
+ "name": {
+ "type": "string"
+ },
+ "context": {
+ "type": [
+ "string",
+ "null"
+ ]
+ },
+ "match": {
+ "enum": [
+ "name_prefix",
+ "name",
+ "description_prefix",
+ "description"
+ ]
+ },
+ "route": {
+ "type": "object",
+ "required": [
+ "key",
+ "params"
+ ],
+ "additionalProperties": false,
+ "properties": {
+ "key": {
+ "type": "string",
+ "pattern": "^[a-z][a-z0-9_]*\\.[a-z][a-z0-9_]*$"
+ },
+ "params": {
+ "type": "object",
+ "additionalProperties": {
+ "type": "string"
+ },
+ "required": [
+ "id"
+ ]
+ }
+ }
+ }
+ }
+ }
+ }
+}
diff --git a/backend/global_search/exceptions.py b/backend/global_search/exceptions.py
new file mode 100644
index 000000000..164b8c527
--- /dev/null
+++ b/backend/global_search/exceptions.py
@@ -0,0 +1,23 @@
+from rest_framework import status
+from rest_framework.exceptions import APIException
+
+
+class SearchUnavailable(APIException):
+ """Search cannot answer because of database errors a retry may clear
+ (unreachable, timed out, locked) or an unusable connection. The client
+ may retry; never "no matches".
+ """
+
+ status_code = status.HTTP_503_SERVICE_UNAVAILABLE
+ default_detail = "Search is temporarily unavailable."
+ default_code = "search_unavailable"
+
+
+class SearchFailed(APIException):
+ """Every searched type failed, and not all with retryable database
+ errors -- most likely a bug, so retrying will not help.
+ """
+
+ status_code = status.HTTP_500_INTERNAL_SERVER_ERROR
+ default_detail = "Search failed."
+ default_code = "search_failed"
diff --git a/backend/global_search/oss_providers.py b/backend/global_search/oss_providers.py
new file mode 100644
index 000000000..2ab6a1796
--- /dev/null
+++ b/backend/global_search/oss_providers.py
@@ -0,0 +1,221 @@
+"""OSS result types.
+
+Each provider starts from the same ``Model.objects.for_user(user)`` its list
+view uses, so search never returns a row the caller could not open from the
+list. Search may narrow further (pipeline type, No Op and frictionless
+adapters), never widen.
+"""
+
+from __future__ import annotations
+
+import dataclasses
+from typing import Any
+
+from adapter_processor_v2.models import AdapterInstance
+from api_v2.models import APIDeployment
+from connector_v2.models import ConnectorInstance
+from django.contrib.contenttypes.models import ContentType
+from django.db.models import (
+ BooleanField,
+ CharField,
+ Exists,
+ F,
+ Model,
+ OuterRef,
+ Q,
+ QuerySet,
+ Subquery,
+ Value,
+)
+from django.db.models.functions import Cast
+from permissions.roles import ResourceRole
+from pipeline_v2.models import Pipeline
+from prompt_studio.prompt_studio_core_v2.models import CustomTool
+from tenant_account_v2.models import ResourceMembership
+from tenant_account_v2.organization_member_service import OrganizationMemberService
+from workflow_manager.workflow_v2.models.workflow import Workflow
+
+from global_search.constants import OSSSearchType
+from global_search.providers import ModelSearchProvider, SearchHit
+
+OWNER_EMAIL = "_search_owner_email"
+PARENT_NAME = "_search_parent_name"
+CAN_EDIT = "_search_can_edit"
+
+# Frontend v1 retires the No Op adapters (spec decision D12, UN-4257);
+# their ids share this prefix.
+NO_OP_ADAPTER_PREFIX = "noOp"
+
+
+def owner_email_subquery(model: type[Model]) -> Subquery:
+ """Earliest live, non-service-account OWNER's e-mail, as one correlated
+ subquery. Same rule as ``HasMembersMixin.owner_email`` without the
+ per-row prefetch.
+ """
+ return Subquery(
+ ResourceMembership.objects.filter(
+ content_type=ContentType.objects.get_for_model(model),
+ object_id=Cast(OuterRef("pk"), output_field=CharField()),
+ role=ResourceRole.OWNER,
+ user__is_service_account=False,
+ )
+ .order_by("created_at", "pk")
+ .values("user__email")[:1]
+ )
+
+
+def _provider(registry_id: str) -> str:
+ """Provider slug from an adapter/connector registry id (``openai|<uuid>``)."""
+ return registry_id.split("|", 1)[0] if registry_id else ""
+
+
+class OwnedModelProvider(ModelSearchProvider):
+ """Context line is the owner's e-mail."""
+
+ def annotate(self, queryset: QuerySet[Any], user: Any) -> QuerySet[Any]:
+ return queryset.annotate(**{OWNER_EMAIL: owner_email_subquery(queryset.model)})
+
+ def context(self, obj: Model) -> str | None:
+ return getattr(obj, OWNER_EMAIL)
+
+
+class WorkflowChildProvider(ModelSearchProvider):
+ """Context line is the parent workflow's name."""
+
+ def annotate(self, queryset: QuerySet[Any], user: Any) -> QuerySet[Any]:
+ return queryset.annotate(**{PARENT_NAME: F("workflow__workflow_name")})
+
+ def context(self, obj: Model) -> str | None:
+ return getattr(obj, PARENT_NAME)
+
+
+class PromptStudioProjectProvider(OwnedModelProvider):
+ type_key = OSSSearchType.PROMPT_STUDIO_PROJECT
+ label = "Prompt Studio projects"
+ order = 10
+ name_fields = ("tool_name",)
+ description_field = "description"
+
+ def visible_queryset(self, user: Any) -> QuerySet[Any]:
+ return CustomTool.objects.for_user(user)
+
+
+class WorkflowProvider(OwnedModelProvider):
+ type_key = OSSSearchType.WORKFLOW
+ label = "Workflows"
+ order = 30
+ name_fields = ("workflow_name",)
+ description_field = "description"
+
+ def visible_queryset(self, user: Any) -> QuerySet[Any]:
+ return Workflow.objects.for_user(user)
+
+
+class APIDeploymentProvider(WorkflowChildProvider):
+ type_key = OSSSearchType.API_DEPLOYMENT
+ label = "API deployments"
+ order = 40
+ # A match on api_name alone still shows display_name: that is the name
+ # users see everywhere else.
+ name_fields = ("display_name", "api_name")
+ description_field = "description"
+
+ def visible_queryset(self, user: Any) -> QuerySet[Any]:
+ return APIDeployment.objects.for_user(user)
+
+
+class _PipelineProvider(WorkflowChildProvider):
+ pipeline_type: str
+ name_fields = ("pipeline_name",)
+
+ def visible_queryset(self, user: Any) -> QuerySet[Any]:
+ return Pipeline.objects.for_user(user).filter(pipeline_type=self.pipeline_type)
+
+
+class ETLPipelineProvider(_PipelineProvider):
+ type_key = OSSSearchType.ETL_PIPELINE
+ label = "ETL pipelines"
+ order = 50
+ pipeline_type = Pipeline.PipelineType.ETL
+
+
+class TaskPipelineProvider(_PipelineProvider):
+ type_key = OSSSearchType.TASK_PIPELINE
+ label = "Task pipelines"
+ order = 60
+ pipeline_type = Pipeline.PipelineType.TASK
+
+
+class AdapterProvider(ModelSearchProvider):
+ type_key = OSSSearchType.ADAPTER
+ label = "Adapters"
+ order = 80
+ name_fields = ("adapter_name",)
+ description_field = "description"
+
+ def visible_queryset(self, user: Any) -> QuerySet[Any]:
+ # Frictionless adapters stay in the adapter list, where any member
+ # may delete them, but nobody can open their detail view: retrieve
+ # refuses them for everyone, admins included (IsFrictionLessAdapter).
+ return AdapterInstance.objects.for_user(user).exclude(
+ Q(adapter_id__startswith=NO_OP_ADAPTER_PREFIX) | Q(is_friction_less=True)
+ )
+
+ @property
+ def info_route_key(self) -> str:
+ return f"{self.type_key}.info"
+
+ def annotate(self, queryset: QuerySet[Any], user: Any) -> QuerySet[Any]:
+ # Same rule as IsFrictionLessAdapter, which gates retrieve: service
+ # accounts, org admins and owners (creator or co-owner). Anyone else
+ # who can list the adapter opens it read-only, via adapter_info.
+ if getattr(
+ user, "is_service_account", False
+ ) or OrganizationMemberService.is_user_organization_admin(user):
+ can_edit = Value(True, output_field=BooleanField())
+ else:
+ can_edit = Exists(
+ ResourceMembership.objects.filter(
+ content_type=ContentType.objects.get_for_model(queryset.model),
+ object_id=Cast(OuterRef("pk"), output_field=CharField()),
+ role=ResourceRole.OWNER,
+ user=user,
+ )
+ )
+ return queryset.annotate(**{CAN_EDIT: can_edit})
+
+ def to_hit(self, obj: Model) -> SearchHit:
+ hit = super().to_hit(obj)
+ if getattr(obj, CAN_EDIT):
+ return hit
+ return dataclasses.replace(hit, route_key=self.info_route_key)
+
+ def context(self, obj: Model) -> str | None:
+ return " · ".join(p for p in (obj.adapter_type, _provider(obj.adapter_id)) if p)
+
+ def route_params(self, obj: Model) -> dict[str, str]:
+ return {"id": str(obj.pk), "adapter_type": obj.adapter_type}
+
+
+class ConnectorProvider(ModelSearchProvider):
+ type_key = OSSSearchType.CONNECTOR
+ label = "Connectors"
+ order = 90
+ name_fields = ("connector_name",)
+
+ def visible_queryset(self, user: Any) -> QuerySet[Any]:
+ return ConnectorInstance.objects.for_user(user)
+
+ def context(self, obj: Model) -> str | None:
+ return _provider(obj.connector_id) or None
+
+
+OSS_PROVIDERS = (
+ PromptStudioProjectProvider,
+ WorkflowProvider,
+ APIDeploymentProvider,
+ ETLPipelineProvider,
+ TaskPipelineProvider,
+ AdapterProvider,
+ ConnectorProvider,
+)
diff --git a/backend/global_search/providers.py b/backend/global_search/providers.py
new file mode 100644
index 000000000..1a00b19f1
--- /dev/null
+++ b/backend/global_search/providers.py
@@ -0,0 +1,104 @@
+"""Search provider contract.
+
+A provider answers one result type. It owns two things the endpoint cannot
+know: which rows the caller may open (it must apply the same visibility rule
+as that type's own list) and how a row becomes a result. Ranking is shared,
+see :mod:`global_search.ranking`.
+"""
+
+from __future__ import annotations
+
+from abc import ABC, abstractmethod
+from dataclasses import dataclass
+from typing import Any
+
+from django.db.models import Model, QuerySet
+
+from global_search.constants import MatchTier
+from global_search.ranking import RANK_ANNOTATION, rank_queryset
+
+
+@dataclass(frozen=True)
+class SearchHit:
+ id: str
+ name: str
+ match: MatchTier
+ # No default, and it must carry the ``id`` the frontend opens.
+ route_params: dict[str, str]
+ context: str | None = None
+ # Overrides the provider's ``route_key`` for this hit, for a type whose
+ # rows open on different screens depending on the caller.
+ route_key: str | None = None
+
+ def __post_init__(self) -> None:
+ if "id" not in self.route_params:
+ raise ValueError(f"SearchHit {self.id!r} has no route id")
+
+
+class SearchProvider(ABC):
+ """One result type in the search response.
+
+ Subclasses set ``type_key`` (stable contract key), ``label`` (display
+ text) and ``order`` (rail position; groups are returned ascending).
+ """
+
+ type_key: str
+ label: str
+ order: int
+
+ @property
+ def route_key(self) -> str:
+ return f"{self.type_key}.detail"
+
+ @abstractmethod
+ def search(self, user: Any, query: str, limit: int) -> list[SearchHit]:
+ """Return up to ``limit + 1`` hits, best first.
+
+ The extra row lets the caller derive ``has_more`` without a COUNT.
+ Must never return a row ``user`` could not open from the type's list.
+ """
+
+
+class ModelSearchProvider(SearchProvider):
+ """Provider backed by one ORM query.
+
+ Subclasses implement :meth:`visible_queryset` with the list view's own
+ visibility rule, and :meth:`to_hit` to shape a row.
+ """
+
+ # Every field matched as "name"; the first is the displayed name.
+ name_fields: tuple[str, ...]
+ description_field: str | None = None
+
+ @abstractmethod
+ def visible_queryset(self, user: Any) -> QuerySet[Any]:
+ """Rows ``user`` can see in this type's list, before search."""
+
+ def annotate(self, queryset: QuerySet[Any], user: Any) -> QuerySet[Any]:
+ """Hook to add context or route columns without per-row queries."""
+ return queryset
+
+ @abstractmethod
+ def context(self, obj: Model) -> str | None:
+ """One display-only line, or ``None``."""
+
+ def route_params(self, obj: Model) -> dict[str, str]:
+ return {"id": str(obj.pk)}
+
+ def search(self, user: Any, query: str, limit: int) -> list[SearchHit]:
+ queryset = rank_queryset(
+ self.visible_queryset(user),
+ query,
+ name_fields=self.name_fields,
+ description_field=self.description_field,
+ )
+ return [self.to_hit(obj) for obj in self.annotate(queryset, user)[: limit + 1]]
+
+ def to_hit(self, obj: Model) -> SearchHit:
+ return SearchHit(
+ id=str(obj.pk),
+ name=getattr(obj, self.name_fields[0]),
+ match=MatchTier(getattr(obj, RANK_ANNOTATION)),
+ context=self.context(obj),
+ route_params=self.route_params(obj),
+ )
diff --git a/backend/global_search/ranking.py b/backend/global_search/ranking.py
new file mode 100644
index 000000000..4320cb14b
--- /dev/null
+++ b/backend/global_search/ranking.py
@@ -0,0 +1,104 @@
+"""Shared ranking: prefix before substring, name before description.
+
+Both an ORM form (one SQL query per type) and an in-memory form (for
+providers whose source is not a single queryset) so every type ranks alike.
+"""
+
+from __future__ import annotations
+
+from collections.abc import Callable, Iterable
+from typing import Any, TypeVar
+
+from django.db.models import Case, IntegerField, Q, QuerySet, Value, When
+
+from global_search.constants import MatchTier
+
+RANK_ANNOTATION = "_search_rank"
+
+T = TypeVar("T")
+
+
+def _any(fields: Iterable[str], lookup: str, query: str) -> Q:
+ condition = Q()
+ for field_name in fields:
+ condition |= Q(**{f"{field_name}__{lookup}": query})
+ return condition
+
+
+def rank_queryset(
+ queryset: QuerySet[Any],
+ query: str,
+ *,
+ name_fields: tuple[str, ...],
+ description_field: str | None = None,
+) -> QuerySet[Any]:
+ """Filter ``queryset`` to rows matching ``query`` and order them by tier,
+ then most recently modified, then pk so equal tiers page deterministically.
+
+ ``icontains``/``istartswith`` escape ``%`` and ``_``, so user input is
+ always matched literally.
+ """
+ tiers = [
+ When(_any(name_fields, "istartswith", query), then=Value(MatchTier.NAME_PREFIX)),
+ When(_any(name_fields, "icontains", query), then=Value(MatchTier.NAME)),
+ ]
+ matches = _any(name_fields, "icontains", query)
+ if description_field:
+ tiers += [
+ When(
+ Q(**{f"{description_field}__istartswith": query}),
+ then=Value(MatchTier.DESCRIPTION_PREFIX),
+ ),
+ When(
+ Q(**{f"{description_field}__icontains": query}),
+ then=Value(MatchTier.DESCRIPTION),
+ ),
+ ]
+ matches |= Q(**{f"{description_field}__icontains": query})
+
+ return (
+ queryset.filter(matches)
+ .annotate(**{RANK_ANNOTATION: Case(*tiers, output_field=IntegerField())})
+ .order_by(RANK_ANNOTATION, "-modified_at", "pk")
+ )
+
+
+def match_tier(
+ query: str, names: Iterable[str | None], description: str | None = None
+) -> MatchTier | None:
+ """In-memory equivalent of the ``rank_queryset`` tiers; ``None`` means no
+ match. Comparison is case-insensitive via ``casefold``.
+ """
+ needle = query.casefold()
+ names = [n.casefold() for n in names if n]
+ if any(n.startswith(needle) for n in names):
+ return MatchTier.NAME_PREFIX
+ if any(needle in n for n in names):
+ return MatchTier.NAME
+ if description:
+ desc = description.casefold()
+ if desc.startswith(needle):
+ return MatchTier.DESCRIPTION_PREFIX
+ if needle in desc:
+ return MatchTier.DESCRIPTION
+ return None
+
+
+def rank_items(
+ items: Iterable[T],
+ query: str,
+ *,
+ names: Callable[[T], Iterable[str | None]],
+ description: Callable[[T], str | None] = lambda _: None,
+ sort_key: Callable[[T], Any],
+) -> list[tuple[T, MatchTier]]:
+ """Filter and order ``items`` like :func:`rank_queryset`. ``sort_key``
+ breaks ties within a tier and must be total, for deterministic order.
+ """
+ ranked = []
+ for item in items:
+ tier = match_tier(query, names(item), description(item))
+ if tier is not None:
+ ranked.append((item, tier))
+ ranked.sort(key=lambda pair: (pair[1], sort_key(pair[0])))
+ return ranked
diff --git a/backend/global_search/registry.py b/backend/global_search/registry.py
new file mode 100644
index 000000000..f8b2c43f3
--- /dev/null
+++ b/backend/global_search/registry.py
@@ -0,0 +1,49 @@
+"""Process-wide provider registry.
+
+OSS registers its providers from ``GlobalSearchConfig.ready()``; cloud apps
+register theirs from their own ``ready()``, so a type exists exactly where
+its app is installed and OSS never names a cloud type.
+"""
+
+from __future__ import annotations
+
+import logging
+import re
+
+from global_search.providers import SearchProvider
+
+logger = logging.getLogger(__name__)
+
+_providers: dict[str, SearchProvider] = {}
+
+# Same pattern the published JSON Schema requires of ``type``.
+_TYPE_KEY = re.compile(r"^[a-z][a-z0-9_]*$")
+
+
+def register_provider(provider: SearchProvider) -> None:
+ """Register ``provider`` under its ``type_key``.
+
+ Re-registering the same class is a no-op, so a repeated ``ready()`` is
+ harmless; a different class claiming a taken key is a wiring bug.
+ """
+ if not _TYPE_KEY.fullmatch(provider.type_key):
+ raise ValueError(f"Invalid search type key '{provider.type_key}'")
+ existing = _providers.get(provider.type_key)
+ if existing is not None:
+ if type(existing) is type(provider):
+ return
+ raise ValueError(
+ f"Search type '{provider.type_key}' is already registered by "
+ f"{type(existing).__name__}"
+ )
+ _providers[provider.type_key] = provider
+ logger.debug("Registered global search provider '%s'", provider.type_key)
+
+
+def unregister_provider(type_key: str) -> None:
+ _providers.pop(type_key, None)
+
+
+def get_providers() -> list[SearchProvider]:
+ """Registered providers in rail order."""
+ return sorted(_providers.values(), key=lambda p: (p.order, p.type_key))
diff --git a/backend/global_search/serializers.py b/backend/global_search/serializers.py
new file mode 100644
index 000000000..dd973c1c5
--- /dev/null
+++ b/backend/global_search/serializers.py
@@ -0,0 +1,75 @@
+"""Request validation, and the response envelope for drf-spectacular.
+
+The view builds plain dicts; the tested contract is
+``contract/search-response.schema.json``.
+"""
+
+from rest_framework import serializers
+
+from global_search.constants import MatchTier, SearchParams
+
+
+class SearchQuerySerializer(serializers.Serializer):
+ q = serializers.CharField(
+ required=False,
+ allow_blank=True,
+ trim_whitespace=True,
+ max_length=SearchParams.MAX_QUERY_LENGTH,
+ default="",
+ help_text=(
+ "Case-insensitive text matched against name and description. "
+ f"Shorter than {SearchParams.MIN_QUERY_LENGTH} characters after "
+ "trimming returns no groups."
+ ),
+ )
+ types = serializers.CharField(
+ required=False,
+ allow_blank=True,
+ default="",
+ help_text=(
+ "Comma-separated type keys to search. Empty means all. Unknown or "
+ "unavailable keys are ignored."
+ ),
+ )
+ limit = serializers.IntegerField(
+ required=False,
+ default=SearchParams.DEFAULT_LIMIT,
+ min_value=1,
+ max_value=SearchParams.MAX_LIMIT,
+ help_text="Results per type.",
+ )
+
+ def validate_types(self, value: str) -> list[str]:
+ return [key for key in (part.strip() for part in value.split(",")) if key]
+
+
+class SearchRouteSerializer(serializers.Serializer):
+ key = serializers.CharField(help_text="Route-registry name, not a URL.")
+ params = serializers.DictField(child=serializers.CharField())
+
+
+class SearchResultSerializer(serializers.Serializer):
+ type = serializers.CharField()
+ id = serializers.CharField()
+ name = serializers.CharField(allow_blank=True)
+ context = serializers.CharField(
+ allow_null=True, help_text="One display-only line; see README for each type."
+ )
+ match = serializers.ChoiceField(choices=[tier.label for tier in MatchTier])
+ route = SearchRouteSerializer()
+
+
+class SearchGroupSerializer(serializers.Serializer):
+ type = serializers.CharField()
+ label = serializers.CharField()
+ has_more = serializers.BooleanField()
+ results = SearchResultSerializer(many=True)
+
+
+class SearchResponseSerializer(serializers.Serializer):
+ query = serializers.CharField(allow_blank=True)
+ groups = SearchGroupSerializer(many=True)
+ failed_types = serializers.ListField(
+ child=serializers.CharField(),
+ help_text="Types whose search failed; their groups are missing.",
+ )
diff --git a/backend/global_search/tests/__init__.py b/backend/global_search/tests/__init__.py
new file mode 100644
index 000000000..e69de29bb
diff --git a/backend/global_search/tests/base.py b/backend/global_search/tests/base.py
new file mode 100644
index 000000000..10cdd8971
--- /dev/null
+++ b/backend/global_search/tests/base.py
@@ -0,0 +1,126 @@
+"""Builders and a request helper shared by the global search tests."""
+
+import secrets
+from collections.abc import Callable
+from typing import Any
+
+from account_v2.models import Organization, User
+from adapter_processor_v2.models import AdapterInstance
+from api_v2.models import APIDeployment
+from connector_v2.models import ConnectorInstance
+from pipeline_v2.models import Pipeline
+from prompt_studio.prompt_studio_core_v2.models import CustomTool
+from rest_framework.response import Response
+from rest_framework.test import APIRequestFactory, force_authenticate
+from workflow_manager.workflow_v2.models.workflow import Workflow
+
+from global_search.constants import OSSSearchType
+from global_search.views import GlobalSearchView
+
+
+def _workflow(org: Organization, user: User, name: str, description: str) -> Workflow:
+ return Workflow.objects.create(
+ workflow_name=name, description=description, organization=org, created_by=user
+ )
+
+
+def _parent(org: Organization, user: User, name: str) -> Workflow:
+ # Random so the parent never matches a test query by accident.
+ return _workflow(org, user, f"parent-{secrets.token_hex(4)}", "")
+
+
+def build_prompt_studio_project(org, user, name, description=""):
+ return CustomTool.objects.create(
+ tool_name=name, description=description, organization=org, created_by=user
+ )
+
+
+def build_workflow(org, user, name, description=""):
+ return _workflow(org, user, name, description)
+
+
+def build_api_deployment(org, user, name, description=""):
+ return APIDeployment.objects.create(
+ display_name=name,
+ api_name=f"api-{secrets.token_hex(4)}",
+ description=description,
+ workflow=_parent(org, user, name),
+ organization=org,
+ created_by=user,
+ )
+
+
+def _pipeline(pipeline_type: str) -> Callable[..., Pipeline]:
+ def build(org, user, name, description=""):
+ return Pipeline.objects.create(
+ pipeline_name=name,
+ pipeline_type=pipeline_type,
+ workflow=_parent(org, user, name),
+ organization=org,
+ created_by=user,
+ )
+
+ return build
+
+
+def build_adapter(org, user, name, description="", **overrides):
+ fields = {
+ "adapter_name": name,
+ "adapter_id": "openai|test",
+ "adapter_type": "LLM",
+ "adapter_metadata": {},
+ "description": description,
+ "organization": org,
+ "created_by": user,
+ }
+ fields.update(overrides)
+ return AdapterInstance.objects.create(**fields)
+
+
+def build_connector(org, user, name, description=""):
+ return ConnectorInstance.objects.create(
+ connector_name=name,
+ connector_id="minio|c799f6e3-2b57-434e-aaac-b5daa415da19",
+ organization=org,
+ created_by=user,
+ )
+
+
+# Every OSS type, keyed by its contract type key.
+OSS_BUILDERS: dict[str, Callable[..., Any]] = {
+ OSSSearchType.PROMPT_STUDIO_PROJECT: build_prompt_studio_project,
+ OSSSearchType.WORKFLOW: build_workflow,
+ OSSSearchType.API_DEPLOYMENT: build_api_deployment,
+ OSSSearchType.ETL_PIPELINE: _pipeline(Pipeline.PipelineType.ETL),
+ OSSSearchType.TASK_PIPELINE: _pipeline(Pipeline.PipelineType.TASK),
+ OSSSearchType.ADAPTER: build_adapter,
+ OSSSearchType.CONNECTOR: build_connector,
+}
+
+# Types whose model has a description column the search matches.
+DESCRIBED_TYPES = {
+ OSSSearchType.PROMPT_STUDIO_PROJECT,
+ OSSSearchType.WORKFLOW,
+ OSSSearchType.API_DEPLOYMENT,
+ OSSSearchType.ADAPTER,
+}
+
+
+def search(user: User, allow_failures: bool = False, **params: Any) -> Response:
+ """Call the view. Unless ``allow_failures``, a 200 must report no failed
+ types, so a crashing provider is never mistaken for one that filtered."""
+ request = APIRequestFactory().get("/search/", params)
+ force_authenticate(request, user=user)
+ response = GlobalSearchView.as_view()(request)
+ if response.status_code == 200 and not allow_failures:
+ assert response.data["failed_types"] == [], response.data["failed_types"]
+ return response
+
+
+def result_ids(response: Response, type_key: str | None = None) -> list[str]:
+ return [
+ result["id"]
+ for group in response.data["groups"]
+ if type_key is None or group["type"] == type_key
+ for result in group["results"]
+ ]
diff --git a/backend/global_search/tests/test_budget.py b/backend/global_search/tests/test_budget.py
new file mode 100644
index 000000000..2b5ef29d7
--- /dev/null
+++ b/backend/global_search/tests/test_budget.py
@@ -0,0 +1,259 @@
+"""Response-time budget on a large organisation.
+
+The contract budget is p95 <= 300 ms server time for all types at limit=5.
+The merge gate is structural: a bounded total of SQL queries (at most three
+per type, summed) that does not change with the result count, and every
+type answering without failure. Wall-clock time depends on the
+runner, so the p95 is always printed but asserted only on request
+(``GLOBAL_SEARCH_ASSERT_P95=1``, against ``GLOBAL_SEARCH_CEILING_MS``).
+
+The fixture size stands in for the largest current organisation until real
+counts are available: 2,000 rows per type, 200 users, 20 groups.
+"""
+
+import os
+import statistics
+import time
+
+from account_v2.models import Organization, User
+from adapter_processor_v2.models import AdapterInstance
+from api_v2.models import APIDeployment
+from connector_v2.models import ConnectorInstance
+from django.contrib.contenttypes.models import ContentType
+from django.db import connection
+from django.test import TestCase
+from django.test.utils import CaptureQueriesContext
+from unittest.mock import patch
+from permissions.roles import ResourceRole
+from pipeline_v2.models import Pipeline
+from prompt_studio.prompt_studio_core_v2.models import CustomTool
+from tenant_account_v2.models import (
+ GroupMembership,
+ OrganizationGroup,
+ OrganizationMember,
+ ResourceGroupShare,
+ ResourceMembership,
+)
+from tenant_account_v2.organization_member_service import OrganizationMemberService
+from utils.user_context import UserContext
+from workflow_manager.workflow_v2.models.workflow import Workflow
+
+from global_search.oss_providers import OSS_PROVIDERS
+from global_search.tests.base import search
+
+ROWS_PER_TYPE = 2_000
+USERS = 200
+GROUPS = 20
+BUDGET_MS = 300
+ASSERT_P95 = os.environ.get("GLOBAL_SEARCH_ASSERT_P95") == "1"
+CEILING_MS = int(os.environ.get("GLOBAL_SEARCH_CEILING_MS", str(BUDGET_MS)))
+# Each type costs one ranked search query plus the organisation point
+# lookups the existing ``for_user()`` makes (the org-scoped manager, and the
+# membership subquery on the non-admin path). Anything per result -- an
+# N+1 -- breaks the bound.
+QUERIES_PER_TYPE = 3
+# Per request, not per type: the caller's admin-role lookup (memoized on the
+# user, so paid once) and the organisation lookup, plus one spare.
+FIXED_OVERHEAD_QUERIES = 3
+# OSS types only, so the bound holds in every distribution; cloud types
+# carry their own query bounds in the cloud repo.
+OSS_TYPES = ",".join(p.type_key for p in OSS_PROVIDERS)
+QUERIES = ["acme", "inv", "report", "zz", "pipe", "2024", "ac", "data"]
+WORDS = ["invoice", "report", "acme", "data", "pipe", "ledger", "claims", "2024"]
+
+
+def _name(prefix: str, i: int) -> str:
+ return f"{WORDS[i % len(WORDS)]}-{prefix}-{i}"
+
+
+class SearchBudgetTests(TestCase):
+ @classmethod
+ def setUpTestData(cls) -> None:
+ cls.org = Organization.objects.create(
+ name="big", display_name="Big", organization_id="org-big"
+ )
+ UserContext.set_organization_identifier(cls.org.organization_id)
+ users = User.objects.bulk_create(
+ User(username=f"u{i}@big.test", email=f"u{i}@big.test") for i in range(USERS)
+ )
+ OrganizationMember.objects.bulk_create(
+ OrganizationMember(organization=cls.org, user=u, role="user") for u in users
+ )
+ groups = OrganizationGroup.objects.bulk_create(
+ OrganizationGroup(organization=cls.org, name=f"g{i}", created_by=users[0])
+ for i in range(GROUPS)
+ )
+ GroupMembership.objects.bulk_create(
+ GroupMembership(group=groups[i % GROUPS], user=u) for i, u in enumerate(users)
+ )
+ cls.member = users[0]
+ cls.admin = users[1]
+ # The role lookup a request pays is counted: only the role-name check
+ # is patched (see setUp), never the predicate that queries it.
+ OrganizationMember.objects.filter(organization=cls.org, user=cls.admin).update(
+ role="admin"
+ )
+
+ common = {"organization": cls.org}
+ n = range(ROWS_PER_TYPE)
+ parents = Workflow.objects.bulk_create(
+ Workflow(workflow_name=f"parent-{i}", **common) for i in range(ROWS_PER_TYPE)
+ )
+ rows = [
+ *Workflow.objects.bulk_create(
+ Workflow(workflow_name=_name("wf", i), description=_name("d", i), **common)
+ for i in n
+ ),
+ *CustomTool.objects.bulk_create(
+ CustomTool(
+ tool_name=_name("tool", i),
+ description=_name("d", i),
+ author="seed",
+ **common,
+ )
+ for i in n
+ ),
+ *APIDeployment.objects.bulk_create(
+ APIDeployment(
+ display_name=_name("api", i),
+ api_name=f"a{i}",
+ api_endpoint=f"deployment/api/org-big/a{i}/",
+ description=_name("d", i),
+ workflow=parents[i],
+ **common,
+ )
+ for i in n
+ ),
+ *Pipeline.objects.bulk_create(
+ Pipeline(
+ pipeline_name=_name(f"p{t}", i),
+ pipeline_type=t,
+ workflow=parents[i],
+ **common,
+ )
+ for t in (Pipeline.PipelineType.ETL, Pipeline.PipelineType.TASK)
+ for i in n
+ ),
+ *AdapterInstance.objects.bulk_create(
+ AdapterInstance(
+ adapter_name=_name("ad", i),
+ adapter_id="openai|seed",
+ adapter_type="LLM",
+ adapter_metadata={},
+ description=_name("d", i),
+ **common,
+ )
+ for i in n
+ ),
+ *ConnectorInstance.objects.bulk_create(
+ ConnectorInstance(connector_name=_name("conn", i), connector_id="minio|seed", **common)
+ for i in n
+ ),
+ ]
+ # One owner per row spread across users, a group share on every 50th
+ # and an org share on every 100th: the non-admin path's real mix.
+ memberships, shares = [], []
+ for i, row in enumerate(rows):
+ ct = ContentType.objects.get_for_model(type(row))
+ memberships.append(
+ ResourceMembership(
+ user=users[i % USERS],
+ role=ResourceRole.OWNER,
+ content_type=ct,
+ object_id=str(row.pk),
+ organization=cls.org,
+ )
+ )
+ if i % 50 == 0:
+ shares.append(
+ ResourceGroupShare(
+ group=groups[i % GROUPS],
+ content_type=ct,
+ object_id=str(row.pk),
+ organization=cls.org,
+ )
+ )
+ if i % 100 == 0:
+ row.shared_to_org = True
+ ResourceMembership.objects.bulk_create(memberships, batch_size=2_000)
+ ResourceGroupShare.objects.bulk_create(shares, batch_size=2_000)
+ for model in (Workflow, CustomTool, APIDeployment, Pipeline, AdapterInstance, ConnectorInstance):
+ model.objects.filter(
+ pk__in=[r.pk for r in rows if isinstance(r, model) and r.shared_to_org]
+ ).update(shared_to_org=True)
+ with connection.cursor() as cursor:
+ cursor.execute("ANALYZE")
+
+ def setUp(self) -> None:
+ UserContext.set_organization_identifier(self.org.organization_id)
+ # Each auth plugin names the admin role differently ("admin" in OSS,
+ # "unstract_admin" in cloud). Pin the name check, which is pure
+ # Python, so the admin subtest measures an admin in every
+ # distribution; the membership query still runs and is counted.
+ name_check = patch(
+ "account_v2.authentication_controller.AuthenticationController"
+ ".is_admin_by_role",
+ new=lambda _self, role: role == "admin",
+ )
+ name_check.start()
+ self.addCleanup(name_check.stop)
+ self.assertTrue(
+ OrganizationMemberService.is_user_organization_admin(
+ User.objects.get(pk=self.admin.pk)
+ )
+ )
+
+ def _queries(self, user: User, limit: int) -> list[str]:
+ """SQL issued for one search, minus the per-provider savepoints the
+ view opens inside a transaction (TestCase always has one)."""
+ # A fresh user object, as each request gets: the admin-role memo
+ # lives on the user, so reusing one would hide that lookup.
+ user = User.objects.get(pk=user.pk)
+ with CaptureQueriesContext(connection) as ctx:
+ response = search(user, q="inv", types=OSS_TYPES, limit=limit)
+ self.assertEqual(response.status_code, 200)
+ # search() already asserts no type failed; every type also matches,
+ # so a type that silently returned nothing would show here.
+ self.assertEqual(
+ {g["type"] for g in response.data["groups"]},
+ {p.type_key for p in OSS_PROVIDERS},
+ )
+ return [
+ q["sql"]
+ for q in ctx.captured_queries
+ if not q["sql"].startswith(("SAVEPOINT", "RELEASE SAVEPOINT"))
+ ]
+
+ def _assert_bounded(self, user: User) -> None:
+ search(user, q="warm", types=OSS_TYPES) # content-type cache warm-up
+ few = self._queries(user, limit=1)
+ many = self._queries(user, limit=20)
+ self.assertLessEqual(
+ len(many),
+ QUERIES_PER_TYPE * len(OSS_PROVIDERS) + FIXED_OVERHEAD_QUERIES,
+ "\n".join(sql[:200] for sql in many),
+ )
+ # An N+1 shows up as more queries for more results.
+ self.assertEqual(len(many), len(few))
+
+ def _p95_ms(self, user: User) -> float:
+ timings = []
+ for _ in range(3):
+ for q in QUERIES:
+ start = time.perf_counter()
+ search(user, q=q, types=OSS_TYPES)
+ timings.append((time.perf_counter() - start) * 1000)
+ return statistics.quantiles(timings, n=20)[-1]
+
+ def test_query_count_is_bounded_per_type(self) -> None:
+ for user in (self.member, self.admin):
+ with self.subTest(user=user.email):
+ self._assert_bounded(user)
+
+ def test_p95_is_measured(self) -> None:
+ for user in (self.member, self.admin):
+ with self.subTest(user=user.email):
+ p95 = self._p95_ms(user)
+ print(f"[global_search budget] {user.email} p95={p95:.1f}ms (budget {BUDGET_MS}ms)")
+ if ASSERT_P95:
+ self.assertLessEqual(p95, CEILING_MS)
diff --git a/backend/global_search/tests/test_contract.py b/backend/global_search/tests/test_contract.py
new file mode 100644
index 000000000..96c734b4c
--- /dev/null
+++ b/backend/global_search/tests/test_contract.py
@@ -0,0 +1,175 @@
+"""The published contract matches what the endpoint serves.
+
+* Real responses validate against the committed JSON Schema.
+* Each OSS type's context line and route params match values copied from
+ the README's type table; the table's order, type, label and route key
+ columns are checked against the registered providers.
+* The endpoint is mounted on the tenant path, and refuses a URL organisation
+ other than the session's.
+"""
+
+import json
+from datetime import timedelta
+from pathlib import Path
+from unittest.mock import patch
+
+import jsonschema
+from django.db import connection
+from account_v2.models import Organization
+from django.test import TestCase
+from django.urls import resolve
+from permissions.roles import ResourceRole
+from permissions.tests.base import CoOwnerOrgTestMixin, make_user
+from tenant_account_v2.models import OrganizationMember
+
+from global_search.constants import OSSSearchType as T
+from global_search.oss_providers import AdapterProvider, OSS_PROVIDERS, WorkflowProvider
+from global_search.providers import SearchProvider
+from global_search.tests.base import OSS_BUILDERS, search
+from global_search.views import GlobalSearchView
+
+APP_DIR = Path(__file__).parents[1]
+SCHEMA = json.loads((APP_DIR / "contract" / "search-response.schema.json").read_text())
+README = (APP_DIR / "README.md").read_text()
+
+
+class SearchContractTests(CoOwnerOrgTestMixin, TestCase):
+ def setUp(self) -> None:
+ self._seed_org()
+
+ def test_responses_validate_against_schema(self) -> None:
+ for type_key, build in OSS_BUILDERS.items():
+ item = build(self.org, self.owner, f"theta-{type_key}", "theta notes")
+ item.memberships.create(user=self.owner, role="owner")
+ response = search(self.admin, q="theta", limit=20)
+ jsonschema.validate(response.data, SCHEMA)
+ self.assertEqual(
+ [g["type"] for g in response.data["groups"]],
+ [p.type_key for p in sorted(OSS_PROVIDERS, key=lambda p: p.order)],
+ )
+
+ def test_context_and_route_params_per_type(self) -> None:
+ items = {}
+ for type_key, build in OSS_BUILDERS.items():
+ items[type_key] = build(self.org, self.owner, f"theta-{type_key}")
+ items[type_key].memberships.create(user=self.owner, role=ResourceRole.OWNER)
+ response = search(self.admin, q="theta", limit=20)
+ results = {g["type"]: g["results"][0] for g in response.data["groups"]}
+
+ def parent(type_key: str) -> str:
+ return items[type_key].workflow.workflow_name
+
+ expected_context = {
+ T.PROMPT_STUDIO_PROJECT: self.owner.email,
+ T.WORKFLOW: self.owner.email,
+ T.API_DEPLOYMENT: parent(T.API_DEPLOYMENT),
+ T.ETL_PIPELINE: parent(T.ETL_PIPELINE),
+ T.TASK_PIPELINE: parent(T.TASK_PIPELINE),
+ T.ADAPTER: "LLM · openai",
+ T.CONNECTOR: "minio",
+ }
+ for type_key, item in items.items():
+ with self.subTest(type=type_key):
+ result = results[type_key]
+ self.assertEqual(result["context"], expected_context[type_key])
+ expected_params = {"id": str(item.pk)}
+ if type_key == T.ADAPTER:
+ expected_params["adapter_type"] = "LLM"
+ self.assertEqual(
+ result["route"],
+ {"key": f"{type_key}.detail", "params": expected_params},
+ )
+
+ def test_owner_context_skips_service_accounts_and_takes_the_earliest(self) -> None:
+ workflow = OSS_BUILDERS[T.WORKFLOW](self.org, self.owner, "theta-owned")
+ service = make_user("[email protected]", is_service_account=True)
+ # The service account is the earliest owner, so only the exclusion
+ # skips it; the co-owner is inserted before the owner, so only the
+ # created_at ordering (not heap order) picks the owner over it.
+ for user in (service, self.coowner, self.owner):
+ workflow.memberships.create(user=user, role=ResourceRole.OWNER)
+ base = workflow.memberships.order_by("created_at").first().created_at
+ for user, minutes in ((service, -2), (self.owner, -1), (self.coowner, 0)):
+ workflow.memberships.filter(user=user).update(
+ created_at=base + timedelta(minutes=minutes)
+ )
+ (group,) = search(self.admin, q="theta", types=T.WORKFLOW).data["groups"]
+ self.assertEqual(group["results"][0]["context"], self.owner.email)
+ # Same rule as the list's "Owned By" column.
+ self.assertEqual(workflow.owner_email(), self.owner.email)
+
+ def test_a_db_error_in_one_provider_does_not_poison_the_others(self) -> None:
+ # TestCase wraps each test in a transaction, as ATOMIC_REQUESTS does.
+ class Broken(SearchProvider):
+ type_key, label, order = "broken", "Broken", 1
+
+ def search(self, user, query, limit):
+ with connection.cursor() as cursor:
+ cursor.execute("SELECT * FROM no_such_table_for_search")
+ return []
+
+ workflow = OSS_BUILDERS[T.WORKFLOW](self.org, self.owner, "theta-flow")
+ with patch(
+ "global_search.views.get_providers",
+ return_value=[Broken(), WorkflowProvider()],
+ ):
+ response = search(self.admin, allow_failures=True, q="theta")
+ self.assertEqual(response.status_code, 200)
+ self.assertEqual(response.data["failed_types"], ["broken"])
+ self.assertEqual(
+ [r["id"] for g in response.data["groups"] for r in g["results"]],
+ [str(workflow.pk)],
+ )
+
+ def test_mounted_on_tenant_path(self) -> None:
+ # Importing the urlconf touches the DB, hence a TestCase.
+ # OrganizationMiddleware strips the org segment before routing, so
+ # /api/v1/unstract/<org>/search/ resolves as this path.
+ match = resolve("/api/v1/unstract/search/")
+ self.assertIs(match.func.view_class, GlobalSearchView)
+
+ def test_url_org_must_match_the_session_org(self) -> None:
+ # Through the real middleware: results come from the session's org,
+ # so a URL naming another org (even one the user belongs to) is
+ # refused rather than answered from the wrong tenant.
+ workflow = OSS_BUILDERS[T.WORKFLOW](self.org, self.owner, "theta-flow")
+ other = Organization.objects.create(
+ name="org-b", display_name="Org B", organization_id="org-b"
+ )
+ OrganizationMember.objects.create(organization=other, user=self.admin, role="admin")
+ # The built-in session auth, so a distribution's auth plugin cannot
+ # 401 the request first; the org check runs after either.
+ no_plugin = patch(
+ "account_v2.custom_auth_middleware.AuthenticationPluginRegistry"
+ ".is_plugin_available",
+ return_value=False,
+ )
+ no_plugin.start()
+ self.addCleanup(no_plugin.stop)
+ self.client.force_login(self.admin)
+ session = self.client.session
+ session["organization"] = self.org.organization_id
+ session.save()
+
+ ok = self.client.get(
+ f"/api/v1/unstract/{self.org.organization_id}/search/", {"q": "theta"}
+ )
+ self.assertEqual(ok.status_code, 200)
+ self.assertIn(
+ str(workflow.pk),
+ [r["id"] for g in ok.json()["groups"] for r in g["results"]],
+ )
+ denied = self.client.get(
+ f"/api/v1/unstract/{other.organization_id}/search/", {"q": "theta"}
+ )
+ self.assertEqual(denied.status_code, 403)
+
+
+def test_readme_lists_every_oss_type():
+ for provider_cls in OSS_PROVIDERS:
+ provider = provider_cls()
+ keys = f"`{provider.route_key}`"
+ if isinstance(provider, AdapterProvider):
+ keys += f" / `{provider.info_route_key}`"
+ row = f"| {provider.order} | `{provider.type_key}` | {provider.label} | {keys} |"
+ assert row in README, f"README type table is missing: {row}"
diff --git a/backend/global_search/tests/test_ranking.py b/backend/global_search/tests/test_ranking.py
new file mode 100644
index 000000000..5cfe71965
--- /dev/null
+++ b/backend/global_search/tests/test_ranking.py
@@ -0,0 +1,92 @@
+"""Ranking against the real database: prefix before substring, name before
+description, and a deterministic order inside a tier.
+"""
+
+from datetime import timedelta
+
+from django.test import TestCase
+from django.utils import timezone
+from permissions.tests.base import CoOwnerOrgTestMixin
+
+from global_search.tests.base import DESCRIBED_TYPES, OSS_BUILDERS, search
+
+
+class SearchRankingTests(CoOwnerOrgTestMixin, TestCase):
+ def setUp(self) -> None:
+ self._seed_org()
+
+ def _results(self, type_key: str, query: str, limit: int = 20) -> list[dict]:
+ response = search(self.admin, q=query, types=type_key, limit=limit)
+ groups = response.data["groups"]
+ return groups[0]["results"] if groups else []
+
+ def test_name_prefix_before_name_substring(self) -> None:
+ for type_key, build in OSS_BUILDERS.items():
+ with self.subTest(type=type_key):
+ # Best tier created first: recency alone would order it last.
+ prefix = build(self.org, self.owner, f"kappa-{type_key}")
+ inner = build(self.org, self.owner, f"old-kappa-{type_key}")
+ results = self._results(type_key, "kappa")
+ self.assertEqual(
+ [r["id"] for r in results], [str(prefix.pk), str(inner.pk)]
+ )
+ self.assertEqual(
+ [r["match"] for r in results], ["name_prefix", "name"]
+ )
+
+ def test_name_before_description(self) -> None:
+ for type_key in sorted(DESCRIBED_TYPES):
+ build = OSS_BUILDERS[type_key]
+ with self.subTest(type=type_key):
+ # Best tier created first: recency alone would reverse it.
+ name_inner = build(self.org, self.owner, f"x-lambda-{type_key}")
+ desc_prefix = build(
+ self.org, self.owner, f"d1-{type_key}", "lambda first"
+ )
+ desc_inner = build(
+ self.org, self.owner, f"d2-{type_key}", "about lambda things"
+ )
+ results = self._results(type_key, "LAMBDA")
+ self.assertEqual(
+ [r["id"] for r in results],
+ [str(name_inner.pk), str(desc_prefix.pk), str(desc_inner.pk)],
+ )
+ self.assertEqual(
+ [r["match"] for r in results],
+ ["name", "description_prefix", "description"],
+ )
+
+ def test_ties_go_to_most_recently_modified(self) -> None:
+ build = OSS_BUILDERS["workflow"]
+ older = build(self.org, self.owner, "sigma-one")
+ newer = build(self.org, self.owner, "sigma-two")
+ # bulk paths keep an explicit modified_at, see BaseModelQuerySet.
+ type(older).objects.filter(pk=older.pk).update(
+ modified_at=timezone.now() - timedelta(days=1)
+ )
+ results = self._results("workflow", "sigma")
+ self.assertEqual([r["id"] for r in results], [str(newer.pk), str(older.pk)])
+
+ def test_has_more_and_limit(self) -> None:
+ build = OSS_BUILDERS["connector"]
+ for i in range(3):
+ build(self.org, self.owner, f"omega-{i}")
+ response = search(self.admin, q="omega", types="connector", limit=2)
+ (group,) = response.data["groups"]
+ self.assertEqual(len(group["results"]), 2)
+ self.assertTrue(group["has_more"])
+
+ response = search(self.admin, q="omega", types="connector", limit=3)
+ self.assertFalse(response.data["groups"][0]["has_more"])
+
+ def test_wildcards_are_literal(self) -> None:
+ build = OSS_BUILDERS["workflow"]
+ build(self.org, self.owner, "plain-name")
+ # Matches "_100" only if "_" were a single-character wildcard.
+ build(self.org, self.owner, "rateX100")
+ literal = build(self.org, self.owner, "rate_100%")
+ response = search(self.admin, q="_100", types="workflow")
+ self.assertEqual(
+ [r["id"] for r in response.data["groups"][0]["results"]], [str(literal.pk)]
+ )
+ self.assertEqual(search(self.admin, q="%%", types="workflow").data["groups"], [])
diff --git a/backend/global_search/tests/test_search_unit.py b/backend/global_search/tests/test_search_unit.py
new file mode 100644
index 000000000..a1d5d607b
--- /dev/null
+++ b/backend/global_search/tests/test_search_unit.py
@@ -0,0 +1,298 @@
+"""DB-free tests: parameters, envelope shape, failure handling, registry and
+in-memory ranking, with faked providers.
+"""
+
+import json
+from pathlib import Path
+from unittest.mock import MagicMock, patch
+
+import jsonschema
+import pytest
+from account_v2.models import User
+from utils.user_context import UserContext
+from django.db import InterfaceError, OperationalError
+
+from global_search.constants import MatchTier
+from global_search.providers import SearchHit, SearchProvider
+from global_search.ranking import match_tier, rank_items
+from global_search.registry import get_providers, register_provider, unregister_provider
+from global_search.tests.base import search
+
+SCHEMA = json.loads(
+ (Path(__file__).parents[1] / "contract" / "search-response.schema.json").read_text()
+)
+
+
+class FakeProvider(SearchProvider):
+ def __init__(self, type_key: str, order: int, names: list[str]):
+ self.type_key = type_key
+ self.label = type_key.title()
+ self.order = order
+ self.names = names
+
+ def search(self, user, query, limit):
+ ranked = rank_items(self.names, query, names=lambda n: [n], sort_key=lambda n: n)
+ return [
+ SearchHit(id=name, name=name, match=tier, context=None, route_params={"id": name})
+ for name, tier in ranked[: limit + 1]
+ ]
+
+
[email protected](autouse=True)
+def no_organization_context():
+ """A 5xx makes Django's error reporter import the URLconf. If an earlier
+ test on the same worker left an organisation in UserContext, the
+ org-scoped managers behind class-level querysets (e.g.
+ ``pipeline_v2.views``) would look it up during that import -- a DB query
+ these tests must not make.
+ """
+ previous = UserContext.get_organization_identifier()
+ UserContext.set_organization_identifier(None)
+ yield
+ UserContext.set_organization_identifier(previous)
+
+
+def user() -> User:
+ # Unsaved: IsAuthenticated only reads ``is_authenticated``.
+ return User(username="[email protected]", email="[email protected]")
+
+
+def providers():
+ fakes = [
+ FakeProvider("beta", 20, ["alpha one", "alpha two", "one alpha"]),
+ FakeProvider("alpha", 10, ["alpha"]),
+ FakeProvider("empty", 30, ["nothing"]),
+ ]
+ with patch("global_search.views.get_providers", return_value=sorted(fakes, key=lambda p: p.order)):
+ yield fakes
+
+
+def test_groups_in_rail_order_with_empty_groups_omitted(user, providers):
+ data = search(user, q="alpha").data
+ jsonschema.validate(data, SCHEMA)
+ assert data["query"] == "alpha"
+ assert [g["type"] for g in data["groups"]] == ["alpha", "beta"]
+ beta = data["groups"][1]
+ assert [r["name"] for r in beta["results"]] == ["alpha one", "alpha two", "one alpha"]
+ assert [r["match"] for r in beta["results"]] == ["name_prefix", "name_prefix", "name"]
+ assert beta["results"][0]["route"] == {"key": "beta.detail", "params": {"id": "alpha one"}}
+
+
+def test_a_failing_provider_drops_only_its_own_group(user, providers, caplog):
+ class Broken(FakeProvider):
+ def search(self, user, query, limit):
+ raise RuntimeError("boom")
+
+ with patch(
+ "global_search.views.get_providers",
+ return_value=[providers[1], Broken("broken", 15, []), providers[0]],
+ ):
+ response = search(user, allow_failures=True, q="alpha")
+ assert response.status_code == 200
+ assert [g["type"] for g in response.data["groups"]] == ["alpha", "beta"]
+ assert response.data["failed_types"] == ["broken"]
+ jsonschema.validate(response.data, SCHEMA)
+ assert "Global search provider 'broken' failed" in caplog.text
+
+
+def _assert_error(response, status: int, code: str) -> None:
+ assert response.status_code == status
+ assert [e["code"] for e in response.data["errors"]] == [code]
+
+
+class _Broken(FakeProvider):
+ def search(self, user, query, limit):
+ raise RuntimeError("boom")
+
+
+class _DbError(FakeProvider):
+ def search(self, user, query, limit):
+ raise OperationalError("canceling statement due to statement timeout")
+
+
+def _connection(*, usable: bool = True, needs_rollback: bool = False) -> MagicMock:
+ """The view's DB connection, in autocommit (the default) with a chosen
+ health, so the escalation rules run without a real failure."""
+ conn = MagicMock(in_atomic_block=False, needs_rollback=needs_rollback)
+ conn.is_usable.return_value = usable
+ return conn
+
+
+def _run(user, providers_list, conn=None, **params):
+ with (
+ patch("global_search.views.connection", conn or _connection()),
+ patch("global_search.views.get_providers", return_value=providers_list),
+ ):
+ return search(user, allow_failures=True, **{"q": "alpha", **params})
+
+
+def test_every_type_failing_with_a_bug_is_a_500(user):
+ response = _run(user, [_Broken("a", 1, []), _Broken("b", 2, [])])
+ _assert_error(response, 500, "search_failed")
+
+
+def test_every_type_failing_on_the_database_is_a_retryable_503(user):
+ response = _run(user, [_DbError("a", 1, []), _DbError("b", 2, [])])
+ _assert_error(response, 503, "search_unavailable")
+
+
+def test_a_failure_that_breaks_the_transaction_fails_the_request(user, providers):
+ # A failed savepoint rollback leaves needs_rollback set; later types
+ # would all fail, so the error must surface rather than be logged per type.
+ conn = _connection(needs_rollback=True)
+ response = _run(user, [_Broken("a", 1, []), providers[0]], conn)
+ _assert_error(response, 503, "search_unavailable")
+
+
+def test_a_lost_connection_fails_the_request_even_if_others_answered(
+ user, providers, caplog
+):
+ # In autocommit nothing marks the connection broken; is_usable() does.
+ conn = _connection(usable=False)
+ response = _run(user, [providers[1], _DbError("lost", 15, [])], conn)
+ _assert_error(response, 503, "search_unavailable")
+ # The 503 itself names no type; the log must.
+ assert "Global search provider 'lost' failed" in caplog.text
+
+
+def test_a_failed_connect_fails_the_request_without_probing_it(user, providers):
+ # Connecting failed, so no connection is open: every later type would
+ # dial the dead database too. is_usable() needs an open connection.
+ conn = _connection()
+ conn.connection = None
+ response = _run(user, [_DbError("down", 5, []), providers[1]], conn)
+ _assert_error(response, 503, "search_unavailable")
+ conn.is_usable.assert_not_called()
+
+
+def test_a_closed_connection_interface_error_fails_the_request(user, providers):
+ # InterfaceError ("connection already closed") is not a DatabaseError in
+ # Django, but it is the canonical unusable-connection error.
+ class Closed(FakeProvider):
+ def search(self, user, query, limit):
+ raise InterfaceError("connection already closed")
+
+ later = MagicMock(wraps=providers[0])
+ later.type_key, later.label, later.order = "alpha", "Alpha", 10
+ conn = _connection(usable=False)
+ response = _run(user, [Closed("closed", 5, []), later], conn)
+ _assert_error(response, 503, "search_unavailable")
+ later.search.assert_not_called()
+
+
+def test_a_bug_alongside_database_errors_is_a_500(user):
+ # Only an all-retryable failure is worth retrying.
+ response = _run(user, [_DbError("a", 1, []), _Broken("b", 2, [])])
+ _assert_error(response, 500, "search_failed")
+
+
+def test_a_db_error_on_a_healthy_connection_drops_only_its_type(user, providers):
+ # A statement timeout, lock timeout or deadlock leaves the connection
+ # usable: the other types still answer.
+ response = _run(user, [providers[1], _DbError("slow", 15, [])])
+ assert response.status_code == 200
+ assert [g["type"] for g in response.data["groups"]] == ["alpha"]
+ assert response.data["failed_types"] == ["slow"]
+
+
+def test_limit_and_has_more(user, providers):
+ beta = search(user, q="alpha", types="beta", limit=2).data["groups"][0]
+ assert len(beta["results"]) == 2
+ assert beta["has_more"] is True
+ assert search(user, q="alpha", types="beta", limit=3).data["groups"][0]["has_more"] is False
+
+
+def test_types_filter_ignores_unknown_keys(user, providers):
+ data = search(user, q="alpha", types=" alpha , agentic_project,,").data
+ assert [g["type"] for g in data["groups"]] == ["alpha"]
+
+
[email protected]("q", ["", "a", " a ", " "])
+def test_short_query_returns_no_groups(user, providers, q):
+ response = search(user, q=q)
+ assert response.status_code == 200
+ assert response.data == {"query": q.strip(), "groups": [], "failed_types": []}
+
+
+def test_missing_query_returns_no_groups(user, providers):
+ response = search(user)
+ assert response.status_code == 200
+ assert response.data["groups"] == []
+
+
+ "params",
+ [{"limit": 0}, {"limit": 21}, {"limit": "x"}, {"q": "a" * 101}],
+)
+def test_invalid_parameters_are_400(user, providers, params):
+ response = search(user, **{"q": "alpha", **params})
+ assert response.status_code == 400
+
+
+def test_query_at_max_length_is_accepted(user, providers):
+ assert search(user, q="a" * 100).status_code == 200
+
+
+def test_anonymous_is_rejected():
+ from django.contrib.auth.models import AnonymousUser
+
+ assert search(AnonymousUser(), q="alpha").status_code in (401, 403)
+
+
+class TestRegistry:
+ def test_sorted_by_order_and_duplicate_class_is_idempotent(self):
+ first, second = FakeProvider("zz_b", 999_002, []), FakeProvider("zz_a", 999_001, [])
+ try:
+ register_provider(first)
+ register_provider(second)
+ register_provider(FakeProvider("zz_a", 999_001, []))
+ keys = [p.type_key for p in get_providers()]
+ assert keys[-2:] == ["zz_a", "zz_b"]
+ finally:
+ unregister_provider("zz_a")
+ unregister_provider("zz_b")
+
+ @pytest.mark.parametrize("bad_key", ["agenticProject", "1abc", "a-b", "", "a.b"])
+ def test_type_key_must_match_the_schema_pattern(self, bad_key):
+ with pytest.raises(ValueError):
+ register_provider(FakeProvider(bad_key, 1, []))
+ assert bad_key not in [p.type_key for p in get_providers()]
+
+ def test_conflicting_class_for_a_key_raises(self):
+ class Other(FakeProvider):
+ pass
+
+ register_provider(FakeProvider("zz_c", 999_003, []))
+ try:
+ with pytest.raises(ValueError):
+ register_provider(Other("zz_c", 1, []))
+ finally:
+ unregister_provider("zz_c")
+
+
+ ("names", "description", "expected"),
+ [
+ (["Invoice parser"], None, MatchTier.NAME_PREFIX),
+ (["My invoices"], None, MatchTier.NAME),
+ (["other", "INVOICE api"], None, MatchTier.NAME_PREFIX),
+ (["x"], "invoice first", MatchTier.DESCRIPTION_PREFIX),
+ (["x"], "about Invoices", MatchTier.DESCRIPTION),
+ (["x", None], None, None),
+ ],
+)
+def test_match_tier(names, description, expected):
+ assert match_tier("invoice", names, description) == expected
+
+
+def test_rank_items_orders_by_tier_then_sort_key():
+ items = [("b", "zz invoice"), ("a", "invoice"), ("c", "invoice")]
+ ranked = rank_items(items, "inv", names=lambda i: [i[1]], sort_key=lambda i: i[0])
+ assert [i[0] for i, _ in ranked] == ["a", "c", "b"]
+
+
+def test_a_hit_must_carry_a_route_id():
+ with pytest.raises(ValueError):
+ SearchHit(id="x", name="x", match=MatchTier.NAME, route_params={})
diff --git a/backend/global_search/tests/test_visibility.py b/backend/global_search/tests/test_visibility.py
new file mode 100644
index 000000000..04bb45746
--- /dev/null
+++ b/backend/global_search/tests/test_visibility.py
@@ -0,0 +1,170 @@
+"""A user never receives a result they cannot open, per resource type.
+
+Each type is checked against the same roles its list honours: owner,
+co-owner, direct viewer, group share, org share, admin and service account
+see it; an org member without access and another org's resources do not.
+"""
+
+from account_v2.models import Organization
+from adapter_processor_v2.models import AdapterInstance
+from django.test import TestCase
+from permissions.permission import IsFrictionLessAdapter
+from permissions.roles import ResourceRole
+from permissions.tests.base import CoOwnerOrgTestMixin, make_user
+from pipeline_v2.models import Pipeline
+from rest_framework.request import Request
+from rest_framework.test import APIRequestFactory
+from tenant_account_v2.models import GroupMembership, OrganizationGroup, OrganizationMember
+from tenant_account_v2.sharing_helpers import set_resource_share_groups
+from utils.user_context import UserContext
+
+from global_search.constants import OSSSearchType
+from global_search.tests.base import (
+ OSS_BUILDERS,
+ _pipeline,
+ build_adapter,
+ result_ids,
+ search,
+)
+
+QUERY = "zeta"
+
+
+class SearchVisibilityTests(CoOwnerOrgTestMixin, TestCase):
+ def setUp(self) -> None:
+ self._seed_org()
+ self.group_member = make_user("[email protected]")
+ OrganizationMember.objects.create(
+ organization=self.org, user=self.group_member, role="user"
+ )
+ self.group = OrganizationGroup.objects.create(
+ organization=self.org, name="Team", created_by=self.owner
+ )
+ GroupMembership.objects.create(group=self.group, user=self.group_member)
+ self.service_account = make_user("[email protected]", is_service_account=True)
+
+ def _private(self, type_key: str):
+ item = OSS_BUILDERS[type_key](self.org, self.owner, f"{QUERY}-{type_key}")
+ item.memberships.create(user=self.owner, role=ResourceRole.OWNER)
+ item.memberships.create(user=self.coowner, role=ResourceRole.OWNER)
+ item.memberships.create(user=self.viewer, role=ResourceRole.VIEWER)
+ return item
+
+ def _found(self, user, type_key: str, item) -> bool:
+ response = search(user, q=QUERY, types=type_key, limit=20)
+ self.assertEqual(response.status_code, 200)
+ return str(item.pk) in result_ids(response, type_key)
+
+ def test_each_type_follows_its_list_visibility(self) -> None:
+ for type_key in OSS_BUILDERS:
+ with self.subTest(type=type_key):
+ item = self._private(type_key)
+
+ for user in (self.owner, self.coowner, self.viewer, self.admin):
+ self.assertTrue(self._found(user, type_key, item), user.email)
+ self.assertTrue(self._found(self.service_account, type_key, item))
+ for user in (self.outsider, self.group_member):
+ self.assertFalse(self._found(user, type_key, item), user.email)
+
+ set_resource_share_groups(item, [self.group.id])
+ self.assertTrue(self._found(self.group_member, type_key, item))
+ self.assertFalse(self._found(self.outsider, type_key, item))
+
+ item.shared_to_org = True
+ item.save()
+ self.assertTrue(self._found(self.outsider, type_key, item))
+
+ def test_other_orgs_resources_never_appear(self) -> None:
+ other = Organization.objects.create(
+ name="org-b", display_name="Org B", organization_id="org-b"
+ )
+ UserContext.set_organization_identifier(other.organization_id)
+ foreign = {}
+ for type_key, build in OSS_BUILDERS.items():
+ item = build(other, self.stranger, f"{QUERY}-{type_key}")
+ item.shared_to_org = True
+ item.save()
+ foreign[type_key] = str(item.pk)
+ UserContext.set_organization_identifier(self.org.organization_id)
+
+ for user in (self.admin, self.service_account, self.outsider):
+ response = search(user, q=QUERY, limit=20)
+ leaked = set(result_ids(response)) & set(foreign.values())
+ self.assertEqual(leaked, set(), user.email)
+
+ def test_frictionless_adapters_are_never_returned(self) -> None:
+ # Listed so any member can delete one, but adapter retrieve refuses
+ # them for everyone, admins included; search would only lead to a 403.
+ adapter = build_adapter(
+ self.org, self.owner, f"{QUERY}-frictionless", is_friction_less=True
+ )
+ adapter.memberships.create(user=self.owner, role=ResourceRole.OWNER)
+ for user in (self.owner, self.admin, self.service_account, self.outsider):
+ self.assertFalse(
+ self._found(user, OSSSearchType.ADAPTER, adapter), user.email
+ )
+
+ def test_adapter_route_follows_who_may_edit_it(self) -> None:
+ # adapter.detail is gated like adapter retrieve: owners, co-owners,
+ # admins and service accounts. Anyone else who can list the adapter
+ # opens the read-only adapter.info.
+ adapter = self._private(OSSSearchType.ADAPTER)
+ set_resource_share_groups(adapter, [self.group.id])
+ adapter.shared_to_org = True
+ adapter.save()
+ expected = {
+ self.owner: "adapter.detail",
+ self.coowner: "adapter.detail",
+ self.admin: "adapter.detail",
+ self.service_account: "adapter.detail",
+ self.viewer: "adapter.info",
+ self.group_member: "adapter.info",
+ self.outsider: "adapter.info",
+ }
+ for user, route_key in expected.items():
+ with self.subTest(user=user.email):
+ response = search(
+ user, q=QUERY, types=OSSSearchType.ADAPTER, limit=20
+ )
+ (result,) = response.data["groups"][0]["results"]
+ self.assertEqual(
+ result["route"],
+ {
+ "key": route_key,
+ "params": {
+ "id": str(adapter.pk),
+ "adapter_type": adapter.adapter_type,
+ },
+ },
+ )
+ # Tied to the real gate, so the two rules cannot drift apart.
+ self.assertEqual(
+ route_key == "adapter.detail", self._may_retrieve(user, adapter)
+ )
+
+ @staticmethod
+ def _may_retrieve(user, adapter) -> bool:
+ request = Request(APIRequestFactory().get("/"))
+ request.user = user
+ return IsFrictionLessAdapter().has_object_permission(
+ request, None, AdapterInstance.objects.get(pk=adapter.pk)
+ )
+
+ def test_no_op_adapters_are_never_returned(self) -> None:
+ adapter = build_adapter(
+ self.org,
+ self.owner,
+ f"{QUERY}-noop",
+ adapter_id="noOpLlm|f673a5a2-90f9-40f5-94c0-9fbc663b7553",
+ shared_to_org=True,
+ )
+ self.assertFalse(self._found(self.admin, OSSSearchType.ADAPTER, adapter))
+
+ def test_default_and_app_pipelines_are_never_returned(self) -> None:
+ for pipeline_type in (Pipeline.PipelineType.DEFAULT, Pipeline.PipelineType.APP):
+ with self.subTest(pipeline_type=pipeline_type.value):
+ pipeline = _pipeline(pipeline_type)(
+ self.org, self.owner, f"{QUERY}-{pipeline_type}"
+ )
+ response = search(self.admin, q=QUERY, limit=20)
+ self.assertNotIn(str(pipeline.pk), result_ids(response))
diff --git a/backend/global_search/urls.py b/backend/global_search/urls.py
new file mode 100644
index 000000000..762a0eea0
--- /dev/null
+++ b/backend/global_search/urls.py
@@ -0,0 +1,7 @@
+from django.urls import path
+
+from global_search.views import GlobalSearchView
+
+urlpatterns = [
+ path("search/", GlobalSearchView.as_view(), name="global_search"),
+]
diff --git a/backend/global_search/views.py b/backend/global_search/views.py
new file mode 100644
index 000000000..57c632fde
--- /dev/null
+++ b/backend/global_search/views.py
@@ -0,0 +1,156 @@
+import logging
+from contextlib import nullcontext
+from typing import Any
+
+from django.db import (
+ Error,
+ InterfaceError,
+ OperationalError,
+ connection,
+ transaction,
+)
+from drf_spectacular.utils import OpenApiResponse, extend_schema
+from rest_framework.permissions import IsAuthenticated
+from rest_framework.request import Request
+from rest_framework.response import Response
+from rest_framework.views import APIView
+
+from global_search.constants import SearchParams
+from global_search.exceptions import SearchFailed, SearchUnavailable
+from global_search.providers import SearchProvider
+from global_search.registry import get_providers
+from global_search.serializers import SearchQuerySerializer, SearchResponseSerializer
+
+logger = logging.getLogger(__name__)
+
+# Database errors a retry may clear (unreachable, timed out, locked,
+# deadlocked), as opposed to a bug.
+_RETRYABLE = (InterfaceError, OperationalError)
+
+
+def _connection_unusable(error: Exception) -> bool:
+ """Whether later types would fail too: a savepoint that could not be
+ rolled back (ATOMIC_REQUESTS), or a database error after which the
+ connection no longer answers. A statement timeout, lock timeout or
+ deadlock is an OperationalError on a healthy connection, so the class
+ alone does not decide; ``is_usable()`` runs only on this error path.
+ """
+ if connection.needs_rollback:
+ return True
+ # django.db.Error, not DatabaseError: InterfaceError ("connection
+ # already closed") derives from Error only.
+ if not isinstance(error, Error):
+ return False
+ if connection.connection is None:
+ # No open connection after a connection-class error: connecting
+ # failed, so every later type would dial a dead database too.
+ return isinstance(error, _RETRYABLE)
+ return not connection.is_usable()
+
+
+def _group(provider: SearchProvider, user: Any, query: str, limit: int) -> dict | None:
+ hits = provider.search(user, query, limit)
+ if not hits:
+ return None
+ return {
+ "type": provider.type_key,
+ "label": provider.label,
+ "has_more": len(hits) > limit,
+ "results": [
+ {
+ "type": provider.type_key,
+ "id": hit.id,
+ "name": hit.name,
+ "context": hit.context,
+ "match": hit.match.label,
+ "route": {
+ "key": hit.route_key or provider.route_key,
+ "params": hit.route_params,
+ },
+ }
+ for hit in hits[:limit]
+ ],
+ }
+
+
+class GlobalSearchView(APIView):
+ """Search every resource type the caller can open, grouped by type."""
+
+ permission_classes = [IsAuthenticated]
+
+ @extend_schema(
+ parameters=[SearchQuerySerializer],
+ responses={
+ 200: SearchResponseSerializer,
+ 400: OpenApiResponse(description="Invalid parameters (validation_error)."),
+ 500: OpenApiResponse(
+ description=(
+ "search_failed: every searched type failed, not all with "
+ "retryable database errors (a bug). Retrying will not help."
+ )
+ ),
+ 503: OpenApiResponse(
+ description=(
+ "search_unavailable: every searched type failed with a "
+ "database error a retry may clear (unreachable, timed out, "
+ "locked), or a failure left the connection unusable. "
+ "Retry; never show as 'no results'."
+ )
+ ),
+ },
+ description=(
+ "Groups come back in rail order with empty groups omitted. Each "
+ "type applies its own list's visibility rule. See "
+ "backend/global_search/README.md for the type vocabulary and the "
+ "response-time budget."
+ ),
+ )
+ def get(self, request: Request) -> Response:
+ params = SearchQuerySerializer(data=request.query_params)
+ params.is_valid(raise_exception=True)
+ query: str = params.validated_data["q"]
+ wanted: list[str] = params.validated_data["types"]
+ limit: int = params.validated_data["limit"]
+
+ groups = []
+ failed_types: list[str] = []
+ attempted = 0
+ errors: list[Exception] = []
+ if len(query) >= SearchParams.MIN_QUERY_LENGTH:
+ for provider in get_providers():
+ if wanted and provider.type_key not in wanted:
+ continue
+ attempted += 1
+ # Under ATOMIC_REQUESTS a failed query aborts the request's
+ # transaction; a savepoint keeps it from poisoning the next type.
+ guard = (
+ transaction.atomic() if connection.in_atomic_block else nullcontext()
+ )
+ try:
+ with guard:
+ group = _group(provider, request.user, query, limit)
+ except Exception as error:
+ logger.exception(
+ "Global search provider '%s' failed", provider.type_key
+ )
+ # Every later type would fail too: fail the request
+ # instead of blaming healthy types. The cause is logged
+ # above, so it is not chained into the handler's log.
+ if _connection_unusable(error):
+ raise SearchUnavailable() from None
+ # Providers can come from other apps (cloud); one failing
+ # type drops its own group and is named in failed_types.
+ failed_types.append(provider.type_key)
+ errors.append(error)
+ continue
+ if group:
+ groups.append(group)
+
+ # Every type failing is never "no matches". Each cause is logged
+ # above; only database errors a retry may clear are worth retrying.
+ if errors and len(errors) == attempted:
+ if all(isinstance(e, _RETRYABLE) for e in errors):
+ raise SearchUnavailable() from None
+ raise SearchFailed() from None
+
+ return Response({"query": query, "groups": groups, "failed_types": failed_types})