LLM-Driven Extraction of Unstructured Data — Built for API Deployments & ETL Pipeline Workflows
Python ai-agentsdata-engineeringdocument-aigenerative-aiidpjson-extractionllmmcp-serverocrpdf-extractionprompt-engineeringstructured-output
UN-4402 [FEAT] Global search API (#2343)
Ritwik Ga0fdc1a3c006
diff --git a/backend/backend/settings/base.py b/backend/backend/settings/base.pyindex 375cc73ce..4ad2ed50a 100644--- a/backend/backend/settings/base.py+++ b/backend/backend/settings/base.py@@ -456,6 +456,7 @@ SHARED_APPS = ("dashboard_metrics","platform_api","global_api_deployment_key",+ "global_search",)TENANT_APPS = []diff --git a/backend/backend/urls_v2.py b/backend/backend/urls_v2.pyindex 4b9636a35..0fef98ea8 100644--- a/backend/backend/urls_v2.py+++ b/backend/backend/urls_v2.py@@ -67,6 +67,7 @@ urlpatterns = [path("metrics/", include("dashboard_metrics.urls")),path("platform-api/", include("platform_api.urls")),path("global-api-deployment/", include("global_api_deployment_key.urls")),+ path("", include("global_search.urls")),]# Organization-scoped MCP server — OFF by default.diff --git a/backend/global_search/README.md b/backend/global_search/README.mdnew file mode 100644index 000000000..fe6d4bbfe--- /dev/null+++ b/backend/global_search/README.md@@ -0,0 +1,156 @@+# Global search++One endpoint that finds anything the caller can open, grouped by type. This+file is the published contract for consumers: P2-T10, the top-bar search+panel, and PB-T18, the portal search, which shares the envelope. The+response JSON Schema is+[`contract/search-response.schema.json`](contract/search-response.schema.json);+tests validate real responses against it and check the type table's order,+type, label and route.key columns against the registered providers.++## Endpoint++```+GET /api/v1/unstract/{org_id}/search/?q=<text>&types=<csv>&limit=<n>+```++Authentication uses the session auth stack shared by the other tenant+endpoints. Service accounts get the same visibility rules their `for_user()`+gives them in each list. The endpoint is additive: no existing endpoint+changes.++## Parameters++| param | default | rules |+|---|---|---|+| `q` | `""` | Trimmed. After trimming, **fewer than 2 characters returns 200 with `groups: []`**. More than 100 characters returns 400. Matches name and description case-insensitively; `%` and `_` match literally. |+| `types` | all | Comma-separated type keys. Unknown or unavailable keys are **ignored**, so one panel can send cloud keys to an OSS backend. |+| `limit` | `5` | Results per type, 1 to 20. A non-integer or out-of-range value returns 400. |++## Response++```json+{+ "query": "invoice",+ "groups": [+ {+ "type": "prompt_studio_project",+ "label": "Prompt Studio projects",+ "has_more": true,+ "results": [+ {+ "type": "prompt_studio_project",+ "id": "6f1c2a4e-1d1b-4c8e-9a51-0c7b2f1e9d10",+ "name": "Invoice parser",+ "context": "[email protected]",+ "match": "name_prefix",+ "route": {+ "key": "prompt_studio_project.detail",+ "params": {"id": "6f1c2a4e-1d1b-4c8e-9a51-0c7b2f1e9d10"}+ }+ }+ ]+ }+ ],+ "failed_types": []+}+```++- Groups are returned in the `order` of the table below. **Empty groups are+ omitted.**+- `has_more` is true when the type has more matches than `limit`. There is+ no total count; the "More" link opens the type's own list.+- `match` names the tier that ranked the result. Results are sorted by tier+ (`name_prefix`, `name`, `description_prefix`, `description`), then by most+ recently modified, then by id.+- `context` is a single display-only line, or `null`. Do not parse it.+- `route.key` is a route-registry name, not a URL. `route.params` values are+ strings. It is one of the keys the table below lists for the result's+ type; adapters have two (see below the table). A client that does not+ know a key should render the result without a link, never guess a+ screen.+- `type` is an open string. A client should render an unknown type with its+ `label`, or skip it.+- `failed_types` lists types whose search failed; their groups are missing+ even if they have matches, so a client can say "some results could not be+ loaded". It is empty on a normal response.+- `route.params` always includes `id`.++Errors use the standard error envelope:++- `400 {"type": "validation_error", "errors": [{"code", "detail", "attr"}]}`+ for bad parameters.+- `503` with code `search_unavailable` when every searched type failed with+ a database error a retry may clear (unreachable, timed out, locked), or+ when a failure left the connection unusable (even if other types had+ already answered). Safe to retry; never show it as "no results". A slow or+ locked query on a healthy connection otherwise only drops its own type+ into `failed_types` while other types still answer.+- `500` with code `search_failed` when every searched type failed and not+ all with such database errors (a bug). Retrying will not help.+- 401 and 403 behave as on every tenant endpoint.++## Types++| order | type | label | route.key | route.params | context |+|---|---|---|---|---|---|+| 10 | `prompt_studio_project` | Prompt Studio projects | `prompt_studio_project.detail` | `id` | owner e-mail |+| 30 | `workflow` | Workflows | `workflow.detail` | `id` | owner e-mail |+| 40 | `api_deployment` | API deployments | `api_deployment.detail` | `id` | parent workflow name |+| 50 | `etl_pipeline` | ETL pipelines | `etl_pipeline.detail` | `id` | parent workflow name |+| 60 | `task_pipeline` | Task pipelines | `task_pipeline.detail` | `id` | parent workflow name |+| 80 | `adapter` | Adapters | `adapter.detail` / `adapter.info` | `id`, `adapter_type` | adapter type · provider |+| 90 | `connector` | Connectors | `connector.detail` | `id` | provider |++An adapter opens on `adapter.detail` only for a caller who may edit it+(owner or co-owner, org admin, service account), the same rule as the+adapter `retrieve` endpoint. Everyone else gets `adapter.info`, with the+same params, a read-only view backed by `adapter/info/<id>/`, which allows+every caller search sends there.++Cloud and on-prem add `agentic_project` (20) and `review_queue` (70). Those+apps register these types themselves, and they are documented in the cloud+repo.++### What is matched and who sees it++Each type starts from the same `for_user()` queryset its list view uses.+That covers owners, co-owners, direct viewers, group shares, org shares,+admins and service accounts. Search rules:++- Names matched: `tool_name`, `workflow_name`, `display_name` and+ `api_name`, `pipeline_name`, `adapter_name`, `connector_name`.+- Descriptions matched: Prompt Studio projects, workflows, API deployments+ and adapters. Pipelines and connectors have no description.+- Pipelines: ETL and Task only. DEFAULT and APP pipelines are excluded.+- Adapters: platform-provided (frictionless) adapters are never returned.+ The adapter list shows them so any member can delete one, but nobody can+ open their detail view. No Op adapters are never returned either.+- Owner e-mail is not matched. The workflow, Prompt Studio, adapter and+ connector list `search` parameters do match it, so the panel and those+ filtered lists can differ slightly.++## Response-time budget++**p95 ≤ 300 ms server time for all types at `limit=5`**, on an organisation+of the largest current size. The fixture used is 2,000 rows per type, 200+users and 20 groups; it will be updated when real largest-org counts are+available.++CI does not gate on latency, because runners are shared. `tests/test_budget.py`+gates on query count instead: a request may issue at most three SQL queries+per type in total (one ranked search per type, plus the organisation point+lookups the existing `for_user()` makes) plus a small fixed overhead, and+the count must not change between `limit=1` and `limit=20`, so an N+1 query+pattern fails the test. The measured p95 is printed on every run and is+asserted only with `GLOBAL_SEARCH_ASSERT_P95=1`, against+`GLOBAL_SEARCH_CEILING_MS` (default 300). No trigram index is used: every+query is filtered by organisation first.++## Adding a type++Subclass `global_search.providers.SearchProvider`, or use+`ModelSearchProvider` for a single queryset. It must apply the type's own+list visibility rule. Register it with+`global_search.registry.register_provider()` from the owning app's+`AppConfig.ready()`, add a row to the table above, and add visibility tests.diff --git a/backend/global_search/__init__.py b/backend/global_search/__init__.pynew file mode 100644index 000000000..e69de29bbdiff --git a/backend/global_search/apps.py b/backend/global_search/apps.pynew file mode 100644index 000000000..bad24c2ec--- /dev/null+++ b/backend/global_search/apps.py@@ -0,0 +1,12 @@+from django.apps import AppConfig+++class GlobalSearchConfig(AppConfig):+ name = "global_search"++ def ready(self) -> None:+ from global_search.oss_providers import OSS_PROVIDERS+ from global_search.registry import register_provider++ for provider_cls in OSS_PROVIDERS:+ register_provider(provider_cls())diff --git a/backend/global_search/constants.py b/backend/global_search/constants.pynew file mode 100644index 000000000..ffec01c2a--- /dev/null+++ b/backend/global_search/constants.py@@ -0,0 +1,45 @@+"""Contract constants for the global search endpoint: parameter bounds,+match tiers and the OSS type keys. Cloud-only type keys are deliberately+absent: cloud apps define and register their own.+"""++from enum import IntEnum+++class SearchParams:+ QUERY = "q"+ TYPES = "types"+ LIMIT = "limit"++ # A shorter query answers an empty result, not an error.+ MIN_QUERY_LENGTH = 2+ MAX_QUERY_LENGTH = 100+ DEFAULT_LIMIT = 5+ MAX_LIMIT = 20+++class MatchTier(IntEnum):+ """Ranking tiers, best first. The value is the sort key; the label is+ what the response's ``match`` field carries.+ """++ NAME_PREFIX = 0+ NAME = 1+ DESCRIPTION_PREFIX = 2+ DESCRIPTION = 3++ @property+ def label(self) -> str:+ return self.name.lower()+++class OSSSearchType:+ """Type keys contributed by OSS."""++ PROMPT_STUDIO_PROJECT = "prompt_studio_project"+ WORKFLOW = "workflow"+ API_DEPLOYMENT = "api_deployment"+ ETL_PIPELINE = "etl_pipeline"+ TASK_PIPELINE = "task_pipeline"+ ADAPTER = "adapter"+ CONNECTOR = "connector"diff --git a/backend/global_search/contract/search-response.schema.json b/backend/global_search/contract/search-response.schema.jsonnew file mode 100644index 000000000..cc5180ed2--- /dev/null+++ b/backend/global_search/contract/search-response.schema.json@@ -0,0 +1,120 @@+{+ "$schema": "https://json-schema.org/draft/2020-12/schema",+ "$id": "unstract/global-search/response/v1",+ "type": "object",+ "required": [+ "query",+ "groups",+ "failed_types"+ ],+ "additionalProperties": false,+ "properties": {+ "query": {+ "type": "string"+ },+ "groups": {+ "type": "array",+ "items": {+ "$ref": "#/$defs/group"+ }+ },+ "failed_types": {+ "type": "array",+ "items": {+ "type": "string"+ }+ }+ },+ "$defs": {+ "group": {+ "type": "object",+ "required": [+ "type",+ "label",+ "has_more",+ "results"+ ],+ "additionalProperties": false,+ "properties": {+ "type": {+ "type": "string",+ "pattern": "^[a-z][a-z0-9_]*$"+ },+ "label": {+ "type": "string"+ },+ "has_more": {+ "type": "boolean"+ },+ "results": {+ "type": "array",+ "minItems": 1,+ "maxItems": 20,+ "items": {+ "$ref": "#/$defs/result"+ }+ }+ }+ },+ "result": {+ "type": "object",+ "required": [+ "type",+ "id",+ "name",+ "context",+ "match",+ "route"+ ],+ "additionalProperties": false,+ "properties": {+ "type": {+ "type": "string"+ },+ "id": {+ "type": "string"+ },+ "name": {+ "type": "string"+ },+ "context": {+ "type": [+ "string",+ "null"+ ]+ },+ "match": {+ "enum": [+ "name_prefix",+ "name",+ "description_prefix",+ "description"+ ]+ },+ "route": {+ "type": "object",+ "required": [+ "key",+ "params"+ ],+ "additionalProperties": false,+ "properties": {+ "key": {+ "type": "string",+ "pattern": "^[a-z][a-z0-9_]*\\.[a-z][a-z0-9_]*$"+ },+ "params": {+ "type": "object",+ "additionalProperties": {+ "type": "string"+ },+ "required": [+ "id"+ ]+ }+ }+ }+ }+ }+ }+}diff --git a/backend/global_search/exceptions.py b/backend/global_search/exceptions.pynew file mode 100644index 000000000..164b8c527--- /dev/null+++ b/backend/global_search/exceptions.py@@ -0,0 +1,23 @@+from rest_framework import status+from rest_framework.exceptions import APIException+++class SearchUnavailable(APIException):+ """Search cannot answer because of database errors a retry may clear+ (unreachable, timed out, locked) or an unusable connection. The client+ may retry; never "no matches".+ """++ status_code = status.HTTP_503_SERVICE_UNAVAILABLE+ default_detail = "Search is temporarily unavailable."+ default_code = "search_unavailable"+++class SearchFailed(APIException):+ """Every searched type failed, and not all with retryable database+ errors -- most likely a bug, so retrying will not help.+ """++ status_code = status.HTTP_500_INTERNAL_SERVER_ERROR+ default_detail = "Search failed."+ default_code = "search_failed"diff --git a/backend/global_search/oss_providers.py b/backend/global_search/oss_providers.pynew file mode 100644index 000000000..2ab6a1796--- /dev/null+++ b/backend/global_search/oss_providers.py@@ -0,0 +1,221 @@+"""OSS result types.++Each provider starts from the same ``Model.objects.for_user(user)`` its list+view uses, so search never returns a row the caller could not open from the+list. Search may narrow further (pipeline type, No Op and frictionless+adapters), never widen.+"""++from __future__ import annotations++import dataclasses+from typing import Any++from adapter_processor_v2.models import AdapterInstance+from api_v2.models import APIDeployment+from connector_v2.models import ConnectorInstance+from django.contrib.contenttypes.models import ContentType+from django.db.models import (+ BooleanField,+ CharField,+ Exists,+ F,+ Model,+ OuterRef,+ Q,+ QuerySet,+ Subquery,+ Value,+)+from django.db.models.functions import Cast+from permissions.roles import ResourceRole+from pipeline_v2.models import Pipeline+from prompt_studio.prompt_studio_core_v2.models import CustomTool+from tenant_account_v2.models import ResourceMembership+from tenant_account_v2.organization_member_service import OrganizationMemberService+from workflow_manager.workflow_v2.models.workflow import Workflow++from global_search.constants import OSSSearchType+from global_search.providers import ModelSearchProvider, SearchHit++OWNER_EMAIL = "_search_owner_email"+PARENT_NAME = "_search_parent_name"+CAN_EDIT = "_search_can_edit"++# Frontend v1 retires the No Op adapters (spec decision D12, UN-4257);+# their ids share this prefix.+NO_OP_ADAPTER_PREFIX = "noOp"+++def owner_email_subquery(model: type[Model]) -> Subquery:+ """Earliest live, non-service-account OWNER's e-mail, as one correlated+ subquery. Same rule as ``HasMembersMixin.owner_email`` without the+ per-row prefetch.+ """+ return Subquery(+ ResourceMembership.objects.filter(+ content_type=ContentType.objects.get_for_model(model),+ object_id=Cast(OuterRef("pk"), output_field=CharField()),+ role=ResourceRole.OWNER,+ user__is_service_account=False,+ )+ .order_by("created_at", "pk")+ .values("user__email")[:1]+ )+++def _provider(registry_id: str) -> str:+ """Provider slug from an adapter/connector registry id (``openai|<uuid>``)."""+ return registry_id.split("|", 1)[0] if registry_id else ""+++class OwnedModelProvider(ModelSearchProvider):+ """Context line is the owner's e-mail."""++ def annotate(self, queryset: QuerySet[Any], user: Any) -> QuerySet[Any]:+ return queryset.annotate(**{OWNER_EMAIL: owner_email_subquery(queryset.model)})++ def context(self, obj: Model) -> str | None:+ return getattr(obj, OWNER_EMAIL)+++class WorkflowChildProvider(ModelSearchProvider):+ """Context line is the parent workflow's name."""++ def annotate(self, queryset: QuerySet[Any], user: Any) -> QuerySet[Any]:+ return queryset.annotate(**{PARENT_NAME: F("workflow__workflow_name")})++ def context(self, obj: Model) -> str | None:+ return getattr(obj, PARENT_NAME)+++class PromptStudioProjectProvider(OwnedModelProvider):+ type_key = OSSSearchType.PROMPT_STUDIO_PROJECT+ label = "Prompt Studio projects"+ order = 10+ name_fields = ("tool_name",)+ description_field = "description"++ def visible_queryset(self, user: Any) -> QuerySet[Any]:+ return CustomTool.objects.for_user(user)+++class WorkflowProvider(OwnedModelProvider):+ type_key = OSSSearchType.WORKFLOW+ label = "Workflows"+ order = 30+ name_fields = ("workflow_name",)+ description_field = "description"++ def visible_queryset(self, user: Any) -> QuerySet[Any]:+ return Workflow.objects.for_user(user)+++class APIDeploymentProvider(WorkflowChildProvider):+ type_key = OSSSearchType.API_DEPLOYMENT+ label = "API deployments"+ order = 40+ # A match on api_name alone still shows display_name: that is the name+ # users see everywhere else.+ name_fields = ("display_name", "api_name")+ description_field = "description"++ def visible_queryset(self, user: Any) -> QuerySet[Any]:+ return APIDeployment.objects.for_user(user)+++class _PipelineProvider(WorkflowChildProvider):+ pipeline_type: str+ name_fields = ("pipeline_name",)++ def visible_queryset(self, user: Any) -> QuerySet[Any]:+ return Pipeline.objects.for_user(user).filter(pipeline_type=self.pipeline_type)+++class ETLPipelineProvider(_PipelineProvider):+ type_key = OSSSearchType.ETL_PIPELINE+ label = "ETL pipelines"+ order = 50+ pipeline_type = Pipeline.PipelineType.ETL+++class TaskPipelineProvider(_PipelineProvider):+ type_key = OSSSearchType.TASK_PIPELINE+ label = "Task pipelines"+ order = 60+ pipeline_type = Pipeline.PipelineType.TASK+++class AdapterProvider(ModelSearchProvider):+ type_key = OSSSearchType.ADAPTER+ label = "Adapters"+ order = 80+ name_fields = ("adapter_name",)+ description_field = "description"++ def visible_queryset(self, user: Any) -> QuerySet[Any]:+ # Frictionless adapters stay in the adapter list, where any member+ # may delete them, but nobody can open their detail view: retrieve+ # refuses them for everyone, admins included (IsFrictionLessAdapter).+ return AdapterInstance.objects.for_user(user).exclude(+ Q(adapter_id__startswith=NO_OP_ADAPTER_PREFIX) | Q(is_friction_less=True)+ )++ @property+ def info_route_key(self) -> str:+ return f"{self.type_key}.info"++ def annotate(self, queryset: QuerySet[Any], user: Any) -> QuerySet[Any]:+ # Same rule as IsFrictionLessAdapter, which gates retrieve: service+ # accounts, org admins and owners (creator or co-owner). Anyone else+ # who can list the adapter opens it read-only, via adapter_info.+ if getattr(+ user, "is_service_account", False+ ) or OrganizationMemberService.is_user_organization_admin(user):+ can_edit = Value(True, output_field=BooleanField())+ else:+ can_edit = Exists(+ ResourceMembership.objects.filter(+ content_type=ContentType.objects.get_for_model(queryset.model),+ object_id=Cast(OuterRef("pk"), output_field=CharField()),+ role=ResourceRole.OWNER,+ user=user,+ )+ )+ return queryset.annotate(**{CAN_EDIT: can_edit})++ def to_hit(self, obj: Model) -> SearchHit:+ hit = super().to_hit(obj)+ if getattr(obj, CAN_EDIT):+ return hit+ return dataclasses.replace(hit, route_key=self.info_route_key)++ def context(self, obj: Model) -> str | None:+ return " · ".join(p for p in (obj.adapter_type, _provider(obj.adapter_id)) if p)++ def route_params(self, obj: Model) -> dict[str, str]:+ return {"id": str(obj.pk), "adapter_type": obj.adapter_type}+++class ConnectorProvider(ModelSearchProvider):+ type_key = OSSSearchType.CONNECTOR+ label = "Connectors"+ order = 90+ name_fields = ("connector_name",)++ def visible_queryset(self, user: Any) -> QuerySet[Any]:+ return ConnectorInstance.objects.for_user(user)++ def context(self, obj: Model) -> str | None:+ return _provider(obj.connector_id) or None+++OSS_PROVIDERS = (+ PromptStudioProjectProvider,+ WorkflowProvider,+ APIDeploymentProvider,+ ETLPipelineProvider,+ TaskPipelineProvider,+ AdapterProvider,+ ConnectorProvider,+)diff --git a/backend/global_search/providers.py b/backend/global_search/providers.pynew file mode 100644index 000000000..1a00b19f1--- /dev/null+++ b/backend/global_search/providers.py@@ -0,0 +1,104 @@+"""Search provider contract.++A provider answers one result type. It owns two things the endpoint cannot+know: which rows the caller may open (it must apply the same visibility rule+as that type's own list) and how a row becomes a result. Ranking is shared,+see :mod:`global_search.ranking`.+"""++from __future__ import annotations++from abc import ABC, abstractmethod+from dataclasses import dataclass+from typing import Any++from django.db.models import Model, QuerySet++from global_search.constants import MatchTier+from global_search.ranking import RANK_ANNOTATION, rank_queryset+++@dataclass(frozen=True)+class SearchHit:+ id: str+ name: str+ match: MatchTier+ # No default, and it must carry the ``id`` the frontend opens.+ route_params: dict[str, str]+ context: str | None = None+ # Overrides the provider's ``route_key`` for this hit, for a type whose+ # rows open on different screens depending on the caller.+ route_key: str | None = None++ def __post_init__(self) -> None:+ if "id" not in self.route_params:+ raise ValueError(f"SearchHit {self.id!r} has no route id")+++class SearchProvider(ABC):+ """One result type in the search response.++ Subclasses set ``type_key`` (stable contract key), ``label`` (display+ text) and ``order`` (rail position; groups are returned ascending).+ """++ type_key: str+ label: str+ order: int++ @property+ def route_key(self) -> str:+ return f"{self.type_key}.detail"++ @abstractmethod+ def search(self, user: Any, query: str, limit: int) -> list[SearchHit]:+ """Return up to ``limit + 1`` hits, best first.++ The extra row lets the caller derive ``has_more`` without a COUNT.+ Must never return a row ``user`` could not open from the type's list.+ """+++class ModelSearchProvider(SearchProvider):+ """Provider backed by one ORM query.++ Subclasses implement :meth:`visible_queryset` with the list view's own+ visibility rule, and :meth:`to_hit` to shape a row.+ """++ # Every field matched as "name"; the first is the displayed name.+ name_fields: tuple[str, ...]+ description_field: str | None = None++ @abstractmethod+ def visible_queryset(self, user: Any) -> QuerySet[Any]:+ """Rows ``user`` can see in this type's list, before search."""++ def annotate(self, queryset: QuerySet[Any], user: Any) -> QuerySet[Any]:+ """Hook to add context or route columns without per-row queries."""+ return queryset++ @abstractmethod+ def context(self, obj: Model) -> str | None:+ """One display-only line, or ``None``."""++ def route_params(self, obj: Model) -> dict[str, str]:+ return {"id": str(obj.pk)}++ def search(self, user: Any, query: str, limit: int) -> list[SearchHit]:+ queryset = rank_queryset(+ self.visible_queryset(user),+ query,+ name_fields=self.name_fields,+ description_field=self.description_field,+ )+ return [self.to_hit(obj) for obj in self.annotate(queryset, user)[: limit + 1]]++ def to_hit(self, obj: Model) -> SearchHit:+ return SearchHit(+ id=str(obj.pk),+ name=getattr(obj, self.name_fields[0]),+ match=MatchTier(getattr(obj, RANK_ANNOTATION)),+ context=self.context(obj),+ route_params=self.route_params(obj),+ )diff --git a/backend/global_search/ranking.py b/backend/global_search/ranking.pynew file mode 100644index 000000000..4320cb14b--- /dev/null+++ b/backend/global_search/ranking.py@@ -0,0 +1,104 @@+"""Shared ranking: prefix before substring, name before description.++Both an ORM form (one SQL query per type) and an in-memory form (for+providers whose source is not a single queryset) so every type ranks alike.+"""++from __future__ import annotations++from collections.abc import Callable, Iterable+from typing import Any, TypeVar++from django.db.models import Case, IntegerField, Q, QuerySet, Value, When++from global_search.constants import MatchTier++RANK_ANNOTATION = "_search_rank"++T = TypeVar("T")+++def _any(fields: Iterable[str], lookup: str, query: str) -> Q:+ condition = Q()+ for field_name in fields:+ condition |= Q(**{f"{field_name}__{lookup}": query})+ return condition+++def rank_queryset(+ queryset: QuerySet[Any],+ query: str,+ *,+ name_fields: tuple[str, ...],+ description_field: str | None = None,+) -> QuerySet[Any]:+ """Filter ``queryset`` to rows matching ``query`` and order them by tier,+ then most recently modified, then pk so equal tiers page deterministically.++ ``icontains``/``istartswith`` escape ``%`` and ``_``, so user input is+ always matched literally.+ """+ tiers = [+ When(_any(name_fields, "istartswith", query), then=Value(MatchTier.NAME_PREFIX)),+ When(_any(name_fields, "icontains", query), then=Value(MatchTier.NAME)),+ ]+ matches = _any(name_fields, "icontains", query)+ if description_field:+ tiers += [+ When(+ Q(**{f"{description_field}__istartswith": query}),+ then=Value(MatchTier.DESCRIPTION_PREFIX),+ ),+ When(+ Q(**{f"{description_field}__icontains": query}),+ then=Value(MatchTier.DESCRIPTION),+ ),+ ]+ matches |= Q(**{f"{description_field}__icontains": query})++ return (+ queryset.filter(matches)+ .annotate(**{RANK_ANNOTATION: Case(*tiers, output_field=IntegerField())})+ .order_by(RANK_ANNOTATION, "-modified_at", "pk")+ )+++def match_tier(+ query: str, names: Iterable[str | None], description: str | None = None+) -> MatchTier | None:+ """In-memory equivalent of the ``rank_queryset`` tiers; ``None`` means no+ match. Comparison is case-insensitive via ``casefold``.+ """+ needle = query.casefold()+ names = [n.casefold() for n in names if n]+ if any(n.startswith(needle) for n in names):+ return MatchTier.NAME_PREFIX+ if any(needle in n for n in names):+ return MatchTier.NAME+ if description:+ desc = description.casefold()+ if desc.startswith(needle):+ return MatchTier.DESCRIPTION_PREFIX+ if needle in desc:+ return MatchTier.DESCRIPTION+ return None+++def rank_items(+ items: Iterable[T],+ query: str,+ *,+ names: Callable[[T], Iterable[str | None]],+ description: Callable[[T], str | None] = lambda _: None,+ sort_key: Callable[[T], Any],+) -> list[tuple[T, MatchTier]]:+ """Filter and order ``items`` like :func:`rank_queryset`. ``sort_key``+ breaks ties within a tier and must be total, for deterministic order.+ """+ ranked = []+ for item in items:+ tier = match_tier(query, names(item), description(item))+ if tier is not None:+ ranked.append((item, tier))+ ranked.sort(key=lambda pair: (pair[1], sort_key(pair[0])))+ return rankeddiff --git a/backend/global_search/registry.py b/backend/global_search/registry.pynew file mode 100644index 000000000..f8b2c43f3--- /dev/null+++ b/backend/global_search/registry.py@@ -0,0 +1,49 @@+"""Process-wide provider registry.++OSS registers its providers from ``GlobalSearchConfig.ready()``; cloud apps+register theirs from their own ``ready()``, so a type exists exactly where+its app is installed and OSS never names a cloud type.+"""++from __future__ import annotations++import logging+import re++from global_search.providers import SearchProvider++logger = logging.getLogger(__name__)++_providers: dict[str, SearchProvider] = {}++# Same pattern the published JSON Schema requires of ``type``.+_TYPE_KEY = re.compile(r"^[a-z][a-z0-9_]*$")+++def register_provider(provider: SearchProvider) -> None:+ """Register ``provider`` under its ``type_key``.++ Re-registering the same class is a no-op, so a repeated ``ready()`` is+ harmless; a different class claiming a taken key is a wiring bug.+ """+ if not _TYPE_KEY.fullmatch(provider.type_key):+ raise ValueError(f"Invalid search type key '{provider.type_key}'")+ existing = _providers.get(provider.type_key)+ if existing is not None:+ if type(existing) is type(provider):+ return+ raise ValueError(+ f"Search type '{provider.type_key}' is already registered by "+ f"{type(existing).__name__}"+ )+ _providers[provider.type_key] = provider+ logger.debug("Registered global search provider '%s'", provider.type_key)+++def unregister_provider(type_key: str) -> None:+ _providers.pop(type_key, None)+++def get_providers() -> list[SearchProvider]:+ """Registered providers in rail order."""+ return sorted(_providers.values(), key=lambda p: (p.order, p.type_key))diff --git a/backend/global_search/serializers.py b/backend/global_search/serializers.pynew file mode 100644index 000000000..dd973c1c5--- /dev/null+++ b/backend/global_search/serializers.py@@ -0,0 +1,75 @@+"""Request validation, and the response envelope for drf-spectacular.++The view builds plain dicts; the tested contract is+``contract/search-response.schema.json``.+"""++from rest_framework import serializers++from global_search.constants import MatchTier, SearchParams+++class SearchQuerySerializer(serializers.Serializer):+ q = serializers.CharField(+ required=False,+ allow_blank=True,+ trim_whitespace=True,+ max_length=SearchParams.MAX_QUERY_LENGTH,+ default="",+ help_text=(+ "Case-insensitive text matched against name and description. "+ f"Shorter than {SearchParams.MIN_QUERY_LENGTH} characters after "+ "trimming returns no groups."+ ),+ )+ types = serializers.CharField(+ required=False,+ allow_blank=True,+ default="",+ help_text=(+ "Comma-separated type keys to search. Empty means all. Unknown or "+ "unavailable keys are ignored."+ ),+ )+ limit = serializers.IntegerField(+ required=False,+ default=SearchParams.DEFAULT_LIMIT,+ min_value=1,+ max_value=SearchParams.MAX_LIMIT,+ help_text="Results per type.",+ )++ def validate_types(self, value: str) -> list[str]:+ return [key for key in (part.strip() for part in value.split(",")) if key]+++class SearchRouteSerializer(serializers.Serializer):+ key = serializers.CharField(help_text="Route-registry name, not a URL.")+ params = serializers.DictField(child=serializers.CharField())+++class SearchResultSerializer(serializers.Serializer):+ type = serializers.CharField()+ id = serializers.CharField()+ name = serializers.CharField(allow_blank=True)+ context = serializers.CharField(+ allow_null=True, help_text="One display-only line; see README for each type."+ )+ match = serializers.ChoiceField(choices=[tier.label for tier in MatchTier])+ route = SearchRouteSerializer()+++class SearchGroupSerializer(serializers.Serializer):+ type = serializers.CharField()+ label = serializers.CharField()+ has_more = serializers.BooleanField()+ results = SearchResultSerializer(many=True)+++class SearchResponseSerializer(serializers.Serializer):+ query = serializers.CharField(allow_blank=True)+ groups = SearchGroupSerializer(many=True)+ failed_types = serializers.ListField(+ child=serializers.CharField(),+ help_text="Types whose search failed; their groups are missing.",+ )diff --git a/backend/global_search/tests/__init__.py b/backend/global_search/tests/__init__.pynew file mode 100644index 000000000..e69de29bbdiff --git a/backend/global_search/tests/base.py b/backend/global_search/tests/base.pynew file mode 100644index 000000000..10cdd8971--- /dev/null+++ b/backend/global_search/tests/base.py@@ -0,0 +1,126 @@+"""Builders and a request helper shared by the global search tests."""++import secrets+from collections.abc import Callable+from typing import Any++from account_v2.models import Organization, User+from adapter_processor_v2.models import AdapterInstance+from api_v2.models import APIDeployment+from connector_v2.models import ConnectorInstance+from pipeline_v2.models import Pipeline+from prompt_studio.prompt_studio_core_v2.models import CustomTool+from rest_framework.response import Response+from rest_framework.test import APIRequestFactory, force_authenticate+from workflow_manager.workflow_v2.models.workflow import Workflow++from global_search.constants import OSSSearchType+from global_search.views import GlobalSearchView+++def _workflow(org: Organization, user: User, name: str, description: str) -> Workflow:+ return Workflow.objects.create(+ workflow_name=name, description=description, organization=org, created_by=user+ )+++def _parent(org: Organization, user: User, name: str) -> Workflow:+ # Random so the parent never matches a test query by accident.+ return _workflow(org, user, f"parent-{secrets.token_hex(4)}", "")+++def build_prompt_studio_project(org, user, name, description=""):+ return CustomTool.objects.create(+ tool_name=name, description=description, organization=org, created_by=user+ )+++def build_workflow(org, user, name, description=""):+ return _workflow(org, user, name, description)+++def build_api_deployment(org, user, name, description=""):+ return APIDeployment.objects.create(+ display_name=name,+ api_name=f"api-{secrets.token_hex(4)}",+ description=description,+ workflow=_parent(org, user, name),+ organization=org,+ created_by=user,+ )+++def _pipeline(pipeline_type: str) -> Callable[..., Pipeline]:+ def build(org, user, name, description=""):+ return Pipeline.objects.create(+ pipeline_name=name,+ pipeline_type=pipeline_type,+ workflow=_parent(org, user, name),+ organization=org,+ created_by=user,+ )++ return build+++def build_adapter(org, user, name, description="", **overrides):+ fields = {+ "adapter_name": name,+ "adapter_id": "openai|test",+ "adapter_type": "LLM",+ "adapter_metadata": {},+ "description": description,+ "organization": org,+ "created_by": user,+ }+ fields.update(overrides)+ return AdapterInstance.objects.create(**fields)+++def build_connector(org, user, name, description=""):+ return ConnectorInstance.objects.create(+ connector_name=name,+ connector_id="minio|c799f6e3-2b57-434e-aaac-b5daa415da19",+ organization=org,+ created_by=user,+ )+++# Every OSS type, keyed by its contract type key.+OSS_BUILDERS: dict[str, Callable[..., Any]] = {+ OSSSearchType.PROMPT_STUDIO_PROJECT: build_prompt_studio_project,+ OSSSearchType.WORKFLOW: build_workflow,+ OSSSearchType.API_DEPLOYMENT: build_api_deployment,+ OSSSearchType.ETL_PIPELINE: _pipeline(Pipeline.PipelineType.ETL),+ OSSSearchType.TASK_PIPELINE: _pipeline(Pipeline.PipelineType.TASK),+ OSSSearchType.ADAPTER: build_adapter,+ OSSSearchType.CONNECTOR: build_connector,+}++# Types whose model has a description column the search matches.+DESCRIBED_TYPES = {+ OSSSearchType.PROMPT_STUDIO_PROJECT,+ OSSSearchType.WORKFLOW,+ OSSSearchType.API_DEPLOYMENT,+ OSSSearchType.ADAPTER,+}+++def search(user: User, allow_failures: bool = False, **params: Any) -> Response:+ """Call the view. Unless ``allow_failures``, a 200 must report no failed+ types, so a crashing provider is never mistaken for one that filtered."""+ request = APIRequestFactory().get("/search/", params)+ force_authenticate(request, user=user)+ response = GlobalSearchView.as_view()(request)+ if response.status_code == 200 and not allow_failures:+ assert response.data["failed_types"] == [], response.data["failed_types"]+ return response+++def result_ids(response: Response, type_key: str | None = None) -> list[str]:+ return [+ result["id"]+ for group in response.data["groups"]+ if type_key is None or group["type"] == type_key+ for result in group["results"]+ ]diff --git a/backend/global_search/tests/test_budget.py b/backend/global_search/tests/test_budget.pynew file mode 100644index 000000000..2b5ef29d7--- /dev/null+++ b/backend/global_search/tests/test_budget.py@@ -0,0 +1,259 @@+"""Response-time budget on a large organisation.++The contract budget is p95 <= 300 ms server time for all types at limit=5.+The merge gate is structural: a bounded total of SQL queries (at most three+per type, summed) that does not change with the result count, and every+type answering without failure. Wall-clock time depends on the+runner, so the p95 is always printed but asserted only on request+(``GLOBAL_SEARCH_ASSERT_P95=1``, against ``GLOBAL_SEARCH_CEILING_MS``).++The fixture size stands in for the largest current organisation until real+counts are available: 2,000 rows per type, 200 users, 20 groups.+"""++import os+import statistics+import time++from account_v2.models import Organization, User+from adapter_processor_v2.models import AdapterInstance+from api_v2.models import APIDeployment+from connector_v2.models import ConnectorInstance+from django.contrib.contenttypes.models import ContentType+from django.db import connection+from django.test import TestCase+from django.test.utils import CaptureQueriesContext+from unittest.mock import patch+from permissions.roles import ResourceRole+from pipeline_v2.models import Pipeline+from prompt_studio.prompt_studio_core_v2.models import CustomTool+from tenant_account_v2.models import (+ GroupMembership,+ OrganizationGroup,+ OrganizationMember,+ ResourceGroupShare,+ ResourceMembership,+)+from tenant_account_v2.organization_member_service import OrganizationMemberService+from utils.user_context import UserContext+from workflow_manager.workflow_v2.models.workflow import Workflow++from global_search.oss_providers import OSS_PROVIDERS+from global_search.tests.base import search++ROWS_PER_TYPE = 2_000+USERS = 200+GROUPS = 20+BUDGET_MS = 300+ASSERT_P95 = os.environ.get("GLOBAL_SEARCH_ASSERT_P95") == "1"+CEILING_MS = int(os.environ.get("GLOBAL_SEARCH_CEILING_MS", str(BUDGET_MS)))+# Each type costs one ranked search query plus the organisation point+# lookups the existing ``for_user()`` makes (the org-scoped manager, and the+# membership subquery on the non-admin path). Anything per result -- an+# N+1 -- breaks the bound.+QUERIES_PER_TYPE = 3+# Per request, not per type: the caller's admin-role lookup (memoized on the+# user, so paid once) and the organisation lookup, plus one spare.+FIXED_OVERHEAD_QUERIES = 3+# OSS types only, so the bound holds in every distribution; cloud types+# carry their own query bounds in the cloud repo.+OSS_TYPES = ",".join(p.type_key for p in OSS_PROVIDERS)+QUERIES = ["acme", "inv", "report", "zz", "pipe", "2024", "ac", "data"]+WORDS = ["invoice", "report", "acme", "data", "pipe", "ledger", "claims", "2024"]+++def _name(prefix: str, i: int) -> str:+ return f"{WORDS[i % len(WORDS)]}-{prefix}-{i}"+++class SearchBudgetTests(TestCase):+ @classmethod+ def setUpTestData(cls) -> None:+ cls.org = Organization.objects.create(+ name="big", display_name="Big", organization_id="org-big"+ )+ UserContext.set_organization_identifier(cls.org.organization_id)+ users = User.objects.bulk_create(+ User(username=f"u{i}@big.test", email=f"u{i}@big.test") for i in range(USERS)+ )+ OrganizationMember.objects.bulk_create(+ OrganizationMember(organization=cls.org, user=u, role="user") for u in users+ )+ groups = OrganizationGroup.objects.bulk_create(+ OrganizationGroup(organization=cls.org, name=f"g{i}", created_by=users[0])+ for i in range(GROUPS)+ )+ GroupMembership.objects.bulk_create(+ GroupMembership(group=groups[i % GROUPS], user=u) for i, u in enumerate(users)+ )+ cls.member = users[0]+ cls.admin = users[1]+ # The role lookup a request pays is counted: only the role-name check+ # is patched (see setUp), never the predicate that queries it.+ OrganizationMember.objects.filter(organization=cls.org, user=cls.admin).update(+ role="admin"+ )++ common = {"organization": cls.org}+ n = range(ROWS_PER_TYPE)+ parents = Workflow.objects.bulk_create(+ Workflow(workflow_name=f"parent-{i}", **common) for i in range(ROWS_PER_TYPE)+ )+ rows = [+ *Workflow.objects.bulk_create(+ Workflow(workflow_name=_name("wf", i), description=_name("d", i), **common)+ for i in n+ ),+ *CustomTool.objects.bulk_create(+ CustomTool(+ tool_name=_name("tool", i),+ description=_name("d", i),+ author="seed",+ **common,+ )+ for i in n+ ),+ *APIDeployment.objects.bulk_create(+ APIDeployment(+ display_name=_name("api", i),+ api_name=f"a{i}",+ api_endpoint=f"deployment/api/org-big/a{i}/",+ description=_name("d", i),+ workflow=parents[i],+ **common,+ )+ for i in n+ ),+ *Pipeline.objects.bulk_create(+ Pipeline(+ pipeline_name=_name(f"p{t}", i),+ pipeline_type=t,+ workflow=parents[i],+ **common,+ )+ for t in (Pipeline.PipelineType.ETL, Pipeline.PipelineType.TASK)+ for i in n+ ),+ *AdapterInstance.objects.bulk_create(+ AdapterInstance(+ adapter_name=_name("ad", i),+ adapter_id="openai|seed",+ adapter_type="LLM",+ adapter_metadata={},+ description=_name("d", i),+ **common,+ )+ for i in n+ ),+ *ConnectorInstance.objects.bulk_create(+ ConnectorInstance(connector_name=_name("conn", i), connector_id="minio|seed", **common)+ for i in n+ ),+ ]+ # One owner per row spread across users, a group share on every 50th+ # and an org share on every 100th: the non-admin path's real mix.+ memberships, shares = [], []+ for i, row in enumerate(rows):+ ct = ContentType.objects.get_for_model(type(row))+ memberships.append(+ ResourceMembership(+ user=users[i % USERS],+ role=ResourceRole.OWNER,+ content_type=ct,+ object_id=str(row.pk),+ organization=cls.org,+ )+ )+ if i % 50 == 0:+ shares.append(+ ResourceGroupShare(+ group=groups[i % GROUPS],+ content_type=ct,+ object_id=str(row.pk),+ organization=cls.org,+ )+ )+ if i % 100 == 0:+ row.shared_to_org = True+ ResourceMembership.objects.bulk_create(memberships, batch_size=2_000)+ ResourceGroupShare.objects.bulk_create(shares, batch_size=2_000)+ for model in (Workflow, CustomTool, APIDeployment, Pipeline, AdapterInstance, ConnectorInstance):+ model.objects.filter(+ pk__in=[r.pk for r in rows if isinstance(r, model) and r.shared_to_org]+ ).update(shared_to_org=True)+ with connection.cursor() as cursor:+ cursor.execute("ANALYZE")++ def setUp(self) -> None:+ UserContext.set_organization_identifier(self.org.organization_id)+ # Each auth plugin names the admin role differently ("admin" in OSS,+ # "unstract_admin" in cloud). Pin the name check, which is pure+ # Python, so the admin subtest measures an admin in every+ # distribution; the membership query still runs and is counted.+ name_check = patch(+ "account_v2.authentication_controller.AuthenticationController"+ ".is_admin_by_role",+ new=lambda _self, role: role == "admin",+ )+ name_check.start()+ self.addCleanup(name_check.stop)+ self.assertTrue(+ OrganizationMemberService.is_user_organization_admin(+ User.objects.get(pk=self.admin.pk)+ )+ )++ def _queries(self, user: User, limit: int) -> list[str]:+ """SQL issued for one search, minus the per-provider savepoints the+ view opens inside a transaction (TestCase always has one)."""+ # A fresh user object, as each request gets: the admin-role memo+ # lives on the user, so reusing one would hide that lookup.+ user = User.objects.get(pk=user.pk)+ with CaptureQueriesContext(connection) as ctx:+ response = search(user, q="inv", types=OSS_TYPES, limit=limit)+ self.assertEqual(response.status_code, 200)+ # search() already asserts no type failed; every type also matches,+ # so a type that silently returned nothing would show here.+ self.assertEqual(+ {g["type"] for g in response.data["groups"]},+ {p.type_key for p in OSS_PROVIDERS},+ )+ return [+ q["sql"]+ for q in ctx.captured_queries+ if not q["sql"].startswith(("SAVEPOINT", "RELEASE SAVEPOINT"))+ ]++ def _assert_bounded(self, user: User) -> None:+ search(user, q="warm", types=OSS_TYPES) # content-type cache warm-up+ few = self._queries(user, limit=1)+ many = self._queries(user, limit=20)+ self.assertLessEqual(+ len(many),+ QUERIES_PER_TYPE * len(OSS_PROVIDERS) + FIXED_OVERHEAD_QUERIES,+ "\n".join(sql[:200] for sql in many),+ )+ # An N+1 shows up as more queries for more results.+ self.assertEqual(len(many), len(few))++ def _p95_ms(self, user: User) -> float:+ timings = []+ for _ in range(3):+ for q in QUERIES:+ start = time.perf_counter()+ search(user, q=q, types=OSS_TYPES)+ timings.append((time.perf_counter() - start) * 1000)+ return statistics.quantiles(timings, n=20)[-1]++ def test_query_count_is_bounded_per_type(self) -> None:+ for user in (self.member, self.admin):+ with self.subTest(user=user.email):+ self._assert_bounded(user)++ def test_p95_is_measured(self) -> None:+ for user in (self.member, self.admin):+ with self.subTest(user=user.email):+ p95 = self._p95_ms(user)+ print(f"[global_search budget] {user.email} p95={p95:.1f}ms (budget {BUDGET_MS}ms)")+ if ASSERT_P95:+ self.assertLessEqual(p95, CEILING_MS)diff --git a/backend/global_search/tests/test_contract.py b/backend/global_search/tests/test_contract.pynew file mode 100644index 000000000..96c734b4c--- /dev/null+++ b/backend/global_search/tests/test_contract.py@@ -0,0 +1,175 @@+"""The published contract matches what the endpoint serves.++* Real responses validate against the committed JSON Schema.+* Each OSS type's context line and route params match values copied from+ the README's type table; the table's order, type, label and route key+ columns are checked against the registered providers.+* The endpoint is mounted on the tenant path, and refuses a URL organisation+ other than the session's.+"""++import json+from datetime import timedelta+from pathlib import Path+from unittest.mock import patch++import jsonschema+from django.db import connection+from account_v2.models import Organization+from django.test import TestCase+from django.urls import resolve+from permissions.roles import ResourceRole+from permissions.tests.base import CoOwnerOrgTestMixin, make_user+from tenant_account_v2.models import OrganizationMember++from global_search.constants import OSSSearchType as T+from global_search.oss_providers import AdapterProvider, OSS_PROVIDERS, WorkflowProvider+from global_search.providers import SearchProvider+from global_search.tests.base import OSS_BUILDERS, search+from global_search.views import GlobalSearchView++APP_DIR = Path(__file__).parents[1]+SCHEMA = json.loads((APP_DIR / "contract" / "search-response.schema.json").read_text())+README = (APP_DIR / "README.md").read_text()+++class SearchContractTests(CoOwnerOrgTestMixin, TestCase):+ def setUp(self) -> None:+ self._seed_org()++ def test_responses_validate_against_schema(self) -> None:+ for type_key, build in OSS_BUILDERS.items():+ item = build(self.org, self.owner, f"theta-{type_key}", "theta notes")+ item.memberships.create(user=self.owner, role="owner")+ response = search(self.admin, q="theta", limit=20)+ jsonschema.validate(response.data, SCHEMA)+ self.assertEqual(+ [g["type"] for g in response.data["groups"]],+ [p.type_key for p in sorted(OSS_PROVIDERS, key=lambda p: p.order)],+ )++ def test_context_and_route_params_per_type(self) -> None:+ items = {}+ for type_key, build in OSS_BUILDERS.items():+ items[type_key] = build(self.org, self.owner, f"theta-{type_key}")+ items[type_key].memberships.create(user=self.owner, role=ResourceRole.OWNER)+ response = search(self.admin, q="theta", limit=20)+ results = {g["type"]: g["results"][0] for g in response.data["groups"]}++ def parent(type_key: str) -> str:+ return items[type_key].workflow.workflow_name++ expected_context = {+ T.PROMPT_STUDIO_PROJECT: self.owner.email,+ T.WORKFLOW: self.owner.email,+ T.API_DEPLOYMENT: parent(T.API_DEPLOYMENT),+ T.ETL_PIPELINE: parent(T.ETL_PIPELINE),+ T.TASK_PIPELINE: parent(T.TASK_PIPELINE),+ T.ADAPTER: "LLM · openai",+ T.CONNECTOR: "minio",+ }+ for type_key, item in items.items():+ with self.subTest(type=type_key):+ result = results[type_key]+ self.assertEqual(result["context"], expected_context[type_key])+ expected_params = {"id": str(item.pk)}+ if type_key == T.ADAPTER:+ expected_params["adapter_type"] = "LLM"+ self.assertEqual(+ result["route"],+ {"key": f"{type_key}.detail", "params": expected_params},+ )++ def test_owner_context_skips_service_accounts_and_takes_the_earliest(self) -> None:+ workflow = OSS_BUILDERS[T.WORKFLOW](self.org, self.owner, "theta-owned")+ service = make_user("[email protected]", is_service_account=True)+ # The service account is the earliest owner, so only the exclusion+ # skips it; the co-owner is inserted before the owner, so only the+ # created_at ordering (not heap order) picks the owner over it.+ for user in (service, self.coowner, self.owner):+ workflow.memberships.create(user=user, role=ResourceRole.OWNER)+ base = workflow.memberships.order_by("created_at").first().created_at+ for user, minutes in ((service, -2), (self.owner, -1), (self.coowner, 0)):+ workflow.memberships.filter(user=user).update(+ created_at=base + timedelta(minutes=minutes)+ )+ (group,) = search(self.admin, q="theta", types=T.WORKFLOW).data["groups"]+ self.assertEqual(group["results"][0]["context"], self.owner.email)+ # Same rule as the list's "Owned By" column.+ self.assertEqual(workflow.owner_email(), self.owner.email)++ def test_a_db_error_in_one_provider_does_not_poison_the_others(self) -> None:+ # TestCase wraps each test in a transaction, as ATOMIC_REQUESTS does.+ class Broken(SearchProvider):+ type_key, label, order = "broken", "Broken", 1++ def search(self, user, query, limit):+ with connection.cursor() as cursor:+ cursor.execute("SELECT * FROM no_such_table_for_search")+ return []++ workflow = OSS_BUILDERS[T.WORKFLOW](self.org, self.owner, "theta-flow")+ with patch(+ "global_search.views.get_providers",+ return_value=[Broken(), WorkflowProvider()],+ ):+ response = search(self.admin, allow_failures=True, q="theta")+ self.assertEqual(response.status_code, 200)+ self.assertEqual(response.data["failed_types"], ["broken"])+ self.assertEqual(+ [r["id"] for g in response.data["groups"] for r in g["results"]],+ [str(workflow.pk)],+ )++ def test_mounted_on_tenant_path(self) -> None:+ # Importing the urlconf touches the DB, hence a TestCase.+ # OrganizationMiddleware strips the org segment before routing, so+ # /api/v1/unstract/<org>/search/ resolves as this path.+ match = resolve("/api/v1/unstract/search/")+ self.assertIs(match.func.view_class, GlobalSearchView)++ def test_url_org_must_match_the_session_org(self) -> None:+ # Through the real middleware: results come from the session's org,+ # so a URL naming another org (even one the user belongs to) is+ # refused rather than answered from the wrong tenant.+ workflow = OSS_BUILDERS[T.WORKFLOW](self.org, self.owner, "theta-flow")+ other = Organization.objects.create(+ name="org-b", display_name="Org B", organization_id="org-b"+ )+ OrganizationMember.objects.create(organization=other, user=self.admin, role="admin")+ # The built-in session auth, so a distribution's auth plugin cannot+ # 401 the request first; the org check runs after either.+ no_plugin = patch(+ "account_v2.custom_auth_middleware.AuthenticationPluginRegistry"+ ".is_plugin_available",+ return_value=False,+ )+ no_plugin.start()+ self.addCleanup(no_plugin.stop)+ self.client.force_login(self.admin)+ session = self.client.session+ session["organization"] = self.org.organization_id+ session.save()++ ok = self.client.get(+ f"/api/v1/unstract/{self.org.organization_id}/search/", {"q": "theta"}+ )+ self.assertEqual(ok.status_code, 200)+ self.assertIn(+ str(workflow.pk),+ [r["id"] for g in ok.json()["groups"] for r in g["results"]],+ )+ denied = self.client.get(+ f"/api/v1/unstract/{other.organization_id}/search/", {"q": "theta"}+ )+ self.assertEqual(denied.status_code, 403)+++def test_readme_lists_every_oss_type():+ for provider_cls in OSS_PROVIDERS:+ provider = provider_cls()+ keys = f"`{provider.route_key}`"+ if isinstance(provider, AdapterProvider):+ keys += f" / `{provider.info_route_key}`"+ row = f"| {provider.order} | `{provider.type_key}` | {provider.label} | {keys} |"+ assert row in README, f"README type table is missing: {row}"diff --git a/backend/global_search/tests/test_ranking.py b/backend/global_search/tests/test_ranking.pynew file mode 100644index 000000000..5cfe71965--- /dev/null+++ b/backend/global_search/tests/test_ranking.py@@ -0,0 +1,92 @@+"""Ranking against the real database: prefix before substring, name before+description, and a deterministic order inside a tier.+"""++from datetime import timedelta++from django.test import TestCase+from django.utils import timezone+from permissions.tests.base import CoOwnerOrgTestMixin++from global_search.tests.base import DESCRIBED_TYPES, OSS_BUILDERS, search+++class SearchRankingTests(CoOwnerOrgTestMixin, TestCase):+ def setUp(self) -> None:+ self._seed_org()++ def _results(self, type_key: str, query: str, limit: int = 20) -> list[dict]:+ response = search(self.admin, q=query, types=type_key, limit=limit)+ groups = response.data["groups"]+ return groups[0]["results"] if groups else []++ def test_name_prefix_before_name_substring(self) -> None:+ for type_key, build in OSS_BUILDERS.items():+ with self.subTest(type=type_key):+ # Best tier created first: recency alone would order it last.+ prefix = build(self.org, self.owner, f"kappa-{type_key}")+ inner = build(self.org, self.owner, f"old-kappa-{type_key}")+ results = self._results(type_key, "kappa")+ self.assertEqual(+ [r["id"] for r in results], [str(prefix.pk), str(inner.pk)]+ )+ self.assertEqual(+ [r["match"] for r in results], ["name_prefix", "name"]+ )++ def test_name_before_description(self) -> None:+ for type_key in sorted(DESCRIBED_TYPES):+ build = OSS_BUILDERS[type_key]+ with self.subTest(type=type_key):+ # Best tier created first: recency alone would reverse it.+ name_inner = build(self.org, self.owner, f"x-lambda-{type_key}")+ desc_prefix = build(+ self.org, self.owner, f"d1-{type_key}", "lambda first"+ )+ desc_inner = build(+ self.org, self.owner, f"d2-{type_key}", "about lambda things"+ )+ results = self._results(type_key, "LAMBDA")+ self.assertEqual(+ [r["id"] for r in results],+ [str(name_inner.pk), str(desc_prefix.pk), str(desc_inner.pk)],+ )+ self.assertEqual(+ [r["match"] for r in results],+ ["name", "description_prefix", "description"],+ )++ def test_ties_go_to_most_recently_modified(self) -> None:+ build = OSS_BUILDERS["workflow"]+ older = build(self.org, self.owner, "sigma-one")+ newer = build(self.org, self.owner, "sigma-two")+ # bulk paths keep an explicit modified_at, see BaseModelQuerySet.+ type(older).objects.filter(pk=older.pk).update(+ modified_at=timezone.now() - timedelta(days=1)+ )+ results = self._results("workflow", "sigma")+ self.assertEqual([r["id"] for r in results], [str(newer.pk), str(older.pk)])++ def test_has_more_and_limit(self) -> None:+ build = OSS_BUILDERS["connector"]+ for i in range(3):+ build(self.org, self.owner, f"omega-{i}")+ response = search(self.admin, q="omega", types="connector", limit=2)+ (group,) = response.data["groups"]+ self.assertEqual(len(group["results"]), 2)+ self.assertTrue(group["has_more"])++ response = search(self.admin, q="omega", types="connector", limit=3)+ self.assertFalse(response.data["groups"][0]["has_more"])++ def test_wildcards_are_literal(self) -> None:+ build = OSS_BUILDERS["workflow"]+ build(self.org, self.owner, "plain-name")+ # Matches "_100" only if "_" were a single-character wildcard.+ build(self.org, self.owner, "rateX100")+ literal = build(self.org, self.owner, "rate_100%")+ response = search(self.admin, q="_100", types="workflow")+ self.assertEqual(+ [r["id"] for r in response.data["groups"][0]["results"]], [str(literal.pk)]+ )+ self.assertEqual(search(self.admin, q="%%", types="workflow").data["groups"], [])diff --git a/backend/global_search/tests/test_search_unit.py b/backend/global_search/tests/test_search_unit.pynew file mode 100644index 000000000..a1d5d607b--- /dev/null+++ b/backend/global_search/tests/test_search_unit.py@@ -0,0 +1,298 @@+"""DB-free tests: parameters, envelope shape, failure handling, registry and+in-memory ranking, with faked providers.+"""++import json+from pathlib import Path+from unittest.mock import MagicMock, patch++import jsonschema+import pytest+from account_v2.models import User+from utils.user_context import UserContext+from django.db import InterfaceError, OperationalError++from global_search.constants import MatchTier+from global_search.providers import SearchHit, SearchProvider+from global_search.ranking import match_tier, rank_items+from global_search.registry import get_providers, register_provider, unregister_provider+from global_search.tests.base import search++SCHEMA = json.loads(+ (Path(__file__).parents[1] / "contract" / "search-response.schema.json").read_text()+)+++class FakeProvider(SearchProvider):+ def __init__(self, type_key: str, order: int, names: list[str]):+ self.type_key = type_key+ self.label = type_key.title()+ self.order = order+ self.names = names++ def search(self, user, query, limit):+ ranked = rank_items(self.names, query, names=lambda n: [n], sort_key=lambda n: n)+ return [+ SearchHit(id=name, name=name, match=tier, context=None, route_params={"id": name})+ for name, tier in ranked[: limit + 1]+ ]++[email protected](autouse=True)+def no_organization_context():+ """A 5xx makes Django's error reporter import the URLconf. If an earlier+ test on the same worker left an organisation in UserContext, the+ org-scoped managers behind class-level querysets (e.g.+ ``pipeline_v2.views``) would look it up during that import -- a DB query+ these tests must not make.+ """+ previous = UserContext.get_organization_identifier()+ UserContext.set_organization_identifier(None)+ yield+ UserContext.set_organization_identifier(previous)+++def user() -> User:+ # Unsaved: IsAuthenticated only reads ``is_authenticated``.+ return User(username="[email protected]", email="[email protected]")+++def providers():+ fakes = [+ FakeProvider("beta", 20, ["alpha one", "alpha two", "one alpha"]),+ FakeProvider("alpha", 10, ["alpha"]),+ FakeProvider("empty", 30, ["nothing"]),+ ]+ with patch("global_search.views.get_providers", return_value=sorted(fakes, key=lambda p: p.order)):+ yield fakes+++def test_groups_in_rail_order_with_empty_groups_omitted(user, providers):+ data = search(user, q="alpha").data+ jsonschema.validate(data, SCHEMA)+ assert data["query"] == "alpha"+ assert [g["type"] for g in data["groups"]] == ["alpha", "beta"]+ beta = data["groups"][1]+ assert [r["name"] for r in beta["results"]] == ["alpha one", "alpha two", "one alpha"]+ assert [r["match"] for r in beta["results"]] == ["name_prefix", "name_prefix", "name"]+ assert beta["results"][0]["route"] == {"key": "beta.detail", "params": {"id": "alpha one"}}+++def test_a_failing_provider_drops_only_its_own_group(user, providers, caplog):+ class Broken(FakeProvider):+ def search(self, user, query, limit):+ raise RuntimeError("boom")++ with patch(+ "global_search.views.get_providers",+ return_value=[providers[1], Broken("broken", 15, []), providers[0]],+ ):+ response = search(user, allow_failures=True, q="alpha")+ assert response.status_code == 200+ assert [g["type"] for g in response.data["groups"]] == ["alpha", "beta"]+ assert response.data["failed_types"] == ["broken"]+ jsonschema.validate(response.data, SCHEMA)+ assert "Global search provider 'broken' failed" in caplog.text+++def _assert_error(response, status: int, code: str) -> None:+ assert response.status_code == status+ assert [e["code"] for e in response.data["errors"]] == [code]+++class _Broken(FakeProvider):+ def search(self, user, query, limit):+ raise RuntimeError("boom")+++class _DbError(FakeProvider):+ def search(self, user, query, limit):+ raise OperationalError("canceling statement due to statement timeout")+++def _connection(*, usable: bool = True, needs_rollback: bool = False) -> MagicMock:+ """The view's DB connection, in autocommit (the default) with a chosen+ health, so the escalation rules run without a real failure."""+ conn = MagicMock(in_atomic_block=False, needs_rollback=needs_rollback)+ conn.is_usable.return_value = usable+ return conn+++def _run(user, providers_list, conn=None, **params):+ with (+ patch("global_search.views.connection", conn or _connection()),+ patch("global_search.views.get_providers", return_value=providers_list),+ ):+ return search(user, allow_failures=True, **{"q": "alpha", **params})+++def test_every_type_failing_with_a_bug_is_a_500(user):+ response = _run(user, [_Broken("a", 1, []), _Broken("b", 2, [])])+ _assert_error(response, 500, "search_failed")+++def test_every_type_failing_on_the_database_is_a_retryable_503(user):+ response = _run(user, [_DbError("a", 1, []), _DbError("b", 2, [])])+ _assert_error(response, 503, "search_unavailable")+++def test_a_failure_that_breaks_the_transaction_fails_the_request(user, providers):+ # A failed savepoint rollback leaves needs_rollback set; later types+ # would all fail, so the error must surface rather than be logged per type.+ conn = _connection(needs_rollback=True)+ response = _run(user, [_Broken("a", 1, []), providers[0]], conn)+ _assert_error(response, 503, "search_unavailable")+++def test_a_lost_connection_fails_the_request_even_if_others_answered(+ user, providers, caplog+):+ # In autocommit nothing marks the connection broken; is_usable() does.+ conn = _connection(usable=False)+ response = _run(user, [providers[1], _DbError("lost", 15, [])], conn)+ _assert_error(response, 503, "search_unavailable")+ # The 503 itself names no type; the log must.+ assert "Global search provider 'lost' failed" in caplog.text+++def test_a_failed_connect_fails_the_request_without_probing_it(user, providers):+ # Connecting failed, so no connection is open: every later type would+ # dial the dead database too. is_usable() needs an open connection.+ conn = _connection()+ conn.connection = None+ response = _run(user, [_DbError("down", 5, []), providers[1]], conn)+ _assert_error(response, 503, "search_unavailable")+ conn.is_usable.assert_not_called()+++def test_a_closed_connection_interface_error_fails_the_request(user, providers):+ # InterfaceError ("connection already closed") is not a DatabaseError in+ # Django, but it is the canonical unusable-connection error.+ class Closed(FakeProvider):+ def search(self, user, query, limit):+ raise InterfaceError("connection already closed")++ later = MagicMock(wraps=providers[0])+ later.type_key, later.label, later.order = "alpha", "Alpha", 10+ conn = _connection(usable=False)+ response = _run(user, [Closed("closed", 5, []), later], conn)+ _assert_error(response, 503, "search_unavailable")+ later.search.assert_not_called()+++def test_a_bug_alongside_database_errors_is_a_500(user):+ # Only an all-retryable failure is worth retrying.+ response = _run(user, [_DbError("a", 1, []), _Broken("b", 2, [])])+ _assert_error(response, 500, "search_failed")+++def test_a_db_error_on_a_healthy_connection_drops_only_its_type(user, providers):+ # A statement timeout, lock timeout or deadlock leaves the connection+ # usable: the other types still answer.+ response = _run(user, [providers[1], _DbError("slow", 15, [])])+ assert response.status_code == 200+ assert [g["type"] for g in response.data["groups"]] == ["alpha"]+ assert response.data["failed_types"] == ["slow"]+++def test_limit_and_has_more(user, providers):+ beta = search(user, q="alpha", types="beta", limit=2).data["groups"][0]+ assert len(beta["results"]) == 2+ assert beta["has_more"] is True+ assert search(user, q="alpha", types="beta", limit=3).data["groups"][0]["has_more"] is False+++def test_types_filter_ignores_unknown_keys(user, providers):+ data = search(user, q="alpha", types=" alpha , agentic_project,,").data+ assert [g["type"] for g in data["groups"]] == ["alpha"]++[email protected]("q", ["", "a", " a ", " "])+def test_short_query_returns_no_groups(user, providers, q):+ response = search(user, q=q)+ assert response.status_code == 200+ assert response.data == {"query": q.strip(), "groups": [], "failed_types": []}+++def test_missing_query_returns_no_groups(user, providers):+ response = search(user)+ assert response.status_code == 200+ assert response.data["groups"] == []+++ "params",+ [{"limit": 0}, {"limit": 21}, {"limit": "x"}, {"q": "a" * 101}],+)+def test_invalid_parameters_are_400(user, providers, params):+ response = search(user, **{"q": "alpha", **params})+ assert response.status_code == 400+++def test_query_at_max_length_is_accepted(user, providers):+ assert search(user, q="a" * 100).status_code == 200+++def test_anonymous_is_rejected():+ from django.contrib.auth.models import AnonymousUser++ assert search(AnonymousUser(), q="alpha").status_code in (401, 403)+++class TestRegistry:+ def test_sorted_by_order_and_duplicate_class_is_idempotent(self):+ first, second = FakeProvider("zz_b", 999_002, []), FakeProvider("zz_a", 999_001, [])+ try:+ register_provider(first)+ register_provider(second)+ register_provider(FakeProvider("zz_a", 999_001, []))+ keys = [p.type_key for p in get_providers()]+ assert keys[-2:] == ["zz_a", "zz_b"]+ finally:+ unregister_provider("zz_a")+ unregister_provider("zz_b")++ @pytest.mark.parametrize("bad_key", ["agenticProject", "1abc", "a-b", "", "a.b"])+ def test_type_key_must_match_the_schema_pattern(self, bad_key):+ with pytest.raises(ValueError):+ register_provider(FakeProvider(bad_key, 1, []))+ assert bad_key not in [p.type_key for p in get_providers()]++ def test_conflicting_class_for_a_key_raises(self):+ class Other(FakeProvider):+ pass++ register_provider(FakeProvider("zz_c", 999_003, []))+ try:+ with pytest.raises(ValueError):+ register_provider(Other("zz_c", 1, []))+ finally:+ unregister_provider("zz_c")+++ ("names", "description", "expected"),+ [+ (["Invoice parser"], None, MatchTier.NAME_PREFIX),+ (["My invoices"], None, MatchTier.NAME),+ (["other", "INVOICE api"], None, MatchTier.NAME_PREFIX),+ (["x"], "invoice first", MatchTier.DESCRIPTION_PREFIX),+ (["x"], "about Invoices", MatchTier.DESCRIPTION),+ (["x", None], None, None),+ ],+)+def test_match_tier(names, description, expected):+ assert match_tier("invoice", names, description) == expected+++def test_rank_items_orders_by_tier_then_sort_key():+ items = [("b", "zz invoice"), ("a", "invoice"), ("c", "invoice")]+ ranked = rank_items(items, "inv", names=lambda i: [i[1]], sort_key=lambda i: i[0])+ assert [i[0] for i, _ in ranked] == ["a", "c", "b"]+++def test_a_hit_must_carry_a_route_id():+ with pytest.raises(ValueError):+ SearchHit(id="x", name="x", match=MatchTier.NAME, route_params={})diff --git a/backend/global_search/tests/test_visibility.py b/backend/global_search/tests/test_visibility.pynew file mode 100644index 000000000..04bb45746--- /dev/null+++ b/backend/global_search/tests/test_visibility.py@@ -0,0 +1,170 @@+"""A user never receives a result they cannot open, per resource type.++Each type is checked against the same roles its list honours: owner,+co-owner, direct viewer, group share, org share, admin and service account+see it; an org member without access and another org's resources do not.+"""++from account_v2.models import Organization+from adapter_processor_v2.models import AdapterInstance+from django.test import TestCase+from permissions.permission import IsFrictionLessAdapter+from permissions.roles import ResourceRole+from permissions.tests.base import CoOwnerOrgTestMixin, make_user+from pipeline_v2.models import Pipeline+from rest_framework.request import Request+from rest_framework.test import APIRequestFactory+from tenant_account_v2.models import GroupMembership, OrganizationGroup, OrganizationMember+from tenant_account_v2.sharing_helpers import set_resource_share_groups+from utils.user_context import UserContext++from global_search.constants import OSSSearchType+from global_search.tests.base import (+ OSS_BUILDERS,+ _pipeline,+ build_adapter,+ result_ids,+ search,+)++QUERY = "zeta"+++class SearchVisibilityTests(CoOwnerOrgTestMixin, TestCase):+ def setUp(self) -> None:+ self._seed_org()+ self.group_member = make_user("[email protected]")+ OrganizationMember.objects.create(+ organization=self.org, user=self.group_member, role="user"+ )+ self.group = OrganizationGroup.objects.create(+ organization=self.org, name="Team", created_by=self.owner+ )+ GroupMembership.objects.create(group=self.group, user=self.group_member)+ self.service_account = make_user("[email protected]", is_service_account=True)++ def _private(self, type_key: str):+ item = OSS_BUILDERS[type_key](self.org, self.owner, f"{QUERY}-{type_key}")+ item.memberships.create(user=self.owner, role=ResourceRole.OWNER)+ item.memberships.create(user=self.coowner, role=ResourceRole.OWNER)+ item.memberships.create(user=self.viewer, role=ResourceRole.VIEWER)+ return item++ def _found(self, user, type_key: str, item) -> bool:+ response = search(user, q=QUERY, types=type_key, limit=20)+ self.assertEqual(response.status_code, 200)+ return str(item.pk) in result_ids(response, type_key)++ def test_each_type_follows_its_list_visibility(self) -> None:+ for type_key in OSS_BUILDERS:+ with self.subTest(type=type_key):+ item = self._private(type_key)++ for user in (self.owner, self.coowner, self.viewer, self.admin):+ self.assertTrue(self._found(user, type_key, item), user.email)+ self.assertTrue(self._found(self.service_account, type_key, item))+ for user in (self.outsider, self.group_member):+ self.assertFalse(self._found(user, type_key, item), user.email)++ set_resource_share_groups(item, [self.group.id])+ self.assertTrue(self._found(self.group_member, type_key, item))+ self.assertFalse(self._found(self.outsider, type_key, item))++ item.shared_to_org = True+ item.save()+ self.assertTrue(self._found(self.outsider, type_key, item))++ def test_other_orgs_resources_never_appear(self) -> None:+ other = Organization.objects.create(+ name="org-b", display_name="Org B", organization_id="org-b"+ )+ UserContext.set_organization_identifier(other.organization_id)+ foreign = {}+ for type_key, build in OSS_BUILDERS.items():+ item = build(other, self.stranger, f"{QUERY}-{type_key}")+ item.shared_to_org = True+ item.save()+ foreign[type_key] = str(item.pk)+ UserContext.set_organization_identifier(self.org.organization_id)++ for user in (self.admin, self.service_account, self.outsider):+ response = search(user, q=QUERY, limit=20)+ leaked = set(result_ids(response)) & set(foreign.values())+ self.assertEqual(leaked, set(), user.email)++ def test_frictionless_adapters_are_never_returned(self) -> None:+ # Listed so any member can delete one, but adapter retrieve refuses+ # them for everyone, admins included; search would only lead to a 403.+ adapter = build_adapter(+ self.org, self.owner, f"{QUERY}-frictionless", is_friction_less=True+ )+ adapter.memberships.create(user=self.owner, role=ResourceRole.OWNER)+ for user in (self.owner, self.admin, self.service_account, self.outsider):+ self.assertFalse(+ self._found(user, OSSSearchType.ADAPTER, adapter), user.email+ )++ def test_adapter_route_follows_who_may_edit_it(self) -> None:+ # adapter.detail is gated like adapter retrieve: owners, co-owners,+ # admins and service accounts. Anyone else who can list the adapter+ # opens the read-only adapter.info.+ adapter = self._private(OSSSearchType.ADAPTER)+ set_resource_share_groups(adapter, [self.group.id])+ adapter.shared_to_org = True+ adapter.save()+ expected = {+ self.owner: "adapter.detail",+ self.coowner: "adapter.detail",+ self.admin: "adapter.detail",+ self.service_account: "adapter.detail",+ self.viewer: "adapter.info",+ self.group_member: "adapter.info",+ self.outsider: "adapter.info",+ }+ for user, route_key in expected.items():+ with self.subTest(user=user.email):+ response = search(+ user, q=QUERY, types=OSSSearchType.ADAPTER, limit=20+ )+ (result,) = response.data["groups"][0]["results"]+ self.assertEqual(+ result["route"],+ {+ "key": route_key,+ "params": {+ "id": str(adapter.pk),+ "adapter_type": adapter.adapter_type,+ },+ },+ )+ # Tied to the real gate, so the two rules cannot drift apart.+ self.assertEqual(+ route_key == "adapter.detail", self._may_retrieve(user, adapter)+ )++ @staticmethod+ def _may_retrieve(user, adapter) -> bool:+ request = Request(APIRequestFactory().get("/"))+ request.user = user+ return IsFrictionLessAdapter().has_object_permission(+ request, None, AdapterInstance.objects.get(pk=adapter.pk)+ )++ def test_no_op_adapters_are_never_returned(self) -> None:+ adapter = build_adapter(+ self.org,+ self.owner,+ f"{QUERY}-noop",+ adapter_id="noOpLlm|f673a5a2-90f9-40f5-94c0-9fbc663b7553",+ shared_to_org=True,+ )+ self.assertFalse(self._found(self.admin, OSSSearchType.ADAPTER, adapter))++ def test_default_and_app_pipelines_are_never_returned(self) -> None:+ for pipeline_type in (Pipeline.PipelineType.DEFAULT, Pipeline.PipelineType.APP):+ with self.subTest(pipeline_type=pipeline_type.value):+ pipeline = _pipeline(pipeline_type)(+ self.org, self.owner, f"{QUERY}-{pipeline_type}"+ )+ response = search(self.admin, q=QUERY, limit=20)+ self.assertNotIn(str(pipeline.pk), result_ids(response))diff --git a/backend/global_search/urls.py b/backend/global_search/urls.pynew file mode 100644index 000000000..762a0eea0--- /dev/null+++ b/backend/global_search/urls.py@@ -0,0 +1,7 @@+from django.urls import path++from global_search.views import GlobalSearchView++urlpatterns = [+ path("search/", GlobalSearchView.as_view(), name="global_search"),+]diff --git a/backend/global_search/views.py b/backend/global_search/views.pynew file mode 100644index 000000000..57c632fde--- /dev/null+++ b/backend/global_search/views.py@@ -0,0 +1,156 @@+import logging+from contextlib import nullcontext+from typing import Any++from django.db import (+ Error,+ InterfaceError,+ OperationalError,+ connection,+ transaction,+)+from drf_spectacular.utils import OpenApiResponse, extend_schema+from rest_framework.permissions import IsAuthenticated+from rest_framework.request import Request+from rest_framework.response import Response+from rest_framework.views import APIView++from global_search.constants import SearchParams+from global_search.exceptions import SearchFailed, SearchUnavailable+from global_search.providers import SearchProvider+from global_search.registry import get_providers+from global_search.serializers import SearchQuerySerializer, SearchResponseSerializer++logger = logging.getLogger(__name__)++# Database errors a retry may clear (unreachable, timed out, locked,+# deadlocked), as opposed to a bug.+_RETRYABLE = (InterfaceError, OperationalError)+++def _connection_unusable(error: Exception) -> bool:+ """Whether later types would fail too: a savepoint that could not be+ rolled back (ATOMIC_REQUESTS), or a database error after which the+ connection no longer answers. A statement timeout, lock timeout or+ deadlock is an OperationalError on a healthy connection, so the class+ alone does not decide; ``is_usable()`` runs only on this error path.+ """+ if connection.needs_rollback:+ return True+ # django.db.Error, not DatabaseError: InterfaceError ("connection+ # already closed") derives from Error only.+ if not isinstance(error, Error):+ return False+ if connection.connection is None:+ # No open connection after a connection-class error: connecting+ # failed, so every later type would dial a dead database too.+ return isinstance(error, _RETRYABLE)+ return not connection.is_usable()+++def _group(provider: SearchProvider, user: Any, query: str, limit: int) -> dict | None:+ hits = provider.search(user, query, limit)+ if not hits:+ return None+ return {+ "type": provider.type_key,+ "label": provider.label,+ "has_more": len(hits) > limit,+ "results": [+ {+ "type": provider.type_key,+ "id": hit.id,+ "name": hit.name,+ "context": hit.context,+ "match": hit.match.label,+ "route": {+ "key": hit.route_key or provider.route_key,+ "params": hit.route_params,+ },+ }+ for hit in hits[:limit]+ ],+ }+++class GlobalSearchView(APIView):+ """Search every resource type the caller can open, grouped by type."""++ permission_classes = [IsAuthenticated]++ @extend_schema(+ parameters=[SearchQuerySerializer],+ responses={+ 200: SearchResponseSerializer,+ 400: OpenApiResponse(description="Invalid parameters (validation_error)."),+ 500: OpenApiResponse(+ description=(+ "search_failed: every searched type failed, not all with "+ "retryable database errors (a bug). Retrying will not help."+ )+ ),+ 503: OpenApiResponse(+ description=(+ "search_unavailable: every searched type failed with a "+ "database error a retry may clear (unreachable, timed out, "+ "locked), or a failure left the connection unusable. "+ "Retry; never show as 'no results'."+ )+ ),+ },+ description=(+ "Groups come back in rail order with empty groups omitted. Each "+ "type applies its own list's visibility rule. See "+ "backend/global_search/README.md for the type vocabulary and the "+ "response-time budget."+ ),+ )+ def get(self, request: Request) -> Response:+ params = SearchQuerySerializer(data=request.query_params)+ params.is_valid(raise_exception=True)+ query: str = params.validated_data["q"]+ wanted: list[str] = params.validated_data["types"]+ limit: int = params.validated_data["limit"]++ groups = []+ failed_types: list[str] = []+ attempted = 0+ errors: list[Exception] = []+ if len(query) >= SearchParams.MIN_QUERY_LENGTH:+ for provider in get_providers():+ if wanted and provider.type_key not in wanted:+ continue+ attempted += 1+ # Under ATOMIC_REQUESTS a failed query aborts the request's+ # transaction; a savepoint keeps it from poisoning the next type.+ guard = (+ transaction.atomic() if connection.in_atomic_block else nullcontext()+ )+ try:+ with guard:+ group = _group(provider, request.user, query, limit)+ except Exception as error:+ logger.exception(+ "Global search provider '%s' failed", provider.type_key+ )+ # Every later type would fail too: fail the request+ # instead of blaming healthy types. The cause is logged+ # above, so it is not chained into the handler's log.+ if _connection_unusable(error):+ raise SearchUnavailable() from None+ # Providers can come from other apps (cloud); one failing+ # type drops its own group and is named in failed_types.+ failed_types.append(provider.type_key)+ errors.append(error)+ continue+ if group:+ groups.append(group)++ # Every type failing is never "no matches". Each cause is logged+ # above; only database errors a retry may clear are worth retrying.+ if errors and len(errors) == attempted:+ if all(isinstance(e, _RETRYABLE) for e in errors):+ raise SearchUnavailable() from None+ raise SearchFailed() from None++ return Response({"query": query, "groups": groups, "failed_types": failed_types})