Build AI methods with your coding agent, then run them anywhere: as an MCP for chatbots, as a webapp for people, or via API for your software.
Feature/consistency on search (#738)
Louis Choquel17ae3622c027
diff --git a/CHANGELOG.md b/CHANGELOG.mdindex 804e99d95..969561077 100644--- a/CHANGELOG.md+++ b/CHANGELOG.md@@ -4,8 +4,18 @@### Added-- **`pipelex login`** — new CLI command for browser-based Pipelex Gateway authentication. Starts a local HTTP server, opens the browser for OAuth login (GitHub or Google), and saves the Gateway API key to `~/.pipelex/.env`. The key never appears in terminal output. Includes a 120-second timeout with helpful fallback instructions.-- Added `app_cli_auth` URL to `URLs` class pointing to `https://app.pipelex.com/auth/cli`.+- **CLI Authentication**: `pipelex login` command that initiates a browser-based OAuth flow (GitHub/Google) to authenticate with the Pipelex Gateway and save the API key locally.+- **Search Filtering**: `from_date`, `to_date`, `include_domains`, and `exclude_domains` options on the `PipeSearch` operator, routed through a new `GatewaySearchWorker`.+- **Expanded Cost Reporting**: Token usage and cost tracking now covers Search, Fetch, Extraction, and Image Generation jobs, in addition to LLMs.+- **New Models**: Added `gpt-5.3-codex` (text, images, pdf inputs) and `nano-banana-2`.++### Changed++- **Search Configuration**: Renamed Linkup model IDs from `linkup/standard` to `linkup-standard` (and `deep` variant), simplified presets to `$standard` and `$deep`.+- **Default Pipeline Execution**: Changed from `in_memory` to `local`.+- **Default Extraction Model**: Changed from `mistral-document-ai-2505` to `azure-document-intelligence`; removed `mistral-document-ai-2505` from the supported Gateway models list.+- **Content Handling**: `TextAndImagesContent` now supports an optional `raw_html` field.+- **Dev Experience**: `Makefile` commands (`generate-mthds-schema`, `update-gateway-models`) run in quiet mode by default.## [v0.19.0] - 2026-03-02diff --git a/docs/home/6-build-reliable-ai-workflows/concepts/native-concepts.md b/docs/home/6-build-reliable-ai-workflows/concepts/native-concepts.mdindex 264255cb6..82a08c686 100644--- a/docs/home/6-build-reliable-ai-workflows/concepts/native-concepts.md+++ b/docs/home/6-build-reliable-ai-workflows/concepts/native-concepts.md@@ -103,8 +103,15 @@ Combines text with one or more images:class TextAndImagesContent(StuffContent):text: Optional[TextContent]images: Optional[List[ImageContent]]+ raw_html: Optional[str]```+**Fields:**++- `text`: The text content+- `images`: A list of images extracted from the content+- `raw_html`: The raw HTML of the fetched page, when requested via `include_raw_html`+**Use for:** Rich content combining text and visuals, extracted document content, reports with diagrams.### PageContentdiff --git a/docs/home/6-build-reliable-ai-workflows/pipes/pipe-operators/PipeSearch.md b/docs/home/6-build-reliable-ai-workflows/pipes/pipe-operators/PipeSearch.mdindex 5b16495f9..97eeb6470 100644--- a/docs/home/6-build-reliable-ai-workflows/pipes/pipe-operators/PipeSearch.md+++ b/docs/home/6-build-reliable-ai-workflows/pipes/pipe-operators/PipeSearch.md@@ -20,7 +20,7 @@ The output is a `SearchResult` (or a concept that refines it), which contains:PipeSearch uses the unified inference backend system to manage search providers. This means you can:- Use different search providers (e.g., Linkup)-- Configure search depth (standard vs deep) through presets+- Choose between standard and deep search models, with presets for each- Route search requests through the same backend system as LLMs and other operatorsCommon search presets:@@ -38,6 +38,12 @@ Common search presets:| `output` | string | The output concept produced by the search. Must be `SearchResult` or a concept that refines `SearchResult`. | Yes || `model` | string | The search model preset to use (e.g., `"$standard"`, `"$deep"`). Defaults to the model specified in the global config. | No || `prompt` | string | The search query. Can be a static string or reference input variables using `$` prefix (e.g., `"What is $topic?"`). | Yes |+| `include_images` | boolean | Whether to include images in search results. Overrides the preset setting. | No |+| `max_results` | integer | Maximum number of results to return. Overrides the preset setting. | No |+| `from_date` | string | Start date filter in YYYY-MM-DD format. Only return results from this date onwards. | No |+| `to_date` | string | End date filter in YYYY-MM-DD format. Only return results up to this date. | No |+| `include_domains` | list of strings | Restrict search to these domains only. | No |+| `exclude_domains` | list of strings | Exclude results from these domains. | No |### Example: Static search query@@ -80,4 +86,20 @@ model = "$deep"prompt = "What are the main details about $topic?"```+### Example: Filtering by date and domain++Use date filters and domain restrictions to narrow search results.++```toml+[pipe.search_recent_from_sources]+type = "PipeSearch"+description = "Search specific sources for recent news"+inputs = { topic = "Text" }+output = "SearchResult"+model = "$standard"+prompt = "What are the latest developments about $topic?"+from_date = "2026-01-01"+include_domains = ["reuters.com", "apnews.com", "bbc.com"]+```+The output of PipeSearch must be `SearchResult` or a concept that refines `SearchResult`. After execution, the output contains the synthesized `answer` and a list of `sources` with their names, URLs, and snippets.diff --git a/docs/home/6-build-reliable-ai-workflows/pipes/pipe-operators/index.md b/docs/home/6-build-reliable-ai-workflows/pipes/pipe-operators/index.mdindex 1c77ba2a8..28c17c495 100644--- a/docs/home/6-build-reliable-ai-workflows/pipes/pipe-operators/index.md+++ b/docs/home/6-build-reliable-ai-workflows/pipes/pipe-operators/index.md@@ -88,4 +88,4 @@ Searches the web and returns structured results with sources.- Web search via configurable providers- Structured results with answer and source citations- Dynamic prompt templates with `$variable` syntax-- Configurable search depth (standard vs deep)+- Standard and deep search modelsdiff --git a/docs/home/7-configuration/config-technical/inference-backend-config.md b/docs/home/7-configuration/config-technical/inference-backend-config.mdindex 9ac2f24f4..ad3ed6528 100644--- a/docs/home/7-configuration/config-technical/inference-backend-config.md+++ b/docs/home/7-configuration/config-technical/inference-backend-config.md@@ -65,12 +65,14 @@ All inference backend configurations are stored in the `.pipelex/inference/` dir│ ├── mistral.toml # Mistral models (LLMs, OCR)│ ├── vertexai.toml # Google Vertex AI models (LLMs)│ ├── fal.toml # FAL models (image generation)+ │ ├── linkup.toml # Linkup models (web search)│ ├── internal.toml # Internal/local models (OCR)│ └── ...└── deck/ # Model deck configurations├── 1_llm_deck.toml # LLM aliases & presets├── 2_img_gen_deck.toml # Image generation config├── 3_extract_deck.toml # Document extraction config+ ├── 4_search_deck.toml # Web search config├── x_custom_llm_deck.toml # Custom LLM waterfalls/overrides└── x_custom_extract_deck.toml # Custom extract waterfalls```@@ -251,6 +253,8 @@ GCP_LOCATION=GCP_CREDENTIALS_FILE_PATH=gcp_credentials.jsonFAL_API_KEY=++LINKUP_API_KEY=# ... (see .env.example for full list)```@@ -275,6 +279,10 @@ api_key = "${MISTRAL_API_KEY}"enabled = trueapi_key = "${FAL_API_KEY}"+[linkup]+enabled = true+api_key = "${LINKUP_API_KEY}"+[internal]enabled = true# No API key needed for internal/local processing@@ -513,6 +521,29 @@ model = "$gen-image-fast" # Uses fast image generation presetmodel = "$gen-image-high-quality" # Uses high quality image generation preset```+### Search Presets++Search presets combine a search model with result options. Defined in `.pipelex/inference/deck/4_search_deck.toml`:++```toml+[search.presets]+standard = { model = "linkup-standard", include_images = false, include_inline_citations = true }+deep = { model = "linkup-deep", include_images = false, include_inline_citations = true }+```++When using search presets in `.mthds` files, prefix them with `$`:++```toml+model = "$standard" # Standard web search+model = "$deep" # More thorough web search+```++Search presets support the following options:++- `model`: The search model to use (e.g., `linkup-standard`, `linkup-deep`)+- `include_images`: Whether to include images in search results+- `include_inline_citations`: Whether to include inline citations in the answer+### Default ChoicesSet default models for different types of AI operations:@@ -527,6 +558,9 @@ choice_default = "@default-extract-document"[img_gen]choice_default = "$gen-image"++[search]+choice_default = "@default-search"```Note the sigil prefixes: `@` for aliases and `$` for presets.diff --git a/pipelex/builder/pipe/pipe_search_spec.py b/pipelex/builder/pipe/pipe_search_spec.pyindex cc65ed109..b0f4810e9 100644--- a/pipelex/builder/pipe/pipe_search_spec.py+++ b/pipelex/builder/pipe/pipe_search_spec.py@@ -30,6 +30,12 @@ class PipeSearchSpec(PipeSpec):examples=list(SearchTalent),)prompt: str = Field(description="A finalized search prompt or prompt template: use `$` prefix for inline variables (e.g., `$topic`).")+ from_date: str | None = Field(+ default=None, description="Start date filter in ISO 8601 format (YYYY-MM-DD). Only return results from this date onwards."+ )+ to_date: str | None = Field(default=None, description="End date filter in ISO 8601 format (YYYY-MM-DD). Only return results up to this date.")+ include_domains: list[str] | None = Field(default=None, description="Restrict search to these domains only (e.g., ['reuters.com', 'bbc.com']).")+ exclude_domains: list[str] | None = Field(default=None, description="Exclude results from these domains.")@field_validator("search_talent", mode="before")@classmethod@@ -63,6 +69,21 @@ class PipeSearchSpec(PipeSpec):search_group.renderables.append(Text()) # Blank linesearch_group.renderables.append(prompt_panel)+ # Show filter fields when set+ filter_lines: list[str] = []+ if self.from_date is not None:+ filter_lines.append(f"From date: {self.from_date}")+ if self.to_date is not None:+ filter_lines.append(f"To date: {self.to_date}")+ if self.include_domains is not None:+ filter_lines.append(f"Include domains: {', '.join(self.include_domains)}")+ if self.exclude_domains is not None:+ filter_lines.append(f"Exclude domains: {', '.join(self.exclude_domains)}")+ if filter_lines:+ search_group.renderables.append(Text()) # Blank line+ for filter_line in filter_lines:+ search_group.renderables.append(Text.from_markup(f"[dim]{filter_line}[/dim]"))+return search_group@override@@ -82,8 +103,8 @@ class PipeSearchSpec(PipeSpec):model=search_choice,include_images=None,max_results=None,- from_date=None,- to_date=None,- include_domains=None,- exclude_domains=None,+ from_date=self.from_date,+ to_date=self.to_date,+ include_domains=self.include_domains,+ exclude_domains=self.exclude_domains,)diff --git a/pipelex/pipe_operators/search/pipe_search_blueprint.py b/pipelex/pipe_operators/search/pipe_search_blueprint.pyindex 80cfae88d..f2920e310 100644--- a/pipelex/pipe_operators/search/pipe_search_blueprint.py+++ b/pipelex/pipe_operators/search/pipe_search_blueprint.py@@ -1,5 +1,7 @@+from datetime import datefrom typing import Literal+from pydantic import field_validatorfrom typing_extensions import overridefrom pipelex.cogt.search.search_setting import SearchModelChoice@@ -24,6 +26,18 @@ class PipeSearchBlueprint(PipeBlueprint):include_domains: list[str] | None = Noneexclude_domains: list[str] | None = None+ @field_validator("from_date", "to_date", mode="before")+ @classmethod+ def validate_date_format(cls, date_value: str | None) -> str | None:+ if date_value is None:+ return date_value+ try:+ date.fromisoformat(date_value)+ except ValueError:+ msg = f"'{date_value}' is not a valid ISO 8601 date (expected YYYY-MM-DD)"+ raise ValueError(msg) from None+ return date_value+@overridedef validate_inputs(self):template_category = TemplateCategory.BASICdiff --git a/pipelex/plugins/gateway/gateway_search_worker.py b/pipelex/plugins/gateway/gateway_search_worker.pyindex e3399ef43..af9ce074b 100644--- a/pipelex/plugins/gateway/gateway_search_worker.py+++ b/pipelex/plugins/gateway/gateway_search_worker.py@@ -76,7 +76,7 @@ class GatewaySearchWorker(SearchWorkerAbstract):params = GatewaySearchRequestParams(query=query,- depth=SearchDepth(self.inference_model.model_id),+ depth=SearchDepth(self.inference_model.model_id.rsplit("/", 1)[-1]),include_images=search_setting.include_images,include_inline_citations=search_setting.include_inline_citations,max_results=search_setting.max_results,@@ -139,7 +139,7 @@ class GatewaySearchWorker(SearchWorkerAbstract):params = GatewaySearchRequestParams(query=query,- depth=SearchDepth(self.inference_model.model_id),+ depth=SearchDepth(self.inference_model.model_id.rsplit("/", 1)[-1]),include_images=search_setting.include_images,include_inline_citations=search_setting.include_inline_citations,max_results=search_setting.max_results,