Skip to content

Build AI methods with your coding agent, then run them anywhere: as an MCP for chatbots, as a webapp for people, or via API for your software.

Merge SearchSourceContent into DocumentContent (#768)

Louis Choquelcc408ed4b332
diff --git a/docs/building-methods/concepts/native-concepts.md b/docs/building-methods/concepts/native-concepts.md
index 331adb396..3effe97a9 100644
--- a/docs/building-methods/concepts/native-concepts.md
+++ b/docs/building-methods/concepts/native-concepts.md
@@ -27,7 +27,7 @@ Here are all the native concepts you can use out of the box:
|-----------------|-------------|---------------------|
| `Text` | A text | `TextContent` |
| `Image` | An image | `ImageContent` |
-| `Document` | A document (PDF, DOCX, PPTX) | `DocumentContent` |
+| `Document` | A document (PDF, DOCX, PPTX, web page) | `DocumentContent` |
| `TextAndImages` | Text with its associated images | `TextAndImagesContent` |
| `Number` | A number | `NumberContent` |
| `Page` | A document page with text, images, and optional page view | `PageContent` |
@@ -80,15 +80,23 @@ Represents a document (PDF, DOCX, PPTX, etc.):
```python
class DocumentContent(StuffContent):
url: str
+ public_url: str | None = None
mime_type: str | None = None
+ filename: str | None = None
+ title: str | None = None
+ snippet: str | None = None
```
**Fields:**
-- `url`: Location of the document file (file path or URL)
+- `url`: Location of the document file, storage URL, or web page URL
+- `public_url`: Optional public-facing URL (when `url` is a private/internal reference)
- `mime_type`: Optional MIME type of the document (e.g., "application/pdf", "application/vnd.openxmlformats-officedocument.wordprocessingml.document")
+- `filename`: Optional filename of the document
+- `title`: Optional title of the document or source
+- `snippet`: Optional text snippet or excerpt from the document
-**Use for:** Contracts, invoices, reports, presentations, any document file.
+**Use for:** Contracts, invoices, reports, presentations, web pages, search source citations, any document file.
### NumberContent
@@ -188,16 +196,13 @@ Represents the result of a web search query. Produced by `PipeSearch`:
```python
class SearchResultContent(StuffContent):
answer: str
- sources: list[SearchSourceContent] = []
+ sources: list[DocumentContent] = []
```
**Fields:**
- `answer`: The synthesized answer text from the search
-- `sources`: A list of source citations, each containing:
- - `name` (str): Source name/title
- - `url` (str): Source URL
- - `snippet` (str | None): Relevant excerpt from the source
+- `sources`: A list of `DocumentContent` source citations (each with `url`, `title`, and `snippet`)
**Use for:** Web search results, research findings, information retrieval with citations.
diff --git a/docs/building-methods/pipes/pipe-operators/PipeExtract.md b/docs/building-methods/pipes/pipe-operators/PipeExtract.md
index d56309d51..7994536a0 100644
--- a/docs/building-methods/pipes/pipe-operators/PipeExtract.md
+++ b/docs/building-methods/pipes/pipe-operators/PipeExtract.md
@@ -1,14 +1,14 @@
---
-description: "PipeExtract performs OCR on images and PDFs to extract text and embedded visuals. Turn documents into structured data for your AI methods."
+description: "PipeExtract extracts structured content from documents (PDF, images, web pages). Turn documents into structured data for your AI methods."
---
# PipeExtract
-The `PipeExtract` operator performs Optical Character Recognition (OCR) on images and PDF documents. It extracts text, embedded images, and provides full-page renderings.
+The `PipeExtract` operator extracts structured content from documents. For PDFs and images, it performs OCR to extract text, embedded images, and full-page renderings. For web pages, it fetches and extracts page content.
## How it works
-`PipeExtract` takes a single input, which must be a `Document` or an `Image` (or a concept that refines one of them). It processes the input and produces a list of `PageContent` objects. Each `PageContent` object encapsulates all the information extracted from one page.
+`PipeExtract` takes a single input, which must be a `Document` or an `Image` (or a concept that refines one of them). The document URL can be a file path, a storage URL, or a web page URL. It processes the input and produces a list of `PageContent` objects. Each `PageContent` object encapsulates all the information extracted from one page.
The output is always a list. If the input is a single image, the output still contains one `Page` item.
@@ -25,30 +25,37 @@ The `PageContent` object has the following structure:
`PipeExtract` is configured in your pipeline's `.mthds` file.
-### OCR Models and Backend System
+### Extraction Models and Backend System
-PipeExtract uses the unified inference backend system to manage OCR models. This means you can:
+PipeExtract uses the unified inference backend system to manage extraction models. This means you can:
-- Use different OCR providers (Mistral OCR, local PDF extraction via internal backend, etc.)
-- Configure OCR models through the same backend system as LLMs and image generation models
-- Use OCR presets for consistent configurations across your pipelines
-- Route OCR requests to different backends based on your routing profile
+- Use different extraction providers (Mistral OCR, Linkup Fetch, local PDF extraction via internal backend, etc.)
+- Configure extraction models through the same backend system as LLMs and image generation models
+- Use extraction presets for consistent configurations across your pipelines
+- Route extraction requests to different backends based on your routing profile
-Common OCR model handles:
+Common extraction model handles:
- `mistral-ocr`: Mistral's OCR model for high-quality text and image extraction
- `pypdfium2-extract-pdf`: Local PDF text extraction (no API calls required)
+- `linkup-fetch`: Web page content extraction from URLs
-OCR presets are defined in your model deck configuration and can include parameters like `max_nb_images` and `image_min_size`.
+Common model aliases:
+
+- `@default-extract-web-page`: Default for web page extraction (Linkup Fetch)
+- `@default-extract-document`: Default for document extraction (Azure Document Intelligence)
+- `@default-text-from-pdf`: Fast local PDF text extraction (pypdfium2)
+
+Extraction presets are defined in your model deck configuration and can include parameters like `max_nb_images` and `image_min_size`.
### MTHDS Parameters
| Parameter | Type | Description | Required |
| --------------------------- | ------- | ---------------------------------------------------------------------------------------------------------------------------------------- | -------- |
| `type` | string | The type of the pipe: `PipeExtract` | Yes |
-| `description` | string | A description of the OCR operation. | Yes |
-| `inputs` | Fixed | The value must be of concept `Document` or `Image` (or a concept that refines one of them). | Yes |
-| `output` | string | The output concept produced by the OCR operation. Use `Page[]`. | Yes |
+| `description` | string | A description of the extraction operation. | Yes |
+| `inputs` | Fixed | The value must be of concept `Document` or `Image` (or a concept that refines one of them). For web page extraction, the Document holds a web URL. | Yes |
+| `output` | string | The output concept produced by the extraction operation. Use `Page[]`. | Yes |
| `max_page_images` | integer or null | Maximum number of images to extract from pages: `null` (or omit) for unlimited, `0` for no images, or a positive integer to limit. Defaults to the value in your Extract model preset. | No |
| `page_views` | boolean | If `true`, a high-fidelity image of each page will be included in the `page_view` field. Defaults to `false`. | No |
| `page_views_dpi` | integer | The resolution (in Dots Per Inch) for the generated page views when processing a PDF. Defaults to `150`. | No |
@@ -80,6 +87,21 @@ The output of `PipeExtract` must be `Page[]`.
To use this pipe, first load a PDF into the `ScannedDocument` concept. After the pipe runs, the output contains a list of `PageContent` objects, where each object has the extracted text and a 200 DPI image of the corresponding page.
+### Example: Extracting a Web Page
+
+This example defines a pipe that extracts content from a web page URL.
+
+```toml
+[pipe.extract_web_article]
+type = "PipeExtract"
+description = "Extract content from a web page"
+inputs = { article_url = "Document" }
+output = "Page[]"
+model = "@default-extract-web-page"
+```
+
+Pass a web URL as the `document_uri` when running the pipe. PipeExtract fetches the page and extracts its content into `Page` objects, following the same pattern as document extraction.
+
## Related Documentation
- [Generic Document Extraction Example](../../../cookbook/extract-generic.md) - Extract markdown from complex PDFs using vision
diff --git a/docs/building-methods/pipes/pipe-operators/PipeSearch.md b/docs/building-methods/pipes/pipe-operators/PipeSearch.md
index f0a85758a..b4edb8e62 100644
--- a/docs/building-methods/pipes/pipe-operators/PipeSearch.md
+++ b/docs/building-methods/pipes/pipe-operators/PipeSearch.md
@@ -13,7 +13,7 @@ The `PipeSearch` operator searches the web using a configurable search provider
The output is a `SearchResult` (or a concept that refines it), which contains:
- `answer`: The synthesized answer text from the search
-- `sources`: A list of sources, each with a `name`, `url`, and optional `snippet`
+- `sources`: A list of sources, each with a `title`, `url`, and optional `snippet`
## Configuration
@@ -106,7 +106,7 @@ from_date = "2026-01-01"
include_domains = ["reuters.com", "apnews.com", "bbc.com"]
```
-When the output concept is compatible with `SearchResult`, the output contains the synthesized `answer` and a list of `sources` with their names, URLs, and snippets. PipeSearch can also feed a structured output concept when your method needs a more specialized downstream shape.
+When the output concept is compatible with `SearchResult`, the output contains the synthesized `answer` and a list of `sources` with their titles, URLs, and snippets. PipeSearch can also feed a structured output concept when your method needs a more specialized downstream shape.
## Related Documentation
diff --git a/docs/features/document-extraction.md b/docs/features/document-extraction.md
index dfc1d6285..e64f04aa6 100644
--- a/docs/features/document-extraction.md
+++ b/docs/features/document-extraction.md
@@ -1,17 +1,17 @@
---
title: "Document Extraction"
-description: "Multi-provider OCR and document processing with Mistral OCR, Azure Document Intelligence, docling, and Deepseek-OCR. PDF processing, layout analysis, and table recognition."
+description: "Multi-provider OCR and document processing with Mistral OCR, Azure Document Intelligence, docling, Deepseek-OCR, and Linkup Fetch. PDF processing, web page extraction, layout analysis, and table recognition."
---
# Document Extraction
-Multi-provider OCR and document processing with a unified interface.
+Multi-provider OCR, document processing, and web content extraction with a unified interface.
## Overview
From simple text extraction to advanced document understanding — Pipelex handles it all. Basic PDF text extraction works out of the box (via pypdfium2), but real documents demand more: OCR for scanned pages, layout analysis for complex structures, image extraction, and VLM-powered understanding.
-Unlike LLM APIs (partly standardized around OpenAI's completions API), the OCR landscape is fragmented. Pipelex solves this with a unified interface: swap providers by changing your PipeExtract config, no code changes required.
+Unlike LLM APIs (partly standardized around OpenAI's completions API), the OCR landscape is fragmented. Pipelex solves this with a unified interface: swap providers by changing your PipeExtract config, no code changes required. For web pages, Linkup Fetch extracts content directly from URLs using the same PipeExtract pattern.
## Supported Providers
@@ -22,6 +22,7 @@ Unlike LLM APIs (partly standardized around OpenAI's completions API), the OCR l
| **docling** | Local SDK | IBM's open-source extraction library with local CPU processing and optional GPU acceleration |
| **Azure Document Intelligence** | Gateway | Enterprise-grade OCR with high accuracy for complex layouts, tables, and handwriting |
| **Deepseek-OCR** | Gateway | Open-source model optimized for markdown extraction from images |
+| **Linkup Fetch** | Cloud API | Web page content extraction — fetches and extracts text from web URLs |
## Key Capabilities
@@ -31,10 +32,11 @@ Unlike LLM APIs (partly standardized around OpenAI's completions API), the OCR l
- **Table recognition** — Automatic table detection and extraction
- **Handwriting support** — Via providers that support handwriting recognition (e.g., Azure Document Intelligence)
- **Multi-page processing** — Batch processing of document pages with per-page results
+- **Web page extraction** — Fetch and extract content from web page URLs via Linkup Fetch
## Documents in LLM Prompts
-Include PDFs directly in your prompts using `@variable` syntax. PipeLLM automatically handles document rendering — single documents, multiple documents, and mixed content combining text, images, and PDFs are all supported.
+Include PDFs directly in your prompts using `@variable` syntax. PipeLLM automatically handles document rendering — single documents, multiple documents, and mixed content combining text, images, and PDFs are all supported. Web page content extracted via PipeExtract follows the same pattern.
## Related Documentation
diff --git a/pipelex/cli/agent_cli/commands/pipe_cmd.py b/pipelex/cli/agent_cli/commands/pipe_cmd.py
index b4f233dcd..c25c110b2 100644
--- a/pipelex/cli/agent_cli/commands/pipe_cmd.py
+++ b/pipelex/cli/agent_cli/commands/pipe_cmd.py
@@ -54,6 +54,7 @@ EXTRACT_TALENT_TO_MODEL: dict[str, str] = {
"pdf-basic-text-extractor": "@default-text-from-pdf",
"image-text-extractor": "@default-extract-image",
"full-document-extractor": "@default-extract-document",
+ "web-page-extractor": "@default-extract-web-page",
}
# Reverse mappings: preset name → talent name (for fuzzy resolution)
diff --git a/pipelex/core/registry_models.py b/pipelex/core/registry_models.py
index 6b6f88baa..50b7fe5e1 100644
--- a/pipelex/core/registry_models.py
+++ b/pipelex/core/registry_models.py
@@ -10,7 +10,7 @@ from pipelex.core.stuffs.json_content import JSONContent
from pipelex.core.stuffs.list_content import ListContent
from pipelex.core.stuffs.number_content import NumberContent
from pipelex.core.stuffs.page_content import PageContent
-from pipelex.core.stuffs.search_result_content import SearchResultContent, SearchSourceContent
+from pipelex.core.stuffs.search_result_content import SearchResultContent
from pipelex.core.stuffs.structured_content import StructuredContent
from pipelex.core.stuffs.stuff import Stuff
from pipelex.core.stuffs.stuff_content import StuffContent
@@ -88,7 +88,6 @@ class CoreRegistryModels(RegistryModels):
PageContent,
JSONContent,
SearchResultContent,
- SearchSourceContent,
]
EXPERIMENTAL: ClassVar[list[ModelType]] = [
diff --git a/pipelex/core/stuffs/document_content.py b/pipelex/core/stuffs/document_content.py
index 74543abf4..bd86f002f 100644
--- a/pipelex/core/stuffs/document_content.py
+++ b/pipelex/core/stuffs/document_content.py
@@ -1,10 +1,12 @@
from pydantic import Field, model_validator
+from rich.text import Text
from typing_extensions import override
from pipelex.cogt.templating.template_category import TemplateCategory
from pipelex.core.stuffs.stuff_content import StuffContent
from pipelex.tools.jinja2.jinja2_rendering import render_jinja2_sync
from pipelex.tools.misc.http_utils import validate_url_resource_exists
+from pipelex.tools.misc.pretty import PrettyPrintable
from pipelex.tools.uri.uri_resolver import extract_filename_from_uri, resolve_uri
from pipelex.types import Self
@@ -15,6 +17,8 @@ class DocumentContent(StuffContent):
public_url: str | None = Field(default=None, description="The public HTTPS URL of the document")
mime_type: str | None = Field(default=None, description="The MIME type of the document")
filename: str | None = Field(default=None, description="The original filename of the document")
+ title: str | None = Field(default=None, description="The title of the document or source")
+ snippet: str | None = Field(default=None, description="A text snippet or excerpt from the document")
@model_validator(mode="after")
def _auto_populate_filename(self) -> Self:
@@ -35,27 +39,57 @@ class DocumentContent(StuffContent):
@property
@override
def short_desc(self) -> str:
+ if self.title:
+ return f"document: {self.title}"
url_desc = resolve_uri(self.url).kind.desc
return f"{url_desc} of a document"
@override
def rendered_plain(self) -> str:
- return self.url
+ parts: list[str] = []
+ if self.title:
+ parts.append(f"- {self.title}: {self.url}")
+ else:
+ parts.append(self.url)
+ if self.snippet:
+ parts.append(f" {self.snippet}")
+ return "\n".join(parts)
@override
def rendered_html(self) -> str:
+ display_text = self.title or self.public_url or self.url
# The |e filter escapes HTML special characters to prevent XSS attacks
template_source = '<a href="{{ url|e }}" class="msg-document">{{ display_text|e }}</a>'
+ if self.snippet:
+ template_source += "<br/><small>{{ snippet|e }}</small>"
+ context: dict[str, str] = {
+ "url": self.public_url or self.url,
+ "display_text": display_text,
+ }
+ if self.snippet:
+ context["snippet"] = self.snippet
return render_jinja2_sync(
template_source=template_source,
template_category=TemplateCategory.HTML,
- templating_context={
- "url": self.public_url or self.url,
- "display_text": self.public_url or self.url,
- },
+ templating_context=context,
)
@override
def rendered_markdown(self, level: int = 1, is_pretty: bool = False) -> str:
- display_text = self.public_url or self.url
- return f"[{display_text}]({self.url})"
+ display_text = self.title or self.public_url or self.url
+ result = f"[{display_text}]({self.url})"
+ if self.snippet:
+ result += f"\n {self.snippet}"
+ return result
+
+ @override
+ def rendered_pretty(self, title: str | None = None, depth: int = 0) -> PrettyPrintable:
+ source_text = Text()
+ if self.title:
+ source_text.append(self.title, style="bold")
+ source_text.append(f" ({self.url})", style="dim cyan")
+ else:
+ source_text.append(self.url, style="dim cyan")
+ if self.snippet:
+ source_text.append(f"\n {self.snippet}", style="dim italic")
+ return source_text
diff --git a/pipelex/core/stuffs/search_result_content.py b/pipelex/core/stuffs/search_result_content.py
index 1c27f8f7c..52d28e2ec 100644
--- a/pipelex/core/stuffs/search_result_content.py
+++ b/pipelex/core/stuffs/search_result_content.py
@@ -6,63 +6,17 @@ from rich.markdown import Markdown
from rich.text import Text
from typing_extensions import override
+from pipelex.core.stuffs.document_content import DocumentContent
from pipelex.core.stuffs.stuff_content import StuffContent
from pipelex.tools.misc.pretty import PrettyPrintable
from pipelex.tools.typing.pydantic_utils import empty_list_factory_of
-class SearchSourceContent(StuffContent):
- """Represents a single search source with name, URL, and optional snippet."""
-
- name: str
- url: str
- snippet: str | None = None
-
- @property
- @override
- def short_desc(self) -> str:
- return f"source: {self.name}"
-
- @override
- def rendered_plain(self) -> str:
- result = f"- {self.name}: {self.url}"
- if self.snippet:
- result += f"\n {self.snippet}"
- return result
-
- @override
- def rendered_html(self) -> str:
- name_escaped = html_module.escape(self.name)
- url_escaped = html_module.escape(self.url)
- result = f'<li><a href="{url_escaped}">{name_escaped}</a>'
- if self.snippet:
- snippet_escaped = html_module.escape(self.snippet)
- result += f"<br/><small>{snippet_escaped}</small>"
- result += "</li>"
- return result
-
- @override
- def rendered_markdown(self, level: int = 1, is_pretty: bool = False) -> str:
- result = f"- [{self.name}]({self.url})"
- if self.snippet:
- result += f"\n {self.snippet}"
- return result
-
- @override
- def rendered_pretty(self, title: str | None = None, depth: int = 0) -> PrettyPrintable:
- source_text = Text()
- source_text.append(self.name, style="bold")
- source_text.append(f" ({self.url})", style="dim cyan")
- if self.snippet:
- source_text.append(f"\n {self.snippet}", style="dim italic")
- return source_text
-
-
class SearchResultContent(StuffContent):
"""Represents the result of a search query with an answer and list of sources."""
answer: str
- sources: list[SearchSourceContent] = Field(default_factory=empty_list_factory_of(SearchSourceContent))
+ sources: list[DocumentContent] = Field(default_factory=empty_list_factory_of(DocumentContent))
@property
@override
@@ -83,7 +37,7 @@ class SearchResultContent(StuffContent):
answer_escaped = html_module.escape(self.answer)
result = f"<div><p>{answer_escaped}</p>"
if self.sources:
- source_items = "".join(source.rendered_html() for source in self.sources)
+ source_items = "".join(f"<li>{source.rendered_html()}</li>" for source in self.sources)
result += f"<h4>Sources</h4><ul>{source_items}</ul>"
result += "</div>"
return result
@@ -93,7 +47,7 @@ class SearchResultContent(StuffContent):
result = self.answer
if self.sources:
result += "\n\n**Sources:**\n\n"
- result += "\n".join(source.rendered_markdown(level=level, is_pretty=is_pretty) for source in self.sources)
+ result += "\n".join(f"- {source.rendered_markdown(level=level, is_pretty=is_pretty)}" for source in self.sources)
return result
@override
diff --git a/pipelex/plugins/gateway/gateway_search_worker.py b/pipelex/plugins/gateway/gateway_search_worker.py
index e2b5b38ef..4a3e2928e 100644
--- a/pipelex/plugins/gateway/gateway_search_worker.py
+++ b/pipelex/plugins/gateway/gateway_search_worker.py
@@ -15,7 +15,8 @@ from pipelex.cogt.search.search_job import SearchJob
from pipelex.cogt.search.search_worker_abstract import SearchWorkerAbstract
from pipelex.cogt.usage.token_category import NbTokensByCategoryDict, TokenCategory
from pipelex.config import get_config
-from pipelex.core.stuffs.search_result_content import SearchResultContent, SearchSourceContent
+from pipelex.core.stuffs.document_content import DocumentContent
+from pipelex.core.stuffs.search_result_content import SearchResultContent
from pipelex.plugins.gateway.gateway_deck import GatewayDeck
from pipelex.plugins.gateway.gateway_exceptions import GatewaySearchResponseError
from pipelex.plugins.gateway.gateway_factory import GatewayFactory
@@ -87,10 +88,12 @@ class GatewaySearchWorker(SearchWorkerAbstract):
result_dict: dict[str, Any] = json.loads(content_str)
sources = [
- SearchSourceContent(
- name=source["name"],
+ DocumentContent(
+ title=source["name"],
url=source["url"],
+ public_url=source["url"],
snippet=source["snippet"],
+ mime_type="text/html",
)
for source in result_dict.get("sources", [])
]
diff --git a/pipelex/plugins/linkup/linkup_search_worker.py b/pipelex/plugins/linkup/linkup_search_worker.py
index 438b8f575..e26ca680f 100644
--- a/pipelex/plugins/linkup/linkup_search_worker.py
+++ b/pipelex/plugins/linkup/linkup_search_worker.py
@@ -9,7 +9,8 @@ from pipelex.cogt.search.search_depth import SearchDepth
from pipelex.cogt.search.search_job import SearchJob
from pipelex.cogt.search.search_worker_abstract import SearchWorkerAbstract
from pipelex.cogt.usage.token_category import TokenCategory
-from pipelex.core.stuffs.search_result_content import SearchResultContent, SearchSourceContent
+from pipelex.core.stuffs.document_content import DocumentContent
+from pipelex.core.stuffs.search_result_content import SearchResultContent
from pipelex.hub import get_secrets_provider
from pipelex.reporting.reporting_protocol import ReportingProtocol
from pipelex.tools.typing.pydantic_utils import BaseModelTypeVar
@@ -56,13 +57,15 @@ class LinkupSearchWorker(SearchWorkerAbstract):
if search_tokens_usage := search_job.job_report.search_tokens_usage:
search_tokens_usage.nb_tokens_by_category = {TokenCategory.INPUT: 1_000_000, TokenCategory.OUTPUT: 1_000_000}
- sources: list[SearchSourceContent] = []
+ sources: list[DocumentContent] = []
for source in response.sources:
sources.append(
- SearchSourceContent(
- name=source.name,
+ DocumentContent(
+ title=source.name,
url=source.url,
+ public_url=source.url,
snippet=source.snippet,
+ mime_type="text/html",
)
)
diff --git a/tests/integration/pipelex/cogt/test_search.py b/tests/integration/pipelex/cogt/test_search.py
index c32240a1a..26a6ca154 100644
--- a/tests/integration/pipelex/cogt/test_search.py
+++ b/tests/integration/pipelex/cogt/test_search.py
@@ -45,7 +45,7 @@ class TestSearch:
assert result.answer, "Expected a non-empty answer"
assert len(result.sources) > 0, "Expected at least one source"
for source in result.sources:
- assert source.name, "Expected source to have a name"
+ assert source.title, "Expected source to have a title"
assert source.url, "Expected source to have a URL"
get_report_delegate().generate_report()
diff --git a/tests/unit/pipelex/builder/pipe/pipe_operator/pipe_extract/test_data.py b/tests/unit/pipelex/builder/pipe/pipe_operator/pipe_extract/test_data.py
index 5d9b45451..3dfd19c7b 100644
--- a/tests/unit/pipelex/builder/pipe/pipe_operator/pipe_extract/test_data.py
+++ b/tests/unit/pipelex/builder/pipe/pipe_operator/pipe_extract/test_data.py
@@ -33,7 +33,6 @@ class PipeExtractTestCases:
output="Page[]",
extract_talent=ExtractTalent.PDF_BASIC_TEXT_EXTRACTOR,
max_page_images=None,
- page_image_captions=True,
page_views=True,
),
PipeExtractBlueprint(
@@ -43,7 +42,7 @@ class PipeExtractTestCases:
output="Page[]",
model="@default-text-from-pdf",
max_page_images=None,
- page_image_captions=True,
+ page_image_captions=None,
page_views=True,
page_views_dpi=None,
),
diff --git a/tests/unit/pipelex/core/stuffs/document_content/test_data.py b/tests/unit/pipelex/core/stuffs/document_content/test_data.py
index 1f1581f0d..67e1991a2 100644
--- a/tests/unit/pipelex/core/stuffs/document_content/test_data.py
+++ b/tests/unit/pipelex/core/stuffs/document_content/test_data.py
@@ -15,6 +15,8 @@ class TestData:
"mime_type": None,
"public_url": None,
"filename": None,
+ "title": None,
+ "snippet": None,
}
# Expected outputs for smart_dump (with optional fields)
@@ -23,6 +25,8 @@ class TestData:
"mime_type": "application/pdf",
"public_url": "Report.pdf",
"filename": None,
+ "title": None,
+ "snippet": None,
}
# Expected outputs for render methods