Build AI methods with your coding agent, then run them anywhere: as an MCP for chatbots, as a webapp for people, or via API for your software.
Linkup Fetch as Extract model (#766)
Louis Choquelc5833b04f2b5
diff --git a/.claude/skills/add-model/SKILL.md b/.claude/skills/add-model/SKILL.mdindex 670ad2dad..d6b05d8fe 100644--- a/.claude/skills/add-model/SKILL.md+++ b/.claude/skills/add-model/SKILL.md@@ -23,9 +23,8 @@ and must be done in order to keep everything consistent.2. **Add to backend TOML(s)** — the model spec in each relevant backend config3. **Sync kit configs** — propagate `.pipelex/` changes into `pipelex/kit/configs/`4. **Add to test profile collections** — so the model is available for test selection-5. **Regenerate test fixtures** — refresh the generated model sets-6. **Run inference tests** — verify the model works end-to-end with real API calls-7. **Gateway (manual)** — the user handles this separately since it's remote config+5. **Run inference tests** — verify the model works end-to-end with real API calls (fixtures are auto-regenerated when `PROF=` is passed)+6. **Gateway (manual)** — the user handles this separately since it's remote config## Step 1: Gather model information@@ -206,20 +205,7 @@ If the model should be part of a named test profile (like `dev` or `coverage`),add it there too. But usually the collections are enough — profiles referencecollections via `@collection_name` or glob patterns like `gpt-*`.-## Step 5: Regenerate test fixtures--Run the fixture regeneration to update the generated model sets:--```bash-make rtm-```--This runs `pipelex-dev preprocess-test-models --generate-fixtures --profile dev`-and updates `tests/integration/pipelex/fixtures/_generated_model_sets.py`.--Verify the generated file includes the new model if the active profile selects it.--## Step 6: Run inference tests against the new model+## Step 5: Run inference tests against the new modelOnce the model is registered and fixtures are regenerated, help the user run theintegration tests to verify the model actually works end-to-end with real API@@ -255,26 +241,26 @@ search_models = []Adjust the `backends` and model list fields based on what type the model is(`llm_models`, `img_gen_models`, `extract_models`, or `search_models`).-### Regenerate fixtures and run tests for EACH backend+### Run tests for EACH backend-For each backend profile, regenerate fixtures and run the inference tests:+For each backend profile, run the inference tests. **No need to call `make rtm`+separately** — all inference test targets (`ti`, `tip`, `tl`, `te`, etc.)+automatically regenerate fixtures when `PROF=` is passed on the command line.```bash# Test on first backend-make rtm PROF=new_model_openai-make ti PROF=new_model_openai TEST=TestLLMInference+make test-inference-fast PROF=new_model_openai TEST=TestLLMInference# Test on second backend-make rtm PROF=new_model_azure-make ti PROF=new_model_azure TEST=TestLLMInference+make test-inference-fast PROF=new_model_azure TEST=TestLLMInference```Use the appropriate test class for the model type:-- **LLM models**: `make ti PROF=<profile> TEST=TestLLMInference`-- **Image gen models**: `make ti PROF=<profile> TEST=TestImageGeneration`-- **Extract models**: `make ti PROF=<profile> TEST=TestExtract`-- **Search models**: `make ti PROF=<profile> TEST=TestSearch`+- **LLM models**: `make test-inference-fast PROF=<profile> TEST=TestLLMInference`+- **Image gen models**: `make test-inference-fast PROF=<profile> TEST=TestImageGeneration`+- **Extract models**: `make test-inference-fast PROF=<profile> TEST=TestExtract`+- **Search models**: `make test-inference-fast PROF=<profile> TEST=TestSearch`The `PROF` parameter selects the test profile (which controls which models aretested), and `TEST` selects the test class or method to run.@@ -283,20 +269,13 @@ These are live API calls, so they require valid API keys for the backend beingtested. If tests fail, check whether the issue is in the model config (wrongmodel_id, missing capabilities) or in the API key / network setup.-### Restore the default profile after testing--Once all backends have been tested, regenerate fixtures back to the standard dev-profile:--```bash-make rtm-```+### Clean up after testingAsk the user if they want the temporary profiles removed from`test_profiles_override.toml`. If they do, clean them up. If not, leave themin place — the file is gitignored so it won't affect anyone else.-## Step 7: Gateway (manual — user action required)+## Step 6: Gateway (manual — user action required)Remind the user:@@ -310,7 +289,7 @@ The local file `.pipelex/inference/backends/pipelex_gateway.toml` only allowsoverriding `sdk` and `structure_method` per model — you cannot define new modelsin it.-## Step 8: Verify everything works+## Step 7: Verify everything worksRun the boot sequence test to make sure config loading succeeds:@@ -334,9 +313,7 @@ Present this checklist to the user at the end so they can confirm everything:- [ ] Model entry added to `<backend>.toml` in `.pipelex/inference/backends/`- [ ] Kit configs synced (`make ukc` + `make ccs`)- [ ] Model added to collections in `.pipelex-dev/test_profiles.toml`-- [ ] Test fixtures regenerated (`make rtm`)-- [ ] Inference tests pass on **every backend** the model was added to (one `make rtm PROF=...` + `make ti` per backend)-- [ ] Default fixtures restored (`make rtm`)+- [ ] Inference tests pass on **every backend** the model was added to (`make test-inference-fast PROF=... TEST=...` per backend — fixtures are auto-regenerated)- [ ] Boot test passes (`make tb`)- [ ] Quality checks pass (`make agent-check`)- [ ] Gateway: user will add the model to the remote gateway config separatelydiff --git a/.claude/skills/test-model/SKILL.md b/.claude/skills/test-model/SKILL.mdindex c0864ab83..010f27bb9 100644--- a/.claude/skills/test-model/SKILL.md+++ b/.claude/skills/test-model/SKILL.md@@ -97,26 +97,27 @@ Set the appropriate model list based on model type:The override file may already contain profiles from previous testing sessions.Don't remove existing profiles — just add or update the one you need.-## Step 3: Regenerate fixtures and run tests+## Step 3: Run tests-Regenerate fixtures for the new profile, then run the appropriate test class.+Run the appropriate test class. **No need to call `make rtm` separately** — all+inference test targets automatically regenerate fixtures when `PROF=` is passed+on the command line.```bash-make rtm PROF=<profile_name>-make ti PROF=<profile_name> TEST=<TestClass>+make test-inference-with-prints PROF=<profile_name> TEST=<TestClass>```Map model type to test class:| Model type | Test class | Make shortcut ||------------|-----------|---------------|-| LLM | `TestLLMInference` | `make ti` |-| Image gen | `TestImageGeneration` | `make ti` |-| Extract | `TestExtract` | `make ti` |-| Search | `TestSearch` | `make ti` |+| LLM | `TestLLMInference` | `make test-inference-with-prints` |+| Image gen | `TestImageGeneration` | `make test-inference-with-prints` |+| Extract | `TestExtract` | `make test-inference-with-prints` |+| Search | `TestSearch` | `make test-inference-with-prints` |-All use `make ti` (test-inference) since all these test classes are marked with-the `inference` pytest marker.+All use `make test-inference-with-prints` since all these test classes are marked+with the `inference` pytest marker.For LLM models specifically, there are additional test classes you can run fordeeper coverage (only if the user wants thorough testing):@@ -138,18 +139,7 @@ deeper coverage (only if the user wants thorough testing):- Capability errors (e.g., vision not supported) → expected skips, not failures- Timeout / rate limit → transient, suggest retrying-## Step 5: Restore default fixtures--After testing, regenerate fixtures back to the default dev profile:--```bash-make rtm-```--This is important — if you leave the fixtures pointing at the temporary profile,-other tests will fail.--## Step 6: Clean up (optional)+## Step 5: Clean up (optional)Ask the user if they want the temporary profile removed from`test_profiles_override.toml`. If yes, remove it. If no, leave it — the file isdiff --git a/.pipelex-dev/test_profiles.toml b/.pipelex-dev/test_profiles.tomlindex 90886c88b..ebdca2079 100644--- a/.pipelex-dev/test_profiles.toml+++ b/.pipelex-dev/test_profiles.toml@@ -47,6 +47,7 @@ all = ["scaleway","vertexai","xai",+ "linkup","internal",]@@ -217,6 +218,9 @@ qwen = ["qwen-image"]linkup = ["linkup-standard", "linkup-deep"][collections.extract]+# --- Web Extraction Models ---+from_web = ["linkup-fetch"]+# --- PDF Extraction Models ---from_pdf = ["pypdfium2-extract-pdf",diff --git a/.pipelex/inference/backends/linkup.toml b/.pipelex/inference/backends/linkup.tomlindex 184e324e9..73ba1d78c 100644--- a/.pipelex/inference/backends/linkup.toml+++ b/.pipelex/inference/backends/linkup.toml@@ -35,3 +35,15 @@ model_id = "deep"inputs = ["text"]outputs = ["sourced-answers", "structured"]costs = { input = 0.05, output = 0 }++################################################################################+# FETCH / EXTRACT MODELS+################################################################################++["linkup-fetch"]+model_type = "text_extractor"+sdk = "linkup_fetch"+model_id = "fetch"+inputs = ["web_page"]+outputs = ["pages"]+costs = { input = 0.005, output = 0 }diff --git a/.pipelex/inference/deck/3_extract_deck.toml b/.pipelex/inference/deck/3_extract_deck.tomlindex afb2e0396..eaf3065be 100644--- a/.pipelex/inference/deck/3_extract_deck.toml+++ b/.pipelex/inference/deck/3_extract_deck.toml@@ -31,6 +31,7 @@ default-extract-document = "azure-document-intelligence"default-extract-image = "azure-document-intelligence"default-text-from-pdf = "pypdfium2-extract-pdf"default-no-inference = "pypdfium2-extract-pdf"+default-extract-web-page = "linkup-fetch"##################################################################################################### Extract Presetsdiff --git a/pipelex/builder/pipe/pipe_extract_spec.py b/pipelex/builder/pipe/pipe_extract_spec.pyindex 43c260c00..f67743cc7 100644--- a/pipelex/builder/pipe/pipe_extract_spec.py+++ b/pipelex/builder/pipe/pipe_extract_spec.py@@ -47,6 +47,8 @@ class PipeExtractSpec(PipeSpec):)page_image_captions: bool | None = Field(default=None, description="Whether to generate captions for detected images using AI.")page_views: bool | None = Field(default=None, description="Whether to include rendered page views in the output.")+ render_js: bool | None = Field(default=None, description="Whether to render JavaScript before fetching (web extraction only).")+ include_raw_html: bool | None = Field(default=None, description="Whether to include raw HTML in the response (web extraction only).")@override@field_validator("output", mode="before")@@ -116,4 +118,6 @@ class PipeExtractSpec(PipeSpec):page_image_captions=self.page_image_captions,page_views=self.page_views,page_views_dpi=None,+ render_js=self.render_js,+ include_raw_html=self.include_raw_html,)diff --git a/pipelex/builder/talents/extract_talent.py b/pipelex/builder/talents/extract_talent.pyindex f2f8a39af..446adf7c3 100644--- a/pipelex/builder/talents/extract_talent.py+++ b/pipelex/builder/talents/extract_talent.py@@ -5,3 +5,4 @@ class ExtractTalent(StrEnum):PDF_BASIC_TEXT_EXTRACTOR = "pdf-basic-text-extractor"IMAGE_TEXT_EXTRACTOR = "image-text-extractor"FULL_DOCUMENT_EXTRACTOR = "full-document-extractor"+ WEB_PAGE_EXTRACTOR = "web-page-extractor"diff --git a/pipelex/cogt/content_generation/content_generator_dry.py b/pipelex/cogt/content_generation/content_generator_dry.pyindex 3687a3a62..983285462 100644--- a/pipelex/cogt/content_generation/content_generator_dry.py+++ b/pipelex/cogt/content_generation/content_generator_dry.py@@ -182,6 +182,7 @@ class ContentGeneratorDry(ContentGeneratorProtocol):text_and_images=TextAndImagesContent(text=TextContent(text=page.text) if page.text else None,images=page_images,+ raw_html=page.raw_html,),))diff --git a/pipelex/cogt/content_generation/generated_content_factory.py b/pipelex/cogt/content_generation/generated_content_factory.pyindex 869074a84..98353c9e0 100644--- a/pipelex/cogt/content_generation/generated_content_factory.py+++ b/pipelex/cogt/content_generation/generated_content_factory.py@@ -1,6 +1,9 @@import base64import hashlib+import httpx++from pipelex import logfrom pipelex.cogt.content_generation.exceptions import NeitherUrlNorDataErrorfrom pipelex.cogt.extract.extract_output import ExtractOutputfrom pipelex.cogt.image.generated_image import GeneratedImageRawDetails@@ -125,16 +128,21 @@ class GeneratedContentFactory:public_url: str | Noneif is_remote_url and get_config().pipelex.storage_config.is_fetch_remote_content_enabled:- actual_bytes = await self._fetch_remote_content(url=url)- storage_key = self._build_storage_key(- primary_id=primary_id,- secondary_id=secondary_id,- data=actual_bytes,- mime_type=mime_type,- image_format=image_format,- )- url = await self.storage_provider.store(data=actual_bytes, key=storage_key)- public_url = await self.storage_provider.public_url(uri=url)+ try:+ actual_bytes = await self._fetch_remote_content(url=url)+ except (httpx.HTTPStatusError, httpx.RequestError) as exc:+ log.warning(f"Failed to fetch a remote image: {exc}")+ public_url = url+ else:+ storage_key = self._build_storage_key(+ primary_id=primary_id,+ secondary_id=secondary_id,+ data=actual_bytes,+ mime_type=mime_type,+ image_format=image_format,+ )+ url = await self.storage_provider.store(data=actual_bytes, key=storage_key)+ public_url = await self.storage_provider.public_url(uri=url)elif not is_remote_url:public_url = await self.storage_provider.public_url(uri=url)else:@@ -170,6 +178,7 @@ class GeneratedContentFactory:text_and_images=TextAndImagesContent(text=TextContent(text=page.text) if page.text else None,images=page_images,+ raw_html=page.raw_html,),))diff --git a/pipelex/cogt/extract/extract_job_components.py b/pipelex/cogt/extract/extract_job_components.pyindex cf44c70ed..a29729f78 100644--- a/pipelex/cogt/extract/extract_job_components.py+++ b/pipelex/cogt/extract/extract_job_components.py@@ -23,6 +23,8 @@ class ExtractJobParams(BaseModel):should_caption_images: boolshould_include_page_views: boolpage_views_dpi: int | None+ render_js: bool | None = None+ include_raw_html: bool | None = None@classmethoddef make_default_extract_job_params(cls) -> "ExtractJobParams":@@ -32,6 +34,8 @@ class ExtractJobParams(BaseModel):image_min_size=None,should_include_page_views=False,page_views_dpi=None,+ render_js=None,+ include_raw_html=None,)diff --git a/pipelex/cogt/extract/extract_output.py b/pipelex/cogt/extract/extract_output.pyindex b21f1cece..554d3251c 100644--- a/pipelex/cogt/extract/extract_output.py+++ b/pipelex/cogt/extract/extract_output.py@@ -11,6 +11,7 @@ class ExtractedImageFromPage(GeneratedImageRawDetails):class Page(CustomBaseModel):text: str | None = None+ raw_html: str | None = Noneextracted_images: list[ExtractedImageFromPage] = Field(default_factory=empty_list_factory_of(ExtractedImageFromPage))diff --git a/pipelex/cogt/extract/extract_worker_abstract.py b/pipelex/cogt/extract/extract_worker_abstract.pyindex cacfe6450..7535fe91a 100644--- a/pipelex/cogt/extract/extract_worker_abstract.py+++ b/pipelex/cogt/extract/extract_worker_abstract.py@@ -38,6 +38,10 @@ class ExtractWorkerAbstract(InferenceWorkerAbstract):def is_pdf_supported(self) -> bool:return self.inference_model.is_pdf_supported_for_extract+ @property+ def is_web_page_supported(self) -> bool:+ return self.inference_model.is_web_page_supported_for_extract+@propertydef is_image_supported(self) -> bool:return self.inference_model.is_image_supported_for_extract@@ -54,8 +58,8 @@ class ExtractWorkerAbstract(InferenceWorkerAbstract):msg = f"Extract engine '{self.inference_model.tag}' does not support image extraction."raise ExtractCapabilityError(msg)elif extract_input.document_uri:- if not self.inference_model.is_pdf_supported_for_extract:- msg = f"Extract engine '{self.inference_model.tag}' does not support PDF extraction."+ if not (self.inference_model.is_pdf_supported_for_extract or self.inference_model.is_web_page_supported_for_extract):+ msg = f"Extract engine '{self.inference_model.tag}' does not support document extraction."raise ExtractCapabilityError(msg)if extract_job.job_params.should_caption_images:if not self.inference_model.is_caption_supported_for_extract:diff --git a/pipelex/cogt/extract/extract_worker_factory.py b/pipelex/cogt/extract/extract_worker_factory.pyindex 6da66aa55..c6b225d4f 100644--- a/pipelex/cogt/extract/extract_worker_factory.py+++ b/pipelex/cogt/extract/extract_worker_factory.py@@ -93,6 +93,24 @@ class ExtractWorkerFactory:inference_model=inference_model,reporting_delegate=reporting_delegate,)+ case "linkup_fetch":+ if importlib.util.find_spec("linkup") is None:+ lib_name = "linkup"+ lib_extra_name = "linkup"+ msg = "The linkup SDK is required in order to use Linkup Fetch extraction models."+ raise MissingDependencyError(+ lib_name,+ lib_extra_name,+ msg,+ )++ from pipelex.plugins.linkup.linkup_extract_worker import LinkupExtractWorker # noqa: PLC0415++ extract_worker = LinkupExtractWorker(+ extra_config=backend.extra_config,+ inference_model=inference_model,+ reporting_delegate=reporting_delegate,+ )case _:msg = f"Plugin '{plugin}' is not supported"raise NotImplementedError(msg)diff --git a/pipelex/cogt/image/generated_image.py b/pipelex/cogt/image/generated_image.pyindex 8bb096f7f..b147d3b27 100644--- a/pipelex/cogt/image/generated_image.py+++ b/pipelex/cogt/image/generated_image.py@@ -38,7 +38,7 @@ class GeneratedImageRawDetails(CustomBaseModel):return None@model_validator(mode="after")- def validate_mime_type_or_image_format(self) -> Self:+ def validate_mime_type_or_image_format_or_actual_url(self) -> Self:if self.mime_type:ImageFormat.raise_if_unsupported_mime_type(self.mime_type)elif self.actual_url_or_prefixed_base64 and (@@ -48,6 +48,10 @@ class GeneratedImageRawDetails(CustomBaseModel):ImageFormat.raise_if_unsupported_mime_type(base64_extracted_mime_type)self.mime_type = base64_extracted_mime_typeself.base64_str = base64_str+ elif self.actual_url:+ # When only a URL is provided (e.g. from web extraction), the mime type+ # will be determined when the image is actually downloaded.+ passelif self.image_format is None:msg = "Either mime_type or image_format must be provided"raise ValueError(msg)diff --git a/pipelex/cogt/model_backends/model_spec.py b/pipelex/cogt/model_backends/model_spec.pyindex 414630059..0b77807a4 100644--- a/pipelex/cogt/model_backends/model_spec.py+++ b/pipelex/cogt/model_backends/model_spec.py@@ -60,6 +60,10 @@ class InferenceModelSpec(ConfigModel):def is_image_supported_for_extract(self) -> bool:return "image" in self.inputs+ @property+ def is_web_page_supported_for_extract(self) -> bool:+ return "web_page" in self.inputs+@propertydef is_caption_supported_for_extract(self) -> bool:return "captions" in self.outputsdiff --git a/pipelex/cogt/search/fetch_job.py b/pipelex/cogt/search/fetch_job.pydeleted file mode 100644index 1beac53b7..000000000--- a/pipelex/cogt/search/fetch_job.py+++ /dev/null@@ -1,35 +0,0 @@-from datetime import datetime--from pydantic import BaseModel-from typing_extensions import override--from pipelex.cogt.inference.inference_job_abstract import InferenceJobAbstract-from pipelex.cogt.model_backends.model_spec import InferenceModelSpec-from pipelex.cogt.search.fetch_report import FetchTokensUsage---class FetchJobReport(BaseModel):- fetch_tokens_usage: FetchTokensUsage | None = None---class FetchJob(InferenceJobAbstract):- job_report: FetchJobReport = FetchJobReport()-- @override- def validate_before_execution(self):- pass-- def fetch_job_before_start(self, inference_model: InferenceModelSpec):- self.job_metadata.started_at = datetime.now()-- self.job_report = FetchJobReport()- self.job_report.fetch_tokens_usage = FetchTokensUsage(- job_metadata=self.job_metadata,- inference_model_name=inference_model.name,- unit_costs=inference_model.costs,- inference_model_id=inference_model.model_id,- nb_tokens_by_category={},- )-- def fetch_job_after_complete(self):- self.job_metadata.completed_at = datetime.now()diff --git a/pipelex/cogt/search/fetch_report.py b/pipelex/cogt/search/fetch_report.pydeleted file mode 100644index 0a68e8287..000000000--- a/pipelex/cogt/search/fetch_report.py+++ /dev/null@@ -1,72 +0,0 @@-from typing import Any--from pydantic import BaseModel--from pipelex.cogt.usage.cost_category import CostCategory, CostsByCategoryDict-from pipelex.cogt.usage.token_category import NbTokensByCategoryDict, TokenCategory-from pipelex.pipeline.job_metadata import JobMetadata-from pipelex.types import StrEnum---class FetchTokenCostReportField(StrEnum):- MODEL_TYPE = "model_type"- FETCH_NAME = "fetch_name"- PLATFORM_FETCH_ID = "platform_fetch_id"- NB_TOKENS_INPUT = "nb_tokens_input"- NB_TOKENS_INPUT_CACHED = "nb_tokens_input_cached"- NB_TOKENS_INPUT_NON_CACHED = "nb_tokens_input_non_cached"- NB_TOKENS_INPUT_JOINED = "nb_tokens_input_joined"- NB_TOKENS_OUTPUT = "nb_tokens_output"- COST_INPUT_CACHED = "cost_input_cached"- COST_INPUT_NON_CACHED = "cost_input_non_cached"- COST_INPUT_JOINED = "cost_input_joined"- COST_OUTPUT = "cost_output"-- @staticmethod- def report_field_for_nb_tokens_by_category(token_category: TokenCategory) -> str:- return f"nb_tokens_{token_category}"-- @staticmethod- def report_field_for_cost_by_category(token_category: CostCategory) -> str:- return f"cost_{token_category}"---class FetchTokenCostReport(BaseModel):- model_type: str = "fetch"- job_metadata: JobMetadata- inference_model_name: str- platform_model_id: str-- nb_tokens_by_category: NbTokensByCategoryDict- costs_by_token_category: CostsByCategoryDict-- def as_flat_dictionary(self) -> dict[str, Any]:- the_dict: dict[str, Any] = {}- dict_for_job_metadata = self.job_metadata.model_dump(serialize_as_any=True)- the_dict.update(dict_for_job_metadata)- dict_for_model: dict[str, Any] = {- FetchTokenCostReportField.MODEL_TYPE: self.model_type,- FetchTokenCostReportField.FETCH_NAME: self.inference_model_name,- FetchTokenCostReportField.PLATFORM_FETCH_ID: self.platform_model_id,- }- the_dict.update(dict_for_model)- dict_for_nb_tokens = {- FetchTokenCostReportField.report_field_for_nb_tokens_by_category(token_category): nb_tokens- for token_category, nb_tokens in self.nb_tokens_by_category.items()- }- the_dict.update(dict_for_nb_tokens)- dict_for_costs = {- FetchTokenCostReportField.report_field_for_cost_by_category(token_category): cost- for token_category, cost in self.costs_by_token_category.items()- }- the_dict.update(dict_for_costs)- return the_dict---class FetchTokensUsage(BaseModel):- model_type: str = "fetch"- job_metadata: JobMetadata- inference_model_name: str- unit_costs: CostsByCategoryDict- inference_model_id: str- nb_tokens_by_category: NbTokensByCategoryDictdiff --git a/pipelex/cogt/search/fetch_worker_abstract.py b/pipelex/cogt/search/fetch_worker_abstract.pydeleted file mode 100644index f19e387d7..000000000--- a/pipelex/cogt/search/fetch_worker_abstract.py+++ /dev/null@@ -1,34 +0,0 @@-from abc import ABC, abstractmethod--from pipelex.core.stuffs.text_and_images_content import TextAndImagesContent-from pipelex.pipeline.job_metadata import JobMetadata-from pipelex.reporting.reporting_protocol import ReportingProtocol---class FetchWorkerAbstract(ABC):- def __init__(self, reporting_delegate: ReportingProtocol | None = None):- self.reporting_delegate = reporting_delegate-- @abstractmethod- async def fetch_url(- self,- url: str,- job_metadata: JobMetadata,- include_raw_html: bool | None = None,- render_js: bool | None = None,- extract_images: bool | None = None,- timeout: float | None = None, # noqa: ASYNC109- ) -> TextAndImagesContent:- """Fetch and clean the content of a web page, returning text and optionally images.-- Args:- url: The URL of the web page to fetch- job_metadata: Job metadata for cost reporting- include_raw_html: Whether to include the raw HTML of the webpage in the response- render_js: Whether to render the JavaScript of the webpage before fetching- extract_images: Whether to extract images from the webpage- timeout: The timeout for the HTTP request, in seconds-- Returns:- TextAndImagesContent with the fetched page content- """diff --git a/pipelex/cogt/search/search_job.py b/pipelex/cogt/search/search_job.pyindex 7977987a5..678c7bd38 100644--- a/pipelex/cogt/search/search_job.py+++ b/pipelex/cogt/search/search_job.py@@ -6,6 +6,15 @@ from typing_extensions import overridefrom pipelex.cogt.inference.inference_job_abstract import InferenceJobAbstractfrom pipelex.cogt.model_backends.model_spec import InferenceModelSpecfrom pipelex.cogt.search.search_report import SearchTokensUsage+from pipelex.cogt.search.search_setting import SearchSetting+++class SearchJobParams(BaseModel):+ search_setting: SearchSetting+ include_domains: list[str] | None = None+ exclude_domains: list[str] | None = None+ from_date: str | None = None+ to_date: str | None = Noneclass SearchJobReport(BaseModel):@@ -13,6 +22,8 @@ class SearchJobReport(BaseModel):class SearchJob(InferenceJobAbstract):+ query: str+ job_params: SearchJobParamsjob_report: SearchJobReport = SearchJobReport()@overridediff --git a/pipelex/cogt/search/search_job_factory.py b/pipelex/cogt/search/search_job_factory.pyindex 6f5efeabe..d6899ade2 100644--- a/pipelex/cogt/search/search_job_factory.py+++ b/pipelex/cogt/search/search_job_factory.py@@ -1,17 +1,30 @@-from pipelex.cogt.search.fetch_job import FetchJob-from pipelex.cogt.search.search_job import SearchJob+from pipelex.cogt.search.search_job import SearchJob, SearchJobParams+from pipelex.cogt.search.search_setting import SearchSettingfrom pipelex.pipeline.job_metadata import JobCategory, JobMetadataclass SearchJobFactory:@classmethod- def make_search_job(cls, job_metadata: JobMetadata) -> SearchJob:+ def make_search_job(+ cls,+ query: str,+ search_setting: SearchSetting,+ job_metadata: JobMetadata,+ include_domains: list[str] | None = None,+ exclude_domains: list[str] | None = None,+ from_date: str | None = None,+ to_date: str | None = None,+ ) -> SearchJob:job_metadata.job_category = JobCategory.SEARCH_JOB- return SearchJob(job_metadata=job_metadata)---class FetchJobFactory:- @classmethod- def make_fetch_job(cls, job_metadata: JobMetadata) -> FetchJob:- job_metadata.job_category = JobCategory.FETCH_JOB- return FetchJob(job_metadata=job_metadata)+ job_params = SearchJobParams(+ search_setting=search_setting,+ include_domains=include_domains,+ exclude_domains=exclude_domains,+ from_date=from_date,+ to_date=to_date,+ )+ return SearchJob(+ job_metadata=job_metadata,+ query=query,+ job_params=job_params,+ )diff --git a/pipelex/cogt/search/search_worker_abstract.py b/pipelex/cogt/search/search_worker_abstract.pyindex de70b2564..02cc83575 100644--- a/pipelex/cogt/search/search_worker_abstract.py+++ b/pipelex/cogt/search/search_worker_abstract.py@@ -1,66 +1,80 @@-from abc import ABC, abstractmethod+from abc import abstractmethodfrom typing import Any-from pipelex.cogt.search.search_setting import SearchSetting+from typing_extensions import override++from pipelex import log+from pipelex.cogt.inference.inference_worker_abstract import InferenceWorkerAbstract+from pipelex.cogt.model_backends.model_spec import InferenceModelSpec+from pipelex.cogt.search.search_job import SearchJobfrom pipelex.core.stuffs.search_result_content import SearchResultContent-from pipelex.pipeline.job_metadata import JobMetadata+from pipelex.pipeline.job_metadata import UnitJobIdfrom pipelex.reporting.reporting_protocol import ReportingProtocol+from pipelex.tools.typing.pydantic_utils import BaseModelTypeVar++class SearchWorkerAbstract(InferenceWorkerAbstract):+ def __init__(+ self,+ inference_model: InferenceModelSpec,+ reporting_delegate: ReportingProtocol | None = None,+ ):+ InferenceWorkerAbstract.__init__(self, reporting_delegate=reporting_delegate)+ self.inference_model = inference_model-class SearchWorkerAbstract(ABC):- def __init__(self, reporting_delegate: ReportingProtocol | None = None):- self.reporting_delegate = reporting_delegate+ @property+ @override+ def desc(self) -> str:+ return f"Search using {self.inference_model.desc}"- @abstractmethodasync def search_sourced_answer(self,- query: str,- search_setting: SearchSetting,- job_metadata: JobMetadata,- include_domains: list[str] | None = None,- exclude_domains: list[str] | None = None,- from_date: str | None = None,- to_date: str | None = None,+ search_job: SearchJob,) -> SearchResultContent:- """Execute a search query and return a sourced answer with sources.+ """Execute a search query and return a sourced answer with sources."""+ log.dev(f"✨ {self.desc} ✨")+ search_job.validate_before_execution()+ search_job.job_metadata.unit_job_id = UnitJobId.SEARCH_SOURCED_ANSWER+ search_job.search_job_before_start(inference_model=self.inference_model)- Args:- query: The search query text- search_setting: Search configuration including model, depth, etc.- job_metadata: Job metadata for cost reporting- include_domains: Optional list of domains to restrict search to- exclude_domains: Optional list of domains to exclude from search- from_date: Optional start date filter (YYYY-MM-DD)- to_date: Optional end date filter (YYYY-MM-DD)+ result = await self._search_sourced_answer(search_job=search_job)- Returns:- SearchResultContent with answer and sources- """+ search_job.search_job_after_complete()+ if self.reporting_delegate:+ self.reporting_delegate.report_inference_job(inference_job=search_job)++ return result- @abstractmethodasync def search_structured(self,- query: str,- search_setting: SearchSetting,- output_schema: type,- job_metadata: JobMetadata,- include_domains: list[str] | None = None,- exclude_domains: list[str] | None = None,- from_date: str | None = None,- to_date: str | None = None,+ search_job: SearchJob,+ schema: type[BaseModelTypeVar],) -> dict[str, Any]:- """Execute a search query and return structured data matching the schema.+ """Execute a search query and return structured data matching the schema."""+ log.dev(f"✨ {self.desc} ✨")+ search_job.validate_before_execution()+ search_job.job_metadata.unit_job_id = UnitJobId.SEARCH_STRUCTURED+ search_job.search_job_before_start(inference_model=self.inference_model)++ result = await self._search_structured(search_job=search_job, schema=schema)++ search_job.search_job_after_complete()+ if self.reporting_delegate:+ self.reporting_delegate.report_inference_job(inference_job=search_job)- Args:- query: The search query text- search_setting: Search configuration including model, depth, etc.- output_schema: Pydantic model class defining the expected output structure- job_metadata: Job metadata for cost reporting- include_domains: Optional list of domains to restrict search to- exclude_domains: Optional list of domains to exclude from search- from_date: Optional start date filter (YYYY-MM-DD)- to_date: Optional end date filter (YYYY-MM-DD)+ return result- Returns:- Dictionary matching the output_schema structure- """+ @abstractmethod+ async def _search_sourced_answer(+ self,+ search_job: SearchJob,+ ) -> SearchResultContent:+ pass++ @abstractmethod+ async def _search_structured(+ self,+ search_job: SearchJob,+ schema: type[BaseModelTypeVar],+ ) -> dict[str, Any]:+ passdiff --git a/pipelex/cogt/search/search_worker_factory.py b/pipelex/cogt/search/search_worker_factory.pyindex 811435198..dc83690fa 100644--- a/pipelex/cogt/search/search_worker_factory.py+++ b/pipelex/cogt/search/search_worker_factory.py@@ -1,5 +1,4 @@from pipelex.cogt.model_backends.model_spec import InferenceModelSpec-from pipelex.cogt.search.fetch_worker_abstract import FetchWorkerAbstractfrom pipelex.cogt.search.search_worker_abstract import SearchWorkerAbstractfrom pipelex.hub import get_models_manager, get_plugin_manager, get_report_delegatefrom pipelex.plugins.plugin import Plugin@@ -27,9 +26,9 @@ class SearchWorkerFactory:search_worker: SearchWorkerAbstractmatch plugin.sdk:case "linkup":- from pipelex.plugins.linkup.linkup_worker import LinkupWorker # noqa: PLC0415+ from pipelex.plugins.linkup.linkup_search_worker import LinkupSearchWorker # noqa: PLC0415- search_worker = LinkupWorker(inference_model=inference_model, reporting_delegate=get_report_delegate())+ search_worker = LinkupSearchWorker(inference_model=inference_model, reporting_delegate=get_report_delegate())case "gateway_search":from pipelex.plugins.gateway.gateway_factory import GatewayFactory # noqa: PLC0415from pipelex.plugins.gateway.gateway_search_worker import GatewaySearchWorker # noqa: PLC0415@@ -46,44 +45,3 @@ class SearchWorkerFactory:raise NotImplementedError(msg)return search_worker-- @classmethod- def make_fetch_worker(- cls,- inference_model: InferenceModelSpec,- ) -> FetchWorkerAbstract:- """Create a fetch worker for the given inference model.-- Discriminates on plugin.sdk to select the appropriate implementation.-- Args:- inference_model: The model spec from the backend configuration.-- Returns:- A FetchWorkerAbstract instance.- """- plugin = Plugin.make_for_inference_model(inference_model=inference_model)- backend = get_models_manager().get_required_inference_backend(inference_model.backend_name)- plugin_sdk_registry = get_plugin_manager().plugin_sdk_registry- fetch_worker: FetchWorkerAbstract- match plugin.sdk:- case "linkup":- from pipelex.plugins.linkup.linkup_worker import LinkupWorker # noqa: PLC0415-- fetch_worker = LinkupWorker(inference_model=inference_model, reporting_delegate=get_report_delegate())- case "gateway_search":- from pipelex.plugins.gateway.gateway_factory import GatewayFactory # noqa: PLC0415- from pipelex.plugins.gateway.gateway_fetch_worker import GatewayFetchWorker # noqa: PLC0415-- sdk_instance = plugin_sdk_registry.get_sdk_instance(plugin=plugin) or plugin_sdk_registry.set_sdk_instance(- plugin=plugin,- sdk_instance=GatewayFactory.make_portkey_client(backend=backend),- )- fetch_worker = GatewayFetchWorker(- sdk_instance=sdk_instance, inference_model=inference_model, reporting_delegate=get_report_delegate()- )- case _:- msg = f"Plugin '{plugin}' is not supported for fetch"- raise NotImplementedError(msg)-- return fetch_workerdiff --git a/pipelex/cogt/usage/cost_registry.py b/pipelex/cogt/usage/cost_registry.pyindex cb62c090b..a427c1ec1 100644--- a/pipelex/cogt/usage/cost_registry.py+++ b/pipelex/cogt/usage/cost_registry.py@@ -12,7 +12,6 @@ from pipelex.cogt.exceptions import CostRegistryErrorfrom pipelex.cogt.extract.extract_report import ExtractTokenCostReport, ExtractTokenCostReportField, ExtractTokensUsagefrom pipelex.cogt.img_gen.img_gen_report import ImgGenTokenCostReport, ImgGenTokenCostReportField, ImgGenTokensUsagefrom pipelex.cogt.llm.llm_report import LLMTokenCostReport, LLMTokenCostReportField, LLMTokensUsage-from pipelex.cogt.search.fetch_report import FetchTokenCostReport, FetchTokenCostReportField, FetchTokensUsagefrom pipelex.cogt.search.search_report import SearchTokenCostReport, SearchTokenCostReportField, SearchTokensUsagefrom pipelex.cogt.usage.cost_category import CostCategory, CostsByCategoryDictfrom pipelex.cogt.usage.costs_per_token import model_cost_per_token@@ -20,8 +19,8 @@ from pipelex.cogt.usage.token_category import TokenCategoryfrom pipelex.hub import get_consolefrom pipelex.tools.typing.pydantic_utils import empty_list_factory_of-TokensUsage = LLMTokensUsage | ImgGenTokensUsage | ExtractTokensUsage | SearchTokensUsage | FetchTokensUsage-TokenCostReport = LLMTokenCostReport | ImgGenTokenCostReport | ExtractTokenCostReport | SearchTokenCostReport | FetchTokenCostReport+TokensUsage = LLMTokensUsage | ImgGenTokensUsage | ExtractTokensUsage | SearchTokensUsage+TokenCostReport = LLMTokenCostReport | ImgGenTokenCostReport | ExtractTokenCostReport | SearchTokenCostReportCostRegistryRoot = list[TokenCostReport]@@ -83,8 +82,7 @@ class CostRegistry(RootModel[CostRegistryRoot]):record.get(report_field.LLM_NAME)or record.get(ImgGenTokenCostReportField.IMG_GEN_NAME)or record.get(ExtractTokenCostReportField.EXTRACT_NAME)- or record.get(SearchTokenCostReportField.SEARCH_NAME)- or record.get(FetchTokenCostReportField.FETCH_NAME, "unknown")+ or record.get(SearchTokenCostReportField.SEARCH_NAME, "unknown"))model_type = record.get(report_field.MODEL_TYPE, "llm")model_types[model_name] = model_type@@ -244,15 +242,6 @@ class CostRegistry(RootModel[CostRegistryRoot]):nb_tokens_by_category=tokens_usage.nb_tokens_by_category,costs_by_token_category=costs_by_token_category,)- if isinstance(tokens_usage, FetchTokensUsage):- return FetchTokenCostReport(- model_type=tokens_usage.model_type,- job_metadata=tokens_usage.job_metadata,- inference_model_name=tokens_usage.inference_model_name,- platform_model_id=tokens_usage.inference_model_id,- nb_tokens_by_category=tokens_usage.nb_tokens_by_category,- costs_by_token_category=costs_by_token_category,- )return ExtractTokenCostReport(model_type=tokens_usage.model_type,job_metadata=tokens_usage.job_metadata,diff --git a/pipelex/kit/configs/inference/backends/linkup.toml b/pipelex/kit/configs/inference/backends/linkup.tomlindex 184e324e9..73ba1d78c 100644--- a/pipelex/kit/configs/inference/backends/linkup.toml+++ b/pipelex/kit/configs/inference/backends/linkup.toml@@ -35,3 +35,15 @@ model_id = "deep"inputs = ["text"]outputs = ["sourced-answers", "structured"]costs = { input = 0.05, output = 0 }++################################################################################+# FETCH / EXTRACT MODELS+################################################################################++["linkup-fetch"]+model_type = "text_extractor"+sdk = "linkup_fetch"+model_id = "fetch"+inputs = ["web_page"]+outputs = ["pages"]+costs = { input = 0.005, output = 0 }diff --git a/pipelex/kit/configs/inference/deck/3_extract_deck.toml b/pipelex/kit/configs/inference/deck/3_extract_deck.tomlindex afb2e0396..eaf3065be 100644--- a/pipelex/kit/configs/inference/deck/3_extract_deck.toml+++ b/pipelex/kit/configs/inference/deck/3_extract_deck.toml@@ -31,6 +31,7 @@ default-extract-document = "azure-document-intelligence"default-extract-image = "azure-document-intelligence"default-text-from-pdf = "pypdfium2-extract-pdf"default-no-inference = "pypdfium2-extract-pdf"+default-extract-web-page = "linkup-fetch"##################################################################################################### Extract Presetsdiff --git a/pipelex/pipe_operators/extract/pipe_extract.py b/pipelex/pipe_operators/extract/pipe_extract.pyindex 74a52b6f2..519421ab0 100644--- a/pipelex/pipe_operators/extract/pipe_extract.py+++ b/pipelex/pipe_operators/extract/pipe_extract.py@@ -42,6 +42,8 @@ class PipeExtract(PipeOperator[PipeExtractOutput]):max_page_images: int | Noneshould_include_page_views: boolpage_views_dpi: int | None+ render_js: bool | None = None+ include_raw_html: bool | None = Noneimage_stuff_name: str | None = Nonedocument_stuff_name: str | None = None@@ -146,6 +148,8 @@ class PipeExtract(PipeOperator[PipeExtractOutput]):page_views_dpi=self.page_views_dpi,max_nb_images=max_nb_images,image_min_size=extract_setting.image_min_size,+ render_js=self.render_js,+ include_raw_html=self.include_raw_html,)extract_input = ExtractInput(image_uri=image_uri,diff --git a/pipelex/pipe_operators/extract/pipe_extract_blueprint.py b/pipelex/pipe_operators/extract/pipe_extract_blueprint.pyindex d569e573f..05eb9e816 100644--- a/pipelex/pipe_operators/extract/pipe_extract_blueprint.py+++ b/pipelex/pipe_operators/extract/pipe_extract_blueprint.py@@ -14,6 +14,8 @@ class PipeExtractBlueprint(PipeBlueprint):page_image_captions: bool | None = Nonepage_views: bool | None = Nonepage_views_dpi: int | None = None+ render_js: bool | None = None+ include_raw_html: bool | None = None@overridedef validate_inputs(self):diff --git a/pipelex/pipe_operators/extract/pipe_extract_factory.py b/pipelex/pipe_operators/extract/pipe_extract_factory.pyindex df66eff4a..ba88e77ae 100644--- a/pipelex/pipe_operators/extract/pipe_extract_factory.py+++ b/pipelex/pipe_operators/extract/pipe_extract_factory.py@@ -67,4 +67,6 @@ class PipeExtractFactory(PipeFactoryProtocol[PipeExtractBlueprint, PipeExtract])should_caption_images=blueprint.page_image_captions or False,should_include_page_views=blueprint.page_views or False,page_views_dpi=blueprint.page_views_dpi,+ render_js=blueprint.render_js,+ include_raw_html=blueprint.include_raw_html,)diff --git a/pipelex/pipe_operators/search/pipe_search.py b/pipelex/pipe_operators/search/pipe_search.pyindex 0ff820734..b3f93be96 100644--- a/pipelex/pipe_operators/search/pipe_search.py+++ b/pipelex/pipe_operators/search/pipe_search.py@@ -7,6 +7,7 @@ from pipelex.cogt.content_generation.dry_run_factory import DryRunFactoryfrom pipelex.cogt.exceptions import ModelChoiceNotFoundErrorfrom pipelex.cogt.model_backends.model_type import ModelTypefrom pipelex.cogt.models.model_deck_check import check_search_choice_with_deck+from pipelex.cogt.search.search_job_factory import SearchJobFactoryfrom pipelex.cogt.search.search_setting import SearchModelChoice, SearchSettingfrom pipelex.cogt.search.search_worker_factory import SearchWorkerFactoryfrom pipelex.cogt.templating.template_blueprint import TemplateBlueprint@@ -111,33 +112,27 @@ class PipeSearch(PipeOperator[PipeSearchOutput]):# 5. Get search worker from factoryworker = SearchWorkerFactory.make_search_worker(inference_model=inference_model)- # 6. Execute search based on output type+ # 6. Create search job+ search_job = SearchJobFactory.make_search_job(+ query=query_text,+ search_setting=search_setting,+ job_metadata=job_metadata,+ include_domains=self.include_domains,+ exclude_domains=self.exclude_domains,+ from_date=self.from_date,+ to_date=self.to_date,+ )++ # 7. Execute search based on output typecontent: StuffContentif not self.is_structured_output:- content = await worker.search_sourced_answer(- query=query_text,- search_setting=search_setting,- job_metadata=job_metadata,- include_domains=self.include_domains,- exclude_domains=self.exclude_domains,- from_date=self.from_date,- to_date=self.to_date,- )+ content = await worker.search_sourced_answer(search_job=search_job)else:output_structure_class = self.output.concept.get_structure_class()- result_dict = await worker.search_structured(- query=query_text,- search_setting=search_setting,- output_schema=output_structure_class,- job_metadata=job_metadata,- include_domains=self.include_domains,- exclude_domains=self.exclude_domains,- from_date=self.from_date,- to_date=self.to_date,- )+ result_dict = await worker.search_structured(search_job=search_job, schema=output_structure_class)content = output_structure_class.model_validate(result_dict)- # 7. Create Stuff, set in working memory, and return output+ # 8. Create Stuff, set in working memory, and return outputoutput_stuff = StuffFactory.make_stuff(name=output_name,concept=self.output.concept,diff --git a/pipelex/pipelex.toml b/pipelex/pipelex.tomlindex 4237c0971..ce231ad8b 100644--- a/pipelex/pipelex.toml+++ b/pipelex/pipelex.toml@@ -72,6 +72,7 @@ gen-image-high-quality = "$gen-image-high-quality"pdf-basic-text-extractor = "@default-text-from-pdf"image-text-extractor = "@default-extract-image"full-document-extractor = "@default-extract-document"+web-page-extractor = "@default-extract-web-page"[pipelex.builder_config.talent_preset_mappings.search]web-search = "$standard"diff --git a/pipelex/pipeline/job_metadata.py b/pipelex/pipeline/job_metadata.pyindex 7631ae5cc..97601e243 100644--- a/pipelex/pipeline/job_metadata.py+++ b/pipelex/pipeline/job_metadata.py@@ -15,7 +15,6 @@ class JobCategory(StrEnum):JINJA2_JOB = "jinja2_job"EXTRACT_JOB = "extract_job"SEARCH_JOB = "search_job"- FETCH_JOB = "fetch_job"class UnitJobId(StrEnum):@@ -25,7 +24,6 @@ class UnitJobId(StrEnum):EXTRACT_PAGES = "extract_pages"SEARCH_SOURCED_ANSWER = "search_sourced_answer"SEARCH_STRUCTURED = "search_structured"- FETCH_URL = "fetch_url"@propertydef model_kind(self) -> str:@@ -38,8 +36,6 @@ class UnitJobId(StrEnum):return "Extract"case UnitJobId.SEARCH_SOURCED_ANSWER | UnitJobId.SEARCH_STRUCTURED:return "Search"- case UnitJobId.FETCH_URL:- return "Fetch"class JobMetadata(BaseModel):diff --git a/pipelex/plugins/gateway/gateway_completions_factory.py b/pipelex/plugins/gateway/gateway_completions_factory.pyindex ab52d754a..eb651de76 100644--- a/pipelex/plugins/gateway/gateway_completions_factory.py+++ b/pipelex/plugins/gateway/gateway_completions_factory.py@@ -30,6 +30,7 @@ from pipelex.plugins.gateway.gateway_exceptions import GatewayExtractResponseErrfrom pipelex.plugins.gateway.gateway_factory import GatewayFactoryfrom pipelex.plugins.gateway.gateway_protocols import GatewayExtractProtocolfrom pipelex.plugins.gateway.gateway_schemas import GatewayExtractPageAzure, GatewayExtractPageDeepseek, GatewayExtractPageMistral+from pipelex.plugins.gateway.gateway_search_schemas import GatewayFetchResultResponsefrom pipelex.plugins.openai.openai_completions_factory import OpenAICompletionsFactoryfrom pipelex.tools.uri.prepared_file import PreparedFileBase64, PreparedFileHttpUrl, PreparedFileLocalPath@@ -133,6 +134,8 @@ class GatewayCompletionsFactory(OpenAICompletionsFactory):return cls._make_extract_output_from_response_azure(response=response)case GatewayExtractProtocol.DEEPSEEK_OCR:return cls._make_extract_output_from_response_deepseek(response=response)+ case GatewayExtractProtocol.LINKUP_FETCH:+ return cls._make_extract_output_from_response_linkup_fetch(response=response)@classmethoddef _extract_pages_from_choices_content(@@ -289,6 +292,64 @@ class GatewayCompletionsFactory(OpenAICompletionsFactory):msg = f"Error parsing Gateway extract response from pages using Deepseek schema: {exc}"raise GatewayExtractResponseError(msg) from exc+ @classmethod+ def _make_extract_output_from_response_linkup_fetch(+ cls,+ response: GenericResponse,+ ) -> ExtractOutput:+ content_str = cls._extract_content_string_from_response(response=response)+ if content_str is None:+ msg = "Gateway fetch response does not contain content in choices[0].message.content"+ raise GatewayExtractResponseError(msg)+ try:+ fetch_result = GatewayFetchResultResponse.model_validate_json(content_str)+ except (ValidationError, ValueError) as exc:+ msg = f"Error parsing Gateway fetch response as GatewayFetchResultResponse: {exc}"+ raise GatewayExtractResponseError(msg) from exc++ extracted_images: list[ExtractedImageFromPage] = []+ if fetch_result.images:+ for image in fetch_result.images:+ extracted_images.append(+ ExtractedImageFromPage(+ size=None,+ actual_url=image.url,+ mime_type=None,+ caption=image.alt or None,+ )+ )++ page = Page(+ text=fetch_result.markdown,+ raw_html=fetch_result.raw_html,+ extracted_images=extracted_images,+ )+ return ExtractOutput(pages={0: page})++ @classmethod+ def _extract_content_string_from_response(+ cls,+ response: GenericResponse,+ ) -> str | None:+ """Extract the string content from choices[0].message.content."""+ try:+ response_dict: dict[str, Any] = response.model_dump(serialize_as_any=True)+ choices = response_dict.get("choices")+ if not choices or not isinstance(choices, list) or len(choices) == 0: # pyright: ignore[reportUnknownArgumentType]+ return None+ first_choice = cast("dict[str, Any]", choices[0])+ message_raw: Any = first_choice.get("message")+ if not message_raw or not isinstance(message_raw, dict):+ return None+ message_dict = cast("dict[str, Any]", message_raw)+ content: Any = message_dict.get("content")+ if not content or not isinstance(content, str):+ return None+ content_str: str = content+ return content_str+ except (KeyError, IndexError, TypeError, AttributeError):+ return None+@overridedef make_extras(self, inference_model: InferenceModelSpec, inference_job: InferenceJobAbstract, output_desc: strdiff --git a/pipelex/plugins/gateway/gateway_extract_worker.py b/pipelex/plugins/gateway/gateway_extract_worker.pyindex 791c05f9a..3c8a3454c 100644--- a/pipelex/plugins/gateway/gateway_extract_worker.py+++ b/pipelex/plugins/gateway/gateway_extract_worker.py@@ -14,11 +14,14 @@ from pipelex.cogt.extract.extract_output import ExtractOutputfrom pipelex.cogt.extract.extract_worker_abstract import ExtractWorkerAbstractfrom pipelex.cogt.inference.inference_constants import InferenceOutputTypefrom pipelex.cogt.model_backends.model_spec import InferenceModelSpec+from pipelex.cogt.usage.token_category import TokenCategoryfrom pipelex.config import get_configfrom pipelex.hub import get_storage_providerfrom pipelex.plugins.gateway.gateway_completions_factory import GatewayCompletionsFactoryfrom pipelex.plugins.gateway.gateway_deck import GatewayDeckfrom pipelex.plugins.gateway.gateway_factory import GatewayFactory+from pipelex.plugins.gateway.gateway_protocols import GatewayExtractProtocol+from pipelex.plugins.gateway.gateway_search_schemas import GatewayFetchRequestParamsfrom pipelex.reporting.reporting_protocol import ReportingProtocolfrom pipelex.tools.uri.uri_resolver import make_base64_url_from_any_uri@@ -66,6 +69,17 @@ class GatewayExtractWorker(ExtractWorkerAbstract):async def _extract_pages(self,extract_job: ExtractJob,+ ) -> ExtractOutput:+ extract_protocol = GatewayExtractProtocol.make_from_model_handle(model_handle=self.inference_model.name)+ match extract_protocol:+ case GatewayExtractProtocol.LINKUP_FETCH:+ return await self._extract_web_fetch(extract_job=extract_job)+ case GatewayExtractProtocol.MISTRAL_DOC_AI | GatewayExtractProtocol.AZURE_DOC_INTEL | GatewayExtractProtocol.DEEPSEEK_OCR:+ return await self._extract_document(extract_job=extract_job)++ async def _extract_document(+ self,+ extract_job: ExtractJob,) -> ExtractOutput:# max_nb_images: None=unlimited, 0=no images, N=limit to N imagesmax_nb_images = extract_job.job_params.max_nb_images@@ -97,6 +111,73 @@ class GatewayExtractWorker(ExtractWorkerAbstract):raise ExtractInputError(msg)return extract_output+ async def _extract_web_fetch(+ self,+ extract_job: ExtractJob,+ ) -> ExtractOutput:+ """Extract content from a web page via the gateway's linkup-fetch endpoint."""+ document_uri = extract_job.extract_input.document_uri+ if not document_uri:+ msg = "GatewayExtractWorker (linkup-fetch) requires a document_uri (web URL) in ExtractInput"+ raise ExtractInputError(msg)++ job_params = extract_job.job_params+ max_nb_images = job_params.max_nb_images+ extract_images = max_nb_images is None or max_nb_images != 0++ fetch_params = GatewayFetchRequestParams(+ url=document_uri,+ render_js=job_params.render_js,+ include_raw_html=job_params.include_raw_html,+ extract_images=extract_images,+ )++ config_id = GatewayDeck.get_config_id(headers=self.inference_model.extra_headers or {})+ log.dev(f"Web fetch via gateway config '{config_id}' for URL: {document_uri}")++ messages: list[dict[str, str]] = [{"role": "user", "content": fetch_params.model_dump_json()}]++ attempt_number = 0+ response: GenericResponse | None = None+ retryer = self._make_retryer()+ try:+ async for attempt in retryer:+ with attempt:+ attempt_number += 1+ response = await self.portkey_client.with_options(config=config_id).post( # pyright: ignore[reportUnknownMemberType, reportUnknownVariableType]+ "/chat/completions",+ model=self.inference_model.model_id,+ messages=messages,+ )+ except portkey_exceptions.APIError as exc:+ error_summary = GatewayFactory.make_error_summary_from_portkey_error(exc)+ msg = f"Web fetch service error for model '{self.inference_model.tag}' after {attempt_number} attempt(s): {error_summary}"+ raise ExtractJobFailureError(msg) from exc++ if response is None:+ msg = f"Could not get a response for model '{self.inference_model.tag}' via Portkey after {attempt_number} attempts"+ raise ExtractJobFailureError(msg)++ if not isinstance(response, GenericResponse):+ msg = "Response is not of type GenericResponse"+ raise TypeError(msg)++ extract_output = GatewayCompletionsFactory.make_extract_output_from_response(inference_model=self.inference_model, response=response)++ # Apply image limit (factory just parses, worker applies limits)+ if max_nb_images is not None and extract_output.pages:+ for page in extract_output.pages.values():+ page.extracted_images = page.extracted_images[:max_nb_images]++ # Per-request cost model: costs are defined per million, so 1 request = 1_000_000+ if extract_tokens_usage := extract_job.job_report.extract_tokens_usage:+ extract_tokens_usage.nb_tokens_by_category = {+ TokenCategory.INPUT: 1_000_000,+ TokenCategory.OUTPUT: 1_000_000,+ }++ return extract_output+async def _extract_base64_url(self,extract_job: ExtractJob,diff --git a/pipelex/plugins/gateway/gateway_factory.py b/pipelex/plugins/gateway/gateway_factory.pyindex 878e43d99..a4dbb8a1e 100644--- a/pipelex/plugins/gateway/gateway_factory.py+++ b/pipelex/plugins/gateway/gateway_factory.py@@ -90,6 +90,8 @@ class GatewayFactory:case GatewayExtractProtocol.DEEPSEEK_OCR:messages_deepseek: list[dict[str, str]] = [{"role": "user", "content": "Convert the document to markdown."}]extra_body["messages"] = messages_deepseek+ case GatewayExtractProtocol.LINKUP_FETCH:+ pass # Fetch params are built directly in GatewayExtractWorker._extract_web_fetchelif isinstance(inference_job, LLMJob) and inference_model.model_id.lower().startswith("mistral-") and inference_job.job_params.seed is None:# Mistral models really want non-null seedextra_body["seed"] = random.randint(0, 1000000)diff --git a/pipelex/plugins/gateway/gateway_fetch_worker.py b/pipelex/plugins/gateway/gateway_fetch_worker.pydeleted file mode 100644index a79b1f19b..000000000--- a/pipelex/plugins/gateway/gateway_fetch_worker.py+++ /dev/null@@ -1,179 +0,0 @@-import json-from typing import Any, cast--from portkey_ai import AsyncPortkey-from portkey_ai.api_resources import exceptions as portkey_exceptions-from portkey_ai.api_resources.utils import GenericResponse-from tenacity import AsyncRetrying, RetryCallState, retry_if_exception, stop_after_attempt, wait_random_exponential-from typing_extensions import override--from pipelex import log-from pipelex.cogt.exceptions import SdkTypeError-from pipelex.cogt.model_backends.model_spec import InferenceModelSpec-from pipelex.cogt.search.fetch_job import FetchJob-from pipelex.cogt.search.fetch_worker_abstract import FetchWorkerAbstract-from pipelex.cogt.search.search_job_factory import FetchJobFactory-from pipelex.cogt.usage.token_category import NbTokensByCategoryDict, TokenCategory-from pipelex.config import get_config-from pipelex.core.stuffs.image_content import ImageContent-from pipelex.core.stuffs.text_and_images_content import TextAndImagesContent-from pipelex.core.stuffs.text_content import TextContent-from pipelex.pipeline.job_metadata import JobMetadata, UnitJobId-from pipelex.plugins.gateway.gateway_deck import GatewayDeck-from pipelex.plugins.gateway.gateway_exceptions import GatewaySearchResponseError-from pipelex.plugins.gateway.gateway_factory import GatewayFactory-from pipelex.plugins.gateway.gateway_search_schemas import GatewayFetchRequestParams-from pipelex.reporting.reporting_protocol import ReportingProtocol---class GatewayFetchWorker(FetchWorkerAbstract):- """Fetch worker that routes through Portkey to the pipelex-relay LinkUp fetch endpoint."""-- def __init__(- self,- sdk_instance: Any,- inference_model: InferenceModelSpec,- reporting_delegate: ReportingProtocol | None = None,- ):- super().__init__(reporting_delegate=reporting_delegate)-- if not isinstance(sdk_instance, AsyncPortkey):- msg = f"Provided sdk_instance for {self.__class__.__name__} is not of type AsyncPortkey: it's a '{type(sdk_instance)}'"- raise SdkTypeError(msg)-- self.portkey_client: AsyncPortkey = sdk_instance- self.inference_model = inference_model- self._tenacity_config = get_config().cogt.tenacity_config-- def _make_retryer(self) -> AsyncRetrying:- """Create a fresh AsyncRetrying instance for each call."""- return AsyncRetrying(- retry=retry_if_exception(self._is_retryable_portkey_error),- before_sleep=self._log_retry,- wait=wait_random_exponential(- multiplier=self._tenacity_config.wait_multiplier,- max=self._tenacity_config.wait_max,- exp_base=self._tenacity_config.wait_exp_base,- ),- reraise=True,- stop=stop_after_attempt(self._tenacity_config.max_retries),- )-- @override- async def fetch_url(- self,- url: str,- job_metadata: JobMetadata,- include_raw_html: bool | None = None,- render_js: bool | None = None,- extract_images: bool | None = None,- timeout: float | None = None, # noqa: ASYNC109- ) -> TextAndImagesContent:- job_metadata.unit_job_id = UnitJobId.FETCH_URL- fetch_job = FetchJobFactory.make_fetch_job(job_metadata=job_metadata)- fetch_job.fetch_job_before_start(inference_model=self.inference_model)-- params = GatewayFetchRequestParams(- url=url,- include_raw_html=include_raw_html,- render_js=render_js,- extract_images=extract_images,- timeout=timeout,- )-- config_id = GatewayDeck.get_config_id(headers=self.inference_model.extra_headers or {})- log.dev(f"Fetch via gateway config '{config_id}'")-- messages: list[dict[str, str]] = [{"role": "user", "content": params.model_dump_json()}]-- attempt_number = 0- response: GenericResponse | None = None- retryer = self._make_retryer()- try:- async for attempt in retryer:- with attempt:- attempt_number += 1- response = await self.portkey_client.with_options(config=config_id).post( # pyright: ignore[reportUnknownMemberType, reportUnknownVariableType]- "/chat/completions",- model="linkup/fetch",- messages=messages,- )- except portkey_exceptions.APIError as exc:- error_summary = GatewayFactory.make_error_summary_from_portkey_error(exc)- msg = f"Fetch service error after {attempt_number} attempt(s): {error_summary}"- raise GatewaySearchResponseError(msg) from exc-- if response is None:- msg = f"Could not get a fetch response via Portkey after {attempt_number} attempts"- raise GatewaySearchResponseError(msg)-- if not isinstance(response, GenericResponse):- msg = "Response is not of type GenericResponse"- raise TypeError(msg)-- self._extract_usage(response=response, fetch_job=fetch_job)- fetch_job.fetch_job_after_complete()- if self.reporting_delegate:- self.reporting_delegate.report_inference_job(inference_job=fetch_job)-- # Extract content from response — use dict access (GenericResponse uses extra="allow")- try:- choice: dict[str, Any] = response.choices[0] # type: ignore[attr-defined] # pyright: ignore[reportAttributeAccessIssue, reportUnknownMemberType]- raw_content = cast("object", choice["message"]["content"])- except (KeyError, IndexError, TypeError) as exc:- msg = "Could not extract content from gateway fetch response"- raise GatewaySearchResponseError(msg) from exc-- if not isinstance(raw_content, str):- msg = f"Expected string content in fetch response, got {type(raw_content)}"- raise GatewaySearchResponseError(msg)- content_str: str = raw_content-- result_dict: dict[str, Any] = json.loads(content_str)-- text = TextContent(text=result_dict["markdown"]) if result_dict.get("markdown") else None-- images: list[ImageContent] | None = None- if result_dict.get("images") is not None:- images = [- ImageContent(- url=image["url"],- caption=image.get("alt"),- )- for image in result_dict["images"]- ]-- return TextAndImagesContent(- text=text,- images=images or None,- raw_html=result_dict.get("raw_html"),- )-- def _extract_usage(self, response: GenericResponse, fetch_job: FetchJob) -> None:- """Extract token usage from the GenericResponse and populate the fetch job report."""- response_dict: dict[str, Any] = response.model_dump(serialize_as_any=True)- fetch_tokens_usage = fetch_job.job_report.fetch_tokens_usage- if (usage_dict := response_dict.get("usage")) and fetch_tokens_usage:- nb_tokens: NbTokensByCategoryDict = {}- if input_tokens := usage_dict.get("prompt_tokens") or usage_dict.get("input_tokens"):- nb_tokens[TokenCategory.INPUT] = input_tokens- if output_tokens := usage_dict.get("completion_tokens") or usage_dict.get("output_tokens"):- nb_tokens[TokenCategory.OUTPUT] = output_tokens- fetch_tokens_usage.nb_tokens_by_category = nb_tokens-- def _is_retryable_portkey_error(self, exc: BaseException) -> bool:- if isinstance(exc, portkey_exceptions.NotFoundError):- msg = str(exc).lower()- return "specified deployment could not be found" in msg- return False-- def _log_retry(self, retry_state: RetryCallState) -> None:- """Called before sleeping between retries."""- if not retry_state.outcome:- log.error("Tenacity retry state outcome is None")- return- exc = retry_state.outcome.exception()- attempt = retry_state.attempt_number- wait_duration = retry_state.next_action.sleep if retry_state.next_action else 0.0- log.dev(f"{self.__class__.__name__} retry #{attempt} for fetch due to '{type(exc).__name__}'.")- log.verbose(f"Wait duration before next attempt: {wait_duration:.4f}s")diff --git a/pipelex/plugins/gateway/gateway_protocols.py b/pipelex/plugins/gateway/gateway_protocols.pyindex 4799780fa..c91e4910f 100644--- a/pipelex/plugins/gateway/gateway_protocols.py+++ b/pipelex/plugins/gateway/gateway_protocols.py@@ -7,6 +7,7 @@ class GatewayExtractProtocol(StrEnum):MISTRAL_DOC_AI = "mistral-doc-ai"AZURE_DOC_INTEL = "azure-doc-intel"DEEPSEEK_OCR = "deepseek-ocr"+ LINKUP_FETCH = "linkup-fetch"@classmethoddef make_from_model_handle(cls, model_handle: str) -> GatewayExtractProtocol:@@ -17,6 +18,8 @@ class GatewayExtractProtocol(StrEnum):return cls.AZURE_DOC_INTELcase "deepseek-ocr":return cls.DEEPSEEK_OCR+ case "linkup-fetch":+ return cls.LINKUP_FETCHcase _:msg = f"Invalid model ID: {model_handle}"raise ValueError(msg)diff --git a/pipelex/plugins/gateway/gateway_search_schemas.py b/pipelex/plugins/gateway/gateway_search_schemas.pyindex adec04b8c..395f348a4 100644--- a/pipelex/plugins/gateway/gateway_search_schemas.py+++ b/pipelex/plugins/gateway/gateway_search_schemas.py@@ -26,3 +26,18 @@ class GatewayFetchRequestParams(BaseModel):render_js: bool | None = Field(default=None, description="Whether to render JavaScript before fetching")extract_images: bool | None = Field(default=None, description="Whether to extract images from the page")timeout: float | None = Field(default=None, description="Request timeout in seconds")+++class GatewayFetchImageResponse(BaseModel):+ """An image extracted from a fetched page."""++ url: str = Field(description="URL of the image")+ alt: str | None = Field(default=None, description="Alt text of the image")+++class GatewayFetchResultResponse(BaseModel):+ """Response from a web page fetch."""++ markdown: str | None = Field(default=None, description="Page content as markdown")+ images: list[GatewayFetchImageResponse] | None = Field(default=None, description="Extracted images")+ raw_html: str | None = Field(default=None, description="Raw HTML of the page")diff --git a/pipelex/plugins/gateway/gateway_search_worker.py b/pipelex/plugins/gateway/gateway_search_worker.pyindex af9ce074b..e2b5b38ef 100644--- a/pipelex/plugins/gateway/gateway_search_worker.py+++ b/pipelex/plugins/gateway/gateway_search_worker.py@@ -12,18 +12,16 @@ from pipelex.cogt.exceptions import SdkTypeErrorfrom pipelex.cogt.model_backends.model_spec import InferenceModelSpecfrom pipelex.cogt.search.search_depth import SearchDepthfrom pipelex.cogt.search.search_job import SearchJob-from pipelex.cogt.search.search_job_factory import SearchJobFactory-from pipelex.cogt.search.search_setting import SearchSettingfrom pipelex.cogt.search.search_worker_abstract import SearchWorkerAbstractfrom pipelex.cogt.usage.token_category import NbTokensByCategoryDict, TokenCategoryfrom pipelex.config import get_configfrom pipelex.core.stuffs.search_result_content import SearchResultContent, SearchSourceContent-from pipelex.pipeline.job_metadata import JobMetadata, UnitJobIdfrom pipelex.plugins.gateway.gateway_deck import GatewayDeckfrom pipelex.plugins.gateway.gateway_exceptions import GatewaySearchResponseErrorfrom pipelex.plugins.gateway.gateway_factory import GatewayFactoryfrom pipelex.plugins.gateway.gateway_search_schemas import GatewaySearchRequestParamsfrom pipelex.reporting.reporting_protocol import ReportingProtocol+from pipelex.tools.typing.pydantic_utils import BaseModelTypeVarclass GatewaySearchWorker(SearchWorkerAbstract):@@ -35,14 +33,13 @@ class GatewaySearchWorker(SearchWorkerAbstract):inference_model: InferenceModelSpec,reporting_delegate: ReportingProtocol | None = None,):- super().__init__(reporting_delegate=reporting_delegate)+ super().__init__(inference_model=inference_model, reporting_delegate=reporting_delegate)if not isinstance(sdk_instance, AsyncPortkey):msg = f"Provided sdk_instance for {self.__class__.__name__} is not of type AsyncPortkey: it's a '{type(sdk_instance)}'"raise SdkTypeError(msg)self.portkey_client: AsyncPortkey = sdk_instance- self.inference_model = inference_modelself._tenacity_config = get_config().cogt.tenacity_configdef _make_retryer(self) -> AsyncRetrying:@@ -60,30 +57,23 @@ class GatewaySearchWorker(SearchWorkerAbstract):)@override- async def search_sourced_answer(+ async def _search_sourced_answer(self,- query: str,- search_setting: SearchSetting,- job_metadata: JobMetadata,- include_domains: list[str] | None = None,- exclude_domains: list[str] | None = None,- from_date: str | None = None,- to_date: str | None = None,+ search_job: SearchJob,) -> SearchResultContent:- job_metadata.unit_job_id = UnitJobId.SEARCH_SOURCED_ANSWER- search_job = SearchJobFactory.make_search_job(job_metadata=job_metadata)- search_job.search_job_before_start(inference_model=self.inference_model)+ job_params = search_job.job_params+ search_setting = job_params.search_settingparams = GatewaySearchRequestParams(- query=query,+ query=search_job.query,depth=SearchDepth(self.inference_model.model_id.rsplit("/", 1)[-1]),include_images=search_setting.include_images,include_inline_citations=search_setting.include_inline_citations,max_results=search_setting.max_results,- include_domains=include_domains,- exclude_domains=exclude_domains,- from_date=from_date,- to_date=to_date,+ include_domains=job_params.include_domains,+ exclude_domains=job_params.exclude_domains,+ from_date=job_params.from_date,+ to_date=job_params.to_date,)response = await self._call_relay(@@ -92,9 +82,6 @@ class GatewaySearchWorker(SearchWorkerAbstract):)self._extract_usage(response=response, search_job=search_job)- search_job.search_job_after_complete()- if self.reporting_delegate:- self.reporting_delegate.report_inference_job(inference_job=search_job)content_str = self._extract_content(response)result_dict: dict[str, Any] = json.loads(content_str)@@ -114,39 +101,27 @@ class GatewaySearchWorker(SearchWorkerAbstract):)@override- async def search_structured(+ async def _search_structured(self,- query: str,- search_setting: SearchSetting,- output_schema: type,- job_metadata: JobMetadata,- include_domains: list[str] | None = None,- exclude_domains: list[str] | None = None,- from_date: str | None = None,- to_date: str | None = None,+ search_job: SearchJob,+ schema: type[BaseModelTypeVar],) -> dict[str, Any]:- job_metadata.unit_job_id = UnitJobId.SEARCH_STRUCTURED- search_job = SearchJobFactory.make_search_job(job_metadata=job_metadata)- search_job.search_job_before_start(inference_model=self.inference_model)+ job_params = search_job.job_params+ search_setting = job_params.search_setting# Convert Pydantic model class to JSON Schema dict for the relay- schema_dict: dict[str, Any]- if hasattr(output_schema, "model_json_schema"):- schema_dict = cast("dict[str, Any]", output_schema.model_json_schema()) # pyright: ignore[reportUnknownMemberType]- else:- msg = f"output_schema must be a Pydantic model class with model_json_schema(), got {output_schema}"- raise SdkTypeError(msg)+ schema_dict: dict[str, Any] = schema.model_json_schema()params = GatewaySearchRequestParams(- query=query,+ query=search_job.query,depth=SearchDepth(self.inference_model.model_id.rsplit("/", 1)[-1]),include_images=search_setting.include_images,include_inline_citations=search_setting.include_inline_citations,max_results=search_setting.max_results,- include_domains=include_domains,- exclude_domains=exclude_domains,- from_date=from_date,- to_date=to_date,+ include_domains=job_params.include_domains,+ exclude_domains=job_params.exclude_domains,+ from_date=job_params.from_date,+ to_date=job_params.to_date,output_schema=schema_dict,)@@ -156,9 +131,6 @@ class GatewaySearchWorker(SearchWorkerAbstract):)self._extract_usage(response=response, search_job=search_job)- search_job.search_job_after_complete()- if self.reporting_delegate:- self.reporting_delegate.report_inference_job(inference_job=search_job)content_str = self._extract_content(response)result: dict[str, Any] = json.loads(content_str)diff --git a/pipelex/plugins/linkup/linkup_extract_worker.py b/pipelex/plugins/linkup/linkup_extract_worker.pynew file mode 100644index 000000000..911a1bd11--- /dev/null+++ b/pipelex/plugins/linkup/linkup_extract_worker.py@@ -0,0 +1,79 @@+from typing import Any++from linkup import LinkupClient, LinkupFetchResponse+from typing_extensions import override++from pipelex.cogt.extract.extract_job import ExtractJob+from pipelex.cogt.extract.extract_output import ExtractedImageFromPage, ExtractOutput, Page+from pipelex.cogt.extract.extract_worker_abstract import ExtractWorkerAbstract+from pipelex.cogt.model_backends.model_spec import InferenceModelSpec+from pipelex.cogt.usage.token_category import TokenCategory+from pipelex.hub import get_secrets_provider+from pipelex.reporting.reporting_protocol import ReportingProtocol+++class LinkupExtractWorker(ExtractWorkerAbstract):+ def __init__(+ self,+ extra_config: dict[str, Any],+ inference_model: InferenceModelSpec,+ reporting_delegate: ReportingProtocol | None = None,+ ) -> None:+ ExtractWorkerAbstract.__init__(+ self,+ extra_config=extra_config,+ inference_model=inference_model,+ reporting_delegate=reporting_delegate,+ )+ api_key = get_secrets_provider().get_secret(secret_id="LINKUP_API_KEY")+ self._linkup_client = LinkupClient(api_key=api_key)++ @override+ async def _extract_pages(+ self,+ extract_job: ExtractJob,+ ) -> ExtractOutput:+ document_uri = extract_job.extract_input.document_uri+ if not document_uri:+ msg = "LinkupExtractWorker requires a document_uri (web URL) in ExtractInput"+ raise ValueError(msg)++ job_params = extract_job.job_params+ extract_images = (job_params.max_nb_images or 0) != 0++ response: LinkupFetchResponse = await self._linkup_client.async_fetch(+ url=document_uri,+ render_js=job_params.render_js,+ include_raw_html=job_params.include_raw_html,+ extract_images=extract_images,+ )++ # Per-request cost model: costs are defined per million, so 1 request = 1_000_000+ if extract_tokens_usage := extract_job.job_report.extract_tokens_usage:+ extract_tokens_usage.nb_tokens_by_category = {+ TokenCategory.INPUT: 1_000_000,+ TokenCategory.OUTPUT: 1_000_000,+ }++ max_images = job_params.max_nb_images+ extracted_images: list[ExtractedImageFromPage] = []+ if response.images:+ for image in response.images:+ if max_images is not None and len(extracted_images) >= max_images:+ break+ extracted_images.append(+ ExtractedImageFromPage(+ size=None,+ actual_url=image.url,+ mime_type=None,+ caption=image.alt or None,+ )+ )++ page = Page(+ text=response.markdown,+ raw_html=response.raw_html,+ extracted_images=extracted_images,+ )++ return ExtractOutput(pages={0: page})diff --git a/pipelex/plugins/linkup/linkup_search_worker.py b/pipelex/plugins/linkup/linkup_search_worker.pynew file mode 100644index 000000000..5554fe6fe--- /dev/null+++ b/pipelex/plugins/linkup/linkup_search_worker.py@@ -0,0 +1,106 @@+from datetime import date+from typing import Any++from linkup import LinkupClient, LinkupSourcedAnswer+from typing_extensions import override++from pipelex.cogt.model_backends.model_spec import InferenceModelSpec+from pipelex.cogt.search.search_depth import SearchDepth+from pipelex.cogt.search.search_job import SearchJob+from pipelex.cogt.search.search_worker_abstract import SearchWorkerAbstract+from pipelex.cogt.usage.token_category import TokenCategory+from pipelex.core.stuffs.search_result_content import SearchResultContent, SearchSourceContent+from pipelex.hub import get_secrets_provider+from pipelex.reporting.reporting_protocol import ReportingProtocol+from pipelex.tools.typing.pydantic_utils import BaseModelTypeVar+++class LinkupSearchWorker(SearchWorkerAbstract):+ def __init__(+ self,+ inference_model: InferenceModelSpec,+ reporting_delegate: ReportingProtocol | None = None,+ ) -> None:+ SearchWorkerAbstract.__init__(self, inference_model=inference_model, reporting_delegate=reporting_delegate)+ api_key = get_secrets_provider().get_secret(secret_id="LINKUP_API_KEY")+ self._linkup_client = LinkupClient(api_key=api_key)++ def _parse_date(self, date_str: str | None) -> date | None:+ if date_str is None:+ return None+ return date.fromisoformat(date_str)++ @override+ async def _search_sourced_answer(+ self,+ search_job: SearchJob,+ ) -> SearchResultContent:+ job_params = search_job.job_params+ search_setting = job_params.search_setting++ depth_value = SearchDepth(self.inference_model.model_id)+ response: LinkupSourcedAnswer = await self._linkup_client.async_search(+ query=search_job.query,+ depth=depth_value, # type: ignore[arg-type] # pyright: ignore[reportArgumentType]+ output_type="sourcedAnswer",+ include_images=search_setting.include_images,+ include_inline_citations=search_setting.include_inline_citations,+ max_results=search_setting.max_results,+ include_domains=job_params.include_domains,+ exclude_domains=job_params.exclude_domains,+ from_date=self._parse_date(job_params.from_date),+ to_date=self._parse_date(job_params.to_date),+ )++ # Per-request cost model: costs are defined per million, so 1 request = 1_000_000+ if search_tokens_usage := search_job.job_report.search_tokens_usage:+ search_tokens_usage.nb_tokens_by_category = {TokenCategory.INPUT: 1_000_000, TokenCategory.OUTPUT: 1_000_000}++ sources: list[SearchSourceContent] = []+ for source in response.sources:+ sources.append(+ SearchSourceContent(+ name=source.name,+ url=source.url,+ snippet=source.snippet,+ )+ )++ return SearchResultContent(+ answer=response.answer,+ sources=sources,+ )++ @override+ async def _search_structured(+ self,+ search_job: SearchJob,+ schema: type[BaseModelTypeVar],+ ) -> dict[str, Any]:+ job_params = search_job.job_params+ search_setting = job_params.search_setting++ depth_value = SearchDepth(self.inference_model.model_id)+ response = await self._linkup_client.async_search(+ query=search_job.query,+ depth=depth_value, # type: ignore[arg-type] # pyright: ignore[reportArgumentType]+ output_type="structured",+ structured_output_schema=schema,+ include_images=search_setting.include_images,+ max_results=search_setting.max_results,+ include_domains=job_params.include_domains,+ exclude_domains=job_params.exclude_domains,+ from_date=self._parse_date(job_params.from_date),+ to_date=self._parse_date(job_params.to_date),+ include_sources=True,+ )++ # Per-request cost model: costs are defined per million, so 1 request = 1_000_000+ if search_tokens_usage := search_job.job_report.search_tokens_usage:+ search_tokens_usage.nb_tokens_by_category = {TokenCategory.INPUT: 1_000_000, TokenCategory.OUTPUT: 1_000_000}++ # If response is a Pydantic model, convert to dict+ if hasattr(response, "model_dump"):+ result: dict[str, Any] = response.model_dump()+ return result+ return dict(response) # type: ignore[arg-type]diff --git a/pipelex/plugins/linkup/linkup_worker.py b/pipelex/plugins/linkup/linkup_worker.pydeleted file mode 100644index 8afdd80ea..000000000--- a/pipelex/plugins/linkup/linkup_worker.py+++ /dev/null@@ -1,180 +0,0 @@-from datetime import date-from typing import Any--from linkup import LinkupClient, LinkupFetchResponse, LinkupSourcedAnswer-from typing_extensions import override--from pipelex.cogt.model_backends.model_spec import InferenceModelSpec-from pipelex.cogt.search.fetch_worker_abstract import FetchWorkerAbstract-from pipelex.cogt.search.search_depth import SearchDepth-from pipelex.cogt.search.search_job_factory import FetchJobFactory, SearchJobFactory-from pipelex.cogt.search.search_setting import SearchSetting-from pipelex.cogt.search.search_worker_abstract import SearchWorkerAbstract-from pipelex.cogt.usage.token_category import TokenCategory-from pipelex.core.stuffs.image_content import ImageContent-from pipelex.core.stuffs.search_result_content import SearchResultContent, SearchSourceContent-from pipelex.core.stuffs.text_and_images_content import TextAndImagesContent-from pipelex.core.stuffs.text_content import TextContent-from pipelex.hub import get_secrets_provider-from pipelex.pipeline.job_metadata import JobMetadata, UnitJobId-from pipelex.reporting.reporting_protocol import ReportingProtocol---class LinkupWorker(SearchWorkerAbstract, FetchWorkerAbstract):- def __init__(- self,- inference_model: InferenceModelSpec,- reporting_delegate: ReportingProtocol | None = None,- ) -> None:- SearchWorkerAbstract.__init__(self, reporting_delegate=reporting_delegate)- FetchWorkerAbstract.__init__(self, reporting_delegate=reporting_delegate)- self.inference_model = inference_model- api_key = get_secrets_provider().get_secret(secret_id="LINKUP_API_KEY")- self._linkup_client = LinkupClient(api_key=api_key)-- def _parse_date(self, date_str: str | None) -> date | None:- if date_str is None:- return None- return date.fromisoformat(date_str)-- @override- async def search_sourced_answer(- self,- query: str,- search_setting: SearchSetting,- job_metadata: JobMetadata,- include_domains: list[str] | None = None,- exclude_domains: list[str] | None = None,- from_date: str | None = None,- to_date: str | None = None,- ) -> SearchResultContent:- job_metadata.unit_job_id = UnitJobId.SEARCH_SOURCED_ANSWER- search_job = SearchJobFactory.make_search_job(job_metadata=job_metadata)- search_job.search_job_before_start(inference_model=self.inference_model)-- depth_value = SearchDepth(self.inference_model.model_id)- response: LinkupSourcedAnswer = await self._linkup_client.async_search(- query=query,- depth=depth_value, # type: ignore[arg-type] # pyright: ignore[reportArgumentType]- output_type="sourcedAnswer",- include_images=search_setting.include_images,- include_inline_citations=search_setting.include_inline_citations,- max_results=search_setting.max_results,- include_domains=include_domains,- exclude_domains=exclude_domains,- from_date=self._parse_date(from_date),- to_date=self._parse_date(to_date),- )-- # Per-request cost model: costs are defined per million, so 1 request = 1_000_000- if search_tokens_usage := search_job.job_report.search_tokens_usage:- search_tokens_usage.nb_tokens_by_category = {TokenCategory.INPUT: 1_000_000, TokenCategory.OUTPUT: 1_000_000}- search_job.search_job_after_complete()- if self.reporting_delegate:- self.reporting_delegate.report_inference_job(inference_job=search_job)-- sources: list[SearchSourceContent] = []- for source in response.sources:- sources.append(- SearchSourceContent(- name=source.name,- url=source.url,- snippet=source.snippet,- )- )-- return SearchResultContent(- answer=response.answer,- sources=sources,- )-- @override- async def search_structured(- self,- query: str,- search_setting: SearchSetting,- output_schema: type,- job_metadata: JobMetadata,- include_domains: list[str] | None = None,- exclude_domains: list[str] | None = None,- from_date: str | None = None,- to_date: str | None = None,- ) -> dict[str, Any]:- job_metadata.unit_job_id = UnitJobId.SEARCH_STRUCTURED- search_job = SearchJobFactory.make_search_job(job_metadata=job_metadata)- search_job.search_job_before_start(inference_model=self.inference_model)-- depth_value = SearchDepth(self.inference_model.model_id)- response = await self._linkup_client.async_search(- query=query,- depth=depth_value, # type: ignore[arg-type] # pyright: ignore[reportArgumentType]- output_type="structured",- structured_output_schema=output_schema,- include_images=search_setting.include_images,- max_results=search_setting.max_results,- include_domains=include_domains,- exclude_domains=exclude_domains,- from_date=self._parse_date(from_date),- to_date=self._parse_date(to_date),- include_sources=True,- )-- # Per-request cost model: costs are defined per million, so 1 request = 1_000_000- if search_tokens_usage := search_job.job_report.search_tokens_usage:- search_tokens_usage.nb_tokens_by_category = {TokenCategory.INPUT: 1_000_000, TokenCategory.OUTPUT: 1_000_000}- search_job.search_job_after_complete()- if self.reporting_delegate:- self.reporting_delegate.report_inference_job(inference_job=search_job)-- # If response is a Pydantic model, convert to dict- if hasattr(response, "model_dump"):- result: dict[str, Any] = response.model_dump()- return result- return dict(response) # type: ignore[arg-type]-- @override- async def fetch_url(- self,- url: str,- job_metadata: JobMetadata,- include_raw_html: bool | None = None,- render_js: bool | None = None,- extract_images: bool | None = None,- timeout: float | None = None, # noqa: ASYNC109- ) -> TextAndImagesContent:- job_metadata.unit_job_id = UnitJobId.FETCH_URL- fetch_job = FetchJobFactory.make_fetch_job(job_metadata=job_metadata)- fetch_job.fetch_job_before_start(inference_model=self.inference_model)-- response: LinkupFetchResponse = await self._linkup_client.async_fetch(- url=url,- include_raw_html=include_raw_html,- render_js=render_js,- extract_images=extract_images,- timeout=timeout,- )-- # Per-request cost model: costs are defined per million, so 1 request = 1_000_000- if fetch_tokens_usage := fetch_job.job_report.fetch_tokens_usage:- fetch_tokens_usage.nb_tokens_by_category = {TokenCategory.INPUT: 1_000_000, TokenCategory.OUTPUT: 1_000_000}- fetch_job.fetch_job_after_complete()- if self.reporting_delegate:- self.reporting_delegate.report_inference_job(inference_job=fetch_job)-- text = TextContent(text=response.markdown) if response.markdown else None-- images: list[ImageContent] | None = None- if response.images is not None:- images = [- ImageContent(- url=image.url,- caption=image.alt or None,- )- for image in response.images- ]-- return TextAndImagesContent(- text=text,- images=images or None,- raw_html=response.raw_html,- )diff --git a/pipelex/reporting/reporting_manager.py b/pipelex/reporting/reporting_manager.pyindex 2512dc127..765d29785 100644--- a/pipelex/reporting/reporting_manager.py+++ b/pipelex/reporting/reporting_manager.py@@ -12,8 +12,6 @@ from pipelex.cogt.img_gen.img_gen_report import ImgGenTokensUsagefrom pipelex.cogt.inference.inference_job_abstract import InferenceJobAbstractfrom pipelex.cogt.llm.llm_job import LLMJobfrom pipelex.cogt.llm.llm_report import LLMTokensUsage-from pipelex.cogt.search.fetch_job import FetchJob-from pipelex.cogt.search.fetch_report import FetchTokensUsagefrom pipelex.cogt.search.search_job import SearchJobfrom pipelex.cogt.search.search_report import SearchTokensUsagefrom pipelex.cogt.usage.cost_registry import CostRegistry@@ -26,7 +24,7 @@ from pipelex.tools.typing.pydantic_utils import empty_list_factory_ofif TYPE_CHECKING:from pathlib import Path-TokensUsage = LLMTokensUsage | ImgGenTokensUsage | ExtractTokensUsage | SearchTokensUsage | FetchTokensUsage+TokensUsage = LLMTokensUsage | ImgGenTokensUsage | ExtractTokensUsage | SearchTokensUsageUsageRegistryRoot = list[TokensUsage]@@ -124,20 +122,6 @@ class ReportingManager(ReportingProtocol):search_token_cost_report = CostRegistry.complete_cost_report(tokens_usage=search_tokens_usage)log.verbose(search_token_cost_report, title="Token Cost report")- def _report_fetch_job(self, fetch_job: FetchJob):- fetch_tokens_usage = fetch_job.job_report.fetch_tokens_usage-- if not fetch_tokens_usage:- log.warning("Fetch job has no fetch_tokens_usage")- return-- pipeline_run_id = fetch_job.job_metadata.pipeline_run_id- self._get_registry(pipeline_run_id).add_tokens_usage(fetch_tokens_usage)-- if self._reporting_config.is_log_costs_to_console:- fetch_token_cost_report = CostRegistry.complete_cost_report(tokens_usage=fetch_tokens_usage)- log.verbose(fetch_token_cost_report, title="Token Cost report")-############################################################# ReportingProtocol############################################################@@ -160,8 +144,6 @@ class ReportingManager(ReportingProtocol):self._report_extract_job(extract_job=inference_job)elif isinstance(inference_job, SearchJob):self._report_search_job(search_job=inference_job)- elif isinstance(inference_job, FetchJob):- self._report_fetch_job(fetch_job=inference_job)else:log.warning(f"ReportingManager does not support reporting for inference job type: {type(inference_job).__name__}")diff --git a/pipelex/system/pipelex_service/pipelex_details.py b/pipelex/system/pipelex_service/pipelex_details.pyindex 262e0920c..31a87604e 100644--- a/pipelex/system/pipelex_service/pipelex_details.py+++ b/pipelex/system/pipelex_service/pipelex_details.py@@ -2,7 +2,7 @@ from pipelex.tools.misc.hash_utils import hash_sha256class PipelexDetails:- REMOTE_CONFIG_URL = "https://pipelex-config.s3.eu-west-3.amazonaws.com/pipelex_remote_config_06.json"+ REMOTE_CONFIG_URL = "https://pipelex-config.s3.eu-west-3.amazonaws.com/pipelex_remote_config_07.json"PIPELEX_GATEWAY_API_KEY_VAR = "PIPELEX_GATEWAY_API_KEY"@classmethoddiff --git a/tests/cases/documents.py b/tests/cases/documents.pyindex 3efdf4c43..aaad5a295 100644--- a/tests/cases/documents.py+++ b/tests/cases/documents.py@@ -38,3 +38,12 @@ class DocumentTestCases:PDF_FILE_URL_1,PDF_FILE_URL_2,]++ # Web URLs+ WEB_URL_1 = "https://books.toscrape.com/catalogue/cravings-recipes-for-what-you-want-to-eat_589/index.html"+ WEB_URL_2 = "https://www.scrapethissite.com/pages/"++ WEB_URLS: ClassVar[list[str]] = [+ WEB_URL_1,+ WEB_URL_2,+ ]diff --git a/tests/integration/pipelex/cogt/test_data.py b/tests/integration/pipelex/cogt/test_data.pyindex 2c6d16001..91fcabcd5 100644--- a/tests/integration/pipelex/cogt/test_data.py+++ b/tests/integration/pipelex/cogt/test_data.py@@ -234,14 +234,6 @@ class SerDeTestLLMCases:]-class FetchTestCases:- """Test cases for fetch integration tests."""-- FETCH_URLS: ClassVar[list[tuple[str, str]]] = [ # topic, url- ("Wikipedia Python", "https://en.wikipedia.org/wiki/Python_(programming_language)"),- ]--class SearchTestCases:"""Test cases for search integration tests."""diff --git a/tests/integration/pipelex/cogt/test_extract.py b/tests/integration/pipelex/cogt/test_extract.pyindex 0e8969f66..58f741915 100644--- a/tests/integration/pipelex/cogt/test_extract.py+++ b/tests/integration/pipelex/cogt/test_extract.py@@ -123,6 +123,50 @@ class TestExtract:pretty_print(extract_output, title="Extract Output")assert extract_output.pages+ @pytest.mark.parametrize("url", DocumentTestCases.WEB_URLS)+ async def test_extract_web_url(+ self,+ generated_content_factory: GeneratedContentFactory,+ job_metadata: JobMetadata,+ extract_combo: ModelCombo,+ url: str,+ ):+ """Test web page extraction using Linkup fetch API."""+ extract_worker = get_extract_worker(extract_handle=extract_combo.handle)+ if not extract_worker.is_web_page_supported:+ msg = f"Web extraction is not supported for this extract worker: '{extract_worker.desc}'"+ pytest.skip(msg)+ extract_job_params = ExtractJobParams(+ max_nb_images=5,+ should_caption_images=False,+ should_include_page_views=False,+ page_views_dpi=None,+ image_min_size=None,+ render_js=False,+ include_raw_html=True,+ )+ extract_job = ExtractJobFactory.make_extract_job(+ extract_input=ExtractInput(document_uri=url),+ extract_job_params=extract_job_params,+ job_metadata=job_metadata,+ )+ extract_output = await extract_worker.extract_pages(extract_job=extract_job)+ assert extract_output.pages+ page = extract_output.pages[0]+ assert page.text, "Web extraction should return markdown text"+ pretty_print(page.text, title="Extracted markdown")+ if page.raw_html:+ pretty_print(page.raw_html[:500], title="Raw HTML (first 500 chars)")+ page_contents = await generated_content_factory.make_page_contents(+ primary_id=job_metadata.user_id,+ secondary_id=job_metadata.pipeline_run_id,+ extract_output=extract_output,+ )+ assert page_contents+ for page_index, page_content in enumerate(page_contents):+ pretty_print(page_content, title=f"Page {page_index}")+ get_report_delegate().generate_report()+@pytest.mark.parametrize("file_path", DocumentTestCases.DOCUMENT_FILE_PATHS)async def test_extract_image_save(self,diff --git a/tests/integration/pipelex/cogt/test_fetch.py b/tests/integration/pipelex/cogt/test_fetch.pydeleted file mode 100644index a470235c2..000000000--- a/tests/integration/pipelex/cogt/test_fetch.py+++ /dev/null@@ -1,64 +0,0 @@-import pytest--from pipelex import pretty_print-from pipelex.cogt.model_backends.model_type import ModelType-from pipelex.cogt.search.search_worker_factory import SearchWorkerFactory-from pipelex.hub import get_model_deck, get_report_delegate-from pipelex.pipeline.job_metadata import JobMetadata-from tests.integration.pipelex.cogt.test_data import FetchTestCases-from tests.integration.pipelex.fixtures.model_combo import ModelCombo--[email protected](loop_scope="class")-class TestFetch:- @pytest.mark.parametrize(- ("topic", "url"),- FetchTestCases.FETCH_URLS,- )- async def test_fetch_url(self, search_combo: ModelCombo, job_metadata: JobMetadata, topic: str, url: str) -> None:- """Verify that fetch_url returns non-empty text content."""- model_deck = get_model_deck()- inference_model = model_deck.get_required_inference_model(model_handle=search_combo.handle, model_type=ModelType.SEARCH)- worker = SearchWorkerFactory.make_fetch_worker(inference_model=inference_model)- result = await worker.fetch_url(url=url, job_metadata=job_metadata)- pretty_print(result, title=f"Fetch Result ({topic})")- assert result.text is not None, "Expected non-empty text content"- assert len(result.text.text) > 0, "Expected text to have content"- assert result.raw_html is None, "Expected raw_html to be None when not requested"- assert result.images is None, "Expected images to be None when not requested"- get_report_delegate().generate_report()-- @pytest.mark.parametrize(- ("topic", "url"),- FetchTestCases.FETCH_URLS,- )- async def test_fetch_url_with_raw_html(self, search_combo: ModelCombo, job_metadata: JobMetadata, topic: str, url: str) -> None:- """Verify that fetch_url returns raw HTML when requested."""- model_deck = get_model_deck()- inference_model = model_deck.get_required_inference_model(model_handle=search_combo.handle, model_type=ModelType.SEARCH)- worker = SearchWorkerFactory.make_fetch_worker(inference_model=inference_model)- result = await worker.fetch_url(url=url, job_metadata=job_metadata, include_raw_html=True)- pretty_print(result, title=f"Fetch Result with HTML ({topic})")- assert result.text is not None, "Expected non-empty text content"- assert result.raw_html is not None, "Expected raw_html to be present when requested"- assert len(result.raw_html) > 0, "Expected raw_html to have content"- get_report_delegate().generate_report()-- @pytest.mark.parametrize(- ("topic", "url"),- FetchTestCases.FETCH_URLS,- )- async def test_fetch_url_with_images(self, search_combo: ModelCombo, job_metadata: JobMetadata, topic: str, url: str) -> None:- """Verify that fetch_url returns images when requested."""- model_deck = get_model_deck()- inference_model = model_deck.get_required_inference_model(model_handle=search_combo.handle, model_type=ModelType.SEARCH)- worker = SearchWorkerFactory.make_fetch_worker(inference_model=inference_model)- result = await worker.fetch_url(url=url, job_metadata=job_metadata, extract_images=True)- pretty_print(result, title=f"Fetch Result with Images ({topic})")- assert result.text is not None, "Expected non-empty text content"- assert result.images is not None, "Expected images to be present when requested"- assert isinstance(result.images, list), "Expected images to be a list"- assert len(result.images) > 0, "Expected images list to be non-empty"- get_report_delegate().generate_report()diff --git a/tests/integration/pipelex/cogt/test_search.py b/tests/integration/pipelex/cogt/test_search.pyindex 4b1847e70..c32240a1a 100644--- a/tests/integration/pipelex/cogt/test_search.py+++ b/tests/integration/pipelex/cogt/test_search.py@@ -3,6 +3,7 @@ from pydantic import BaseModelfrom pipelex import pretty_printfrom pipelex.cogt.model_backends.model_type import ModelType+from pipelex.cogt.search.search_job_factory import SearchJobFactoryfrom pipelex.cogt.search.search_setting import SearchSettingfrom pipelex.cogt.search.search_worker_factory import SearchWorkerFactoryfrom pipelex.hub import get_model_deck, get_report_delegate@@ -34,11 +35,12 @@ class TestSearch:model=search_combo.handle,max_results=2,)- result = await worker.search_sourced_answer(+ search_job = SearchJobFactory.make_search_job(query=query,search_setting=search_setting,job_metadata=job_metadata,)+ result = await worker.search_sourced_answer(search_job=search_job)pretty_print(result, title=f"Sourced Answer Result ({topic})")assert result.answer, "Expected a non-empty answer"assert len(result.sources) > 0, "Expected at least one source"@@ -59,12 +61,12 @@ class TestSearch:search_setting = SearchSetting(model=search_combo.handle,)- result = await worker.search_structured(+ search_job = SearchJobFactory.make_search_job(query=query,search_setting=search_setting,- output_schema=TopicSummary,job_metadata=job_metadata,)+ result = await worker.search_structured(search_job=search_job, schema=TopicSummary)pretty_print(result, title=f"Structured Search Result ({topic})")assert isinstance(result, dict), "Expected a dict result"assert "data" in result, "Expected 'data' key in result"