Skip to content

Build AI methods with your coding agent, then run them anywhere: as an MCP for chatbots, as a webapp for people, or via API for your software.

Linkup Fetch as Extract model (#766)

Louis Choquelc5833b04f2b5
diff --git a/.claude/skills/add-model/SKILL.md b/.claude/skills/add-model/SKILL.md
index 670ad2dad..d6b05d8fe 100644
--- a/.claude/skills/add-model/SKILL.md
+++ b/.claude/skills/add-model/SKILL.md
@@ -23,9 +23,8 @@ and must be done in order to keep everything consistent.
2. **Add to backend TOML(s)** — the model spec in each relevant backend config
3. **Sync kit configs** — propagate `.pipelex/` changes into `pipelex/kit/configs/`
4. **Add to test profile collections** — so the model is available for test selection
-5. **Regenerate test fixtures** — refresh the generated model sets
-6. **Run inference tests** — verify the model works end-to-end with real API calls
-7. **Gateway (manual)** — the user handles this separately since it's remote config
+5. **Run inference tests** — verify the model works end-to-end with real API calls (fixtures are auto-regenerated when `PROF=` is passed)
+6. **Gateway (manual)** — the user handles this separately since it's remote config
## Step 1: Gather model information
@@ -206,20 +205,7 @@ If the model should be part of a named test profile (like `dev` or `coverage`),
add it there too. But usually the collections are enough — profiles reference
collections via `@collection_name` or glob patterns like `gpt-*`.
-## Step 5: Regenerate test fixtures
-
-Run the fixture regeneration to update the generated model sets:
-
-```bash
-make rtm
-```
-
-This runs `pipelex-dev preprocess-test-models --generate-fixtures --profile dev`
-and updates `tests/integration/pipelex/fixtures/_generated_model_sets.py`.
-
-Verify the generated file includes the new model if the active profile selects it.
-
-## Step 6: Run inference tests against the new model
+## Step 5: Run inference tests against the new model
Once the model is registered and fixtures are regenerated, help the user run the
integration tests to verify the model actually works end-to-end with real API
@@ -255,26 +241,26 @@ search_models = []
Adjust the `backends` and model list fields based on what type the model is
(`llm_models`, `img_gen_models`, `extract_models`, or `search_models`).
-### Regenerate fixtures and run tests for EACH backend
+### Run tests for EACH backend
-For each backend profile, regenerate fixtures and run the inference tests:
+For each backend profile, run the inference tests. **No need to call `make rtm`
+separately** — all inference test targets (`ti`, `tip`, `tl`, `te`, etc.)
+automatically regenerate fixtures when `PROF=` is passed on the command line.
```bash
# Test on first backend
-make rtm PROF=new_model_openai
-make ti PROF=new_model_openai TEST=TestLLMInference
+make test-inference-fast PROF=new_model_openai TEST=TestLLMInference
# Test on second backend
-make rtm PROF=new_model_azure
-make ti PROF=new_model_azure TEST=TestLLMInference
+make test-inference-fast PROF=new_model_azure TEST=TestLLMInference
```
Use the appropriate test class for the model type:
-- **LLM models**: `make ti PROF=<profile> TEST=TestLLMInference`
-- **Image gen models**: `make ti PROF=<profile> TEST=TestImageGeneration`
-- **Extract models**: `make ti PROF=<profile> TEST=TestExtract`
-- **Search models**: `make ti PROF=<profile> TEST=TestSearch`
+- **LLM models**: `make test-inference-fast PROF=<profile> TEST=TestLLMInference`
+- **Image gen models**: `make test-inference-fast PROF=<profile> TEST=TestImageGeneration`
+- **Extract models**: `make test-inference-fast PROF=<profile> TEST=TestExtract`
+- **Search models**: `make test-inference-fast PROF=<profile> TEST=TestSearch`
The `PROF` parameter selects the test profile (which controls which models are
tested), and `TEST` selects the test class or method to run.
@@ -283,20 +269,13 @@ These are live API calls, so they require valid API keys for the backend being
tested. If tests fail, check whether the issue is in the model config (wrong
model_id, missing capabilities) or in the API key / network setup.
-### Restore the default profile after testing
-
-Once all backends have been tested, regenerate fixtures back to the standard dev
-profile:
-
-```bash
-make rtm
-```
+### Clean up after testing
Ask the user if they want the temporary profiles removed from
`test_profiles_override.toml`. If they do, clean them up. If not, leave them
in place — the file is gitignored so it won't affect anyone else.
-## Step 7: Gateway (manual — user action required)
+## Step 6: Gateway (manual — user action required)
Remind the user:
@@ -310,7 +289,7 @@ The local file `.pipelex/inference/backends/pipelex_gateway.toml` only allows
overriding `sdk` and `structure_method` per model — you cannot define new models
in it.
-## Step 8: Verify everything works
+## Step 7: Verify everything works
Run the boot sequence test to make sure config loading succeeds:
@@ -334,9 +313,7 @@ Present this checklist to the user at the end so they can confirm everything:
- [ ] Model entry added to `<backend>.toml` in `.pipelex/inference/backends/`
- [ ] Kit configs synced (`make ukc` + `make ccs`)
- [ ] Model added to collections in `.pipelex-dev/test_profiles.toml`
-- [ ] Test fixtures regenerated (`make rtm`)
-- [ ] Inference tests pass on **every backend** the model was added to (one `make rtm PROF=...` + `make ti` per backend)
-- [ ] Default fixtures restored (`make rtm`)
+- [ ] Inference tests pass on **every backend** the model was added to (`make test-inference-fast PROF=... TEST=...` per backend — fixtures are auto-regenerated)
- [ ] Boot test passes (`make tb`)
- [ ] Quality checks pass (`make agent-check`)
- [ ] Gateway: user will add the model to the remote gateway config separately
diff --git a/.claude/skills/test-model/SKILL.md b/.claude/skills/test-model/SKILL.md
index c0864ab83..010f27bb9 100644
--- a/.claude/skills/test-model/SKILL.md
+++ b/.claude/skills/test-model/SKILL.md
@@ -97,26 +97,27 @@ Set the appropriate model list based on model type:
The override file may already contain profiles from previous testing sessions.
Don't remove existing profiles — just add or update the one you need.
-## Step 3: Regenerate fixtures and run tests
+## Step 3: Run tests
-Regenerate fixtures for the new profile, then run the appropriate test class.
+Run the appropriate test class. **No need to call `make rtm` separately** — all
+inference test targets automatically regenerate fixtures when `PROF=` is passed
+on the command line.
```bash
-make rtm PROF=<profile_name>
-make ti PROF=<profile_name> TEST=<TestClass>
+make test-inference-with-prints PROF=<profile_name> TEST=<TestClass>
```
Map model type to test class:
| Model type | Test class | Make shortcut |
|------------|-----------|---------------|
-| LLM | `TestLLMInference` | `make ti` |
-| Image gen | `TestImageGeneration` | `make ti` |
-| Extract | `TestExtract` | `make ti` |
-| Search | `TestSearch` | `make ti` |
+| LLM | `TestLLMInference` | `make test-inference-with-prints` |
+| Image gen | `TestImageGeneration` | `make test-inference-with-prints` |
+| Extract | `TestExtract` | `make test-inference-with-prints` |
+| Search | `TestSearch` | `make test-inference-with-prints` |
-All use `make ti` (test-inference) since all these test classes are marked with
-the `inference` pytest marker.
+All use `make test-inference-with-prints` since all these test classes are marked
+with the `inference` pytest marker.
For LLM models specifically, there are additional test classes you can run for
deeper coverage (only if the user wants thorough testing):
@@ -138,18 +139,7 @@ deeper coverage (only if the user wants thorough testing):
- Capability errors (e.g., vision not supported) → expected skips, not failures
- Timeout / rate limit → transient, suggest retrying
-## Step 5: Restore default fixtures
-
-After testing, regenerate fixtures back to the default dev profile:
-
-```bash
-make rtm
-```
-
-This is important — if you leave the fixtures pointing at the temporary profile,
-other tests will fail.
-
-## Step 6: Clean up (optional)
+## Step 5: Clean up (optional)
Ask the user if they want the temporary profile removed from
`test_profiles_override.toml`. If yes, remove it. If no, leave it — the file is
diff --git a/.pipelex-dev/test_profiles.toml b/.pipelex-dev/test_profiles.toml
index 90886c88b..ebdca2079 100644
--- a/.pipelex-dev/test_profiles.toml
+++ b/.pipelex-dev/test_profiles.toml
@@ -47,6 +47,7 @@ all = [
"scaleway",
"vertexai",
"xai",
+ "linkup",
"internal",
]
@@ -217,6 +218,9 @@ qwen = ["qwen-image"]
linkup = ["linkup-standard", "linkup-deep"]
[collections.extract]
+# --- Web Extraction Models ---
+from_web = ["linkup-fetch"]
+
# --- PDF Extraction Models ---
from_pdf = [
"pypdfium2-extract-pdf",
diff --git a/.pipelex/inference/backends/linkup.toml b/.pipelex/inference/backends/linkup.toml
index 184e324e9..73ba1d78c 100644
--- a/.pipelex/inference/backends/linkup.toml
+++ b/.pipelex/inference/backends/linkup.toml
@@ -35,3 +35,15 @@ model_id = "deep"
inputs = ["text"]
outputs = ["sourced-answers", "structured"]
costs = { input = 0.05, output = 0 }
+
+################################################################################
+# FETCH / EXTRACT MODELS
+################################################################################
+
+["linkup-fetch"]
+model_type = "text_extractor"
+sdk = "linkup_fetch"
+model_id = "fetch"
+inputs = ["web_page"]
+outputs = ["pages"]
+costs = { input = 0.005, output = 0 }
diff --git a/.pipelex/inference/deck/3_extract_deck.toml b/.pipelex/inference/deck/3_extract_deck.toml
index afb2e0396..eaf3065be 100644
--- a/.pipelex/inference/deck/3_extract_deck.toml
+++ b/.pipelex/inference/deck/3_extract_deck.toml
@@ -31,6 +31,7 @@ default-extract-document = "azure-document-intelligence"
default-extract-image = "azure-document-intelligence"
default-text-from-pdf = "pypdfium2-extract-pdf"
default-no-inference = "pypdfium2-extract-pdf"
+default-extract-web-page = "linkup-fetch"
####################################################################################################
# Extract Presets
diff --git a/pipelex/builder/pipe/pipe_extract_spec.py b/pipelex/builder/pipe/pipe_extract_spec.py
index 43c260c00..f67743cc7 100644
--- a/pipelex/builder/pipe/pipe_extract_spec.py
+++ b/pipelex/builder/pipe/pipe_extract_spec.py
@@ -47,6 +47,8 @@ class PipeExtractSpec(PipeSpec):
)
page_image_captions: bool | None = Field(default=None, description="Whether to generate captions for detected images using AI.")
page_views: bool | None = Field(default=None, description="Whether to include rendered page views in the output.")
+ render_js: bool | None = Field(default=None, description="Whether to render JavaScript before fetching (web extraction only).")
+ include_raw_html: bool | None = Field(default=None, description="Whether to include raw HTML in the response (web extraction only).")
@override
@field_validator("output", mode="before")
@@ -116,4 +118,6 @@ class PipeExtractSpec(PipeSpec):
page_image_captions=self.page_image_captions,
page_views=self.page_views,
page_views_dpi=None,
+ render_js=self.render_js,
+ include_raw_html=self.include_raw_html,
)
diff --git a/pipelex/builder/talents/extract_talent.py b/pipelex/builder/talents/extract_talent.py
index f2f8a39af..446adf7c3 100644
--- a/pipelex/builder/talents/extract_talent.py
+++ b/pipelex/builder/talents/extract_talent.py
@@ -5,3 +5,4 @@ class ExtractTalent(StrEnum):
PDF_BASIC_TEXT_EXTRACTOR = "pdf-basic-text-extractor"
IMAGE_TEXT_EXTRACTOR = "image-text-extractor"
FULL_DOCUMENT_EXTRACTOR = "full-document-extractor"
+ WEB_PAGE_EXTRACTOR = "web-page-extractor"
diff --git a/pipelex/cogt/content_generation/content_generator_dry.py b/pipelex/cogt/content_generation/content_generator_dry.py
index 3687a3a62..983285462 100644
--- a/pipelex/cogt/content_generation/content_generator_dry.py
+++ b/pipelex/cogt/content_generation/content_generator_dry.py
@@ -182,6 +182,7 @@ class ContentGeneratorDry(ContentGeneratorProtocol):
text_and_images=TextAndImagesContent(
text=TextContent(text=page.text) if page.text else None,
images=page_images,
+ raw_html=page.raw_html,
),
)
)
diff --git a/pipelex/cogt/content_generation/generated_content_factory.py b/pipelex/cogt/content_generation/generated_content_factory.py
index 869074a84..98353c9e0 100644
--- a/pipelex/cogt/content_generation/generated_content_factory.py
+++ b/pipelex/cogt/content_generation/generated_content_factory.py
@@ -1,6 +1,9 @@
import base64
import hashlib
+import httpx
+
+from pipelex import log
from pipelex.cogt.content_generation.exceptions import NeitherUrlNorDataError
from pipelex.cogt.extract.extract_output import ExtractOutput
from pipelex.cogt.image.generated_image import GeneratedImageRawDetails
@@ -125,16 +128,21 @@ class GeneratedContentFactory:
public_url: str | None
if is_remote_url and get_config().pipelex.storage_config.is_fetch_remote_content_enabled:
- actual_bytes = await self._fetch_remote_content(url=url)
- storage_key = self._build_storage_key(
- primary_id=primary_id,
- secondary_id=secondary_id,
- data=actual_bytes,
- mime_type=mime_type,
- image_format=image_format,
- )
- url = await self.storage_provider.store(data=actual_bytes, key=storage_key)
- public_url = await self.storage_provider.public_url(uri=url)
+ try:
+ actual_bytes = await self._fetch_remote_content(url=url)
+ except (httpx.HTTPStatusError, httpx.RequestError) as exc:
+ log.warning(f"Failed to fetch a remote image: {exc}")
+ public_url = url
+ else:
+ storage_key = self._build_storage_key(
+ primary_id=primary_id,
+ secondary_id=secondary_id,
+ data=actual_bytes,
+ mime_type=mime_type,
+ image_format=image_format,
+ )
+ url = await self.storage_provider.store(data=actual_bytes, key=storage_key)
+ public_url = await self.storage_provider.public_url(uri=url)
elif not is_remote_url:
public_url = await self.storage_provider.public_url(uri=url)
else:
@@ -170,6 +178,7 @@ class GeneratedContentFactory:
text_and_images=TextAndImagesContent(
text=TextContent(text=page.text) if page.text else None,
images=page_images,
+ raw_html=page.raw_html,
),
)
)
diff --git a/pipelex/cogt/extract/extract_job_components.py b/pipelex/cogt/extract/extract_job_components.py
index cf44c70ed..a29729f78 100644
--- a/pipelex/cogt/extract/extract_job_components.py
+++ b/pipelex/cogt/extract/extract_job_components.py
@@ -23,6 +23,8 @@ class ExtractJobParams(BaseModel):
should_caption_images: bool
should_include_page_views: bool
page_views_dpi: int | None
+ render_js: bool | None = None
+ include_raw_html: bool | None = None
@classmethod
def make_default_extract_job_params(cls) -> "ExtractJobParams":
@@ -32,6 +34,8 @@ class ExtractJobParams(BaseModel):
image_min_size=None,
should_include_page_views=False,
page_views_dpi=None,
+ render_js=None,
+ include_raw_html=None,
)
diff --git a/pipelex/cogt/extract/extract_output.py b/pipelex/cogt/extract/extract_output.py
index b21f1cece..554d3251c 100644
--- a/pipelex/cogt/extract/extract_output.py
+++ b/pipelex/cogt/extract/extract_output.py
@@ -11,6 +11,7 @@ class ExtractedImageFromPage(GeneratedImageRawDetails):
class Page(CustomBaseModel):
text: str | None = None
+ raw_html: str | None = None
extracted_images: list[ExtractedImageFromPage] = Field(default_factory=empty_list_factory_of(ExtractedImageFromPage))
diff --git a/pipelex/cogt/extract/extract_worker_abstract.py b/pipelex/cogt/extract/extract_worker_abstract.py
index cacfe6450..7535fe91a 100644
--- a/pipelex/cogt/extract/extract_worker_abstract.py
+++ b/pipelex/cogt/extract/extract_worker_abstract.py
@@ -38,6 +38,10 @@ class ExtractWorkerAbstract(InferenceWorkerAbstract):
def is_pdf_supported(self) -> bool:
return self.inference_model.is_pdf_supported_for_extract
+ @property
+ def is_web_page_supported(self) -> bool:
+ return self.inference_model.is_web_page_supported_for_extract
+
@property
def is_image_supported(self) -> bool:
return self.inference_model.is_image_supported_for_extract
@@ -54,8 +58,8 @@ class ExtractWorkerAbstract(InferenceWorkerAbstract):
msg = f"Extract engine '{self.inference_model.tag}' does not support image extraction."
raise ExtractCapabilityError(msg)
elif extract_input.document_uri:
- if not self.inference_model.is_pdf_supported_for_extract:
- msg = f"Extract engine '{self.inference_model.tag}' does not support PDF extraction."
+ if not (self.inference_model.is_pdf_supported_for_extract or self.inference_model.is_web_page_supported_for_extract):
+ msg = f"Extract engine '{self.inference_model.tag}' does not support document extraction."
raise ExtractCapabilityError(msg)
if extract_job.job_params.should_caption_images:
if not self.inference_model.is_caption_supported_for_extract:
diff --git a/pipelex/cogt/extract/extract_worker_factory.py b/pipelex/cogt/extract/extract_worker_factory.py
index 6da66aa55..c6b225d4f 100644
--- a/pipelex/cogt/extract/extract_worker_factory.py
+++ b/pipelex/cogt/extract/extract_worker_factory.py
@@ -93,6 +93,24 @@ class ExtractWorkerFactory:
inference_model=inference_model,
reporting_delegate=reporting_delegate,
)
+ case "linkup_fetch":
+ if importlib.util.find_spec("linkup") is None:
+ lib_name = "linkup"
+ lib_extra_name = "linkup"
+ msg = "The linkup SDK is required in order to use Linkup Fetch extraction models."
+ raise MissingDependencyError(
+ lib_name,
+ lib_extra_name,
+ msg,
+ )
+
+ from pipelex.plugins.linkup.linkup_extract_worker import LinkupExtractWorker # noqa: PLC0415
+
+ extract_worker = LinkupExtractWorker(
+ extra_config=backend.extra_config,
+ inference_model=inference_model,
+ reporting_delegate=reporting_delegate,
+ )
case _:
msg = f"Plugin '{plugin}' is not supported"
raise NotImplementedError(msg)
diff --git a/pipelex/cogt/image/generated_image.py b/pipelex/cogt/image/generated_image.py
index 8bb096f7f..b147d3b27 100644
--- a/pipelex/cogt/image/generated_image.py
+++ b/pipelex/cogt/image/generated_image.py
@@ -38,7 +38,7 @@ class GeneratedImageRawDetails(CustomBaseModel):
return None
@model_validator(mode="after")
- def validate_mime_type_or_image_format(self) -> Self:
+ def validate_mime_type_or_image_format_or_actual_url(self) -> Self:
if self.mime_type:
ImageFormat.raise_if_unsupported_mime_type(self.mime_type)
elif self.actual_url_or_prefixed_base64 and (
@@ -48,6 +48,10 @@ class GeneratedImageRawDetails(CustomBaseModel):
ImageFormat.raise_if_unsupported_mime_type(base64_extracted_mime_type)
self.mime_type = base64_extracted_mime_type
self.base64_str = base64_str
+ elif self.actual_url:
+ # When only a URL is provided (e.g. from web extraction), the mime type
+ # will be determined when the image is actually downloaded.
+ pass
elif self.image_format is None:
msg = "Either mime_type or image_format must be provided"
raise ValueError(msg)
diff --git a/pipelex/cogt/model_backends/model_spec.py b/pipelex/cogt/model_backends/model_spec.py
index 414630059..0b77807a4 100644
--- a/pipelex/cogt/model_backends/model_spec.py
+++ b/pipelex/cogt/model_backends/model_spec.py
@@ -60,6 +60,10 @@ class InferenceModelSpec(ConfigModel):
def is_image_supported_for_extract(self) -> bool:
return "image" in self.inputs
+ @property
+ def is_web_page_supported_for_extract(self) -> bool:
+ return "web_page" in self.inputs
+
@property
def is_caption_supported_for_extract(self) -> bool:
return "captions" in self.outputs
diff --git a/pipelex/cogt/search/fetch_job.py b/pipelex/cogt/search/fetch_job.py
deleted file mode 100644
index 1beac53b7..000000000
--- a/pipelex/cogt/search/fetch_job.py
+++ /dev/null
@@ -1,35 +0,0 @@
-from datetime import datetime
-
-from pydantic import BaseModel
-from typing_extensions import override
-
-from pipelex.cogt.inference.inference_job_abstract import InferenceJobAbstract
-from pipelex.cogt.model_backends.model_spec import InferenceModelSpec
-from pipelex.cogt.search.fetch_report import FetchTokensUsage
-
-
-class FetchJobReport(BaseModel):
- fetch_tokens_usage: FetchTokensUsage | None = None
-
-
-class FetchJob(InferenceJobAbstract):
- job_report: FetchJobReport = FetchJobReport()
-
- @override
- def validate_before_execution(self):
- pass
-
- def fetch_job_before_start(self, inference_model: InferenceModelSpec):
- self.job_metadata.started_at = datetime.now()
-
- self.job_report = FetchJobReport()
- self.job_report.fetch_tokens_usage = FetchTokensUsage(
- job_metadata=self.job_metadata,
- inference_model_name=inference_model.name,
- unit_costs=inference_model.costs,
- inference_model_id=inference_model.model_id,
- nb_tokens_by_category={},
- )
-
- def fetch_job_after_complete(self):
- self.job_metadata.completed_at = datetime.now()
diff --git a/pipelex/cogt/search/fetch_report.py b/pipelex/cogt/search/fetch_report.py
deleted file mode 100644
index 0a68e8287..000000000
--- a/pipelex/cogt/search/fetch_report.py
+++ /dev/null
@@ -1,72 +0,0 @@
-from typing import Any
-
-from pydantic import BaseModel
-
-from pipelex.cogt.usage.cost_category import CostCategory, CostsByCategoryDict
-from pipelex.cogt.usage.token_category import NbTokensByCategoryDict, TokenCategory
-from pipelex.pipeline.job_metadata import JobMetadata
-from pipelex.types import StrEnum
-
-
-class FetchTokenCostReportField(StrEnum):
- MODEL_TYPE = "model_type"
- FETCH_NAME = "fetch_name"
- PLATFORM_FETCH_ID = "platform_fetch_id"
- NB_TOKENS_INPUT = "nb_tokens_input"
- NB_TOKENS_INPUT_CACHED = "nb_tokens_input_cached"
- NB_TOKENS_INPUT_NON_CACHED = "nb_tokens_input_non_cached"
- NB_TOKENS_INPUT_JOINED = "nb_tokens_input_joined"
- NB_TOKENS_OUTPUT = "nb_tokens_output"
- COST_INPUT_CACHED = "cost_input_cached"
- COST_INPUT_NON_CACHED = "cost_input_non_cached"
- COST_INPUT_JOINED = "cost_input_joined"
- COST_OUTPUT = "cost_output"
-
- @staticmethod
- def report_field_for_nb_tokens_by_category(token_category: TokenCategory) -> str:
- return f"nb_tokens_{token_category}"
-
- @staticmethod
- def report_field_for_cost_by_category(token_category: CostCategory) -> str:
- return f"cost_{token_category}"
-
-
-class FetchTokenCostReport(BaseModel):
- model_type: str = "fetch"
- job_metadata: JobMetadata
- inference_model_name: str
- platform_model_id: str
-
- nb_tokens_by_category: NbTokensByCategoryDict
- costs_by_token_category: CostsByCategoryDict
-
- def as_flat_dictionary(self) -> dict[str, Any]:
- the_dict: dict[str, Any] = {}
- dict_for_job_metadata = self.job_metadata.model_dump(serialize_as_any=True)
- the_dict.update(dict_for_job_metadata)
- dict_for_model: dict[str, Any] = {
- FetchTokenCostReportField.MODEL_TYPE: self.model_type,
- FetchTokenCostReportField.FETCH_NAME: self.inference_model_name,
- FetchTokenCostReportField.PLATFORM_FETCH_ID: self.platform_model_id,
- }
- the_dict.update(dict_for_model)
- dict_for_nb_tokens = {
- FetchTokenCostReportField.report_field_for_nb_tokens_by_category(token_category): nb_tokens
- for token_category, nb_tokens in self.nb_tokens_by_category.items()
- }
- the_dict.update(dict_for_nb_tokens)
- dict_for_costs = {
- FetchTokenCostReportField.report_field_for_cost_by_category(token_category): cost
- for token_category, cost in self.costs_by_token_category.items()
- }
- the_dict.update(dict_for_costs)
- return the_dict
-
-
-class FetchTokensUsage(BaseModel):
- model_type: str = "fetch"
- job_metadata: JobMetadata
- inference_model_name: str
- unit_costs: CostsByCategoryDict
- inference_model_id: str
- nb_tokens_by_category: NbTokensByCategoryDict
diff --git a/pipelex/cogt/search/fetch_worker_abstract.py b/pipelex/cogt/search/fetch_worker_abstract.py
deleted file mode 100644
index f19e387d7..000000000
--- a/pipelex/cogt/search/fetch_worker_abstract.py
+++ /dev/null
@@ -1,34 +0,0 @@
-from abc import ABC, abstractmethod
-
-from pipelex.core.stuffs.text_and_images_content import TextAndImagesContent
-from pipelex.pipeline.job_metadata import JobMetadata
-from pipelex.reporting.reporting_protocol import ReportingProtocol
-
-
-class FetchWorkerAbstract(ABC):
- def __init__(self, reporting_delegate: ReportingProtocol | None = None):
- self.reporting_delegate = reporting_delegate
-
- @abstractmethod
- async def fetch_url(
- self,
- url: str,
- job_metadata: JobMetadata,
- include_raw_html: bool | None = None,
- render_js: bool | None = None,
- extract_images: bool | None = None,
- timeout: float | None = None, # noqa: ASYNC109
- ) -> TextAndImagesContent:
- """Fetch and clean the content of a web page, returning text and optionally images.
-
- Args:
- url: The URL of the web page to fetch
- job_metadata: Job metadata for cost reporting
- include_raw_html: Whether to include the raw HTML of the webpage in the response
- render_js: Whether to render the JavaScript of the webpage before fetching
- extract_images: Whether to extract images from the webpage
- timeout: The timeout for the HTTP request, in seconds
-
- Returns:
- TextAndImagesContent with the fetched page content
- """
diff --git a/pipelex/cogt/search/search_job.py b/pipelex/cogt/search/search_job.py
index 7977987a5..678c7bd38 100644
--- a/pipelex/cogt/search/search_job.py
+++ b/pipelex/cogt/search/search_job.py
@@ -6,6 +6,15 @@ from typing_extensions import override
from pipelex.cogt.inference.inference_job_abstract import InferenceJobAbstract
from pipelex.cogt.model_backends.model_spec import InferenceModelSpec
from pipelex.cogt.search.search_report import SearchTokensUsage
+from pipelex.cogt.search.search_setting import SearchSetting
+
+
+class SearchJobParams(BaseModel):
+ search_setting: SearchSetting
+ include_domains: list[str] | None = None
+ exclude_domains: list[str] | None = None
+ from_date: str | None = None
+ to_date: str | None = None
class SearchJobReport(BaseModel):
@@ -13,6 +22,8 @@ class SearchJobReport(BaseModel):
class SearchJob(InferenceJobAbstract):
+ query: str
+ job_params: SearchJobParams
job_report: SearchJobReport = SearchJobReport()
@override
diff --git a/pipelex/cogt/search/search_job_factory.py b/pipelex/cogt/search/search_job_factory.py
index 6f5efeabe..d6899ade2 100644
--- a/pipelex/cogt/search/search_job_factory.py
+++ b/pipelex/cogt/search/search_job_factory.py
@@ -1,17 +1,30 @@
-from pipelex.cogt.search.fetch_job import FetchJob
-from pipelex.cogt.search.search_job import SearchJob
+from pipelex.cogt.search.search_job import SearchJob, SearchJobParams
+from pipelex.cogt.search.search_setting import SearchSetting
from pipelex.pipeline.job_metadata import JobCategory, JobMetadata
class SearchJobFactory:
@classmethod
- def make_search_job(cls, job_metadata: JobMetadata) -> SearchJob:
+ def make_search_job(
+ cls,
+ query: str,
+ search_setting: SearchSetting,
+ job_metadata: JobMetadata,
+ include_domains: list[str] | None = None,
+ exclude_domains: list[str] | None = None,
+ from_date: str | None = None,
+ to_date: str | None = None,
+ ) -> SearchJob:
job_metadata.job_category = JobCategory.SEARCH_JOB
- return SearchJob(job_metadata=job_metadata)
-
-
-class FetchJobFactory:
- @classmethod
- def make_fetch_job(cls, job_metadata: JobMetadata) -> FetchJob:
- job_metadata.job_category = JobCategory.FETCH_JOB
- return FetchJob(job_metadata=job_metadata)
+ job_params = SearchJobParams(
+ search_setting=search_setting,
+ include_domains=include_domains,
+ exclude_domains=exclude_domains,
+ from_date=from_date,
+ to_date=to_date,
+ )
+ return SearchJob(
+ job_metadata=job_metadata,
+ query=query,
+ job_params=job_params,
+ )
diff --git a/pipelex/cogt/search/search_worker_abstract.py b/pipelex/cogt/search/search_worker_abstract.py
index de70b2564..02cc83575 100644
--- a/pipelex/cogt/search/search_worker_abstract.py
+++ b/pipelex/cogt/search/search_worker_abstract.py
@@ -1,66 +1,80 @@
-from abc import ABC, abstractmethod
+from abc import abstractmethod
from typing import Any
-from pipelex.cogt.search.search_setting import SearchSetting
+from typing_extensions import override
+
+from pipelex import log
+from pipelex.cogt.inference.inference_worker_abstract import InferenceWorkerAbstract
+from pipelex.cogt.model_backends.model_spec import InferenceModelSpec
+from pipelex.cogt.search.search_job import SearchJob
from pipelex.core.stuffs.search_result_content import SearchResultContent
-from pipelex.pipeline.job_metadata import JobMetadata
+from pipelex.pipeline.job_metadata import UnitJobId
from pipelex.reporting.reporting_protocol import ReportingProtocol
+from pipelex.tools.typing.pydantic_utils import BaseModelTypeVar
+
+class SearchWorkerAbstract(InferenceWorkerAbstract):
+ def __init__(
+ self,
+ inference_model: InferenceModelSpec,
+ reporting_delegate: ReportingProtocol | None = None,
+ ):
+ InferenceWorkerAbstract.__init__(self, reporting_delegate=reporting_delegate)
+ self.inference_model = inference_model
-class SearchWorkerAbstract(ABC):
- def __init__(self, reporting_delegate: ReportingProtocol | None = None):
- self.reporting_delegate = reporting_delegate
+ @property
+ @override
+ def desc(self) -> str:
+ return f"Search using {self.inference_model.desc}"
- @abstractmethod
async def search_sourced_answer(
self,
- query: str,
- search_setting: SearchSetting,
- job_metadata: JobMetadata,
- include_domains: list[str] | None = None,
- exclude_domains: list[str] | None = None,
- from_date: str | None = None,
- to_date: str | None = None,
+ search_job: SearchJob,
) -> SearchResultContent:
- """Execute a search query and return a sourced answer with sources.
+ """Execute a search query and return a sourced answer with sources."""
+ log.dev(f"✨ {self.desc} ✨")
+ search_job.validate_before_execution()
+ search_job.job_metadata.unit_job_id = UnitJobId.SEARCH_SOURCED_ANSWER
+ search_job.search_job_before_start(inference_model=self.inference_model)
- Args:
- query: The search query text
- search_setting: Search configuration including model, depth, etc.
- job_metadata: Job metadata for cost reporting
- include_domains: Optional list of domains to restrict search to
- exclude_domains: Optional list of domains to exclude from search
- from_date: Optional start date filter (YYYY-MM-DD)
- to_date: Optional end date filter (YYYY-MM-DD)
+ result = await self._search_sourced_answer(search_job=search_job)
- Returns:
- SearchResultContent with answer and sources
- """
+ search_job.search_job_after_complete()
+ if self.reporting_delegate:
+ self.reporting_delegate.report_inference_job(inference_job=search_job)
+
+ return result
- @abstractmethod
async def search_structured(
self,
- query: str,
- search_setting: SearchSetting,
- output_schema: type,
- job_metadata: JobMetadata,
- include_domains: list[str] | None = None,
- exclude_domains: list[str] | None = None,
- from_date: str | None = None,
- to_date: str | None = None,
+ search_job: SearchJob,
+ schema: type[BaseModelTypeVar],
) -> dict[str, Any]:
- """Execute a search query and return structured data matching the schema.
+ """Execute a search query and return structured data matching the schema."""
+ log.dev(f"✨ {self.desc} ✨")
+ search_job.validate_before_execution()
+ search_job.job_metadata.unit_job_id = UnitJobId.SEARCH_STRUCTURED
+ search_job.search_job_before_start(inference_model=self.inference_model)
+
+ result = await self._search_structured(search_job=search_job, schema=schema)
+
+ search_job.search_job_after_complete()
+ if self.reporting_delegate:
+ self.reporting_delegate.report_inference_job(inference_job=search_job)
- Args:
- query: The search query text
- search_setting: Search configuration including model, depth, etc.
- output_schema: Pydantic model class defining the expected output structure
- job_metadata: Job metadata for cost reporting
- include_domains: Optional list of domains to restrict search to
- exclude_domains: Optional list of domains to exclude from search
- from_date: Optional start date filter (YYYY-MM-DD)
- to_date: Optional end date filter (YYYY-MM-DD)
+ return result
- Returns:
- Dictionary matching the output_schema structure
- """
+ @abstractmethod
+ async def _search_sourced_answer(
+ self,
+ search_job: SearchJob,
+ ) -> SearchResultContent:
+ pass
+
+ @abstractmethod
+ async def _search_structured(
+ self,
+ search_job: SearchJob,
+ schema: type[BaseModelTypeVar],
+ ) -> dict[str, Any]:
+ pass
diff --git a/pipelex/cogt/search/search_worker_factory.py b/pipelex/cogt/search/search_worker_factory.py
index 811435198..dc83690fa 100644
--- a/pipelex/cogt/search/search_worker_factory.py
+++ b/pipelex/cogt/search/search_worker_factory.py
@@ -1,5 +1,4 @@
from pipelex.cogt.model_backends.model_spec import InferenceModelSpec
-from pipelex.cogt.search.fetch_worker_abstract import FetchWorkerAbstract
from pipelex.cogt.search.search_worker_abstract import SearchWorkerAbstract
from pipelex.hub import get_models_manager, get_plugin_manager, get_report_delegate
from pipelex.plugins.plugin import Plugin
@@ -27,9 +26,9 @@ class SearchWorkerFactory:
search_worker: SearchWorkerAbstract
match plugin.sdk:
case "linkup":
- from pipelex.plugins.linkup.linkup_worker import LinkupWorker # noqa: PLC0415
+ from pipelex.plugins.linkup.linkup_search_worker import LinkupSearchWorker # noqa: PLC0415
- search_worker = LinkupWorker(inference_model=inference_model, reporting_delegate=get_report_delegate())
+ search_worker = LinkupSearchWorker(inference_model=inference_model, reporting_delegate=get_report_delegate())
case "gateway_search":
from pipelex.plugins.gateway.gateway_factory import GatewayFactory # noqa: PLC0415
from pipelex.plugins.gateway.gateway_search_worker import GatewaySearchWorker # noqa: PLC0415
@@ -46,44 +45,3 @@ class SearchWorkerFactory:
raise NotImplementedError(msg)
return search_worker
-
- @classmethod
- def make_fetch_worker(
- cls,
- inference_model: InferenceModelSpec,
- ) -> FetchWorkerAbstract:
- """Create a fetch worker for the given inference model.
-
- Discriminates on plugin.sdk to select the appropriate implementation.
-
- Args:
- inference_model: The model spec from the backend configuration.
-
- Returns:
- A FetchWorkerAbstract instance.
- """
- plugin = Plugin.make_for_inference_model(inference_model=inference_model)
- backend = get_models_manager().get_required_inference_backend(inference_model.backend_name)
- plugin_sdk_registry = get_plugin_manager().plugin_sdk_registry
- fetch_worker: FetchWorkerAbstract
- match plugin.sdk:
- case "linkup":
- from pipelex.plugins.linkup.linkup_worker import LinkupWorker # noqa: PLC0415
-
- fetch_worker = LinkupWorker(inference_model=inference_model, reporting_delegate=get_report_delegate())
- case "gateway_search":
- from pipelex.plugins.gateway.gateway_factory import GatewayFactory # noqa: PLC0415
- from pipelex.plugins.gateway.gateway_fetch_worker import GatewayFetchWorker # noqa: PLC0415
-
- sdk_instance = plugin_sdk_registry.get_sdk_instance(plugin=plugin) or plugin_sdk_registry.set_sdk_instance(
- plugin=plugin,
- sdk_instance=GatewayFactory.make_portkey_client(backend=backend),
- )
- fetch_worker = GatewayFetchWorker(
- sdk_instance=sdk_instance, inference_model=inference_model, reporting_delegate=get_report_delegate()
- )
- case _:
- msg = f"Plugin '{plugin}' is not supported for fetch"
- raise NotImplementedError(msg)
-
- return fetch_worker
diff --git a/pipelex/cogt/usage/cost_registry.py b/pipelex/cogt/usage/cost_registry.py
index cb62c090b..a427c1ec1 100644
--- a/pipelex/cogt/usage/cost_registry.py
+++ b/pipelex/cogt/usage/cost_registry.py
@@ -12,7 +12,6 @@ from pipelex.cogt.exceptions import CostRegistryError
from pipelex.cogt.extract.extract_report import ExtractTokenCostReport, ExtractTokenCostReportField, ExtractTokensUsage
from pipelex.cogt.img_gen.img_gen_report import ImgGenTokenCostReport, ImgGenTokenCostReportField, ImgGenTokensUsage
from pipelex.cogt.llm.llm_report import LLMTokenCostReport, LLMTokenCostReportField, LLMTokensUsage
-from pipelex.cogt.search.fetch_report import FetchTokenCostReport, FetchTokenCostReportField, FetchTokensUsage
from pipelex.cogt.search.search_report import SearchTokenCostReport, SearchTokenCostReportField, SearchTokensUsage
from pipelex.cogt.usage.cost_category import CostCategory, CostsByCategoryDict
from pipelex.cogt.usage.costs_per_token import model_cost_per_token
@@ -20,8 +19,8 @@ from pipelex.cogt.usage.token_category import TokenCategory
from pipelex.hub import get_console
from pipelex.tools.typing.pydantic_utils import empty_list_factory_of
-TokensUsage = LLMTokensUsage | ImgGenTokensUsage | ExtractTokensUsage | SearchTokensUsage | FetchTokensUsage
-TokenCostReport = LLMTokenCostReport | ImgGenTokenCostReport | ExtractTokenCostReport | SearchTokenCostReport | FetchTokenCostReport
+TokensUsage = LLMTokensUsage | ImgGenTokensUsage | ExtractTokensUsage | SearchTokensUsage
+TokenCostReport = LLMTokenCostReport | ImgGenTokenCostReport | ExtractTokenCostReport | SearchTokenCostReport
CostRegistryRoot = list[TokenCostReport]
@@ -83,8 +82,7 @@ class CostRegistry(RootModel[CostRegistryRoot]):
record.get(report_field.LLM_NAME)
or record.get(ImgGenTokenCostReportField.IMG_GEN_NAME)
or record.get(ExtractTokenCostReportField.EXTRACT_NAME)
- or record.get(SearchTokenCostReportField.SEARCH_NAME)
- or record.get(FetchTokenCostReportField.FETCH_NAME, "unknown")
+ or record.get(SearchTokenCostReportField.SEARCH_NAME, "unknown")
)
model_type = record.get(report_field.MODEL_TYPE, "llm")
model_types[model_name] = model_type
@@ -244,15 +242,6 @@ class CostRegistry(RootModel[CostRegistryRoot]):
nb_tokens_by_category=tokens_usage.nb_tokens_by_category,
costs_by_token_category=costs_by_token_category,
)
- if isinstance(tokens_usage, FetchTokensUsage):
- return FetchTokenCostReport(
- model_type=tokens_usage.model_type,
- job_metadata=tokens_usage.job_metadata,
- inference_model_name=tokens_usage.inference_model_name,
- platform_model_id=tokens_usage.inference_model_id,
- nb_tokens_by_category=tokens_usage.nb_tokens_by_category,
- costs_by_token_category=costs_by_token_category,
- )
return ExtractTokenCostReport(
model_type=tokens_usage.model_type,
job_metadata=tokens_usage.job_metadata,
diff --git a/pipelex/kit/configs/inference/backends/linkup.toml b/pipelex/kit/configs/inference/backends/linkup.toml
index 184e324e9..73ba1d78c 100644
--- a/pipelex/kit/configs/inference/backends/linkup.toml
+++ b/pipelex/kit/configs/inference/backends/linkup.toml
@@ -35,3 +35,15 @@ model_id = "deep"
inputs = ["text"]
outputs = ["sourced-answers", "structured"]
costs = { input = 0.05, output = 0 }
+
+################################################################################
+# FETCH / EXTRACT MODELS
+################################################################################
+
+["linkup-fetch"]
+model_type = "text_extractor"
+sdk = "linkup_fetch"
+model_id = "fetch"
+inputs = ["web_page"]
+outputs = ["pages"]
+costs = { input = 0.005, output = 0 }
diff --git a/pipelex/kit/configs/inference/deck/3_extract_deck.toml b/pipelex/kit/configs/inference/deck/3_extract_deck.toml
index afb2e0396..eaf3065be 100644
--- a/pipelex/kit/configs/inference/deck/3_extract_deck.toml
+++ b/pipelex/kit/configs/inference/deck/3_extract_deck.toml
@@ -31,6 +31,7 @@ default-extract-document = "azure-document-intelligence"
default-extract-image = "azure-document-intelligence"
default-text-from-pdf = "pypdfium2-extract-pdf"
default-no-inference = "pypdfium2-extract-pdf"
+default-extract-web-page = "linkup-fetch"
####################################################################################################
# Extract Presets
diff --git a/pipelex/pipe_operators/extract/pipe_extract.py b/pipelex/pipe_operators/extract/pipe_extract.py
index 74a52b6f2..519421ab0 100644
--- a/pipelex/pipe_operators/extract/pipe_extract.py
+++ b/pipelex/pipe_operators/extract/pipe_extract.py
@@ -42,6 +42,8 @@ class PipeExtract(PipeOperator[PipeExtractOutput]):
max_page_images: int | None
should_include_page_views: bool
page_views_dpi: int | None
+ render_js: bool | None = None
+ include_raw_html: bool | None = None
image_stuff_name: str | None = None
document_stuff_name: str | None = None
@@ -146,6 +148,8 @@ class PipeExtract(PipeOperator[PipeExtractOutput]):
page_views_dpi=self.page_views_dpi,
max_nb_images=max_nb_images,
image_min_size=extract_setting.image_min_size,
+ render_js=self.render_js,
+ include_raw_html=self.include_raw_html,
)
extract_input = ExtractInput(
image_uri=image_uri,
diff --git a/pipelex/pipe_operators/extract/pipe_extract_blueprint.py b/pipelex/pipe_operators/extract/pipe_extract_blueprint.py
index d569e573f..05eb9e816 100644
--- a/pipelex/pipe_operators/extract/pipe_extract_blueprint.py
+++ b/pipelex/pipe_operators/extract/pipe_extract_blueprint.py
@@ -14,6 +14,8 @@ class PipeExtractBlueprint(PipeBlueprint):
page_image_captions: bool | None = None
page_views: bool | None = None
page_views_dpi: int | None = None
+ render_js: bool | None = None
+ include_raw_html: bool | None = None
@override
def validate_inputs(self):
diff --git a/pipelex/pipe_operators/extract/pipe_extract_factory.py b/pipelex/pipe_operators/extract/pipe_extract_factory.py
index df66eff4a..ba88e77ae 100644
--- a/pipelex/pipe_operators/extract/pipe_extract_factory.py
+++ b/pipelex/pipe_operators/extract/pipe_extract_factory.py
@@ -67,4 +67,6 @@ class PipeExtractFactory(PipeFactoryProtocol[PipeExtractBlueprint, PipeExtract])
should_caption_images=blueprint.page_image_captions or False,
should_include_page_views=blueprint.page_views or False,
page_views_dpi=blueprint.page_views_dpi,
+ render_js=blueprint.render_js,
+ include_raw_html=blueprint.include_raw_html,
)
diff --git a/pipelex/pipe_operators/search/pipe_search.py b/pipelex/pipe_operators/search/pipe_search.py
index 0ff820734..b3f93be96 100644
--- a/pipelex/pipe_operators/search/pipe_search.py
+++ b/pipelex/pipe_operators/search/pipe_search.py
@@ -7,6 +7,7 @@ from pipelex.cogt.content_generation.dry_run_factory import DryRunFactory
from pipelex.cogt.exceptions import ModelChoiceNotFoundError
from pipelex.cogt.model_backends.model_type import ModelType
from pipelex.cogt.models.model_deck_check import check_search_choice_with_deck
+from pipelex.cogt.search.search_job_factory import SearchJobFactory
from pipelex.cogt.search.search_setting import SearchModelChoice, SearchSetting
from pipelex.cogt.search.search_worker_factory import SearchWorkerFactory
from pipelex.cogt.templating.template_blueprint import TemplateBlueprint
@@ -111,33 +112,27 @@ class PipeSearch(PipeOperator[PipeSearchOutput]):
# 5. Get search worker from factory
worker = SearchWorkerFactory.make_search_worker(inference_model=inference_model)
- # 6. Execute search based on output type
+ # 6. Create search job
+ search_job = SearchJobFactory.make_search_job(
+ query=query_text,
+ search_setting=search_setting,
+ job_metadata=job_metadata,
+ include_domains=self.include_domains,
+ exclude_domains=self.exclude_domains,
+ from_date=self.from_date,
+ to_date=self.to_date,
+ )
+
+ # 7. Execute search based on output type
content: StuffContent
if not self.is_structured_output:
- content = await worker.search_sourced_answer(
- query=query_text,
- search_setting=search_setting,
- job_metadata=job_metadata,
- include_domains=self.include_domains,
- exclude_domains=self.exclude_domains,
- from_date=self.from_date,
- to_date=self.to_date,
- )
+ content = await worker.search_sourced_answer(search_job=search_job)
else:
output_structure_class = self.output.concept.get_structure_class()
- result_dict = await worker.search_structured(
- query=query_text,
- search_setting=search_setting,
- output_schema=output_structure_class,
- job_metadata=job_metadata,
- include_domains=self.include_domains,
- exclude_domains=self.exclude_domains,
- from_date=self.from_date,
- to_date=self.to_date,
- )
+ result_dict = await worker.search_structured(search_job=search_job, schema=output_structure_class)
content = output_structure_class.model_validate(result_dict)
- # 7. Create Stuff, set in working memory, and return output
+ # 8. Create Stuff, set in working memory, and return output
output_stuff = StuffFactory.make_stuff(
name=output_name,
concept=self.output.concept,
diff --git a/pipelex/pipelex.toml b/pipelex/pipelex.toml
index 4237c0971..ce231ad8b 100644
--- a/pipelex/pipelex.toml
+++ b/pipelex/pipelex.toml
@@ -72,6 +72,7 @@ gen-image-high-quality = "$gen-image-high-quality"
pdf-basic-text-extractor = "@default-text-from-pdf"
image-text-extractor = "@default-extract-image"
full-document-extractor = "@default-extract-document"
+web-page-extractor = "@default-extract-web-page"
[pipelex.builder_config.talent_preset_mappings.search]
web-search = "$standard"
diff --git a/pipelex/pipeline/job_metadata.py b/pipelex/pipeline/job_metadata.py
index 7631ae5cc..97601e243 100644
--- a/pipelex/pipeline/job_metadata.py
+++ b/pipelex/pipeline/job_metadata.py
@@ -15,7 +15,6 @@ class JobCategory(StrEnum):
JINJA2_JOB = "jinja2_job"
EXTRACT_JOB = "extract_job"
SEARCH_JOB = "search_job"
- FETCH_JOB = "fetch_job"
class UnitJobId(StrEnum):
@@ -25,7 +24,6 @@ class UnitJobId(StrEnum):
EXTRACT_PAGES = "extract_pages"
SEARCH_SOURCED_ANSWER = "search_sourced_answer"
SEARCH_STRUCTURED = "search_structured"
- FETCH_URL = "fetch_url"
@property
def model_kind(self) -> str:
@@ -38,8 +36,6 @@ class UnitJobId(StrEnum):
return "Extract"
case UnitJobId.SEARCH_SOURCED_ANSWER | UnitJobId.SEARCH_STRUCTURED:
return "Search"
- case UnitJobId.FETCH_URL:
- return "Fetch"
class JobMetadata(BaseModel):
diff --git a/pipelex/plugins/gateway/gateway_completions_factory.py b/pipelex/plugins/gateway/gateway_completions_factory.py
index ab52d754a..eb651de76 100644
--- a/pipelex/plugins/gateway/gateway_completions_factory.py
+++ b/pipelex/plugins/gateway/gateway_completions_factory.py
@@ -30,6 +30,7 @@ from pipelex.plugins.gateway.gateway_exceptions import GatewayExtractResponseErr
from pipelex.plugins.gateway.gateway_factory import GatewayFactory
from pipelex.plugins.gateway.gateway_protocols import GatewayExtractProtocol
from pipelex.plugins.gateway.gateway_schemas import GatewayExtractPageAzure, GatewayExtractPageDeepseek, GatewayExtractPageMistral
+from pipelex.plugins.gateway.gateway_search_schemas import GatewayFetchResultResponse
from pipelex.plugins.openai.openai_completions_factory import OpenAICompletionsFactory
from pipelex.tools.uri.prepared_file import PreparedFileBase64, PreparedFileHttpUrl, PreparedFileLocalPath
@@ -133,6 +134,8 @@ class GatewayCompletionsFactory(OpenAICompletionsFactory):
return cls._make_extract_output_from_response_azure(response=response)
case GatewayExtractProtocol.DEEPSEEK_OCR:
return cls._make_extract_output_from_response_deepseek(response=response)
+ case GatewayExtractProtocol.LINKUP_FETCH:
+ return cls._make_extract_output_from_response_linkup_fetch(response=response)
@classmethod
def _extract_pages_from_choices_content(
@@ -289,6 +292,64 @@ class GatewayCompletionsFactory(OpenAICompletionsFactory):
msg = f"Error parsing Gateway extract response from pages using Deepseek schema: {exc}"
raise GatewayExtractResponseError(msg) from exc
+ @classmethod
+ def _make_extract_output_from_response_linkup_fetch(
+ cls,
+ response: GenericResponse,
+ ) -> ExtractOutput:
+ content_str = cls._extract_content_string_from_response(response=response)
+ if content_str is None:
+ msg = "Gateway fetch response does not contain content in choices[0].message.content"
+ raise GatewayExtractResponseError(msg)
+ try:
+ fetch_result = GatewayFetchResultResponse.model_validate_json(content_str)
+ except (ValidationError, ValueError) as exc:
+ msg = f"Error parsing Gateway fetch response as GatewayFetchResultResponse: {exc}"
+ raise GatewayExtractResponseError(msg) from exc
+
+ extracted_images: list[ExtractedImageFromPage] = []
+ if fetch_result.images:
+ for image in fetch_result.images:
+ extracted_images.append(
+ ExtractedImageFromPage(
+ size=None,
+ actual_url=image.url,
+ mime_type=None,
+ caption=image.alt or None,
+ )
+ )
+
+ page = Page(
+ text=fetch_result.markdown,
+ raw_html=fetch_result.raw_html,
+ extracted_images=extracted_images,
+ )
+ return ExtractOutput(pages={0: page})
+
+ @classmethod
+ def _extract_content_string_from_response(
+ cls,
+ response: GenericResponse,
+ ) -> str | None:
+ """Extract the string content from choices[0].message.content."""
+ try:
+ response_dict: dict[str, Any] = response.model_dump(serialize_as_any=True)
+ choices = response_dict.get("choices")
+ if not choices or not isinstance(choices, list) or len(choices) == 0: # pyright: ignore[reportUnknownArgumentType]
+ return None
+ first_choice = cast("dict[str, Any]", choices[0])
+ message_raw: Any = first_choice.get("message")
+ if not message_raw or not isinstance(message_raw, dict):
+ return None
+ message_dict = cast("dict[str, Any]", message_raw)
+ content: Any = message_dict.get("content")
+ if not content or not isinstance(content, str):
+ return None
+ content_str: str = content
+ return content_str
+ except (KeyError, IndexError, TypeError, AttributeError):
+ return None
+
@override
def make_extras(
self, inference_model: InferenceModelSpec, inference_job: InferenceJobAbstract, output_desc: str
diff --git a/pipelex/plugins/gateway/gateway_extract_worker.py b/pipelex/plugins/gateway/gateway_extract_worker.py
index 791c05f9a..3c8a3454c 100644
--- a/pipelex/plugins/gateway/gateway_extract_worker.py
+++ b/pipelex/plugins/gateway/gateway_extract_worker.py
@@ -14,11 +14,14 @@ from pipelex.cogt.extract.extract_output import ExtractOutput
from pipelex.cogt.extract.extract_worker_abstract import ExtractWorkerAbstract
from pipelex.cogt.inference.inference_constants import InferenceOutputType
from pipelex.cogt.model_backends.model_spec import InferenceModelSpec
+from pipelex.cogt.usage.token_category import TokenCategory
from pipelex.config import get_config
from pipelex.hub import get_storage_provider
from pipelex.plugins.gateway.gateway_completions_factory import GatewayCompletionsFactory
from pipelex.plugins.gateway.gateway_deck import GatewayDeck
from pipelex.plugins.gateway.gateway_factory import GatewayFactory
+from pipelex.plugins.gateway.gateway_protocols import GatewayExtractProtocol
+from pipelex.plugins.gateway.gateway_search_schemas import GatewayFetchRequestParams
from pipelex.reporting.reporting_protocol import ReportingProtocol
from pipelex.tools.uri.uri_resolver import make_base64_url_from_any_uri
@@ -66,6 +69,17 @@ class GatewayExtractWorker(ExtractWorkerAbstract):
async def _extract_pages(
self,
extract_job: ExtractJob,
+ ) -> ExtractOutput:
+ extract_protocol = GatewayExtractProtocol.make_from_model_handle(model_handle=self.inference_model.name)
+ match extract_protocol:
+ case GatewayExtractProtocol.LINKUP_FETCH:
+ return await self._extract_web_fetch(extract_job=extract_job)
+ case GatewayExtractProtocol.MISTRAL_DOC_AI | GatewayExtractProtocol.AZURE_DOC_INTEL | GatewayExtractProtocol.DEEPSEEK_OCR:
+ return await self._extract_document(extract_job=extract_job)
+
+ async def _extract_document(
+ self,
+ extract_job: ExtractJob,
) -> ExtractOutput:
# max_nb_images: None=unlimited, 0=no images, N=limit to N images
max_nb_images = extract_job.job_params.max_nb_images
@@ -97,6 +111,73 @@ class GatewayExtractWorker(ExtractWorkerAbstract):
raise ExtractInputError(msg)
return extract_output
+ async def _extract_web_fetch(
+ self,
+ extract_job: ExtractJob,
+ ) -> ExtractOutput:
+ """Extract content from a web page via the gateway's linkup-fetch endpoint."""
+ document_uri = extract_job.extract_input.document_uri
+ if not document_uri:
+ msg = "GatewayExtractWorker (linkup-fetch) requires a document_uri (web URL) in ExtractInput"
+ raise ExtractInputError(msg)
+
+ job_params = extract_job.job_params
+ max_nb_images = job_params.max_nb_images
+ extract_images = max_nb_images is None or max_nb_images != 0
+
+ fetch_params = GatewayFetchRequestParams(
+ url=document_uri,
+ render_js=job_params.render_js,
+ include_raw_html=job_params.include_raw_html,
+ extract_images=extract_images,
+ )
+
+ config_id = GatewayDeck.get_config_id(headers=self.inference_model.extra_headers or {})
+ log.dev(f"Web fetch via gateway config '{config_id}' for URL: {document_uri}")
+
+ messages: list[dict[str, str]] = [{"role": "user", "content": fetch_params.model_dump_json()}]
+
+ attempt_number = 0
+ response: GenericResponse | None = None
+ retryer = self._make_retryer()
+ try:
+ async for attempt in retryer:
+ with attempt:
+ attempt_number += 1
+ response = await self.portkey_client.with_options(config=config_id).post( # pyright: ignore[reportUnknownMemberType, reportUnknownVariableType]
+ "/chat/completions",
+ model=self.inference_model.model_id,
+ messages=messages,
+ )
+ except portkey_exceptions.APIError as exc:
+ error_summary = GatewayFactory.make_error_summary_from_portkey_error(exc)
+ msg = f"Web fetch service error for model '{self.inference_model.tag}' after {attempt_number} attempt(s): {error_summary}"
+ raise ExtractJobFailureError(msg) from exc
+
+ if response is None:
+ msg = f"Could not get a response for model '{self.inference_model.tag}' via Portkey after {attempt_number} attempts"
+ raise ExtractJobFailureError(msg)
+
+ if not isinstance(response, GenericResponse):
+ msg = "Response is not of type GenericResponse"
+ raise TypeError(msg)
+
+ extract_output = GatewayCompletionsFactory.make_extract_output_from_response(inference_model=self.inference_model, response=response)
+
+ # Apply image limit (factory just parses, worker applies limits)
+ if max_nb_images is not None and extract_output.pages:
+ for page in extract_output.pages.values():
+ page.extracted_images = page.extracted_images[:max_nb_images]
+
+ # Per-request cost model: costs are defined per million, so 1 request = 1_000_000
+ if extract_tokens_usage := extract_job.job_report.extract_tokens_usage:
+ extract_tokens_usage.nb_tokens_by_category = {
+ TokenCategory.INPUT: 1_000_000,
+ TokenCategory.OUTPUT: 1_000_000,
+ }
+
+ return extract_output
+
async def _extract_base64_url(
self,
extract_job: ExtractJob,
diff --git a/pipelex/plugins/gateway/gateway_factory.py b/pipelex/plugins/gateway/gateway_factory.py
index 878e43d99..a4dbb8a1e 100644
--- a/pipelex/plugins/gateway/gateway_factory.py
+++ b/pipelex/plugins/gateway/gateway_factory.py
@@ -90,6 +90,8 @@ class GatewayFactory:
case GatewayExtractProtocol.DEEPSEEK_OCR:
messages_deepseek: list[dict[str, str]] = [{"role": "user", "content": "Convert the document to markdown."}]
extra_body["messages"] = messages_deepseek
+ case GatewayExtractProtocol.LINKUP_FETCH:
+ pass # Fetch params are built directly in GatewayExtractWorker._extract_web_fetch
elif isinstance(inference_job, LLMJob) and inference_model.model_id.lower().startswith("mistral-") and inference_job.job_params.seed is None:
# Mistral models really want non-null seed
extra_body["seed"] = random.randint(0, 1000000)
diff --git a/pipelex/plugins/gateway/gateway_fetch_worker.py b/pipelex/plugins/gateway/gateway_fetch_worker.py
deleted file mode 100644
index a79b1f19b..000000000
--- a/pipelex/plugins/gateway/gateway_fetch_worker.py
+++ /dev/null
@@ -1,179 +0,0 @@
-import json
-from typing import Any, cast
-
-from portkey_ai import AsyncPortkey
-from portkey_ai.api_resources import exceptions as portkey_exceptions
-from portkey_ai.api_resources.utils import GenericResponse
-from tenacity import AsyncRetrying, RetryCallState, retry_if_exception, stop_after_attempt, wait_random_exponential
-from typing_extensions import override
-
-from pipelex import log
-from pipelex.cogt.exceptions import SdkTypeError
-from pipelex.cogt.model_backends.model_spec import InferenceModelSpec
-from pipelex.cogt.search.fetch_job import FetchJob
-from pipelex.cogt.search.fetch_worker_abstract import FetchWorkerAbstract
-from pipelex.cogt.search.search_job_factory import FetchJobFactory
-from pipelex.cogt.usage.token_category import NbTokensByCategoryDict, TokenCategory
-from pipelex.config import get_config
-from pipelex.core.stuffs.image_content import ImageContent
-from pipelex.core.stuffs.text_and_images_content import TextAndImagesContent
-from pipelex.core.stuffs.text_content import TextContent
-from pipelex.pipeline.job_metadata import JobMetadata, UnitJobId
-from pipelex.plugins.gateway.gateway_deck import GatewayDeck
-from pipelex.plugins.gateway.gateway_exceptions import GatewaySearchResponseError
-from pipelex.plugins.gateway.gateway_factory import GatewayFactory
-from pipelex.plugins.gateway.gateway_search_schemas import GatewayFetchRequestParams
-from pipelex.reporting.reporting_protocol import ReportingProtocol
-
-
-class GatewayFetchWorker(FetchWorkerAbstract):
- """Fetch worker that routes through Portkey to the pipelex-relay LinkUp fetch endpoint."""
-
- def __init__(
- self,
- sdk_instance: Any,
- inference_model: InferenceModelSpec,
- reporting_delegate: ReportingProtocol | None = None,
- ):
- super().__init__(reporting_delegate=reporting_delegate)
-
- if not isinstance(sdk_instance, AsyncPortkey):
- msg = f"Provided sdk_instance for {self.__class__.__name__} is not of type AsyncPortkey: it's a '{type(sdk_instance)}'"
- raise SdkTypeError(msg)
-
- self.portkey_client: AsyncPortkey = sdk_instance
- self.inference_model = inference_model
- self._tenacity_config = get_config().cogt.tenacity_config
-
- def _make_retryer(self) -> AsyncRetrying:
- """Create a fresh AsyncRetrying instance for each call."""
- return AsyncRetrying(
- retry=retry_if_exception(self._is_retryable_portkey_error),
- before_sleep=self._log_retry,
- wait=wait_random_exponential(
- multiplier=self._tenacity_config.wait_multiplier,
- max=self._tenacity_config.wait_max,
- exp_base=self._tenacity_config.wait_exp_base,
- ),
- reraise=True,
- stop=stop_after_attempt(self._tenacity_config.max_retries),
- )
-
- @override
- async def fetch_url(
- self,
- url: str,
- job_metadata: JobMetadata,
- include_raw_html: bool | None = None,
- render_js: bool | None = None,
- extract_images: bool | None = None,
- timeout: float | None = None, # noqa: ASYNC109
- ) -> TextAndImagesContent:
- job_metadata.unit_job_id = UnitJobId.FETCH_URL
- fetch_job = FetchJobFactory.make_fetch_job(job_metadata=job_metadata)
- fetch_job.fetch_job_before_start(inference_model=self.inference_model)
-
- params = GatewayFetchRequestParams(
- url=url,
- include_raw_html=include_raw_html,
- render_js=render_js,
- extract_images=extract_images,
- timeout=timeout,
- )
-
- config_id = GatewayDeck.get_config_id(headers=self.inference_model.extra_headers or {})
- log.dev(f"Fetch via gateway config '{config_id}'")
-
- messages: list[dict[str, str]] = [{"role": "user", "content": params.model_dump_json()}]
-
- attempt_number = 0
- response: GenericResponse | None = None
- retryer = self._make_retryer()
- try:
- async for attempt in retryer:
- with attempt:
- attempt_number += 1
- response = await self.portkey_client.with_options(config=config_id).post( # pyright: ignore[reportUnknownMemberType, reportUnknownVariableType]
- "/chat/completions",
- model="linkup/fetch",
- messages=messages,
- )
- except portkey_exceptions.APIError as exc:
- error_summary = GatewayFactory.make_error_summary_from_portkey_error(exc)
- msg = f"Fetch service error after {attempt_number} attempt(s): {error_summary}"
- raise GatewaySearchResponseError(msg) from exc
-
- if response is None:
- msg = f"Could not get a fetch response via Portkey after {attempt_number} attempts"
- raise GatewaySearchResponseError(msg)
-
- if not isinstance(response, GenericResponse):
- msg = "Response is not of type GenericResponse"
- raise TypeError(msg)
-
- self._extract_usage(response=response, fetch_job=fetch_job)
- fetch_job.fetch_job_after_complete()
- if self.reporting_delegate:
- self.reporting_delegate.report_inference_job(inference_job=fetch_job)
-
- # Extract content from response — use dict access (GenericResponse uses extra="allow")
- try:
- choice: dict[str, Any] = response.choices[0] # type: ignore[attr-defined] # pyright: ignore[reportAttributeAccessIssue, reportUnknownMemberType]
- raw_content = cast("object", choice["message"]["content"])
- except (KeyError, IndexError, TypeError) as exc:
- msg = "Could not extract content from gateway fetch response"
- raise GatewaySearchResponseError(msg) from exc
-
- if not isinstance(raw_content, str):
- msg = f"Expected string content in fetch response, got {type(raw_content)}"
- raise GatewaySearchResponseError(msg)
- content_str: str = raw_content
-
- result_dict: dict[str, Any] = json.loads(content_str)
-
- text = TextContent(text=result_dict["markdown"]) if result_dict.get("markdown") else None
-
- images: list[ImageContent] | None = None
- if result_dict.get("images") is not None:
- images = [
- ImageContent(
- url=image["url"],
- caption=image.get("alt"),
- )
- for image in result_dict["images"]
- ]
-
- return TextAndImagesContent(
- text=text,
- images=images or None,
- raw_html=result_dict.get("raw_html"),
- )
-
- def _extract_usage(self, response: GenericResponse, fetch_job: FetchJob) -> None:
- """Extract token usage from the GenericResponse and populate the fetch job report."""
- response_dict: dict[str, Any] = response.model_dump(serialize_as_any=True)
- fetch_tokens_usage = fetch_job.job_report.fetch_tokens_usage
- if (usage_dict := response_dict.get("usage")) and fetch_tokens_usage:
- nb_tokens: NbTokensByCategoryDict = {}
- if input_tokens := usage_dict.get("prompt_tokens") or usage_dict.get("input_tokens"):
- nb_tokens[TokenCategory.INPUT] = input_tokens
- if output_tokens := usage_dict.get("completion_tokens") or usage_dict.get("output_tokens"):
- nb_tokens[TokenCategory.OUTPUT] = output_tokens
- fetch_tokens_usage.nb_tokens_by_category = nb_tokens
-
- def _is_retryable_portkey_error(self, exc: BaseException) -> bool:
- if isinstance(exc, portkey_exceptions.NotFoundError):
- msg = str(exc).lower()
- return "specified deployment could not be found" in msg
- return False
-
- def _log_retry(self, retry_state: RetryCallState) -> None:
- """Called before sleeping between retries."""
- if not retry_state.outcome:
- log.error("Tenacity retry state outcome is None")
- return
- exc = retry_state.outcome.exception()
- attempt = retry_state.attempt_number
- wait_duration = retry_state.next_action.sleep if retry_state.next_action else 0.0
- log.dev(f"{self.__class__.__name__} retry #{attempt} for fetch due to '{type(exc).__name__}'.")
- log.verbose(f"Wait duration before next attempt: {wait_duration:.4f}s")
diff --git a/pipelex/plugins/gateway/gateway_protocols.py b/pipelex/plugins/gateway/gateway_protocols.py
index 4799780fa..c91e4910f 100644
--- a/pipelex/plugins/gateway/gateway_protocols.py
+++ b/pipelex/plugins/gateway/gateway_protocols.py
@@ -7,6 +7,7 @@ class GatewayExtractProtocol(StrEnum):
MISTRAL_DOC_AI = "mistral-doc-ai"
AZURE_DOC_INTEL = "azure-doc-intel"
DEEPSEEK_OCR = "deepseek-ocr"
+ LINKUP_FETCH = "linkup-fetch"
@classmethod
def make_from_model_handle(cls, model_handle: str) -> GatewayExtractProtocol:
@@ -17,6 +18,8 @@ class GatewayExtractProtocol(StrEnum):
return cls.AZURE_DOC_INTEL
case "deepseek-ocr":
return cls.DEEPSEEK_OCR
+ case "linkup-fetch":
+ return cls.LINKUP_FETCH
case _:
msg = f"Invalid model ID: {model_handle}"
raise ValueError(msg)
diff --git a/pipelex/plugins/gateway/gateway_search_schemas.py b/pipelex/plugins/gateway/gateway_search_schemas.py
index adec04b8c..395f348a4 100644
--- a/pipelex/plugins/gateway/gateway_search_schemas.py
+++ b/pipelex/plugins/gateway/gateway_search_schemas.py
@@ -26,3 +26,18 @@ class GatewayFetchRequestParams(BaseModel):
render_js: bool | None = Field(default=None, description="Whether to render JavaScript before fetching")
extract_images: bool | None = Field(default=None, description="Whether to extract images from the page")
timeout: float | None = Field(default=None, description="Request timeout in seconds")
+
+
+class GatewayFetchImageResponse(BaseModel):
+ """An image extracted from a fetched page."""
+
+ url: str = Field(description="URL of the image")
+ alt: str | None = Field(default=None, description="Alt text of the image")
+
+
+class GatewayFetchResultResponse(BaseModel):
+ """Response from a web page fetch."""
+
+ markdown: str | None = Field(default=None, description="Page content as markdown")
+ images: list[GatewayFetchImageResponse] | None = Field(default=None, description="Extracted images")
+ raw_html: str | None = Field(default=None, description="Raw HTML of the page")
diff --git a/pipelex/plugins/gateway/gateway_search_worker.py b/pipelex/plugins/gateway/gateway_search_worker.py
index af9ce074b..e2b5b38ef 100644
--- a/pipelex/plugins/gateway/gateway_search_worker.py
+++ b/pipelex/plugins/gateway/gateway_search_worker.py
@@ -12,18 +12,16 @@ from pipelex.cogt.exceptions import SdkTypeError
from pipelex.cogt.model_backends.model_spec import InferenceModelSpec
from pipelex.cogt.search.search_depth import SearchDepth
from pipelex.cogt.search.search_job import SearchJob
-from pipelex.cogt.search.search_job_factory import SearchJobFactory
-from pipelex.cogt.search.search_setting import SearchSetting
from pipelex.cogt.search.search_worker_abstract import SearchWorkerAbstract
from pipelex.cogt.usage.token_category import NbTokensByCategoryDict, TokenCategory
from pipelex.config import get_config
from pipelex.core.stuffs.search_result_content import SearchResultContent, SearchSourceContent
-from pipelex.pipeline.job_metadata import JobMetadata, UnitJobId
from pipelex.plugins.gateway.gateway_deck import GatewayDeck
from pipelex.plugins.gateway.gateway_exceptions import GatewaySearchResponseError
from pipelex.plugins.gateway.gateway_factory import GatewayFactory
from pipelex.plugins.gateway.gateway_search_schemas import GatewaySearchRequestParams
from pipelex.reporting.reporting_protocol import ReportingProtocol
+from pipelex.tools.typing.pydantic_utils import BaseModelTypeVar
class GatewaySearchWorker(SearchWorkerAbstract):
@@ -35,14 +33,13 @@ class GatewaySearchWorker(SearchWorkerAbstract):
inference_model: InferenceModelSpec,
reporting_delegate: ReportingProtocol | None = None,
):
- super().__init__(reporting_delegate=reporting_delegate)
+ super().__init__(inference_model=inference_model, reporting_delegate=reporting_delegate)
if not isinstance(sdk_instance, AsyncPortkey):
msg = f"Provided sdk_instance for {self.__class__.__name__} is not of type AsyncPortkey: it's a '{type(sdk_instance)}'"
raise SdkTypeError(msg)
self.portkey_client: AsyncPortkey = sdk_instance
- self.inference_model = inference_model
self._tenacity_config = get_config().cogt.tenacity_config
def _make_retryer(self) -> AsyncRetrying:
@@ -60,30 +57,23 @@ class GatewaySearchWorker(SearchWorkerAbstract):
)
@override
- async def search_sourced_answer(
+ async def _search_sourced_answer(
self,
- query: str,
- search_setting: SearchSetting,
- job_metadata: JobMetadata,
- include_domains: list[str] | None = None,
- exclude_domains: list[str] | None = None,
- from_date: str | None = None,
- to_date: str | None = None,
+ search_job: SearchJob,
) -> SearchResultContent:
- job_metadata.unit_job_id = UnitJobId.SEARCH_SOURCED_ANSWER
- search_job = SearchJobFactory.make_search_job(job_metadata=job_metadata)
- search_job.search_job_before_start(inference_model=self.inference_model)
+ job_params = search_job.job_params
+ search_setting = job_params.search_setting
params = GatewaySearchRequestParams(
- query=query,
+ query=search_job.query,
depth=SearchDepth(self.inference_model.model_id.rsplit("/", 1)[-1]),
include_images=search_setting.include_images,
include_inline_citations=search_setting.include_inline_citations,
max_results=search_setting.max_results,
- include_domains=include_domains,
- exclude_domains=exclude_domains,
- from_date=from_date,
- to_date=to_date,
+ include_domains=job_params.include_domains,
+ exclude_domains=job_params.exclude_domains,
+ from_date=job_params.from_date,
+ to_date=job_params.to_date,
)
response = await self._call_relay(
@@ -92,9 +82,6 @@ class GatewaySearchWorker(SearchWorkerAbstract):
)
self._extract_usage(response=response, search_job=search_job)
- search_job.search_job_after_complete()
- if self.reporting_delegate:
- self.reporting_delegate.report_inference_job(inference_job=search_job)
content_str = self._extract_content(response)
result_dict: dict[str, Any] = json.loads(content_str)
@@ -114,39 +101,27 @@ class GatewaySearchWorker(SearchWorkerAbstract):
)
@override
- async def search_structured(
+ async def _search_structured(
self,
- query: str,
- search_setting: SearchSetting,
- output_schema: type,
- job_metadata: JobMetadata,
- include_domains: list[str] | None = None,
- exclude_domains: list[str] | None = None,
- from_date: str | None = None,
- to_date: str | None = None,
+ search_job: SearchJob,
+ schema: type[BaseModelTypeVar],
) -> dict[str, Any]:
- job_metadata.unit_job_id = UnitJobId.SEARCH_STRUCTURED
- search_job = SearchJobFactory.make_search_job(job_metadata=job_metadata)
- search_job.search_job_before_start(inference_model=self.inference_model)
+ job_params = search_job.job_params
+ search_setting = job_params.search_setting
# Convert Pydantic model class to JSON Schema dict for the relay
- schema_dict: dict[str, Any]
- if hasattr(output_schema, "model_json_schema"):
- schema_dict = cast("dict[str, Any]", output_schema.model_json_schema()) # pyright: ignore[reportUnknownMemberType]
- else:
- msg = f"output_schema must be a Pydantic model class with model_json_schema(), got {output_schema}"
- raise SdkTypeError(msg)
+ schema_dict: dict[str, Any] = schema.model_json_schema()
params = GatewaySearchRequestParams(
- query=query,
+ query=search_job.query,
depth=SearchDepth(self.inference_model.model_id.rsplit("/", 1)[-1]),
include_images=search_setting.include_images,
include_inline_citations=search_setting.include_inline_citations,
max_results=search_setting.max_results,
- include_domains=include_domains,
- exclude_domains=exclude_domains,
- from_date=from_date,
- to_date=to_date,
+ include_domains=job_params.include_domains,
+ exclude_domains=job_params.exclude_domains,
+ from_date=job_params.from_date,
+ to_date=job_params.to_date,
output_schema=schema_dict,
)
@@ -156,9 +131,6 @@ class GatewaySearchWorker(SearchWorkerAbstract):
)
self._extract_usage(response=response, search_job=search_job)
- search_job.search_job_after_complete()
- if self.reporting_delegate:
- self.reporting_delegate.report_inference_job(inference_job=search_job)
content_str = self._extract_content(response)
result: dict[str, Any] = json.loads(content_str)
diff --git a/pipelex/plugins/linkup/linkup_extract_worker.py b/pipelex/plugins/linkup/linkup_extract_worker.py
new file mode 100644
index 000000000..911a1bd11
--- /dev/null
+++ b/pipelex/plugins/linkup/linkup_extract_worker.py
@@ -0,0 +1,79 @@
+from typing import Any
+
+from linkup import LinkupClient, LinkupFetchResponse
+from typing_extensions import override
+
+from pipelex.cogt.extract.extract_job import ExtractJob
+from pipelex.cogt.extract.extract_output import ExtractedImageFromPage, ExtractOutput, Page
+from pipelex.cogt.extract.extract_worker_abstract import ExtractWorkerAbstract
+from pipelex.cogt.model_backends.model_spec import InferenceModelSpec
+from pipelex.cogt.usage.token_category import TokenCategory
+from pipelex.hub import get_secrets_provider
+from pipelex.reporting.reporting_protocol import ReportingProtocol
+
+
+class LinkupExtractWorker(ExtractWorkerAbstract):
+ def __init__(
+ self,
+ extra_config: dict[str, Any],
+ inference_model: InferenceModelSpec,
+ reporting_delegate: ReportingProtocol | None = None,
+ ) -> None:
+ ExtractWorkerAbstract.__init__(
+ self,
+ extra_config=extra_config,
+ inference_model=inference_model,
+ reporting_delegate=reporting_delegate,
+ )
+ api_key = get_secrets_provider().get_secret(secret_id="LINKUP_API_KEY")
+ self._linkup_client = LinkupClient(api_key=api_key)
+
+ @override
+ async def _extract_pages(
+ self,
+ extract_job: ExtractJob,
+ ) -> ExtractOutput:
+ document_uri = extract_job.extract_input.document_uri
+ if not document_uri:
+ msg = "LinkupExtractWorker requires a document_uri (web URL) in ExtractInput"
+ raise ValueError(msg)
+
+ job_params = extract_job.job_params
+ extract_images = (job_params.max_nb_images or 0) != 0
+
+ response: LinkupFetchResponse = await self._linkup_client.async_fetch(
+ url=document_uri,
+ render_js=job_params.render_js,
+ include_raw_html=job_params.include_raw_html,
+ extract_images=extract_images,
+ )
+
+ # Per-request cost model: costs are defined per million, so 1 request = 1_000_000
+ if extract_tokens_usage := extract_job.job_report.extract_tokens_usage:
+ extract_tokens_usage.nb_tokens_by_category = {
+ TokenCategory.INPUT: 1_000_000,
+ TokenCategory.OUTPUT: 1_000_000,
+ }
+
+ max_images = job_params.max_nb_images
+ extracted_images: list[ExtractedImageFromPage] = []
+ if response.images:
+ for image in response.images:
+ if max_images is not None and len(extracted_images) >= max_images:
+ break
+ extracted_images.append(
+ ExtractedImageFromPage(
+ size=None,
+ actual_url=image.url,
+ mime_type=None,
+ caption=image.alt or None,
+ )
+ )
+
+ page = Page(
+ text=response.markdown,
+ raw_html=response.raw_html,
+ extracted_images=extracted_images,
+ )
+
+ return ExtractOutput(pages={0: page})
diff --git a/pipelex/plugins/linkup/linkup_search_worker.py b/pipelex/plugins/linkup/linkup_search_worker.py
new file mode 100644
index 000000000..5554fe6fe
--- /dev/null
+++ b/pipelex/plugins/linkup/linkup_search_worker.py
@@ -0,0 +1,106 @@
+from datetime import date
+from typing import Any
+
+from linkup import LinkupClient, LinkupSourcedAnswer
+from typing_extensions import override
+
+from pipelex.cogt.model_backends.model_spec import InferenceModelSpec
+from pipelex.cogt.search.search_depth import SearchDepth
+from pipelex.cogt.search.search_job import SearchJob
+from pipelex.cogt.search.search_worker_abstract import SearchWorkerAbstract
+from pipelex.cogt.usage.token_category import TokenCategory
+from pipelex.core.stuffs.search_result_content import SearchResultContent, SearchSourceContent
+from pipelex.hub import get_secrets_provider
+from pipelex.reporting.reporting_protocol import ReportingProtocol
+from pipelex.tools.typing.pydantic_utils import BaseModelTypeVar
+
+
+class LinkupSearchWorker(SearchWorkerAbstract):
+ def __init__(
+ self,
+ inference_model: InferenceModelSpec,
+ reporting_delegate: ReportingProtocol | None = None,
+ ) -> None:
+ SearchWorkerAbstract.__init__(self, inference_model=inference_model, reporting_delegate=reporting_delegate)
+ api_key = get_secrets_provider().get_secret(secret_id="LINKUP_API_KEY")
+ self._linkup_client = LinkupClient(api_key=api_key)
+
+ def _parse_date(self, date_str: str | None) -> date | None:
+ if date_str is None:
+ return None
+ return date.fromisoformat(date_str)
+
+ @override
+ async def _search_sourced_answer(
+ self,
+ search_job: SearchJob,
+ ) -> SearchResultContent:
+ job_params = search_job.job_params
+ search_setting = job_params.search_setting
+
+ depth_value = SearchDepth(self.inference_model.model_id)
+ response: LinkupSourcedAnswer = await self._linkup_client.async_search(
+ query=search_job.query,
+ depth=depth_value, # type: ignore[arg-type] # pyright: ignore[reportArgumentType]
+ output_type="sourcedAnswer",
+ include_images=search_setting.include_images,
+ include_inline_citations=search_setting.include_inline_citations,
+ max_results=search_setting.max_results,
+ include_domains=job_params.include_domains,
+ exclude_domains=job_params.exclude_domains,
+ from_date=self._parse_date(job_params.from_date),
+ to_date=self._parse_date(job_params.to_date),
+ )
+
+ # Per-request cost model: costs are defined per million, so 1 request = 1_000_000
+ if search_tokens_usage := search_job.job_report.search_tokens_usage:
+ search_tokens_usage.nb_tokens_by_category = {TokenCategory.INPUT: 1_000_000, TokenCategory.OUTPUT: 1_000_000}
+
+ sources: list[SearchSourceContent] = []
+ for source in response.sources:
+ sources.append(
+ SearchSourceContent(
+ name=source.name,
+ url=source.url,
+ snippet=source.snippet,
+ )
+ )
+
+ return SearchResultContent(
+ answer=response.answer,
+ sources=sources,
+ )
+
+ @override
+ async def _search_structured(
+ self,
+ search_job: SearchJob,
+ schema: type[BaseModelTypeVar],
+ ) -> dict[str, Any]:
+ job_params = search_job.job_params
+ search_setting = job_params.search_setting
+
+ depth_value = SearchDepth(self.inference_model.model_id)
+ response = await self._linkup_client.async_search(
+ query=search_job.query,
+ depth=depth_value, # type: ignore[arg-type] # pyright: ignore[reportArgumentType]
+ output_type="structured",
+ structured_output_schema=schema,
+ include_images=search_setting.include_images,
+ max_results=search_setting.max_results,
+ include_domains=job_params.include_domains,
+ exclude_domains=job_params.exclude_domains,
+ from_date=self._parse_date(job_params.from_date),
+ to_date=self._parse_date(job_params.to_date),
+ include_sources=True,
+ )
+
+ # Per-request cost model: costs are defined per million, so 1 request = 1_000_000
+ if search_tokens_usage := search_job.job_report.search_tokens_usage:
+ search_tokens_usage.nb_tokens_by_category = {TokenCategory.INPUT: 1_000_000, TokenCategory.OUTPUT: 1_000_000}
+
+ # If response is a Pydantic model, convert to dict
+ if hasattr(response, "model_dump"):
+ result: dict[str, Any] = response.model_dump()
+ return result
+ return dict(response) # type: ignore[arg-type]
diff --git a/pipelex/plugins/linkup/linkup_worker.py b/pipelex/plugins/linkup/linkup_worker.py
deleted file mode 100644
index 8afdd80ea..000000000
--- a/pipelex/plugins/linkup/linkup_worker.py
+++ /dev/null
@@ -1,180 +0,0 @@
-from datetime import date
-from typing import Any
-
-from linkup import LinkupClient, LinkupFetchResponse, LinkupSourcedAnswer
-from typing_extensions import override
-
-from pipelex.cogt.model_backends.model_spec import InferenceModelSpec
-from pipelex.cogt.search.fetch_worker_abstract import FetchWorkerAbstract
-from pipelex.cogt.search.search_depth import SearchDepth
-from pipelex.cogt.search.search_job_factory import FetchJobFactory, SearchJobFactory
-from pipelex.cogt.search.search_setting import SearchSetting
-from pipelex.cogt.search.search_worker_abstract import SearchWorkerAbstract
-from pipelex.cogt.usage.token_category import TokenCategory
-from pipelex.core.stuffs.image_content import ImageContent
-from pipelex.core.stuffs.search_result_content import SearchResultContent, SearchSourceContent
-from pipelex.core.stuffs.text_and_images_content import TextAndImagesContent
-from pipelex.core.stuffs.text_content import TextContent
-from pipelex.hub import get_secrets_provider
-from pipelex.pipeline.job_metadata import JobMetadata, UnitJobId
-from pipelex.reporting.reporting_protocol import ReportingProtocol
-
-
-class LinkupWorker(SearchWorkerAbstract, FetchWorkerAbstract):
- def __init__(
- self,
- inference_model: InferenceModelSpec,
- reporting_delegate: ReportingProtocol | None = None,
- ) -> None:
- SearchWorkerAbstract.__init__(self, reporting_delegate=reporting_delegate)
- FetchWorkerAbstract.__init__(self, reporting_delegate=reporting_delegate)
- self.inference_model = inference_model
- api_key = get_secrets_provider().get_secret(secret_id="LINKUP_API_KEY")
- self._linkup_client = LinkupClient(api_key=api_key)
-
- def _parse_date(self, date_str: str | None) -> date | None:
- if date_str is None:
- return None
- return date.fromisoformat(date_str)
-
- @override
- async def search_sourced_answer(
- self,
- query: str,
- search_setting: SearchSetting,
- job_metadata: JobMetadata,
- include_domains: list[str] | None = None,
- exclude_domains: list[str] | None = None,
- from_date: str | None = None,
- to_date: str | None = None,
- ) -> SearchResultContent:
- job_metadata.unit_job_id = UnitJobId.SEARCH_SOURCED_ANSWER
- search_job = SearchJobFactory.make_search_job(job_metadata=job_metadata)
- search_job.search_job_before_start(inference_model=self.inference_model)
-
- depth_value = SearchDepth(self.inference_model.model_id)
- response: LinkupSourcedAnswer = await self._linkup_client.async_search(
- query=query,
- depth=depth_value, # type: ignore[arg-type] # pyright: ignore[reportArgumentType]
- output_type="sourcedAnswer",
- include_images=search_setting.include_images,
- include_inline_citations=search_setting.include_inline_citations,
- max_results=search_setting.max_results,
- include_domains=include_domains,
- exclude_domains=exclude_domains,
- from_date=self._parse_date(from_date),
- to_date=self._parse_date(to_date),
- )
-
- # Per-request cost model: costs are defined per million, so 1 request = 1_000_000
- if search_tokens_usage := search_job.job_report.search_tokens_usage:
- search_tokens_usage.nb_tokens_by_category = {TokenCategory.INPUT: 1_000_000, TokenCategory.OUTPUT: 1_000_000}
- search_job.search_job_after_complete()
- if self.reporting_delegate:
- self.reporting_delegate.report_inference_job(inference_job=search_job)
-
- sources: list[SearchSourceContent] = []
- for source in response.sources:
- sources.append(
- SearchSourceContent(
- name=source.name,
- url=source.url,
- snippet=source.snippet,
- )
- )
-
- return SearchResultContent(
- answer=response.answer,
- sources=sources,
- )
-
- @override
- async def search_structured(
- self,
- query: str,
- search_setting: SearchSetting,
- output_schema: type,
- job_metadata: JobMetadata,
- include_domains: list[str] | None = None,
- exclude_domains: list[str] | None = None,
- from_date: str | None = None,
- to_date: str | None = None,
- ) -> dict[str, Any]:
- job_metadata.unit_job_id = UnitJobId.SEARCH_STRUCTURED
- search_job = SearchJobFactory.make_search_job(job_metadata=job_metadata)
- search_job.search_job_before_start(inference_model=self.inference_model)
-
- depth_value = SearchDepth(self.inference_model.model_id)
- response = await self._linkup_client.async_search(
- query=query,
- depth=depth_value, # type: ignore[arg-type] # pyright: ignore[reportArgumentType]
- output_type="structured",
- structured_output_schema=output_schema,
- include_images=search_setting.include_images,
- max_results=search_setting.max_results,
- include_domains=include_domains,
- exclude_domains=exclude_domains,
- from_date=self._parse_date(from_date),
- to_date=self._parse_date(to_date),
- include_sources=True,
- )
-
- # Per-request cost model: costs are defined per million, so 1 request = 1_000_000
- if search_tokens_usage := search_job.job_report.search_tokens_usage:
- search_tokens_usage.nb_tokens_by_category = {TokenCategory.INPUT: 1_000_000, TokenCategory.OUTPUT: 1_000_000}
- search_job.search_job_after_complete()
- if self.reporting_delegate:
- self.reporting_delegate.report_inference_job(inference_job=search_job)
-
- # If response is a Pydantic model, convert to dict
- if hasattr(response, "model_dump"):
- result: dict[str, Any] = response.model_dump()
- return result
- return dict(response) # type: ignore[arg-type]
-
- @override
- async def fetch_url(
- self,
- url: str,
- job_metadata: JobMetadata,
- include_raw_html: bool | None = None,
- render_js: bool | None = None,
- extract_images: bool | None = None,
- timeout: float | None = None, # noqa: ASYNC109
- ) -> TextAndImagesContent:
- job_metadata.unit_job_id = UnitJobId.FETCH_URL
- fetch_job = FetchJobFactory.make_fetch_job(job_metadata=job_metadata)
- fetch_job.fetch_job_before_start(inference_model=self.inference_model)
-
- response: LinkupFetchResponse = await self._linkup_client.async_fetch(
- url=url,
- include_raw_html=include_raw_html,
- render_js=render_js,
- extract_images=extract_images,
- timeout=timeout,
- )
-
- # Per-request cost model: costs are defined per million, so 1 request = 1_000_000
- if fetch_tokens_usage := fetch_job.job_report.fetch_tokens_usage:
- fetch_tokens_usage.nb_tokens_by_category = {TokenCategory.INPUT: 1_000_000, TokenCategory.OUTPUT: 1_000_000}
- fetch_job.fetch_job_after_complete()
- if self.reporting_delegate:
- self.reporting_delegate.report_inference_job(inference_job=fetch_job)
-
- text = TextContent(text=response.markdown) if response.markdown else None
-
- images: list[ImageContent] | None = None
- if response.images is not None:
- images = [
- ImageContent(
- url=image.url,
- caption=image.alt or None,
- )
- for image in response.images
- ]
-
- return TextAndImagesContent(
- text=text,
- images=images or None,
- raw_html=response.raw_html,
- )
diff --git a/pipelex/reporting/reporting_manager.py b/pipelex/reporting/reporting_manager.py
index 2512dc127..765d29785 100644
--- a/pipelex/reporting/reporting_manager.py
+++ b/pipelex/reporting/reporting_manager.py
@@ -12,8 +12,6 @@ from pipelex.cogt.img_gen.img_gen_report import ImgGenTokensUsage
from pipelex.cogt.inference.inference_job_abstract import InferenceJobAbstract
from pipelex.cogt.llm.llm_job import LLMJob
from pipelex.cogt.llm.llm_report import LLMTokensUsage
-from pipelex.cogt.search.fetch_job import FetchJob
-from pipelex.cogt.search.fetch_report import FetchTokensUsage
from pipelex.cogt.search.search_job import SearchJob
from pipelex.cogt.search.search_report import SearchTokensUsage
from pipelex.cogt.usage.cost_registry import CostRegistry
@@ -26,7 +24,7 @@ from pipelex.tools.typing.pydantic_utils import empty_list_factory_of
if TYPE_CHECKING:
from pathlib import Path
-TokensUsage = LLMTokensUsage | ImgGenTokensUsage | ExtractTokensUsage | SearchTokensUsage | FetchTokensUsage
+TokensUsage = LLMTokensUsage | ImgGenTokensUsage | ExtractTokensUsage | SearchTokensUsage
UsageRegistryRoot = list[TokensUsage]
@@ -124,20 +122,6 @@ class ReportingManager(ReportingProtocol):
search_token_cost_report = CostRegistry.complete_cost_report(tokens_usage=search_tokens_usage)
log.verbose(search_token_cost_report, title="Token Cost report")
- def _report_fetch_job(self, fetch_job: FetchJob):
- fetch_tokens_usage = fetch_job.job_report.fetch_tokens_usage
-
- if not fetch_tokens_usage:
- log.warning("Fetch job has no fetch_tokens_usage")
- return
-
- pipeline_run_id = fetch_job.job_metadata.pipeline_run_id
- self._get_registry(pipeline_run_id).add_tokens_usage(fetch_tokens_usage)
-
- if self._reporting_config.is_log_costs_to_console:
- fetch_token_cost_report = CostRegistry.complete_cost_report(tokens_usage=fetch_tokens_usage)
- log.verbose(fetch_token_cost_report, title="Token Cost report")
-
############################################################
# ReportingProtocol
############################################################
@@ -160,8 +144,6 @@ class ReportingManager(ReportingProtocol):
self._report_extract_job(extract_job=inference_job)
elif isinstance(inference_job, SearchJob):
self._report_search_job(search_job=inference_job)
- elif isinstance(inference_job, FetchJob):
- self._report_fetch_job(fetch_job=inference_job)
else:
log.warning(f"ReportingManager does not support reporting for inference job type: {type(inference_job).__name__}")
diff --git a/pipelex/system/pipelex_service/pipelex_details.py b/pipelex/system/pipelex_service/pipelex_details.py
index 262e0920c..31a87604e 100644
--- a/pipelex/system/pipelex_service/pipelex_details.py
+++ b/pipelex/system/pipelex_service/pipelex_details.py
@@ -2,7 +2,7 @@ from pipelex.tools.misc.hash_utils import hash_sha256
class PipelexDetails:
- REMOTE_CONFIG_URL = "https://pipelex-config.s3.eu-west-3.amazonaws.com/pipelex_remote_config_06.json"
+ REMOTE_CONFIG_URL = "https://pipelex-config.s3.eu-west-3.amazonaws.com/pipelex_remote_config_07.json"
PIPELEX_GATEWAY_API_KEY_VAR = "PIPELEX_GATEWAY_API_KEY"
@classmethod
diff --git a/tests/cases/documents.py b/tests/cases/documents.py
index 3efdf4c43..aaad5a295 100644
--- a/tests/cases/documents.py
+++ b/tests/cases/documents.py
@@ -38,3 +38,12 @@ class DocumentTestCases:
PDF_FILE_URL_1,
PDF_FILE_URL_2,
]
+
+ # Web URLs
+ WEB_URL_1 = "https://books.toscrape.com/catalogue/cravings-recipes-for-what-you-want-to-eat_589/index.html"
+ WEB_URL_2 = "https://www.scrapethissite.com/pages/"
+
+ WEB_URLS: ClassVar[list[str]] = [
+ WEB_URL_1,
+ WEB_URL_2,
+ ]
diff --git a/tests/integration/pipelex/cogt/test_data.py b/tests/integration/pipelex/cogt/test_data.py
index 2c6d16001..91fcabcd5 100644
--- a/tests/integration/pipelex/cogt/test_data.py
+++ b/tests/integration/pipelex/cogt/test_data.py
@@ -234,14 +234,6 @@ class SerDeTestLLMCases:
]
-class FetchTestCases:
- """Test cases for fetch integration tests."""
-
- FETCH_URLS: ClassVar[list[tuple[str, str]]] = [ # topic, url
- ("Wikipedia Python", "https://en.wikipedia.org/wiki/Python_(programming_language)"),
- ]
-
-
class SearchTestCases:
"""Test cases for search integration tests."""
diff --git a/tests/integration/pipelex/cogt/test_extract.py b/tests/integration/pipelex/cogt/test_extract.py
index 0e8969f66..58f741915 100644
--- a/tests/integration/pipelex/cogt/test_extract.py
+++ b/tests/integration/pipelex/cogt/test_extract.py
@@ -123,6 +123,50 @@ class TestExtract:
pretty_print(extract_output, title="Extract Output")
assert extract_output.pages
+ @pytest.mark.parametrize("url", DocumentTestCases.WEB_URLS)
+ async def test_extract_web_url(
+ self,
+ generated_content_factory: GeneratedContentFactory,
+ job_metadata: JobMetadata,
+ extract_combo: ModelCombo,
+ url: str,
+ ):
+ """Test web page extraction using Linkup fetch API."""
+ extract_worker = get_extract_worker(extract_handle=extract_combo.handle)
+ if not extract_worker.is_web_page_supported:
+ msg = f"Web extraction is not supported for this extract worker: '{extract_worker.desc}'"
+ pytest.skip(msg)
+ extract_job_params = ExtractJobParams(
+ max_nb_images=5,
+ should_caption_images=False,
+ should_include_page_views=False,
+ page_views_dpi=None,
+ image_min_size=None,
+ render_js=False,
+ include_raw_html=True,
+ )
+ extract_job = ExtractJobFactory.make_extract_job(
+ extract_input=ExtractInput(document_uri=url),
+ extract_job_params=extract_job_params,
+ job_metadata=job_metadata,
+ )
+ extract_output = await extract_worker.extract_pages(extract_job=extract_job)
+ assert extract_output.pages
+ page = extract_output.pages[0]
+ assert page.text, "Web extraction should return markdown text"
+ pretty_print(page.text, title="Extracted markdown")
+ if page.raw_html:
+ pretty_print(page.raw_html[:500], title="Raw HTML (first 500 chars)")
+ page_contents = await generated_content_factory.make_page_contents(
+ primary_id=job_metadata.user_id,
+ secondary_id=job_metadata.pipeline_run_id,
+ extract_output=extract_output,
+ )
+ assert page_contents
+ for page_index, page_content in enumerate(page_contents):
+ pretty_print(page_content, title=f"Page {page_index}")
+ get_report_delegate().generate_report()
+
@pytest.mark.parametrize("file_path", DocumentTestCases.DOCUMENT_FILE_PATHS)
async def test_extract_image_save(
self,
diff --git a/tests/integration/pipelex/cogt/test_fetch.py b/tests/integration/pipelex/cogt/test_fetch.py
deleted file mode 100644
index a470235c2..000000000
--- a/tests/integration/pipelex/cogt/test_fetch.py
+++ /dev/null
@@ -1,64 +0,0 @@
-import pytest
-
-from pipelex import pretty_print
-from pipelex.cogt.model_backends.model_type import ModelType
-from pipelex.cogt.search.search_worker_factory import SearchWorkerFactory
-from pipelex.hub import get_model_deck, get_report_delegate
-from pipelex.pipeline.job_metadata import JobMetadata
-from tests.integration.pipelex.cogt.test_data import FetchTestCases
-from tests.integration.pipelex.fixtures.model_combo import ModelCombo
-
-
[email protected](loop_scope="class")
-class TestFetch:
- @pytest.mark.parametrize(
- ("topic", "url"),
- FetchTestCases.FETCH_URLS,
- )
- async def test_fetch_url(self, search_combo: ModelCombo, job_metadata: JobMetadata, topic: str, url: str) -> None:
- """Verify that fetch_url returns non-empty text content."""
- model_deck = get_model_deck()
- inference_model = model_deck.get_required_inference_model(model_handle=search_combo.handle, model_type=ModelType.SEARCH)
- worker = SearchWorkerFactory.make_fetch_worker(inference_model=inference_model)
- result = await worker.fetch_url(url=url, job_metadata=job_metadata)
- pretty_print(result, title=f"Fetch Result ({topic})")
- assert result.text is not None, "Expected non-empty text content"
- assert len(result.text.text) > 0, "Expected text to have content"
- assert result.raw_html is None, "Expected raw_html to be None when not requested"
- assert result.images is None, "Expected images to be None when not requested"
- get_report_delegate().generate_report()
-
- @pytest.mark.parametrize(
- ("topic", "url"),
- FetchTestCases.FETCH_URLS,
- )
- async def test_fetch_url_with_raw_html(self, search_combo: ModelCombo, job_metadata: JobMetadata, topic: str, url: str) -> None:
- """Verify that fetch_url returns raw HTML when requested."""
- model_deck = get_model_deck()
- inference_model = model_deck.get_required_inference_model(model_handle=search_combo.handle, model_type=ModelType.SEARCH)
- worker = SearchWorkerFactory.make_fetch_worker(inference_model=inference_model)
- result = await worker.fetch_url(url=url, job_metadata=job_metadata, include_raw_html=True)
- pretty_print(result, title=f"Fetch Result with HTML ({topic})")
- assert result.text is not None, "Expected non-empty text content"
- assert result.raw_html is not None, "Expected raw_html to be present when requested"
- assert len(result.raw_html) > 0, "Expected raw_html to have content"
- get_report_delegate().generate_report()
-
- @pytest.mark.parametrize(
- ("topic", "url"),
- FetchTestCases.FETCH_URLS,
- )
- async def test_fetch_url_with_images(self, search_combo: ModelCombo, job_metadata: JobMetadata, topic: str, url: str) -> None:
- """Verify that fetch_url returns images when requested."""
- model_deck = get_model_deck()
- inference_model = model_deck.get_required_inference_model(model_handle=search_combo.handle, model_type=ModelType.SEARCH)
- worker = SearchWorkerFactory.make_fetch_worker(inference_model=inference_model)
- result = await worker.fetch_url(url=url, job_metadata=job_metadata, extract_images=True)
- pretty_print(result, title=f"Fetch Result with Images ({topic})")
- assert result.text is not None, "Expected non-empty text content"
- assert result.images is not None, "Expected images to be present when requested"
- assert isinstance(result.images, list), "Expected images to be a list"
- assert len(result.images) > 0, "Expected images list to be non-empty"
- get_report_delegate().generate_report()
diff --git a/tests/integration/pipelex/cogt/test_search.py b/tests/integration/pipelex/cogt/test_search.py
index 4b1847e70..c32240a1a 100644
--- a/tests/integration/pipelex/cogt/test_search.py
+++ b/tests/integration/pipelex/cogt/test_search.py
@@ -3,6 +3,7 @@ from pydantic import BaseModel
from pipelex import pretty_print
from pipelex.cogt.model_backends.model_type import ModelType
+from pipelex.cogt.search.search_job_factory import SearchJobFactory
from pipelex.cogt.search.search_setting import SearchSetting
from pipelex.cogt.search.search_worker_factory import SearchWorkerFactory
from pipelex.hub import get_model_deck, get_report_delegate
@@ -34,11 +35,12 @@ class TestSearch:
model=search_combo.handle,
max_results=2,
)
- result = await worker.search_sourced_answer(
+ search_job = SearchJobFactory.make_search_job(
query=query,
search_setting=search_setting,
job_metadata=job_metadata,
)
+ result = await worker.search_sourced_answer(search_job=search_job)
pretty_print(result, title=f"Sourced Answer Result ({topic})")
assert result.answer, "Expected a non-empty answer"
assert len(result.sources) > 0, "Expected at least one source"
@@ -59,12 +61,12 @@ class TestSearch:
search_setting = SearchSetting(
model=search_combo.handle,
)
- result = await worker.search_structured(
+ search_job = SearchJobFactory.make_search_job(
query=query,
search_setting=search_setting,
- output_schema=TopicSummary,
job_metadata=job_metadata,
)
+ result = await worker.search_structured(search_job=search_job, schema=TopicSummary)
pretty_print(result, title=f"Structured Search Result ({topic})")
assert isinstance(result, dict), "Expected a dict result"
assert "data" in result, "Expected 'data' key in result"