Build AI methods with your coding agent, then run them anywhere: as an MCP for chatbots, as a webapp for people, or via API for your software.
Feature/s3 storage (#559)
Louis Choquel82d64fc5a0ac
diff --git a/.blackboxrules b/.blackboxrulesindex df84a2fb8..30316939c 100644--- a/.blackboxrules+++ b/.blackboxrules@@ -257,6 +257,18 @@ This document outlines the core coding standards, best practices, and quality co"""```+## Standards related to developing the Pipelex codebase++### Main config++- The main config model is defined using `ConfigModel` classes, derived from `pydantic BaseModel`+- The model is defined in `pipelex/system/configuration/configs.py`, some of the submodels being defined in their respective sub-packages+- When adding new configs, place them where it makes most sense, ask the user if you need arbitrage+- As per our python standards, use StrEnum for multiple-value enums. In that case they must not be strict pydantic fields, i.e. add `= Field(strict=False)`+- **Important**: NEVER EVER set default values for config attributes in the class definition. All the default values are defined in the main config file `pipelex/pipelex.toml`. The only exception si for Optional values which must be set to `None` in the class definition.+- If (and only if) you add some config that will clearly make sense for client projects to override, for instance if it's a case of user preference, then you can also add a copy of the settings to the project override config file `.pipelex/pipelex.toml` but leaving them commented out, as an invitation to override.+- The different `pipelex.toml` files and the python model `configs.py` must be up to date with each other in terms of structure and attributes, otherwise the loading of teh config fails. To check quickly that you're good, just run `make tb` which tests the boot sequence, which includes the config loading.+## Writing tests### Unit test generalitiesdiff --git a/.cursor/rules/pipelex_standards.mdc b/.cursor/rules/pipelex_standards.mdcnew file mode 100644index 000000000..2b61bd55a--- /dev/null+++ b/.cursor/rules/pipelex_standards.mdc@@ -0,0 +1,15 @@+---+alwaysApply: true+description: Standards related to developing the Pipelex codebase+---+# Standards related to developing the Pipelex codebase++## Main config++- The main config model is defined using `ConfigModel` classes, derived from `pydantic BaseModel`+- The model is defined in `pipelex/system/configuration/configs.py`, some of the submodels being defined in their respective sub-packages+- When adding new configs, place them where it makes most sense, ask the user if you need arbitrage+- As per our python standards, use StrEnum for multiple-value enums. In that case they must not be strict pydantic fields, i.e. add `= Field(strict=False)`+- **Important**: NEVER EVER set default values for config attributes in the class definition. All the default values are defined in the main config file `pipelex/pipelex.toml`. The only exception si for Optional values which must be set to `None` in the class definition.+- If (and only if) you add some config that will clearly make sense for client projects to override, for instance if it's a case of user preference, then you can also add a copy of the settings to the project override config file `.pipelex/pipelex.toml` but leaving them commented out, as an invitation to override.+- The different `pipelex.toml` files and the python model `configs.py` must be up to date with each other in terms of structure and attributes, otherwise the loading of teh config fails. To check quickly that you're good, just run `make tb` which tests the boot sequence, which includes the config loading.diff --git a/.github/copilot-instructions.md b/.github/copilot-instructions.mdindex df84a2fb8..30316939c 100644--- a/.github/copilot-instructions.md+++ b/.github/copilot-instructions.md@@ -257,6 +257,18 @@ This document outlines the core coding standards, best practices, and quality co"""```+## Standards related to developing the Pipelex codebase++### Main config++- The main config model is defined using `ConfigModel` classes, derived from `pydantic BaseModel`+- The model is defined in `pipelex/system/configuration/configs.py`, some of the submodels being defined in their respective sub-packages+- When adding new configs, place them where it makes most sense, ask the user if you need arbitrage+- As per our python standards, use StrEnum for multiple-value enums. In that case they must not be strict pydantic fields, i.e. add `= Field(strict=False)`+- **Important**: NEVER EVER set default values for config attributes in the class definition. All the default values are defined in the main config file `pipelex/pipelex.toml`. The only exception si for Optional values which must be set to `None` in the class definition.+- If (and only if) you add some config that will clearly make sense for client projects to override, for instance if it's a case of user preference, then you can also add a copy of the settings to the project override config file `.pipelex/pipelex.toml` but leaving them commented out, as an invitation to override.+- The different `pipelex.toml` files and the python model `configs.py` must be up to date with each other in terms of structure and attributes, otherwise the loading of teh config fails. To check quickly that you're good, just run `make tb` which tests the boot sequence, which includes the config loading.+## Writing tests### Unit test generalitiesdiff --git a/.pipelex/inference/backends/pipelex_gateway_models.md b/.pipelex/inference/backends/pipelex_gateway_models.mdindex 22f9ba7a6..569c4fccb 100644--- a/.pipelex/inference/backends/pipelex_gateway_models.md+++ b/.pipelex/inference/backends/pipelex_gateway_models.md@@ -1,7 +1,7 @@# Pipelex Gateway - Available Models> **AUTO-GENERATED FILE** - Do not edit manually.-> Last updated: 2026-01-06T15:33:39Z+> Last updated: 2026-01-07T00:44:26Z>> Run `pipelex-dev update-gateway-models` or `make ugm` to regenerate.diff --git a/.pipelex/pipelex.toml b/.pipelex/pipelex.tomlindex 62f9e95ef..48f08435d 100644--- a/.pipelex/pipelex.toml+++ b/.pipelex/pipelex.toml@@ -28,12 +28,34 @@[pipelex.storage_config]-# Uncomment to change the storage method: "local" (default) or "in_memory"+# Storage method: "local" (default), "in_memory", "s3", or "gcp"+# method = "local"++# Whether to fetch remote HTTP URLs and store them locally# is_fetch_remote_content_enabled = true++# --- Local Storage Configuration ---+# Uncomment to customize local storage settings:+[pipelex.storage_config.local]# uri_format = "{primary_id}/{secondary_id}/{hash}.{extension}"-# method = "local"# local_storage_path = ".pipelex/storage"+# --- AWS S3 Storage Configuration ---+# Uncomment to use S3 storage (requires boto3: `pip install pipelex[s3]`):+[pipelex.storage_config.s3]+# uri_format = "{primary_id}/{secondary_id}/{hash}.{extension}"+# bucket_name = "your-bucket-name"+# region = "us-east-1"+# signed_urls_lifespan_seconds = 3600 # Set to "disabled" for public URLs++# --- Google Cloud Storage Configuration ---+# Uncomment to use GCP storage (requires google-cloud-storage: `pip install pipelex[gcp-storage]`):+[pipelex.storage_config.gcp]+# uri_format = "{primary_id}/{secondary_id}/{hash}.{extension}"+# bucket_name = "your-bucket-name"+# project_id = "your-project-id"+# signed_urls_lifespan_seconds = 3600 # Set to "disabled" for public URLs+[pipelex.scan_config]# Uncomment to customize the excluded directories for scanning# excluded_dirs = [diff --git a/.windsurfrules.md b/.windsurfrules.mdindex df84a2fb8..30316939c 100644--- a/.windsurfrules.md+++ b/.windsurfrules.md@@ -257,6 +257,18 @@ This document outlines the core coding standards, best practices, and quality co"""```+## Standards related to developing the Pipelex codebase++### Main config++- The main config model is defined using `ConfigModel` classes, derived from `pydantic BaseModel`+- The model is defined in `pipelex/system/configuration/configs.py`, some of the submodels being defined in their respective sub-packages+- When adding new configs, place them where it makes most sense, ask the user if you need arbitrage+- As per our python standards, use StrEnum for multiple-value enums. In that case they must not be strict pydantic fields, i.e. add `= Field(strict=False)`+- **Important**: NEVER EVER set default values for config attributes in the class definition. All the default values are defined in the main config file `pipelex/pipelex.toml`. The only exception si for Optional values which must be set to `None` in the class definition.+- If (and only if) you add some config that will clearly make sense for client projects to override, for instance if it's a case of user preference, then you can also add a copy of the settings to the project override config file `.pipelex/pipelex.toml` but leaving them commented out, as an invitation to override.+- The different `pipelex.toml` files and the python model `configs.py` must be up to date with each other in terms of structure and attributes, otherwise the loading of teh config fails. To check quickly that you're good, just run `make tb` which tests the boot sequence, which includes the config loading.+## Writing tests### Unit test generalitiesdiff --git a/CHANGELOG.md b/CHANGELOG.mdindex a4d55f4dd..37134df77 100644--- a/CHANGELOG.md+++ b/CHANGELOG.md@@ -6,7 +6,7 @@- **`PipeCompose` Construct Mode** — New mode for deterministically building `StructuredContent` objects without an LLM. Compose fields from working memory variables, fixed values, templates, and nested structures.-- **Content Storage System** — Configurable storage for generated artifacts (images, extracted pages). Supports `local` filesystem (`.pipelex/storage/`) and `in_memory` backends. Content referenced via stable `pipelex-storage://` URIs.+- **Content Storage System** — Configurable storage for generated artifacts (images, extracted pages). Supports `local` filesystem (`.pipelex/storage/`), `in_memory`, **AWS S3** (`pip install pipelex[s3]`), and **Google Cloud Storage** (`pip install pipelex[gcp-storage]`). Cloud providers support both public URLs and time-limited signed URLs. Content referenced via stable `pipelex-storage://` URIs.- **Hugging Face Backend** — Support for Hugging Face Inference API, including `qwen-image` text-to-image model.@@ -49,6 +49,8 @@- **Automatic Data URL Normalization**: Pipeline pre-processing step that converts large `data:` URLs in `ImageContent` to `pipelex-storage://` URIs for improved performance. Configurable via `is_normalize_data_urls_to_storage` in `pipelex.toml`.- **`PreparedImage` Abstraction**: New models (`PreparedImageHttpUrl`, `PreparedImageBase64`) representing images ready for LLM provider APIs.- Comprehensive unit and integration tests for URI resolver, data URL normalization, and `pipelex-storage://` image handling.+- **Cloud Storage Providers**: Added support for AWS S3 (`pip install pipelex[s3]`) and Google Cloud Storage (`pip install pipelex[gcp-storage]`), both supporting public URLs and time-limited signed URLs. These can now be used to store images generated by image generation models, images extracted from PDFs and any image you provide as input. Thanks to these, we now manipulate http URLs which are way more efficient for API calls and distributed computing, compared to base64 data URLs. Configured in `pipelex.toml` under the `[pipelex.storage_config]` section, using a top-level `method` setting with provider-specific tables (e.g., `[pipelex.storage_config.local]`, `[pipelex.storage_config.s3]`).+- Agent rules: Added `pipelex_standards.md` outlining standards for the Pipelex configuration system, also included as rules for AI development agents.- Pipelex Gateway service with terms of service management via `.pipelex/pipelex_service.toml` and interactive acceptance flow in `pipelex init`- Configurable retry logic with exponential backoff for inference API calls, configurable in `pipelex.toml` under `[cogt.tenacity_config]`- Context manager support for the `Pipelex` class (`with Pipelex.make(): ...`) for graceful shutdowndiff --git a/CLAUDE.md b/CLAUDE.mdindex df84a2fb8..30316939c 100644--- a/CLAUDE.md+++ b/CLAUDE.md@@ -257,6 +257,18 @@ This document outlines the core coding standards, best practices, and quality co"""```+## Standards related to developing the Pipelex codebase++### Main config++- The main config model is defined using `ConfigModel` classes, derived from `pydantic BaseModel`+- The model is defined in `pipelex/system/configuration/configs.py`, some of the submodels being defined in their respective sub-packages+- When adding new configs, place them where it makes most sense, ask the user if you need arbitrage+- As per our python standards, use StrEnum for multiple-value enums. In that case they must not be strict pydantic fields, i.e. add `= Field(strict=False)`+- **Important**: NEVER EVER set default values for config attributes in the class definition. All the default values are defined in the main config file `pipelex/pipelex.toml`. The only exception si for Optional values which must be set to `None` in the class definition.+- If (and only if) you add some config that will clearly make sense for client projects to override, for instance if it's a case of user preference, then you can also add a copy of the settings to the project override config file `.pipelex/pipelex.toml` but leaving them commented out, as an invitation to override.+- The different `pipelex.toml` files and the python model `configs.py` must be up to date with each other in terms of structure and attributes, otherwise the loading of teh config fails. To check quickly that you're good, just run `make tb` which tests the boot sequence, which includes the config loading.+## Writing tests### Unit test generalitiesdiff --git a/pipelex/cogt/content_generation/generated_content_factory.py b/pipelex/cogt/content_generation/generated_content_factory.pyindex 9b4dc6c61..1ec3b0559 100644--- a/pipelex/cogt/content_generation/generated_content_factory.py+++ b/pipelex/cogt/content_generation/generated_content_factory.py@@ -107,7 +107,7 @@ class GeneratedContentFactory:mime_type=raw_details.mime_type or base64_extracted_mime_type,output_format=output_format,)- url = self.storage_provider.store(data=actual_bytes, key=storage_key)+ url = await self.storage_provider.store(data=actual_bytes, key=storage_key)is_remote_url = Falseelse:msg = "No URL or bytes found"@@ -133,10 +133,10 @@ class GeneratedContentFactory:mime_type=mime_type,output_format=output_format,)- url = self.storage_provider.store(data=actual_bytes, key=storage_key)- display_link = self.storage_provider.display_link(uri=url)+ url = await self.storage_provider.store(data=actual_bytes, key=storage_key)+ display_link = await self.storage_provider.display_link(uri=url)elif not is_remote_url:- display_link = self.storage_provider.display_link(uri=url)+ display_link = await self.storage_provider.display_link(uri=url)else:display_link = urldiff --git a/pipelex/cogt/exceptions.py b/pipelex/cogt/exceptions.pyindex e4eefaa24..a710b3fac 100644--- a/pipelex/cogt/exceptions.py+++ b/pipelex/cogt/exceptions.py@@ -164,19 +164,6 @@ class ImgGenGeneratedTypeError(ImgGenGenerationError):pass-class MissingDependencyError(CogtError):- """Raised when a required dependency is not installed."""-- def __init__(self, dependency_name: str, extra_name: str, message: str | None = None):- self.dependency_name = dependency_name- self.extra_name = extra_name- error_msg = f"Required dependency '{dependency_name}' is not installed."- if message:- error_msg += f" {message}"- error_msg += f" Please install it with 'pip install pipelex[{extra_name}]'."- super().__init__(error_msg)--class ExtractCapabilityError(CogtError):passdiff --git a/pipelex/cogt/extract/extract_worker_factory.py b/pipelex/cogt/extract/extract_worker_factory.pyindex 9db33e0ad..6da66aa55 100644--- a/pipelex/cogt/extract/extract_worker_factory.py+++ b/pipelex/cogt/extract/extract_worker_factory.py@@ -1,8 +1,8 @@import importlib.util-from pipelex.cogt.exceptions import MissingDependencyErrorfrom pipelex.cogt.extract.extract_worker_abstract import ExtractWorkerAbstractfrom pipelex.cogt.model_backends.model_spec import InferenceModelSpec+from pipelex.exceptions import MissingDependencyErrorfrom pipelex.hub import get_models_manager, get_plugin_managerfrom pipelex.plugins.plugin_sdk_registry import Pluginfrom pipelex.reporting.reporting_protocol import ReportingProtocoldiff --git a/pipelex/cogt/image/prompt_image_utils.py b/pipelex/cogt/image/prompt_image_utils.pyindex cc933019f..e29ed0959 100644--- a/pipelex/cogt/image/prompt_image_utils.py+++ b/pipelex/cogt/image/prompt_image_utils.py@@ -75,7 +75,7 @@ async def prepare_prompt_image(case ResolvedPipelexStorage():storage = get_storage_provider()- raw_bytes = storage.load(uri=prompt_image.resolved.storage_uri)+ raw_bytes = await storage.load(uri=prompt_image.resolved.storage_uri)prepared = PreparedImageBase64(base64_data=base64.b64encode(raw_bytes).decode("ascii"),file_type=detect_file_type_from_bytes(raw_bytes),@@ -153,7 +153,7 @@ async def prepare_prompt_image_as_base64(prompt_image: PromptImage) -> PreparedIcase ResolvedPipelexStorage():storage = get_storage_provider()- raw_bytes = storage.load(uri=prompt_image.resolved.storage_uri)+ raw_bytes = await storage.load(uri=prompt_image.resolved.storage_uri)return PreparedImageBase64(base64_data=base64.b64encode(raw_bytes).decode("ascii"),file_type=detect_file_type_from_bytes(raw_bytes),diff --git a/pipelex/cogt/img_gen/img_gen_worker_factory.py b/pipelex/cogt/img_gen/img_gen_worker_factory.pyindex 6e388d2bd..e9e4243d3 100644--- a/pipelex/cogt/img_gen/img_gen_worker_factory.py+++ b/pipelex/cogt/img_gen/img_gen_worker_factory.py@@ -1,9 +1,9 @@import importlib.utilfrom typing import TYPE_CHECKING-from pipelex.cogt.exceptions import MissingDependencyErrorfrom pipelex.cogt.img_gen.img_gen_worker_abstract import ImgGenWorkerAbstractfrom pipelex.cogt.model_backends.model_spec import InferenceModelSpec+from pipelex.exceptions import MissingDependencyErrorfrom pipelex.hub import get_models_manager, get_plugin_managerfrom pipelex.plugins.blackboxai.blackboxai_completions_factory import BlackboxaiCompletionsFactoryfrom pipelex.plugins.plugin_sdk_registry import Plugindiff --git a/pipelex/cogt/llm/llm_worker_factory.py b/pipelex/cogt/llm/llm_worker_factory.pyindex 17835ea34..1b5a77e42 100644--- a/pipelex/cogt/llm/llm_worker_factory.py+++ b/pipelex/cogt/llm/llm_worker_factory.py@@ -1,8 +1,8 @@import importlib.util-from pipelex.cogt.exceptions import MissingDependencyErrorfrom pipelex.cogt.llm.llm_worker_internal_abstract import LLMWorkerInternalAbstractfrom pipelex.cogt.model_backends.model_spec import InferenceModelSpec+from pipelex.exceptions import MissingDependencyErrorfrom pipelex.hub import get_models_manager, get_plugin_managerfrom pipelex.plugins.plugin_sdk_registry import Pluginfrom pipelex.reporting.reporting_protocol import ReportingProtocoldiff --git a/pipelex/cogt/model_backends/model_lists.py b/pipelex/cogt/model_backends/model_lists.pyindex 339df1fd2..dab311d93 100644--- a/pipelex/cogt/model_backends/model_lists.py+++ b/pipelex/cogt/model_backends/model_lists.py@@ -3,7 +3,7 @@ from __future__ import annotationsimport importlib.utilfrom pipelex.cli.exceptions import PipelexCLIError-from pipelex.cogt.exceptions import MissingDependencyError+from pipelex.exceptions import MissingDependencyErrorfrom pipelex.hub import get_console, get_models_managerdiff --git a/pipelex/exceptions.py b/pipelex/exceptions.pyindex c23a72e00..c26c8704f 100644--- a/pipelex/exceptions.py+++ b/pipelex/exceptions.py@@ -51,6 +51,7 @@ from pipelex.system.exceptions import (ConfigValidationError,CredentialsError,FatalError,+ MissingDependencyError,NestedKeyConflictError,ToolError,TracebackMessageError,@@ -118,6 +119,7 @@ __all__ = ["CredentialsError","TracebackMessageError","FatalError",+ "MissingDependencyError","ConfigValidationError","ConfigModelError",]diff --git a/pipelex/kit/agent_rules/pipelex_standards.md b/pipelex/kit/agent_rules/pipelex_standards.mdnew file mode 100644index 000000000..8feb1b7c7--- /dev/null+++ b/pipelex/kit/agent_rules/pipelex_standards.md@@ -0,0 +1,11 @@+# Standards related to developing the Pipelex codebase++## Main config++- The main config model is defined using `ConfigModel` classes, derived from `pydantic BaseModel`+- The model is defined in `pipelex/system/configuration/configs.py`, some of the submodels being defined in their respective sub-packages+- When adding new configs, place them where it makes most sense, ask the user if you need arbitrage+- As per our python standards, use StrEnum for multiple-value enums. In that case they must not be strict pydantic fields, i.e. add `= Field(strict=False)`+- **Important**: NEVER EVER set default values for config attributes in the class definition. All the default values are defined in the main config file `pipelex/pipelex.toml`. The only exception si for Optional values which must be set to `None` in the class definition.+- If (and only if) you add some config that will clearly make sense for client projects to override, for instance if it's a case of user preference, then you can also add a copy of the settings to the project override config file `.pipelex/pipelex.toml` but leaving them commented out, as an invitation to override.+- The different `pipelex.toml` files and the python model `configs.py` must be up to date with each other in terms of structure and attributes, otherwise the loading of teh config fails. To check quickly that you're good, just run `make tb` which tests the boot sequence, which includes the config loading.diff --git a/pipelex/kit/configs/inference/backends/pipelex_gateway_models.md b/pipelex/kit/configs/inference/backends/pipelex_gateway_models.mdindex 22f9ba7a6..569c4fccb 100644--- a/pipelex/kit/configs/inference/backends/pipelex_gateway_models.md+++ b/pipelex/kit/configs/inference/backends/pipelex_gateway_models.md@@ -1,7 +1,7 @@# Pipelex Gateway - Available Models> **AUTO-GENERATED FILE** - Do not edit manually.-> Last updated: 2026-01-06T15:33:39Z+> Last updated: 2026-01-07T00:44:26Z>> Run `pipelex-dev update-gateway-models` or `make ugm` to regenerate.diff --git a/pipelex/kit/configs/pipelex.toml b/pipelex/kit/configs/pipelex.tomlindex 62f9e95ef..48f08435d 100644--- a/pipelex/kit/configs/pipelex.toml+++ b/pipelex/kit/configs/pipelex.toml@@ -28,12 +28,34 @@[pipelex.storage_config]-# Uncomment to change the storage method: "local" (default) or "in_memory"+# Storage method: "local" (default), "in_memory", "s3", or "gcp"+# method = "local"++# Whether to fetch remote HTTP URLs and store them locally# is_fetch_remote_content_enabled = true++# --- Local Storage Configuration ---+# Uncomment to customize local storage settings:+[pipelex.storage_config.local]# uri_format = "{primary_id}/{secondary_id}/{hash}.{extension}"-# method = "local"# local_storage_path = ".pipelex/storage"+# --- AWS S3 Storage Configuration ---+# Uncomment to use S3 storage (requires boto3: `pip install pipelex[s3]`):+[pipelex.storage_config.s3]+# uri_format = "{primary_id}/{secondary_id}/{hash}.{extension}"+# bucket_name = "your-bucket-name"+# region = "us-east-1"+# signed_urls_lifespan_seconds = 3600 # Set to "disabled" for public URLs++# --- Google Cloud Storage Configuration ---+# Uncomment to use GCP storage (requires google-cloud-storage: `pip install pipelex[gcp-storage]`):+[pipelex.storage_config.gcp]+# uri_format = "{primary_id}/{secondary_id}/{hash}.{extension}"+# bucket_name = "your-bucket-name"+# project_id = "your-project-id"+# signed_urls_lifespan_seconds = 3600 # Set to "disabled" for public URLs+[pipelex.scan_config]# Uncomment to customize the excluded directories for scanning# excluded_dirs = [diff --git a/pipelex/kit/index.toml b/pipelex/kit/index.tomlindex 5c420bc28..5b4eb5041 100644--- a/pipelex/kit/index.toml+++ b/pipelex/kit/index.toml@@ -10,6 +10,7 @@ default_set = "pipelex_language"coding_standards = ["commands.md","python_standards.md",+ "pipelex_standards.md","pytest_standards.md","docs.md","tdd.md",@@ -18,6 +19,7 @@ pipelex_language = ["write_pipelex.md", "run_pipelex.md", "llms.md"]all = ["commands.md","python_standards.md",+ "pipelex_standards.md","write_pipelex.md","run_pipelex.md","docs.md",@@ -33,6 +35,10 @@ alwaysApply = falsedescription = "Python coding standards and best practices"globs = ["**/*.py"]+[agent_rules.cursor.files.pipelex_standards.front_matter]+description = "Standards related to developing the Pipelex codebase"+alwaysApply = true+[agent_rules.cursor.files.pytest_standards.front_matter]description = "Guidelines for writing unit tests"globs = ["tests/**/*.py"]diff --git a/pipelex/pipelex.py b/pipelex/pipelex.pyindex 0d10ae932..e100ad4be 100644--- a/pipelex/pipelex.py+++ b/pipelex/pipelex.py@@ -1,5 +1,4 @@import types-from pathlib import Pathfrom typing import TYPE_CHECKING, Any, castfrom kajson.class_registry import ClassRegistry@@ -46,7 +45,7 @@ from pipelex.reporting.reporting_manager import ReportingManagerfrom pipelex.reporting.reporting_protocol import ReportingNoOp, ReportingProtocolfrom pipelex.system.configuration.config_loader import config_managerfrom pipelex.system.configuration.config_root import ConfigRoot-from pipelex.system.configuration.configs import ConfigPaths, PipelexConfig, StorageMethod+from pipelex.system.configuration.configs import ConfigPaths, PipelexConfigfrom pipelex.system.pipelex_service.exceptions import (GatewayTermsNotAcceptedError,)@@ -70,9 +69,8 @@ from pipelex.test_extras.registry_test_models import TestRegistryModelsfrom pipelex.tools.misc.package_utils import get_package_infofrom pipelex.tools.secrets.env_secrets_provider import EnvSecretsProviderfrom pipelex.tools.secrets.secrets_provider_abstract import SecretsProviderAbstract-from pipelex.tools.storage.in_memory_storage_provider import InMemoryStorageProvider-from pipelex.tools.storage.local_storage_provider import LocalStorageProviderfrom pipelex.tools.storage.storage_provider_abstract import StorageProviderAbstract+from pipelex.tools.storage.storage_provider_factory import make_storage_provider_from_configfrom pipelex.types import Selffrom pipelex.urls import URLs@@ -210,13 +208,8 @@ If you need help, drop by our Discord: we're happy to assist: {URLs.discord}.self.kajson_manager = KajsonManager(class_registry=self.class_registry)self.pipelex_hub.set_secrets_provider(secrets_provider=secrets_provider)if storage_provider is None:- match get_config().pipelex.storage_config.method:- case StorageMethod.LOCAL:- log.dev(f"Using local storage at: {get_config().pipelex.storage_config.storage_path}")- storage_provider = LocalStorageProvider(root_path=Path(get_config().pipelex.storage_config.storage_path))- case StorageMethod.IN_MEMORY:- log.dev("Using in-memory storage")- storage_provider = InMemoryStorageProvider()+ storage_config = get_config().pipelex.storage_config+ storage_provider = make_storage_provider_from_config(storage_config)self.pipelex_hub.set_storage_provider(storage_provider)self.library_manager = library_manager or LibraryManager()diff --git a/pipelex/pipelex.toml b/pipelex/pipelex.tomlindex 3bcbf0f33..9df3ac8b4 100644--- a/pipelex/pipelex.toml+++ b/pipelex/pipelex.toml@@ -4,10 +4,27 @@[pipelex.storage_config]is_fetch_remote_content_enabled = true+method = "in_memory" # local, in_memory, s3, gcp++[pipelex.storage_config.local]uri_format = "{primary_id}/{secondary_id}/{hash}.{extension}"-method = "local"local_storage_path = ".pipelex/storage"+[pipelex.storage_config.in_memory]+uri_format = "{primary_id}/{secondary_id}/{hash}.{extension}"++[pipelex.storage_config.s3]+uri_format = "{primary_id}/{secondary_id}/{hash}.{extension}"+bucket_name = ""+region = ""+signed_urls_lifespan_seconds = 3600++[pipelex.storage_config.gcp]+uri_format = "{primary_id}/{secondary_id}/{hash}.{extension}"+bucket_name = ""+project_id = ""+signed_urls_lifespan_seconds = 3600+[pipelex.observer_config]observer_dir = "results/observer"diff --git a/pipelex/pipeline/input_normalizer.py b/pipelex/pipeline/input_normalizer.pyindex 57123d54a..648ec05e0 100644--- a/pipelex/pipeline/input_normalizer.py+++ b/pipelex/pipeline/input_normalizer.py@@ -21,13 +21,14 @@ from pipelex.tools.uri.resolved_uri import ResolvedBase64DataUrlfrom pipelex.tools.uri.uri_resolver import resolve_uri-def normalize_data_urls_to_storage(working_memory: WorkingMemory) -> WorkingMemory:+async def normalize_data_urls_to_storage(working_memory: WorkingMemory) -> WorkingMemory:"""Convert all data URLs in ImageContent to pipelex-storage:// URIs.Scans all stuffs in working memory and for any ImageContent with a data:...;base64,...URL, stores the data and replaces the URL with a pipelex-storage:// URI.This handles:+- Direct ImageContent- ListContent containing ImageContent items- StructuredContent with nested ImageContent fields (recursive)@@ -42,14 +43,14 @@ def normalize_data_urls_to_storage(working_memory: WorkingMemory) -> WorkingMemofor stuff in working_memory.root.values():content = stuff.content- normalized_content, changed = _normalize_value(value=content, storage=storage)+ normalized_content, changed = await _normalize_value(value=content, storage=storage)if changed:stuff.content = normalized_contentreturn working_memory-def _normalize_value(+async def _normalize_value(value: Any,storage: StorageProviderAbstract,) -> tuple[Any, bool]:@@ -64,26 +65,26 @@ def _normalize_value("""# Handle ImageContent directlyif isinstance(value, ImageContent):- normalized = _normalize_image_content(image_content=value, storage=storage)+ normalized = await _normalize_image_content(image_content=value, storage=storage)return normalized, normalized is not value# Handle StructuredContent (recursively process all fields)if isinstance(value, StructuredContent):- return _normalize_structured_content(structured_content=value, storage=storage)+ return await _normalize_structured_content(structured_content=value, storage=storage)# Handle ListContentif isinstance(value, ListContent):- return _normalize_list_content(list_content=value, storage=storage) # pyright: ignore[reportUnknownArgumentType]+ return await _normalize_list_content(list_content=value, storage=storage) # pyright: ignore[reportUnknownArgumentType]# Handle plain lists (might contain ImageContent or StructuredContent)if isinstance(value, list):- return _normalize_list(items=value, storage=storage) # pyright: ignore[reportUnknownArgumentType]+ return await _normalize_list(items=value, storage=storage) # pyright: ignore[reportUnknownArgumentType]# Other types don't need normalizationreturn value, False-def _normalize_structured_content(+async def _normalize_structured_content(structured_content: StructuredContent,storage: StorageProviderAbstract,) -> tuple[StructuredContent, bool]:@@ -100,7 +101,7 @@ def _normalize_structured_content(has_changes = Falsefor field_name, field_value in structured_content:- normalized_value, changed = _normalize_value(value=field_value, storage=storage)+ normalized_value, changed = await _normalize_value(value=field_value, storage=storage)if changed:updates[field_name] = normalized_valuehas_changes = True@@ -113,7 +114,7 @@ def _normalize_structured_content(return structured_content.model_copy(update=updates), True-def _normalize_list_content(+async def _normalize_list_content(list_content: ListContent[Any],storage: StorageProviderAbstract,) -> tuple[ListContent[Any], bool]:@@ -130,7 +131,7 @@ def _normalize_list_content(if not raw_items:return list_content, False- normalized_items, has_changes = _normalize_list(items=raw_items, storage=storage) # pyright: ignore[reportUnknownArgumentType]+ normalized_items, has_changes = await _normalize_list(items=raw_items, storage=storage) # pyright: ignore[reportUnknownArgumentType]if not has_changes:return list_content, False@@ -144,7 +145,7 @@ def _normalize_list_content(return ListContent(items=normalized_items), True-def _normalize_list(+async def _normalize_list(items: list[Any],storage: StorageProviderAbstract,) -> tuple[list[Any], bool]:@@ -161,7 +162,7 @@ def _normalize_list(has_changes = Falsefor item in items:- normalized_item, changed = _normalize_value(value=item, storage=storage)+ normalized_item, changed = await _normalize_value(value=item, storage=storage)normalized_items.append(normalized_item)if changed:has_changes = True@@ -169,7 +170,7 @@ def _normalize_list(return normalized_items, has_changes-def _normalize_image_content(+async def _normalize_image_content(image_content: ImageContent,storage: StorageProviderAbstract,) -> ImageContent:@@ -193,7 +194,7 @@ def _normalize_image_content(raw_bytes = base64.b64decode(resolved_uri.base64_data)file_type = detect_file_type_from_bytes(raw_bytes)key = f"normalized/{shortuuid.uuid()}.{file_type.extension}"- storage_uri = storage.store(data=raw_bytes, key=key)+ storage_uri = await storage.store(data=raw_bytes, key=key)return ImageContent(url=storage_uri,diff --git a/pipelex/pipeline/pipeline_run_setup.py b/pipelex/pipeline/pipeline_run_setup.pyindex 0e56efb0d..83a535e95 100644--- a/pipelex/pipeline/pipeline_run_setup.py+++ b/pipelex/pipeline/pipeline_run_setup.py@@ -163,7 +163,7 @@ async def pipeline_run_setup(# Normalize data URLs to pipelex-storage:// URIs if configuredif working_memory and get_config().pipelex.pipeline_execution_config.is_normalize_data_urls_to_storage:- working_memory = normalize_data_urls_to_storage(working_memory)+ working_memory = await normalize_data_urls_to_storage(working_memory)if pipe_run_mode is None:if run_mode_from_env := get_optional_env(key=FORCE_DRY_RUN_MODE_ENV_KEY):diff --git a/pipelex/plugins/anthropic/anthropic_list.py b/pipelex/plugins/anthropic/anthropic_list.pyindex 4400b9740..b81a35e3c 100644--- a/pipelex/plugins/anthropic/anthropic_list.py+++ b/pipelex/plugins/anthropic/anthropic_list.py@@ -7,7 +7,7 @@ from rich import boxfrom rich.table import Tablefrom pipelex.cli.exceptions import PipelexCLIError-from pipelex.cogt.exceptions import MissingDependencyError+from pipelex.exceptions import MissingDependencyErrorfrom pipelex.hub import get_consolefrom pipelex.plugins.plugin_sdk_registry import Plugindiff --git a/pipelex/plugins/bedrock/bedrock_list.py b/pipelex/plugins/bedrock/bedrock_list.pyindex 73c411f52..03b777ebb 100644--- a/pipelex/plugins/bedrock/bedrock_list.py+++ b/pipelex/plugins/bedrock/bedrock_list.py@@ -7,8 +7,8 @@ from rich import boxfrom rich.table import Tablefrom pipelex.cli.exceptions import PipelexCLIError-from pipelex.cogt.exceptions import MissingDependencyErrorfrom pipelex.config import get_config+from pipelex.exceptions import MissingDependencyErrorfrom pipelex.hub import get_consolefrom pipelex.plugins.plugin_sdk_registry import Pluginfrom pipelex.tools.aws.aws_config import AwsCredentialsErrordiff --git a/pipelex/plugins/bedrock/bedrock_llms.py b/pipelex/plugins/bedrock/bedrock_llms.pyindex b92cf87ab..f6b229570 100644--- a/pipelex/plugins/bedrock/bedrock_llms.py+++ b/pipelex/plugins/bedrock/bedrock_llms.py@@ -1,8 +1,8 @@from typing import Any, cast-from pipelex.cogt.exceptions import MissingDependencyErrorfrom pipelex.cogt.model_backends.backend import InferenceBackendfrom pipelex.config import get_config+from pipelex.exceptions import MissingDependencyErrorfrom pipelex.plugins.plugin_sdk_registry import Pluginfrom pipelex.tools.aws.aws_config import AwsCredentialsErrordiff --git a/pipelex/plugins/google/google_list.py b/pipelex/plugins/google/google_list.pyindex 2e83d576b..409e1efb7 100644--- a/pipelex/plugins/google/google_list.py+++ b/pipelex/plugins/google/google_list.py@@ -6,7 +6,7 @@ from typing import TYPE_CHECKING, Anyfrom rich import boxfrom rich.table import Table-from pipelex.cogt.exceptions import MissingDependencyError+from pipelex.exceptions import MissingDependencyErrorfrom pipelex.hub import get_consoleif TYPE_CHECKING:diff --git a/pipelex/plugins/mistral/mistral_list.py b/pipelex/plugins/mistral/mistral_list.pyindex c5b5ab0a3..8afbd0ef6 100644--- a/pipelex/plugins/mistral/mistral_list.py+++ b/pipelex/plugins/mistral/mistral_list.py@@ -6,7 +6,7 @@ from typing import Anyfrom rich import boxfrom rich.table import Table-from pipelex.cogt.exceptions import MissingDependencyError+from pipelex.exceptions import MissingDependencyErrorfrom pipelex.hub import get_consolediff --git a/pipelex/plugins/openai/vertexai_factory.py b/pipelex/plugins/openai/vertexai_factory.pyindex 0ece3bd9b..a6db2a4f4 100644--- a/pipelex/plugins/openai/vertexai_factory.py+++ b/pipelex/plugins/openai/vertexai_factory.py@@ -1,8 +1,8 @@from typing import Any-from pipelex.cogt.exceptions import CogtError, MissingDependencyError+from pipelex.cogt.exceptions import CogtErrorfrom pipelex.system.configuration.config_model import ConfigModel-from pipelex.system.exceptions import CredentialsError+from pipelex.system.exceptions import CredentialsError, MissingDependencyErrorfrom pipelex.tools.misc.json_utils import load_json_dict_from_pathfrom pipelex.tools.misc.placeholder import value_is_placeholderfrom pipelex.types import StrEnumdiff --git a/pipelex/system/configuration/configs.py b/pipelex/system/configuration/configs.pyindex 20e1158fc..07c9ca218 100644--- a/pipelex/system/configuration/configs.py+++ b/pipelex/system/configuration/configs.py@@ -1,7 +1,7 @@from typing import castimport shortuuid-from pydantic import Field, field_validator, model_validator+from pydantic import Field, field_validatorfrom pipelex.base_exceptions import PipelexConfigErrorfrom pipelex.cogt.config_cogt import Cogt@@ -13,7 +13,8 @@ from pipelex.system.configuration.config_model import ConfigModelfrom pipelex.system.configuration.config_root import ConfigRootfrom pipelex.tools.aws.aws_config import AwsConfigfrom pipelex.tools.log.log_config import LogConfig-from pipelex.types import Self, StrEnum+from pipelex.tools.storage.storage_config import StorageConfig+from pipelex.types import StrEnumclass ConfigPaths:@@ -34,11 +35,6 @@ class ConfigPaths:OVERRIDES_DECK_FILE_PATH = f"{MODEL_DECKS_DIR_PATH}/{OVERRIDES_DECK_FILE_NAME}"-class StorageMethod(StrEnum):- LOCAL = "local"- IN_MEMORY = "in_memory"--class ValidationErrorReaction(StrEnum):RAISE = "raise"LOG = "log"@@ -62,31 +58,6 @@ class ValidationErrorConfig(ConfigModel):)-class StorageConfig(ConfigModel):- is_fetch_remote_content_enabled: bool- uri_format: str- method: StorageMethod = Field(strict=False)- local_storage_path: str | None = None-- @model_validator(mode="after")- def validate_storage_config(self) -> Self:- match self.method:- case StorageMethod.LOCAL:- if not self.local_storage_path:- msg = "local_storage_path is required when method is local"- raise PipelexConfigError(msg)- case StorageMethod.IN_MEMORY:- pass- return self-- @property- def storage_path(self) -> str:- if not self.local_storage_path:- msg = "local_storage_path is required when method is local"- raise PipelexConfigError(msg)- return self.local_storage_path--class PipeRunConfig(ConfigModel):pipe_stack_limit: intdiff --git a/pipelex/system/exceptions.py b/pipelex/system/exceptions.pyindex 6d2f9938d..52fa72d47 100644--- a/pipelex/system/exceptions.py+++ b/pipelex/system/exceptions.py@@ -13,6 +13,19 @@ class NestedKeyConflictError(ToolError):"""Raised when attempting to create nested keys under a non-dict value."""+class MissingDependencyError(PipelexError):+ """Raised when a required dependency is not installed."""++ def __init__(self, dependency_name: str, extra_name: str, message: str | None = None):+ self.dependency_name = dependency_name+ self.extra_name = extra_name+ error_msg = f"Required dependency '{dependency_name}' is not installed."+ if message:+ error_msg += f" {message}"+ error_msg += f" Please install it with 'pip install pipelex[{extra_name}]'."+ super().__init__(error_msg)++class CredentialsError(PipelexError):passdiff --git a/pipelex/tools/storage/exceptions.py b/pipelex/tools/storage/exceptions.pyindex b7e3f2131..9b62409df 100644--- a/pipelex/tools/storage/exceptions.py+++ b/pipelex/tools/storage/exceptions.py@@ -1,3 +1,4 @@+from pipelex.base_exceptions import PipelexConfigErrorfrom pipelex.system.exceptions import ToolError@@ -5,6 +6,10 @@ class StorageError(ToolError):"""Base exception for storage-related errors."""+class StorageConfigError(StorageError, PipelexConfigError):+ """Raised when a storage configuration is invalid."""++class StorageFileNotFoundError(StorageError):"""Raised when a requested file does not exist in storage."""@@ -15,3 +20,15 @@ class StorageInvalidUriError(StorageError):class StorageInvalidKeyError(StorageError):"""Raised when a storage key is invalid (e.g., contains scheme prefix)."""+++class StorageS3Error(StorageError):+ """Raised when an S3 storage operation fails."""+++class StorageGcpError(StorageError):+ """Raised when a GCP storage operation fails."""+++class StorageGcpCredentialsError(StorageGcpError):+ """Raised when GCP credentials file is not found or invalid."""diff --git a/pipelex/tools/storage/gcp_storage_provider.py b/pipelex/tools/storage/gcp_storage_provider.pynew file mode 100644index 000000000..a300104eb--- /dev/null+++ b/pipelex/tools/storage/gcp_storage_provider.py@@ -0,0 +1,220 @@+import asyncio+import importlib.util+from datetime import timedelta+from pathlib import Path+from typing import Any++from typing_extensions import override++from pipelex.system.exceptions import MissingDependencyError+from pipelex.tools.storage.exceptions import (+ StorageFileNotFoundError,+ StorageGcpCredentialsError,+ StorageGcpError,+)+from pipelex.tools.storage.storage_provider_abstract import StorageProviderAbstract++# GCS signed URL signing process version (v4 is recommended, v2 is deprecated)+GCS_SIGNED_URL_VERSION = "v4"+++class GcpStorageProvider(StorageProviderAbstract):+ """Storage provider implementation for Google Cloud Storage.++ Files are stored in a GCS bucket with keys being path strings.+ Uses asyncio.to_thread to wrap sync GCS operations.+ """++ def __init__(+ self,+ bucket_name: str,+ project_id: str,+ credentials_file_path: str,+ signed_urls_lifespan: int | None,+ ) -> None:+ """Initialize the GCP storage provider.++ Args:+ bucket_name: The GCS bucket name.+ project_id: The GCP project ID.+ credentials_file_path: Path to the service account credentials JSON file.+ signed_urls_lifespan: Lifespan in seconds for signed URLs, or None if disabled.+ """+ self._bucket_name = bucket_name+ self._project_id = project_id+ self._credentials_file_path = credentials_file_path+ self._signed_urls_lifespan = signed_urls_lifespan+ self._bucket: Any = None++ def _get_bucket(self) -> Any:+ """Get or create the GCS bucket client (lazy initialization).++ Returns:+ The GCS bucket object.++ Raises:+ MissingDependencyError: If google-cloud-storage is not installed.+ StorageGcpCredentialsError: If credentials file is not found.+ """+ if self._bucket is None:+ if importlib.util.find_spec("google.cloud.storage") is None:+ lib_name = "google-cloud-storage"+ lib_extra_name = "gcp-storage"+ msg = "google-cloud-storage is required for GCP storage."+ raise MissingDependencyError(+ lib_name,+ lib_extra_name,+ msg,+ )++ from google.cloud import storage # type: ignore[import-untyped] # noqa: PLC0415++ credentials_path = Path(self._credentials_file_path)+ if not credentials_path.exists():+ msg = f"GCP credentials file not found: '{self._credentials_file_path}'"+ raise StorageGcpCredentialsError(msg)++ client = storage.Client.from_service_account_json( # pyright: ignore[reportUnknownVariableType,reportUnknownMemberType]+ self._credentials_file_path,+ project=self._project_id,+ )+ self._bucket = client.bucket(self._bucket_name) # pyright: ignore[reportUnknownMemberType]+ return self._bucket # pyright: ignore[reportUnknownVariableType,reportUnknownMemberType]++ def _load_sync(self, key: str) -> bytes:+ """Synchronous implementation of load for use with to_thread.++ Args:+ key: Storage key (without scheme prefix).++ Returns:+ The object contents as bytes.++ Raises:+ StorageFileNotFoundError: If the object does not exist.+ StorageGcpError: If the GCS operation fails.+ """+ bucket = self._get_bucket()++ from google.api_core.exceptions import ( # type: ignore[import-untyped] # noqa: PLC0415 - optional dependency, lazy import+ GoogleAPIError,+ NotFound,+ )++ try:+ blob = bucket.blob(key)+ data: bytes = blob.download_as_bytes()+ return data+ except NotFound as exc: # pyright: ignore[reportUnknownVariableType]+ msg = f"Object not found in GCS: '{key}'"+ raise StorageFileNotFoundError(msg) from exc+ except GoogleAPIError as exc: # pyright: ignore[reportUnknownVariableType]+ msg = f"Failed to load object from GCS: '{key}': {exc}"+ raise StorageGcpError(msg) from exc++ @override+ async def _load(self, key: str) -> bytes:+ """Load bytes from a GCS object.++ Args:+ key: Storage key (without scheme prefix).++ Returns:+ The object contents as bytes.++ Raises:+ StorageFileNotFoundError: If the object does not exist.+ StorageGcpError: If the GCS operation fails.+ """+ return await asyncio.to_thread(self._load_sync, key)++ def _store_sync(self, data: bytes, key: str, content_type: str | None) -> None:+ """Synchronous implementation of store for use with to_thread.++ Args:+ data: The bytes to store.+ key: Storage key (without scheme prefix).+ content_type: Optional MIME type for the object.++ Raises:+ StorageGcpError: If the GCS operation fails.+ """+ bucket = self._get_bucket()++ from google.api_core.exceptions import GoogleAPIError # type: ignore[import-untyped] # noqa: PLC0415 - optional dependency, lazy import++ try:+ blob = bucket.blob(key)+ blob.upload_from_string(data, content_type=content_type)+ except GoogleAPIError as exc: # pyright: ignore[reportUnknownVariableType]+ msg = f"Failed to store object to GCS: '{key}': {exc}"+ raise StorageGcpError(msg) from exc++ @override+ async def _store(self, data: bytes, *, key: str, content_type: str | None) -> None:+ """Store bytes to a GCS object.++ Args:+ data: The bytes to store.+ key: Storage key (without scheme prefix).+ content_type: Optional MIME type for the object.++ Raises:+ StorageGcpError: If the GCS operation fails.+ """+ await asyncio.to_thread(self._store_sync, data, key, content_type)++ def _make_public_url(self, key: str) -> str:+ """Build a public URL for a GCS object.++ Args:+ key: Storage key (without scheme prefix).++ Returns:+ Public URL for the object.+ """+ return f"https://storage.googleapis.com/{self._bucket_name}/{key}"++ def _generate_signed_url_sync(self, key: str) -> str | None:+ """Synchronous implementation of signed URL generation for use with to_thread.++ Args:+ key: Storage key (without scheme prefix).++ Returns:+ Signed URL or public URL if signing fails.+ """+ bucket = self._get_bucket()++ from google.api_core.exceptions import GoogleAPIError # type: ignore[import-untyped] # noqa: PLC0415 - optional dependency, lazy import++ try:+ blob = bucket.blob(key)+ signed_url: str = blob.generate_signed_url(+ version=GCS_SIGNED_URL_VERSION,+ expiration=timedelta(seconds=self._signed_urls_lifespan or 0),+ method="GET",+ )+ return signed_url+ except (GoogleAPIError, ValueError, TypeError, AttributeError):+ # GoogleAPIError: GCS API failures+ # ValueError/TypeError: invalid arguments+ # AttributeError: credentials that cannot sign (e.g., Application Default Credentials)+ return self._make_public_url(key)++ @override+ async def display_link(self, uri: str) -> str | None:+ """Return a URL for this storage URI.++ Args:+ uri: Full URI including pipelex-storage:// scheme.++ Returns:+ Signed URL if signed_urls_lifespan is configured, otherwise a public URL.+ """+ key = self._strip_scheme(uri)++ if self._signed_urls_lifespan is None:+ return self._make_public_url(key)++ return await asyncio.to_thread(self._generate_signed_url_sync, key)diff --git a/pipelex/tools/storage/in_memory_storage_provider.py b/pipelex/tools/storage/in_memory_storage_provider.pyindex 1047b8696..0ededa282 100644--- a/pipelex/tools/storage/in_memory_storage_provider.py+++ b/pipelex/tools/storage/in_memory_storage_provider.py@@ -2,8 +2,8 @@ from pydantic import RootModelfrom typing_extensions import overridefrom pipelex import log-from pipelex.tools.storage.exceptions import StorageFileNotFoundError, StorageInvalidKeyError-from pipelex.tools.storage.storage_provider_abstract import PIPELEX_STORAGE_SCHEME, StorageProviderAbstract+from pipelex.tools.storage.exceptions import StorageFileNotFoundError+from pipelex.tools.storage.storage_provider_abstract import StorageProviderAbstractInMemoryStorageRoot = dict[str, bytes]@@ -13,58 +13,39 @@ class InMemoryStorageProvider(RootModel[InMemoryStorageRoot], StorageProviderAbsroot: InMemoryStorageRoot = {}- def _strip_scheme(self, uri: str) -> str:- """Extract key from URI, raising error if invalid."""- if not uri.startswith(PIPELEX_STORAGE_SCHEME):- msg = f"Invalid URI '{uri}': must start with '{PIPELEX_STORAGE_SCHEME}'"- raise StorageFileNotFoundError(msg)- return uri.removeprefix(PIPELEX_STORAGE_SCHEME)-- def _add_scheme(self, key: str) -> str:- """Build URI from key, raising error if key already has scheme."""- if key.startswith(PIPELEX_STORAGE_SCHEME):- msg = f"Key should not include scheme prefix: '{key}'"- raise StorageInvalidKeyError(msg)- return f"{PIPELEX_STORAGE_SCHEME}{key}"-@override- def load(self, uri: str) -> bytes:- """Load bytes from memory at the given URI.+ async def _load(self, key: str) -> bytes:+ """Load bytes from memory.Args:- uri: Full URI including PIPELEX_STORAGE_SCHEME prefix.+ key: Storage key (without scheme prefix).Returns:The stored bytes.Raises:- StorageFileNotFoundError: If no data exists for the URI.+ StorageFileNotFoundError: If no data exists for the key."""- key = self._strip_scheme(uri)if key not in self.root:msg = f"File not found: '{key}'"raise StorageFileNotFoundError(msg)- log.dev(f"Loaded data from URI: '{uri}'")+ log.dev(f"Loaded data from key: '{key}'")return self.root[key]@override- def store(self, data: bytes, key: str) -> str:+ async def _store(self, data: bytes, *, key: str, content_type: str | None) -> None:"""Store bytes in memory.Args:data: The bytes to store.key: Storage key (without scheme prefix).-- Returns:- Full URI with pipelex-storage:// scheme.+ content_type: Ignored for in-memory storage."""- uri = self._add_scheme(key)self.root[key] = data- return uri@override- def display_link(self, uri: str) -> str | None:+ async def display_link(self, uri: str) -> str | None:"""In-memory storage cannot generate a display link.Args:diff --git a/pipelex/tools/storage/local_storage_provider.py b/pipelex/tools/storage/local_storage_provider.pyindex 6fab368ff..de9851476 100644--- a/pipelex/tools/storage/local_storage_provider.py+++ b/pipelex/tools/storage/local_storage_provider.py@@ -1,9 +1,10 @@from pathlib import Path+import aiofilesfrom typing_extensions import override-from pipelex.tools.storage.exceptions import StorageFileNotFoundError, StorageInvalidKeyError, StorageInvalidUriError-from pipelex.tools.storage.storage_provider_abstract import PIPELEX_STORAGE_SCHEME, StorageProviderAbstract+from pipelex.tools.storage.exceptions import StorageFileNotFoundError, StorageInvalidUriError+from pipelex.tools.storage.storage_provider_abstract import StorageProviderAbstractclass LocalStorageProvider(StorageProviderAbstract):@@ -20,20 +21,6 @@ class LocalStorageProvider(StorageProviderAbstract):"""self._root_path = root_path- def _strip_scheme(self, uri: str) -> str:- """Extract key from URI, raising error if invalid."""- if not uri.startswith(PIPELEX_STORAGE_SCHEME):- msg = f"Invalid URI '{uri}': must start with '{PIPELEX_STORAGE_SCHEME}'"- raise StorageFileNotFoundError(msg)- return uri.removeprefix(PIPELEX_STORAGE_SCHEME)-- def _add_scheme(self, key: str) -> str:- """Build URI from key, raising error if key already has scheme."""- if key.startswith(PIPELEX_STORAGE_SCHEME):- msg = f"Key should not include scheme prefix: '{key}'"- raise StorageInvalidKeyError(msg)- return f"{PIPELEX_STORAGE_SCHEME}{key}"-def _validate_key(self, key: str) -> Path:"""Validate the key and return the resolved absolute path.@@ -64,11 +51,11 @@ class LocalStorageProvider(StorageProviderAbstract):return resolved_path@override- def load(self, uri: str) -> bytes:- """Load bytes from a file at the given URI.+ async def _load(self, key: str) -> bytes:+ """Load bytes from a file.Args:- uri: Full URI including PIPELEX_STORAGE_SCHEME prefix.+ key: Storage key (relative path, without scheme prefix).Returns:The file contents as bytes.@@ -77,42 +64,37 @@ class LocalStorageProvider(StorageProviderAbstract):StorageFileNotFoundError: If the file does not exist.StorageInvalidUriError: If the key is invalid."""- key = self._strip_scheme(uri)file_path = self._validate_key(key)if not file_path.exists():msg = f"File not found: '{key}'"raise StorageFileNotFoundError(msg)- return file_path.read_bytes()+ async with aiofiles.open(file_path, "rb") as file_handle: # pyright: ignore[reportUnknownMemberType]+ return await file_handle.read()@override- def store(self, data: bytes, key: str) -> str:+ async def _store(self, data: bytes, *, key: str, content_type: str | None) -> None:"""Store bytes to a file.Args:data: The bytes to store.key: Storage key (relative path, without scheme prefix).-- Returns:- Full URI with pipelex-storage:// scheme.+ content_type: Ignored for local storage.Raises:- StorageInvalidKeyError: If the key contains scheme prefix.StorageInvalidUriError: If the key is invalid."""- uri = self._add_scheme(key)file_path = self._validate_key(key)# Create parent directories if they don't existfile_path.parent.mkdir(parents=True, exist_ok=True)- file_path.write_bytes(data)-- return uri+ async with aiofiles.open(file_path, "wb") as file_handle: # pyright: ignore[reportUnknownMemberType]+ await file_handle.write(data)@override- def display_link(self, uri: str) -> str:+ async def display_link(self, uri: str) -> str:"""Return a file:// URI for this storage URI.Args:diff --git a/pipelex/tools/storage/s3_storage_provider.py b/pipelex/tools/storage/s3_storage_provider.pynew file mode 100644index 000000000..0b6445458--- /dev/null+++ b/pipelex/tools/storage/s3_storage_provider.py@@ -0,0 +1,218 @@+import importlib.util+import inspect+from typing import Any++from typing_extensions import override++from pipelex.system.exceptions import MissingDependencyError+from pipelex.tools.storage.exceptions import (+ StorageFileNotFoundError,+ StorageS3Error,+)+from pipelex.tools.storage.storage_provider_abstract import StorageProviderAbstract+++class S3StorageProvider(StorageProviderAbstract):+ """Storage provider implementation for AWS S3 storage.++ Files are stored in an S3 bucket with keys being path strings.+ Uses aioboto3 for async S3 operations.+ """++ def __init__(+ self,+ bucket_name: str,+ region: str,+ signed_urls_lifespan: int | None,+ ) -> None:+ """Initialize the S3 storage provider.++ Args:+ bucket_name: The S3 bucket name.+ region: The AWS region.+ signed_urls_lifespan: Lifespan in seconds for signed URLs, or None if disabled.+ """+ self._bucket_name = bucket_name+ self._region = region+ self._signed_urls_lifespan = signed_urls_lifespan+ self._session: Any = None++ def _check_dependency(self) -> None:+ """Check if aioboto3 is installed.++ Raises:+ MissingDependencyError: If aioboto3 is not installed.+ """+ if importlib.util.find_spec("aioboto3") is None:+ lib_name = "aioboto3"+ lib_extra_name = "s3"+ msg = "aioboto3 is required for S3 storage."+ raise MissingDependencyError(+ lib_name,+ lib_extra_name,+ msg,+ )++ def _get_session(self) -> Any:+ """Get or create the aioboto3 session (lazy initialization).++ Returns:+ The aioboto3 Session.+ """+ self._check_dependency()++ if self._session is None:+ import aioboto3 # noqa: PLC0415 - optional dependency, lazy import++ self._session = aioboto3.Session() # pyright: ignore[reportUnknownMemberType]+ return self._session++ def _get_client_config(self) -> dict[str, Any]:+ """Get the configuration for creating S3 clients.++ Returns:+ Dictionary of client configuration parameters.+ """+ from botocore.config import Config # noqa: PLC0415 - optional dependency, lazy import++ endpoint_url = f"https://s3.{self._region}.amazonaws.com"+ config = Config(signature_version="s3v4") # pyright: ignore[reportUnknownArgumentType]++ return {+ "service_name": "s3",+ "region_name": self._region,+ "endpoint_url": endpoint_url,+ "config": config,+ }++ @override+ async def _load(self, key: str) -> bytes:+ """Load bytes from an S3 object.++ Args:+ key: Storage key (without scheme prefix).++ Returns:+ The object contents as bytes.++ Raises:+ StorageFileNotFoundError: If the object does not exist.+ StorageS3Error: If the S3 operation fails.+ """+ from botocore.exceptions import ( # noqa: PLC0415 - optional dependency, lazy import+ ClientError,+ EndpointConnectionError,+ NoCredentialsError,+ )++ session = self._get_session()+ client_config = self._get_client_config()++ async with session.client(**client_config) as client: # pyright: ignore[reportUnknownMemberType]+ try:+ response = await client.get_object(Bucket=self._bucket_name, Key=key)+ async with response["Body"] as stream:+ data: bytes = await stream.read()+ return data+ except client.exceptions.NoSuchKey as exc:+ msg = f"Object not found in S3: '{key}'"+ raise StorageFileNotFoundError(msg) from exc+ except client.exceptions.NoSuchBucket as exc:+ msg = f"Bucket not found in S3: '{self._bucket_name}'"+ raise StorageS3Error(msg) from exc+ except ClientError as exc:+ error_code = (exc.response.get("Error") or {}).get("Code", "Unknown")+ if error_code == "NoSuchKey":+ msg = f"Object not found in S3: '{key}'"+ raise StorageFileNotFoundError(msg) from exc+ msg = f"S3 ClientError ({error_code}) for key '{key}'"+ raise StorageS3Error(msg) from exc+ except (NoCredentialsError, EndpointConnectionError) as exc:+ msg = f"S3 connectivity/credentials error for key '{key}'"+ raise StorageS3Error(msg) from exc++ @override+ async def _store(self, data: bytes, *, key: str, content_type: str | None) -> None:+ """Store bytes to an S3 object.++ Args:+ data: The bytes to store.+ key: Storage key (without scheme prefix).+ content_type: Optional MIME type for the object.++ Raises:+ StorageS3Error: If the S3 operation fails.+ """+ from botocore.exceptions import ( # noqa: PLC0415 - optional dependency, lazy import+ ClientError,+ EndpointConnectionError,+ NoCredentialsError,+ )++ session = self._get_session()+ client_config = self._get_client_config()++ async with session.client(**client_config) as client: # pyright: ignore[reportUnknownMemberType]+ try:+ put_params: dict[str, Any] = {+ "Bucket": self._bucket_name,+ "Key": key,+ "Body": data,+ }+ if content_type:+ put_params["ContentType"] = content_type+ await client.put_object(**put_params)+ except client.exceptions.NoSuchBucket as exc:+ msg = f"Bucket not found in S3: '{self._bucket_name}'"+ raise StorageS3Error(msg) from exc+ except ClientError as exc:+ error_code = (exc.response.get("Error") or {}).get("Code", "Unknown")+ msg = f"S3 ClientError ({error_code}) for key '{key}'"+ raise StorageS3Error(msg) from exc+ except (NoCredentialsError, EndpointConnectionError) as exc:+ msg = f"S3 connectivity/credentials error for key '{key}'"+ raise StorageS3Error(msg) from exc++ def _make_public_url(self, key: str) -> str:+ """Build a public URL for an S3 object.++ Args:+ key: Storage key (without scheme prefix).++ Returns:+ Public URL for the object.+ """+ return f"https://{self._bucket_name}.s3.{self._region}.amazonaws.com/{key}"++ @override+ async def display_link(self, uri: str) -> str | None:+ """Return a URL for this storage URI.++ Args:+ uri: Full URI including pipelex-storage:// scheme.++ Returns:+ Presigned URL if signed_urls_lifespan is configured, otherwise a public URL.+ """+ from botocore.exceptions import ClientError # noqa: PLC0415 - optional dependency, lazy import++ key = self._strip_scheme(uri)++ if self._signed_urls_lifespan is None:+ return self._make_public_url(key)++ session = self._get_session()+ client_config = self._get_client_config()++ async with session.client(**client_config) as client: # pyright: ignore[reportUnknownMemberType]+ try:+ # generate_presigned_url may be sync or async depending on aioboto3 version+ maybe_url = client.generate_presigned_url(+ "get_object",+ Params={"Bucket": self._bucket_name, "Key": key},+ ExpiresIn=self._signed_urls_lifespan,+ )+ presigned_url: str = await maybe_url if inspect.isawaitable(maybe_url) else maybe_url+ return presigned_url+ except ClientError:+ return self._make_public_url(key)diff --git a/pipelex/tools/storage/storage_config.py b/pipelex/tools/storage/storage_config.pynew file mode 100644index 000000000..c07bf45d1--- /dev/null+++ b/pipelex/tools/storage/storage_config.py@@ -0,0 +1,160 @@+from typing import Literal++from pydantic import Field, model_validator++from pipelex.system.configuration.config_model import ConfigModel+from pipelex.tools.storage.exceptions import StorageConfigError+from pipelex.types import Self, StrEnum+from pipelex.urls import URLs+++class StorageMethod(StrEnum):+ LOCAL = "local"+ IN_MEMORY = "in_memory"+ S3 = "s3"+ GCP = "gcp"+++class StorageLocalConfig(ConfigModel):+ uri_format: str+ local_storage_path: str+++class StorageInMemoryConfig(ConfigModel):+ uri_format: str+++class StorageS3Config(ConfigModel):+ uri_format: str+ bucket_name: str+ region: str+ signed_urls_lifespan_seconds: int | Literal["disabled"]++ @property+ def signed_urls_lifespan(self) -> int | None:+ """Return signed URL lifespan in seconds, or None if disabled."""+ if self.signed_urls_lifespan_seconds == "disabled":+ return None+ return self.signed_urls_lifespan_seconds++ def lazy_validate(self):+ error_msgs: list[str] = []++ if self.uri_format == "":+ error_msgs.append("- set a value for uri_format")+ elif "{hash}" not in self.uri_format:+ error_msgs.append("- uri_format must contain a {hash} placeholder")++ if self.bucket_name == "":+ error_msgs.append("- set a value for bucket_name")+ elif "." in self.bucket_name:+ error_msgs.append("- bucket_name cannot contain a dot")+ elif "/" in self.bucket_name:+ error_msgs.append("- bucket_name cannot contain a slash")++ if self.region == "":+ error_msgs.append("- set a value for region")++ if error_msgs:+ msg = "You have enabled storage on S3 so you need a proper S3 config.\n\nTo fix you S3 config:\n"+ msg += "\n".join(error_msgs)+ msg += f"\nThis can be done in the .pipelex/pipelex.toml file. More details can be found in the documentation: {URLs.documentation}"+ raise StorageConfigError(msg)+++class StorageGcpConfig(ConfigModel):+ uri_format: str+ bucket_name: str+ project_id: str+ signed_urls_lifespan_seconds: int | Literal["disabled"]++ @property+ def signed_urls_lifespan(self) -> int | None:+ """Return signed URL lifespan in seconds, or None if disabled."""+ if self.signed_urls_lifespan_seconds == "disabled":+ return None+ return self.signed_urls_lifespan_seconds++ def lazy_validate(self):+ error_msgs: list[str] = []++ if self.uri_format == "":+ error_msgs.append("set a value for uri_format")+ elif "hash" not in self.uri_format:+ error_msgs.append("uri_format must contain a {hash} placeholder")++ if self.bucket_name == "":+ error_msgs.append("set a value for bucket_name")+ elif "." in self.bucket_name:+ error_msgs.append("bucket_name cannot contain a dot")+ elif "/" in self.bucket_name:+ error_msgs.append("bucket_name cannot contain a slash")++ if self.project_id == "":+ error_msgs.append("set a value for project_id")++ if error_msgs:+ msg = "You have enabled storage on GCP so you need a proper GCP config:\n"+ msg += "\n".join(error_msgs)+ raise StorageConfigError(msg)+++class StorageConfig(ConfigModel):+ is_fetch_remote_content_enabled: bool+ method: StorageMethod = Field(strict=False)+ local: StorageLocalConfig | None = None+ in_memory: StorageInMemoryConfig | None = None+ s3: StorageS3Config | None = None+ gcp: StorageGcpConfig | None = None++ @model_validator(mode="after")+ def validate_storage_config(self) -> Self:+ match self.method:+ case StorageMethod.LOCAL:+ if not self.local:+ msg = "local config is required when method is local"+ raise StorageConfigError(msg)+ case StorageMethod.IN_MEMORY:+ if not self.in_memory:+ msg = "in_memory config is required when method is in_memory"+ raise StorageConfigError(msg)+ case StorageMethod.S3:+ if not self.s3:+ msg = "s3 config is required when method is s3"+ raise StorageConfigError(msg)+ case StorageMethod.GCP:+ if not self.gcp:+ msg = "gcp config is required when method is gcp"+ raise StorageConfigError(msg)+ return self++ @property+ def storage_path(self) -> str:+ if not self.local:+ msg = "local config is required when method is local"+ raise StorageConfigError(msg)+ return self.local.local_storage_path++ @property+ def uri_format(self) -> str:+ match self.method:+ case StorageMethod.LOCAL:+ if not self.local:+ msg = "local config is required to access uri_format"+ raise StorageConfigError(msg)+ return self.local.uri_format+ case StorageMethod.IN_MEMORY:+ if not self.in_memory:+ msg = "in_memory config is required to access uri_format"+ raise StorageConfigError(msg)+ return self.in_memory.uri_format+ case StorageMethod.S3:+ if not self.s3:+ msg = "s3 config is required to access uri_format"+ raise StorageConfigError(msg)+ return self.s3.uri_format+ case StorageMethod.GCP:+ if not self.gcp:+ msg = "gcp config is required to access uri_format"+ raise StorageConfigError(msg)+ return self.gcp.uri_formatdiff --git a/pipelex/tools/storage/storage_provider_abstract.py b/pipelex/tools/storage/storage_provider_abstract.pyindex ca2e3c2df..a025066ef 100644--- a/pipelex/tools/storage/storage_provider_abstract.py+++ b/pipelex/tools/storage/storage_provider_abstract.py@@ -4,8 +4,56 @@ PIPELEX_STORAGE_SCHEME = "pipelex-storage://"class StorageProviderAbstract(ABC):- @abstractmethod- def load(self, uri: str) -> bytes:+ """Abstract base class for storage providers.++ Provides common URI scheme handling (strip/add scheme) and defines+ the interface for concrete storage providers.++ Subclasses must implement:+ - _load(key): Load data by key (without scheme)+ - _store(data, key, content_type): Store data and return URI+ - display_link(uri): Return human-readable link for URI+ """++ def _strip_scheme(self, uri: str) -> str:+ """Extract key from URI, raising error if invalid.++ Args:+ uri: Full URI including PIPELEX_STORAGE_SCHEME prefix.++ Returns:+ The key part without the scheme prefix.++ Raises:+ StorageFileNotFoundError: If URI doesn't start with the expected scheme.+ """+ from pipelex.tools.storage.exceptions import StorageFileNotFoundError # noqa: PLC0415++ if not uri.startswith(PIPELEX_STORAGE_SCHEME):+ msg = f"Invalid URI '{uri}': must start with '{PIPELEX_STORAGE_SCHEME}'"+ raise StorageFileNotFoundError(msg)+ return uri.removeprefix(PIPELEX_STORAGE_SCHEME)++ def _add_scheme(self, key: str) -> str:+ """Build URI from key, raising error if key already has scheme.++ Args:+ key: Storage key without scheme prefix.++ Returns:+ Full URI with PIPELEX_STORAGE_SCHEME prefix.++ Raises:+ StorageInvalidKeyError: If key already contains the scheme prefix.+ """+ from pipelex.tools.storage.exceptions import StorageInvalidKeyError # noqa: PLC0415++ if key.startswith(PIPELEX_STORAGE_SCHEME):+ msg = f"Key should not include scheme prefix: '{key}'"+ raise StorageInvalidKeyError(msg)+ return f"{PIPELEX_STORAGE_SCHEME}{key}"++ async def load(self, uri: str) -> bytes:"""Load data from storage.Args:@@ -14,21 +62,47 @@ class StorageProviderAbstract(ABC):Returns:The stored bytes."""+ key = self._strip_scheme(uri)+ return await self._load(key)@abstractmethod- def store(self, data: bytes, key: str) -> str:+ async def _load(self, key: str) -> bytes:+ """Load data from storage by key.++ Args:+ key: Storage key (without scheme prefix).++ Returns:+ The stored bytes.+ """++ async def store(self, data: bytes, key: str, content_type: str | None = None) -> str:"""Store data and return full URI with scheme.Args:data: The bytes to store.key: Storage key (without scheme prefix).+ content_type: Optional MIME type for the data (used by cloud providers).Returns:Full URI with PIPELEX_STORAGE_SCHEME prefix."""+ uri = self._add_scheme(key)+ await self._store(data=data, key=key, content_type=content_type)+ return uri++ @abstractmethod+ async def _store(self, data: bytes, *, key: str, content_type: str | None) -> None:+ """Store data in storage.++ Args:+ data: The bytes to store.+ key: Storage key (without scheme prefix).+ content_type: Optional MIME type for the data.+ """@abstractmethod- def display_link(self, uri: str) -> str | None:+ async def display_link(self, uri: str) -> str | None:"""Return human-readable link for this URI.Args:diff --git a/pipelex/tools/storage/storage_provider_factory.py b/pipelex/tools/storage/storage_provider_factory.pynew file mode 100644index 000000000..4e3d93d02--- /dev/null+++ b/pipelex/tools/storage/storage_provider_factory.py@@ -0,0 +1,56 @@+from pathlib import Path++from pipelex import log+from pipelex.hub import get_secrets_provider+from pipelex.tools.storage.exceptions import StorageConfigError+from pipelex.tools.storage.gcp_storage_provider import GcpStorageProvider+from pipelex.tools.storage.in_memory_storage_provider import InMemoryStorageProvider+from pipelex.tools.storage.local_storage_provider import LocalStorageProvider+from pipelex.tools.storage.s3_storage_provider import S3StorageProvider+from pipelex.tools.storage.storage_config import StorageConfig, StorageMethod+from pipelex.tools.storage.storage_provider_abstract import StorageProviderAbstract+++def make_storage_provider_from_config(storage_config: StorageConfig) -> StorageProviderAbstract:+ """Create a storage provider based on the provided storage configuration.++ Args:+ storage_config: The storage configuration specifying method and provider-specific settings.++ Returns:+ A configured storage provider instance.++ Raises:+ PipelexConfigError: If required provider-specific config is missing.+ """+ match storage_config.method:+ case StorageMethod.LOCAL:+ log.verbose(f"Using local storage at: {storage_config.storage_path}")+ return LocalStorageProvider(root_path=Path(storage_config.storage_path))+ case StorageMethod.IN_MEMORY:+ log.verbose("Using in-memory storage")+ return InMemoryStorageProvider()+ case StorageMethod.S3:+ if storage_config.s3 is None:+ msg = "S3 config is required when method is s3"+ raise StorageConfigError(msg)++ storage_config.s3.lazy_validate()+ log.verbose(f"Using S3 storage: bucket={storage_config.s3.bucket_name}, region={storage_config.s3.region}")+ return S3StorageProvider(+ bucket_name=storage_config.s3.bucket_name,+ region=storage_config.s3.region,+ signed_urls_lifespan=storage_config.s3.signed_urls_lifespan,+ )+ case StorageMethod.GCP:+ if storage_config.gcp is None:+ msg = "GCP config is required when method is gcp"+ raise StorageConfigError(msg)+ storage_config.gcp.lazy_validate()+ log.verbose(f"Using GCP storage: bucket={storage_config.gcp.bucket_name}, project={storage_config.gcp.project_id}")+ return GcpStorageProvider(+ bucket_name=storage_config.gcp.bucket_name,+ project_id=storage_config.gcp.project_id,+ credentials_file_path=get_secrets_provider().get_required_secret(secret_id="GCP_CREDENTIALS_FILE_PATH"),+ signed_urls_lifespan=storage_config.gcp.signed_urls_lifespan,+ )diff --git a/pyproject.toml b/pyproject.tomlindex add5de857..731f4199f 100644--- a/pyproject.toml+++ b/pyproject.toml@@ -61,10 +61,12 @@ anthropic = ["anthropic>=0.60.0"]bedrock = ["boto3>=1.34.131", "aioboto3>=13.4.0"]docling = ["docling>=2.64.0"]fal = ["fal-client>=0.4.1"]+gcp-storage = ["google-cloud-storage>=2.10.0"]google = ["google-auth-oauthlib>=1.2.1"]google-genai = ["google-genai", "instructor[google-genai]"]huggingface = ["huggingface_hub>=0.23,<1.0.0"]mistralai = ["mistralai==1.5.2"]+s3 = ["boto3>=1.34.131", "aioboto3>=13.4.0"]docs = ["mkdocs==1.6.1","mkdocs-glightbox==0.4.0",@@ -74,6 +76,7 @@ docs = [dev = ["boto3-stubs>=1.35.24",+ "moto[s3]>=5.0.0","mypy>=1.11.2","pyright>=1.1.405","pylint>=3.3.8",diff --git a/tests/integration/pipelex/cogt/image/test_data_url_image_flow.py b/tests/integration/pipelex/cogt/image/test_data_url_image_flow.pyindex 36edb4e29..3f9470a20 100644--- a/tests/integration/pipelex/cogt/image/test_data_url_image_flow.py+++ b/tests/integration/pipelex/cogt/image/test_data_url_image_flow.py@@ -84,7 +84,7 @@ class TestDataUrlImageFlow:working_memory = WorkingMemoryFactory.make_from_single_stuff(stuff=stuff)# Normalize- normalized_memory = normalize_data_urls_to_storage(working_memory)+ normalized_memory = await normalize_data_urls_to_storage(working_memory)# Verify the URL was converted to pipelex-storage://normalized_stuff = normalized_memory.get_stuff("test_image")@@ -92,7 +92,7 @@ class TestDataUrlImageFlow:assert normalized_stuff.content.url.startswith("pipelex-storage://")# Verify the data is stored correctly and matches original- stored_bytes = provider.load(uri=normalized_stuff.content.url)+ stored_bytes = await provider.load(uri=normalized_stuff.content.url)# Extract base64 data from the data URL for comparisonbase64_data = data_url.split(",", 1)[1]expected_bytes = base64.b64decode(base64_data)@@ -121,7 +121,7 @@ class TestDataUrlImageFlow:working_memory = WorkingMemoryFactory.make_from_single_stuff(stuff=stuff)# Normalize to storage- normalized_memory = normalize_data_urls_to_storage(working_memory)+ normalized_memory = await normalize_data_urls_to_storage(working_memory)normalized_stuff = normalized_memory.get_stuff("test_image")assert isinstance(normalized_stuff.content, ImageContent)storage_uri = normalized_stuff.content.url@@ -162,7 +162,7 @@ class TestDataUrlImageFlow:working_memory = WorkingMemoryFactory.make_from_single_stuff(stuff=stuff)# Normalize- normalized_memory = normalize_data_urls_to_storage(working_memory)+ normalized_memory = await normalize_data_urls_to_storage(working_memory)# Verify all images were convertednormalized_stuff = normalized_memory.get_stuff("image_list")@@ -193,7 +193,7 @@ class TestDataUrlImageFlow:working_memory = WorkingMemoryFactory.make_from_single_stuff(stuff=stuff)# Normalize- normalized_memory = normalize_data_urls_to_storage(working_memory)+ normalized_memory = await normalize_data_urls_to_storage(working_memory)# Verify URL was NOT changednormalized_stuff = normalized_memory.get_stuff("test_image")@@ -223,7 +223,7 @@ class TestDataUrlImageFlow:working_memory = WorkingMemoryFactory.make_from_single_stuff(stuff=stuff)# Normalize- normalized_memory = normalize_data_urls_to_storage(working_memory)+ normalized_memory = await normalize_data_urls_to_storage(working_memory)# Verify the embedded image URL was converted to pipelex-storage://normalized_stuff = normalized_memory.get_stuff("test_article")@@ -234,7 +234,7 @@ class TestDataUrlImageFlow:assert normalized_article.image.url.startswith("pipelex-storage://")# Verify the stored data is correct- stored_bytes = provider.load(uri=normalized_article.image.url)+ stored_bytes = await provider.load(uri=normalized_article.image.url)expected_bytes = base64.b64decode(ImageTestCases.MINIMAL_PNG_BASE64)assert stored_bytes == expected_bytes@@ -263,7 +263,7 @@ class TestDataUrlImageFlow:working_memory = WorkingMemoryFactory.make_from_single_stuff(stuff=stuff)# Normalize- normalized_memory = normalize_data_urls_to_storage(working_memory)+ normalized_memory = await normalize_data_urls_to_storage(working_memory)# Verify all image URLs were convertednormalized_stuff = normalized_memory.get_stuff("nested_article")@@ -278,10 +278,10 @@ class TestDataUrlImageFlow:assert normalized_nested.related_image.url.startswith("pipelex-storage://")# Verify both stored data are correct- nested_stored = provider.load(uri=normalized_nested.main_article.image.url)+ nested_stored = await provider.load(uri=normalized_nested.main_article.image.url)assert nested_stored == base64.b64decode(ImageTestCases.MINIMAL_PNG_BASE64)- related_stored = provider.load(uri=normalized_nested.related_image.url)+ related_stored = await provider.load(uri=normalized_nested.related_image.url)assert related_stored == base64.b64decode(ImageTestCases.MINIMAL_JPEG_BASE64)async def test_normalization_with_list_in_structured_content(@@ -313,7 +313,7 @@ class TestDataUrlImageFlow:working_memory = WorkingMemoryFactory.make_from_single_stuff(stuff=stuff)# Normalize- normalized_memory = normalize_data_urls_to_storage(working_memory)+ normalized_memory = await normalize_data_urls_to_storage(working_memory)# Verify all images in the list were convertednormalized_stuff = normalized_memory.get_stuff("gallery")diff --git a/tests/integration/pipelex/cogt/image/test_pipelex_storage_image_flow.py b/tests/integration/pipelex/cogt/image/test_pipelex_storage_image_flow.pyindex 1791306b9..ca3bfd534 100644--- a/tests/integration/pipelex/cogt/image/test_pipelex_storage_image_flow.py+++ b/tests/integration/pipelex/cogt/image/test_pipelex_storage_image_flow.py@@ -11,6 +11,7 @@ These tests verify the complete flow:import base64import pytest+import pytest_asynciofrom pytest_mock import MockerFixturefrom pipelex.cogt.image.prepared_image import PreparedImageBase64@@ -27,14 +28,14 @@ from pipelex.tools.uri.uri_resolver import resolve_uriTEST_IMAGE_PATH = "tests/data/images/eiffel_tower.png"-def storage_provider_with_test_image() -> tuple[InMemoryStorageProvider, str]:+@pytest_asyncio.fixture+async def storage_provider_with_test_image() -> tuple[InMemoryStorageProvider, str]:"""Create an in-memory storage with a real test image and return (provider, uri)."""provider = InMemoryStorageProvider()# Load a real test imageimage_bytes = load_binary(path=TEST_IMAGE_PATH)key = "pipeline_run_123/generated_image.png"- uri = provider.store(data=image_bytes, key=key)+ uri = await provider.store(data=image_bytes, key=key)return provider, uridiff --git a/tests/integration/pipelex/cogt/test_img_gen.py b/tests/integration/pipelex/cogt/test_img_gen.pyindex d06f1d854..22ec7d96d 100644--- a/tests/integration/pipelex/cogt/test_img_gen.py+++ b/tests/integration/pipelex/cogt/test_img_gen.py@@ -30,8 +30,7 @@ class TestImageGeneration:generated_content_factory: GeneratedContentFactory,):pretty_print(f"Testing image generation with handle '{img_gen_handle}', output format '{img_gen_job_params.output_format}'")- pretty_print(positive_text, title="Positive text")- pretty_print(negative_text, title="Negative text")+ pretty_print(f"Positive text: {positive_text}\nNegative text: {negative_text}", title="Prompts")img_gen_worker_async = get_img_gen_worker(img_gen_handle=img_gen_handle)img_gen_job = ImgGenJobFactory.make_img_gen_job_from_prompt_contents(positive_text=positive_text,diff --git a/tests/integration/pipelex/fixtures/generator_fixtures.py b/tests/integration/pipelex/fixtures/generator_fixtures.pyindex 918838a50..125af854e 100644--- a/tests/integration/pipelex/fixtures/generator_fixtures.py+++ b/tests/integration/pipelex/fixtures/generator_fixtures.py@@ -5,13 +5,53 @@ import pytestfrom pipelex.cogt.content_generation.content_generator import ContentGeneratorfrom pipelex.cogt.content_generation.content_generator_protocol import ContentGeneratorProtocolfrom pipelex.cogt.content_generation.generated_content_factory import GeneratedContentFactory-from pipelex.tools.storage.local_storage_provider import LocalStorageProvider+from pipelex.config import get_config+from pipelex.tools.storage.storage_config import StorageMethod+from pipelex.tools.storage.storage_provider_factory import make_storage_provider_from_config++S3_TEST_BUCKET = "pipelex-storage-test"+GCP_TEST_BUCKET = "pipelex-storage-test"+TEST_SIGNED_URLS_LIFESPAN = 300@pytest.fixturedef generated_content_factory(tmp_path: Path) -> GeneratedContentFactory:- """Create a GeneratedContentFactory with a local storage provider."""- storage_provider = LocalStorageProvider(root_path=tmp_path)+ """Create a GeneratedContentFactory with storage provider based on config.++ Applies test overrides for bucket names and signed URL lifespans.+ """+ storage_config = get_config().pipelex.storage_config+ match storage_config.method:+ case StorageMethod.S3:+ assert storage_config.s3 is not None+ patched_s3_config = storage_config.s3.model_copy(+ update={+ "bucket_name": S3_TEST_BUCKET,+ "signed_urls_lifespan_seconds": TEST_SIGNED_URLS_LIFESPAN,+ }+ )+ patched_storage_config = storage_config.model_copy(update={"s3": patched_s3_config})+ case StorageMethod.GCP:+ assert storage_config.gcp is not None+ patched_gcp_config = storage_config.gcp.model_copy(+ update={+ "bucket_name": GCP_TEST_BUCKET,+ "signed_urls_lifespan_seconds": TEST_SIGNED_URLS_LIFESPAN,+ }+ )+ patched_storage_config = storage_config.model_copy(update={"gcp": patched_gcp_config})+ case StorageMethod.LOCAL:+ assert storage_config.local is not None+ patched_local_config = storage_config.local.model_copy(+ update={+ "local_storage_path": tmp_path,+ }+ )+ patched_storage_config = storage_config.model_copy(update={"local": patched_local_config})+ case StorageMethod.IN_MEMORY:+ patched_storage_config = storage_config++ storage_provider = make_storage_provider_from_config(patched_storage_config)return GeneratedContentFactory(storage_provider=storage_provider)diff --git a/tests/integration/pipelex/fixtures/img_gen_fixtures.py b/tests/integration/pipelex/fixtures/img_gen_fixtures.pyindex 4bbbc7c45..e62511022 100644--- a/tests/integration/pipelex/fixtures/img_gen_fixtures.py+++ b/tests/integration/pipelex/fixtures/img_gen_fixtures.py@@ -83,7 +83,7 @@ def img_gen_handle(request: pytest.FixtureRequest) -> str:ImgGenJobParams(aspect_ratio=AspectRatio.SQUARE,background=Background.OPAQUE,- nb_steps=10,+ nb_steps=8,guidance_scale=2.5,is_moderated=None,safety_tolerance=1,diff --git a/tests/integration/pipelex/tools/storage/conftest.py b/tests/integration/pipelex/tools/storage/conftest.pynew file mode 100644index 000000000..bc50ddcbc--- /dev/null+++ b/tests/integration/pipelex/tools/storage/conftest.py@@ -0,0 +1,248 @@+from pathlib import Path+from typing import Any, Literal, cast+from unittest.mock import AsyncMock, MagicMock++import pytest+from pytest_mock import MockerFixture++from pipelex.cogt.content_generation.generated_content_factory import GeneratedContentFactory+from pipelex.config import get_config+from pipelex.tools.storage.gcp_storage_provider import GcpStorageProvider+from pipelex.tools.storage.in_memory_storage_provider import InMemoryStorageProvider+from pipelex.tools.storage.local_storage_provider import LocalStorageProvider+from pipelex.tools.storage.s3_storage_provider import S3StorageProvider+from pipelex.tools.storage.storage_provider_abstract import StorageProviderAbstract++StorageMethodLiteral = Literal["local", "in_memory", "s3", "gcp"]++# Constants for S3 mocking+S3_TEST_BUCKET = "test-pipelex-bucket"+S3_TEST_REGION = "us-east-1"++# Constants for GCP mocking+GCP_TEST_BUCKET = "test-gcp-pipelex-bucket"+GCP_TEST_PROJECT = "test-project"++[email protected](params=["local", "in_memory", "s3", "gcp"])+def storage_method(request: pytest.FixtureRequest) -> StorageMethodLiteral:+ """Parametrized fixture that yields each storage method."""+ return cast("StorageMethodLiteral", request.param)+++def s3_mock(mocker: MockerFixture) -> dict[str, Any]:+ """Mock aioboto3 for S3 tests.++ Creates a mock that simulates S3 behavior using an in-memory dict.+ """+ # Storage for mock data+ mock_data_storage: dict[str, bytes] = {}++ def create_mock_stream(key: str) -> AsyncMock:+ """Create a mock async stream for response body."""+ stream = AsyncMock()+ stream.read = AsyncMock(side_effect=lambda: mock_data_storage.get(key, b""))+ stream.__aenter__ = AsyncMock(return_value=stream)+ stream.__aexit__ = AsyncMock(return_value=None)+ return stream++ # Create mock exceptions+ mock_exceptions = MagicMock()+ mock_exceptions.NoSuchKey = type("NoSuchKey", (Exception,), {})+ mock_exceptions.NoSuchBucket = type("NoSuchBucket", (Exception,), {})+ mock_exceptions.ClientError = type("ClientError", (Exception,), {})++ def mock_get_object(Bucket: str, Key: str) -> dict[str, Any]: # noqa: ARG001, N803+ if Key not in mock_data_storage:+ msg = f"Key {Key} not found"+ raise mock_exceptions.NoSuchKey(msg)+ return {"Body": create_mock_stream(Key)}++ def mock_put_object(Bucket: str, Key: str, Body: bytes, **kwargs: Any) -> dict[str, Any]: # noqa: ARG001, N803+ mock_data_storage[Key] = Body+ return {}++ def mock_generate_presigned_url(+ method: str, # noqa: ARG001+ Params: dict[str, str], # noqa: N803+ ExpiresIn: int, # noqa: ARG001, N803+ ) -> str:+ key = Params.get("Key", "")+ return f"https://{S3_TEST_BUCKET}.s3.{S3_TEST_REGION}.amazonaws.com/{key}?X-Amz-Signature=mock"++ # Create mock client with sync functions wrapped in AsyncMock+ mock_client = AsyncMock()+ mock_client.get_object = AsyncMock(side_effect=mock_get_object)+ mock_client.put_object = AsyncMock(side_effect=mock_put_object)+ mock_client.generate_presigned_url = AsyncMock(side_effect=mock_generate_presigned_url)+ mock_client.exceptions = mock_exceptions++ # Create async context manager for client+ mock_client_context = AsyncMock()+ mock_client_context.__aenter__ = AsyncMock(return_value=mock_client)+ mock_client_context.__aexit__ = AsyncMock(return_value=None)++ # Create mock session+ mock_session = MagicMock()+ mock_session.client = MagicMock(return_value=mock_client_context)++ # Patch aioboto3.Session+ mocker.patch("aioboto3.Session", return_value=mock_session)++ return {+ "session": mock_session,+ "client": mock_client,+ "storage": mock_data_storage,+ }+++def gcp_mock(tmp_path: Path, mocker: MockerFixture) -> dict[str, Any]:+ """Mock GCP storage for tests.++ Creates a mock that simulates GCS behavior using an in-memory dict.+ """+ # Import the module first so it can be patched+ from google.cloud import storage # type: ignore[import-untyped] # noqa: PLC0415++ # Create mock credentials file+ credentials_path = tmp_path / "gcp_credentials.json"+ credentials_path.write_text('{"type": "service_account"}')++ # Storage for mock data+ mock_data_storage: dict[str, bytes] = {}++ # Create mock blob that behaves like a real blob+ def create_mock_blob(key: str) -> MagicMock:+ blob = MagicMock()+ blob.exists.side_effect = lambda: key in mock_data_storage+ blob.download_as_bytes.side_effect = lambda: mock_data_storage[key]+ blob.upload_from_string.side_effect = lambda data, content_type=None: mock_data_storage.__setitem__(key, data) # noqa: ARG005 # pyright: ignore[reportUnknownLambdaType,reportUnknownArgumentType]+ blob.generate_signed_url.return_value = f"https://storage.googleapis.com/{GCP_TEST_BUCKET}/{key}?signed=true"+ return blob++ # Create mock bucket+ mock_bucket = MagicMock()+ mock_bucket.blob.side_effect = create_mock_blob++ # Create mock client+ mock_client = MagicMock()+ mock_client.bucket.return_value = mock_bucket++ # Patch the Client.from_service_account_json method directly+ mocker.patch.object(+ storage.Client,+ "from_service_account_json",+ return_value=mock_client,+ )++ return {+ "credentials_path": str(credentials_path),+ "storage": mock_data_storage,+ }+++def storage_provider(+ storage_method: StorageMethodLiteral,+ tmp_path: Path,+ request: pytest.FixtureRequest,+) -> StorageProviderAbstract:+ """Create a storage provider based on the storage method."""+ match storage_method:+ case "local":+ return LocalStorageProvider(root_path=tmp_path)+ case "in_memory":+ return InMemoryStorageProvider()+ case "s3":+ # Request the s3_mock fixture to set up aioboto3 mocks+ request.getfixturevalue("s3_mock")+ return S3StorageProvider(+ bucket_name=S3_TEST_BUCKET,+ region=S3_TEST_REGION,+ signed_urls_lifespan=None,+ )+ case "gcp":+ # Request the gcp_mock fixture to set up mocks+ gcp_mock_data = request.getfixturevalue("gcp_mock")+ return GcpStorageProvider(+ bucket_name=GCP_TEST_BUCKET,+ project_id=GCP_TEST_PROJECT,+ credentials_file_path=gcp_mock_data["credentials_path"],+ signed_urls_lifespan=None,+ )+++def uri_format(storage_method: StorageMethodLiteral) -> str:+ """Get the URI format for the given storage method from the config."""+ storage_config = get_config().pipelex.storage_config+ match storage_method:+ case "local":+ assert storage_config.local is not None+ return storage_config.local.uri_format+ case "in_memory":+ assert storage_config.in_memory is not None+ return storage_config.in_memory.uri_format+ case "s3":+ assert storage_config.s3 is not None+ return storage_config.s3.uri_format+ case "gcp":+ assert storage_config.gcp is not None+ return storage_config.gcp.uri_format+++def generated_content_factory(storage_provider: StorageProviderAbstract) -> GeneratedContentFactory:+ """Create a GeneratedContentFactory with the parametrized storage provider.++ This fixture is useful for testing generated image flows with different storage methods.+ Future S3/GCP implementations should also work with this pattern.+ """+ return GeneratedContentFactory(storage_provider=storage_provider)+++def mock_fetch_remote_content_enabled(mocker: MockerFixture) -> None:+ """Mock the config to enable fetching remote content.++ When `is_fetch_remote_content_enabled` is True, remote HTTP URLs are fetched+ and stored locally instead of being passed through.+ """+ original_config = get_config()+ mocker.patch.object(+ original_config.pipelex.storage_config,+ "is_fetch_remote_content_enabled",+ True,+ )+++def mock_fetch_remote_content_disabled(mocker: MockerFixture) -> None:+ """Mock the config to disable fetching remote content.++ When `is_fetch_remote_content_enabled` is False, remote HTTP URLs are passed+ through without being fetched and stored.+ """+ original_config = get_config()+ mocker.patch.object(+ original_config.pipelex.storage_config,+ "is_fetch_remote_content_enabled",+ False,+ )+++def storage_provider_patched(+ storage_provider: StorageProviderAbstract,+ mocker: MockerFixture,+) -> StorageProviderAbstract:+ """Patch get_storage_provider to return the parametrized storage provider.++ This is useful for tests that need to verify the full flow where internal+ code calls get_storage_provider() to obtain the storage provider.+ """+ mocker.patch("pipelex.pipeline.input_normalizer.get_storage_provider", return_value=storage_provider)+ mocker.patch("pipelex.cogt.image.prompt_image_utils.get_storage_provider", return_value=storage_provider)+ return storage_providerdiff --git a/tests/integration/pipelex/tools/storage/test_generated_image_storage.py b/tests/integration/pipelex/tools/storage/test_generated_image_storage.pynew file mode 100644index 000000000..41caee1ff--- /dev/null+++ b/tests/integration/pipelex/tools/storage/test_generated_image_storage.py@@ -0,0 +1,152 @@+"""Integration tests for generated images with storage providers.++These tests verify the flow: GeneratedImageRawDetails -> GeneratedContentFactory -> Storage -> Retrieval++Future storage implementations (S3, GCP) must pass these tests to ensure+compatibility with AI-generated image handling.+"""++import base64++import pytest+from pytest_mock import MockerFixture++from pipelex.cogt.content_generation.generated_content_factory import GeneratedContentFactory+from pipelex.cogt.image.generated_image import GeneratedImageRawDetails+from pipelex.cogt.image.image_size import ImageSize+from pipelex.cogt.image.prepared_image import PreparedImageBase64+from pipelex.cogt.image.prompt_image_factory import PromptImageFactory+from pipelex.cogt.image.prompt_image_utils import prepare_prompt_image+from pipelex.tools.misc.file_utils import load_binary+from pipelex.tools.storage.storage_provider_abstract import PIPELEX_STORAGE_SCHEME, StorageProviderAbstract+from tests.cases import ImageTestCases++[email protected](loop_scope="class")+class TestGeneratedImageStorage:+ """Integration tests for generated images with storage providers.++ These tests verify the flow: GeneratedImageRawDetails -> GeneratedContentFactory -> Storage -> Retrieval++ Future storage implementations (S3, GCP) must pass these tests to ensure+ compatibility with AI-generated image handling.+ """++ async def test_generated_image_from_bytes(+ self,+ generated_content_factory: GeneratedContentFactory,+ mocker: MockerFixture,+ storage_provider: StorageProviderAbstract,+ ) -> None:+ """Test GeneratedContentFactory stores images from raw bytes correctly.++ This simulates the flow when an image generation API returns raw bytes+ (e.g., from a PIL image or binary response).+ """+ # Mock get_storage_provider for the retrieval step+ mocker.patch("pipelex.cogt.image.prompt_image_utils.get_storage_provider", return_value=storage_provider)++ # Load a real test image as bytes+ image_bytes = load_binary(path=ImageTestCases.IMAGE_FILE_PATH_LOGO_TINY)++ # Create GeneratedImageRawDetails with raw bytes+ raw_details = GeneratedImageRawDetails(+ size=ImageSize(width=18, height=18),+ actual_bytes=image_bytes,+ mime_type="image/png",+ )++ # Generate ImageContent via factory+ image_content = await generated_content_factory.make_image_content(+ primary_id="test_pipeline",+ secondary_id="test_step",+ raw_details=raw_details,+ )++ # Verify the URL is a pipelex-storage:// URI+ assert image_content.url.startswith(PIPELEX_STORAGE_SCHEME)++ # Verify roundtrip: storage -> PreparedImage+ prompt_image = PromptImageFactory.make_prompt_image(uri=image_content.url)+ prepared = await prepare_prompt_image(+ prompt_image=prompt_image,+ is_http_url_enabled=False,+ )++ assert isinstance(prepared, PreparedImageBase64)+ # Verify the data matches the original+ decoded = base64.b64decode(prepared.base64_data)+ assert decoded == image_bytes++ async def test_generated_image_from_base64(+ self,+ generated_content_factory: GeneratedContentFactory,+ mocker: MockerFixture,+ storage_provider: StorageProviderAbstract,+ ) -> None:+ """Test GeneratedContentFactory stores images from base64 strings correctly.++ This simulates the flow when an image generation API returns a base64-encoded+ image string (common with OpenAI, Stability AI, etc.).+ """+ # Mock get_storage_provider for the retrieval step+ mocker.patch("pipelex.cogt.image.prompt_image_utils.get_storage_provider", return_value=storage_provider)++ # Create GeneratedImageRawDetails with base64 string+ raw_details = GeneratedImageRawDetails(+ size=ImageSize(width=1, height=1),+ base64_str=ImageTestCases.MINIMAL_PNG_BASE64,+ mime_type="image/png",+ )++ # Generate ImageContent via factory+ image_content = await generated_content_factory.make_image_content(+ primary_id="test_pipeline",+ secondary_id="test_step",+ raw_details=raw_details,+ )++ # Verify the URL is a pipelex-storage:// URI+ assert image_content.url.startswith(PIPELEX_STORAGE_SCHEME)++ # Verify roundtrip: storage -> PreparedImage+ prompt_image = PromptImageFactory.make_prompt_image(uri=image_content.url)+ prepared = await prepare_prompt_image(+ prompt_image=prompt_image,+ is_http_url_enabled=False,+ )++ assert isinstance(prepared, PreparedImageBase64)+ assert prepared.base64_data == ImageTestCases.MINIMAL_PNG_BASE64++ async def test_generated_image_uri_format_applied(+ self,+ generated_content_factory: GeneratedContentFactory,+ ) -> None:+ """Test that the uri_format from config is correctly applied.++ Each storage method has its own uri_format configured. This test verifies+ that the format is used when building storage keys.+ """+ # Create GeneratedImageRawDetails+ raw_details = GeneratedImageRawDetails(+ size=ImageSize(width=1, height=1),+ base64_str=ImageTestCases.MINIMAL_PNG_BASE64,+ mime_type="image/png",+ )++ # Generate ImageContent+ image_content = await generated_content_factory.make_image_content(+ primary_id="pipeline_abc",+ secondary_id="step_xyz",+ raw_details=raw_details,+ )++ # Extract the key from the URI (remove scheme prefix)+ storage_key = image_content.url.removeprefix(PIPELEX_STORAGE_SCHEME)++ # Verify the key follows the uri_format pattern+ # The format is "{primary_id}/{secondary_id}/{hash}.{extension}"+ assert "pipeline_abc" in storage_key+ assert "step_xyz" in storage_key+ assert storage_key.endswith(".png")diff --git a/tests/integration/pipelex/tools/storage/test_storage_config.py b/tests/integration/pipelex/tools/storage/test_storage_config.pynew file mode 100644index 000000000..1a2e58096--- /dev/null+++ b/tests/integration/pipelex/tools/storage/test_storage_config.py@@ -0,0 +1,43 @@+from pipelex.config import get_config+from pipelex.tools.storage.storage_config import StorageMethod+++class TestStorageConfigIntegration:+ """Integration tests for storage config values."""++ def test_config_has_local_uri_format(self) -> None:+ """Test that local storage config has uri_format defined."""+ storage_config = get_config().pipelex.storage_config+ assert storage_config.local is not None+ assert storage_config.local.uri_format is not None+ assert "{primary_id}" in storage_config.local.uri_format+ assert "{secondary_id}" in storage_config.local.uri_format+ assert "{hash}" in storage_config.local.uri_format+ assert "{extension}" in storage_config.local.uri_format++ def test_config_has_in_memory_uri_format(self) -> None:+ """Test that in-memory storage config has uri_format defined."""+ storage_config = get_config().pipelex.storage_config+ assert storage_config.in_memory is not None+ assert storage_config.in_memory.uri_format is not None+ assert "{primary_id}" in storage_config.in_memory.uri_format+ assert "{secondary_id}" in storage_config.in_memory.uri_format+ assert "{hash}" in storage_config.in_memory.uri_format+ assert "{extension}" in storage_config.in_memory.uri_format++ def test_uri_format_property_returns_correct_format_for_local(self) -> None:+ """Test that the uri_format property returns the correct format when method is local."""+ storage_config = get_config().pipelex.storage_config+ match storage_config.method:+ case StorageMethod.LOCAL:+ assert storage_config.local is not None+ assert storage_config.uri_format == storage_config.local.uri_format+ case StorageMethod.IN_MEMORY:+ assert storage_config.in_memory is not None+ assert storage_config.uri_format == storage_config.in_memory.uri_format+ case StorageMethod.S3:+ assert storage_config.s3 is not None+ assert storage_config.uri_format == storage_config.s3.uri_format+ case StorageMethod.GCP:+ assert storage_config.gcp is not None+ assert storage_config.uri_format == storage_config.gcp.uri_formatdiff --git a/tests/integration/pipelex/tools/storage/test_storage_provider_contract.py b/tests/integration/pipelex/tools/storage/test_storage_provider_contract.pynew file mode 100644index 000000000..6647f6659--- /dev/null+++ b/tests/integration/pipelex/tools/storage/test_storage_provider_contract.py@@ -0,0 +1,159 @@+"""Tests that verify storage providers implement the full contract.++These tests document the requirements that any storage provider (including+future S3/GCP implementations) must satisfy to work with ImageContent.++When implementing a new storage provider, ensure all these tests pass.++Storage Provider Contract Requirements:++- store(data: bytes, key: str) -> str: Must return a pipelex-storage:// URI+- load(uri: str) -> bytes: Must load data from a pipelex-storage:// URI+- display_link(uri: str) -> str | None: Must return a human-readable link or None+"""++import pytest++from pipelex.tools.misc.file_utils import load_binary+from pipelex.tools.storage.storage_provider_abstract import PIPELEX_STORAGE_SCHEME, StorageProviderAbstract+from tests.cases import ImageTestCases++[email protected](loop_scope="class")+class TestStorageProviderContract:+ """Tests that verify storage providers implement the full contract.++ These tests document the requirements that any storage provider (including+ future S3/GCP implementations) must satisfy to work with ImageContent.++ When implementing a new storage provider, ensure all these tests pass.+ """++ async def test_store_returns_pipelex_storage_uri(+ self,+ storage_provider: StorageProviderAbstract,+ ) -> None:+ """Contract: store() must return a URI with pipelex-storage:// scheme.++ This is critical because the entire image flow depends on recognizing+ pipelex-storage:// URIs to route them to the storage provider.+ """+ test_data = b"contract test data"+ key = "contract/test.bin"++ uri = await storage_provider.store(data=test_data, key=key)++ assert uri.startswith(PIPELEX_STORAGE_SCHEME)+ assert uri == f"{PIPELEX_STORAGE_SCHEME}{key}"++ async def test_load_retrieves_stored_data(+ self,+ storage_provider: StorageProviderAbstract,+ ) -> None:+ """Contract: load() must retrieve exactly the data that was stored.++ Data integrity is essential for image handling. The bytes retrieved+ must be identical to the bytes stored.+ """+ # Use real image data for a realistic test+ image_bytes = load_binary(path=ImageTestCases.IMAGE_FILE_PATH_LOGO_TINY)+ key = "contract/image.png"++ uri = await storage_provider.store(data=image_bytes, key=key)+ retrieved = await storage_provider.load(uri=uri)++ assert retrieved == image_bytes++ async def test_display_link_returns_string_or_none(+ self,+ storage_provider: StorageProviderAbstract,+ ) -> None:+ """Contract: display_link() must return either a string or None.++ - Local storage: Returns file:// URI for terminal clickability+ - In-memory storage: Returns None (no persistent location)+ - S3/GCP storage: Should return https:// URL or signed URL++ This behavior must be documented for each storage type.+ """+ test_data = b"display link test"+ key = "contract/display.bin"++ uri = await storage_provider.store(data=test_data, key=key)+ display = await storage_provider.display_link(uri=uri)++ # Must be string or None+ assert display is None or isinstance(display, str)++ # If string, it should be a valid link format+ if display is not None:+ assert len(display) > 0++ async def test_handles_nested_key_paths(+ self,+ storage_provider: StorageProviderAbstract,+ ) -> None:+ """Contract: store() must handle nested key paths with forward slashes.++ The uri_format uses paths like "{primary_id}/{secondary_id}/{hash}.{extension}"+ which creates nested directories/paths. Storage providers must support this.+ """+ test_data = b"nested path test"+ key = "level1/level2/level3/file.png"++ uri = await storage_provider.store(data=test_data, key=key)+ retrieved = await storage_provider.load(uri=uri)++ assert retrieved == test_data++ async def test_handles_overwrite(+ self,+ storage_provider: StorageProviderAbstract,+ ) -> None:+ """Contract: store() with same key must overwrite existing data.++ This is important for regenerating images or retrying failed operations.+ """+ key = "contract/overwrite.png"+ original = b"original data"+ updated = b"updated data"++ await storage_provider.store(data=original, key=key)+ uri = await storage_provider.store(data=updated, key=key)+ retrieved = await storage_provider.load(uri=uri)++ assert retrieved == updated++ async def test_handles_empty_data(+ self,+ storage_provider: StorageProviderAbstract,+ ) -> None:+ """Contract: store() and load() must handle empty bytes.++ While unusual for images, edge cases should be handled gracefully.+ """+ key = "contract/empty.bin"++ uri = await storage_provider.store(data=b"", key=key)+ retrieved = await storage_provider.load(uri=uri)++ assert retrieved == b""++ async def test_handles_large_binary_data(+ self,+ storage_provider: StorageProviderAbstract,+ ) -> None:+ """Contract: store() and load() must handle larger binary data.++ Images can be several megabytes. This test uses ~256KB to verify+ the provider handles non-trivial data sizes.+ """+ # ~256KB of binary data+ large_data = bytes(range(256)) * 1000+ key = "contract/large.bin"++ uri = await storage_provider.store(data=large_data, key=key)+ retrieved = await storage_provider.load(uri=uri)++ assert retrieved == large_data+ assert len(retrieved) == len(large_data)diff --git a/tests/integration/pipelex/tools/storage/test_storage_providers.py b/tests/integration/pipelex/tools/storage/test_storage_providers.pynew file mode 100644index 000000000..0bc6d4efb--- /dev/null+++ b/tests/integration/pipelex/tools/storage/test_storage_providers.py@@ -0,0 +1,127 @@+import pytest++from pipelex.tools.storage.storage_provider_abstract import PIPELEX_STORAGE_SCHEME, StorageProviderAbstract++[email protected](loop_scope="class")+class TestStorageProviders:+ """Integration tests for both Local and InMemory storage providers."""++ async def test_store_and_load_roundtrip(self, storage_provider: StorageProviderAbstract) -> None:+ """Test that data can be stored and loaded correctly."""+ test_data = b"Hello, World! \x00\x01\x02\xff"+ key = "test/roundtrip.bin"++ returned_uri = await storage_provider.store(data=test_data, key=key)+ loaded_data = await storage_provider.load(uri=returned_uri)++ assert loaded_data == test_data+ assert returned_uri == f"{PIPELEX_STORAGE_SCHEME}{key}"++ async def test_uri_format_from_config(+ self,+ storage_provider: StorageProviderAbstract,+ uri_format: str,+ ) -> None:+ """Test that the URI format from config works correctly for building storage keys."""+ test_data = b"config format test"+ primary_id = "pipeline_123"+ secondary_id = "step_456"+ hash_value = "abc123def456"+ extension = "png"++ key = uri_format.format(+ primary_id=primary_id,+ secondary_id=secondary_id,+ hash=hash_value,+ extension=extension,+ )++ returned_uri = await storage_provider.store(data=test_data, key=key)+ loaded_data = await storage_provider.load(uri=returned_uri)++ assert loaded_data == test_data+ assert primary_id in key+ assert secondary_id in key+ assert hash_value in key+ assert extension in key++ async def test_store_binary_image_data(self, storage_provider: StorageProviderAbstract) -> None:+ """Test storing and retrieving binary image data (PNG header)."""+ png_header = b"\x89PNG\r\n\x1a\n\x00\x00\x00\rIHDR"+ key = "images/test_image.png"++ returned_uri = await storage_provider.store(data=png_header, key=key)+ loaded_data = await storage_provider.load(uri=returned_uri)++ assert loaded_data == png_header+ assert loaded_data[:8] == b"\x89PNG\r\n\x1a\n"++ async def test_store_multiple_files_isolated(self, storage_provider: StorageProviderAbstract) -> None:+ """Test that multiple files are stored and retrieved independently."""+ data1 = b"first file content"+ data2 = b"second file content"+ data3 = b"third file content"++ uri1 = await storage_provider.store(data=data1, key="multi/file1.bin")+ uri2 = await storage_provider.store(data=data2, key="multi/file2.bin")+ uri3 = await storage_provider.store(data=data3, key="multi/file3.bin")++ assert await storage_provider.load(uri=uri1) == data1+ assert await storage_provider.load(uri=uri2) == data2+ assert await storage_provider.load(uri=uri3) == data3++ async def test_store_with_nested_path(self, storage_provider: StorageProviderAbstract) -> None:+ """Test storing data in a deeply nested path structure."""+ test_data = b"nested content"+ key = "level1/level2/level3/deep_file.bin"++ returned_uri = await storage_provider.store(data=test_data, key=key)+ loaded_data = await storage_provider.load(uri=returned_uri)++ assert loaded_data == test_data++ async def test_store_empty_bytes(self, storage_provider: StorageProviderAbstract) -> None:+ """Test storing and loading empty bytes."""+ key = "empty/file.bin"++ returned_uri = await storage_provider.store(data=b"", key=key)+ loaded_data = await storage_provider.load(uri=returned_uri)++ assert loaded_data == b""++ async def test_store_large_binary_data(self, storage_provider: StorageProviderAbstract) -> None:+ """Test storing and loading larger binary data (~256KB)."""+ test_data = bytes(range(256)) * 1000+ key = "large/file.bin"++ returned_uri = await storage_provider.store(data=test_data, key=key)+ loaded_data = await storage_provider.load(uri=returned_uri)++ assert loaded_data == test_data+ assert len(loaded_data) == len(test_data)++ async def test_overwrite_existing_data(self, storage_provider: StorageProviderAbstract) -> None:+ """Test that storing with the same key overwrites the previous data."""+ key = "overwrite/test.bin"+ original_data = b"original"+ updated_data = b"updated content"++ await storage_provider.store(data=original_data, key=key)+ returned_uri = await storage_provider.store(data=updated_data, key=key)+ loaded_data = await storage_provider.load(uri=returned_uri)++ assert loaded_data == updated_data++ async def test_display_link_returns_value_or_none(self, storage_provider: StorageProviderAbstract) -> None:+ """Test that display_link returns either a valid link or None."""+ test_data = b"display link test"+ key = "display/test.bin"++ returned_uri = await storage_provider.store(data=test_data, key=key)+ display = await storage_provider.display_link(uri=returned_uri)++ # display_link can return None (in-memory) or a string (local file://)+ if display is not None:+ assert isinstance(display, str)+ assert len(display) > 0diff --git a/tests/integration/pipelex/tools/storage/test_user_provided_image_storage.py b/tests/integration/pipelex/tools/storage/test_user_provided_image_storage.pynew file mode 100644index 000000000..2fb43cf3e--- /dev/null+++ b/tests/integration/pipelex/tools/storage/test_user_provided_image_storage.py@@ -0,0 +1,122 @@+"""Integration tests for user-provided images with storage providers.++These tests verify the flow: User Input -> normalize_data_urls_to_storage -> Storage -> Retrieval++Future storage implementations (S3, GCP) must pass these tests to ensure+compatibility with user-provided image handling.+"""++import base64++import pytest++from pipelex.cogt.image.prepared_image import PreparedImageBase64+from pipelex.cogt.image.prompt_image import PromptImageUri+from pipelex.cogt.image.prompt_image_factory import PromptImageFactory+from pipelex.cogt.image.prompt_image_utils import prepare_prompt_image+from pipelex.core.concepts.concept_factory import ConceptFactory+from pipelex.core.concepts.native.concept_native import NativeConceptCode+from pipelex.core.memory.working_memory_factory import WorkingMemoryFactory+from pipelex.core.stuffs.image_content import ImageContent+from pipelex.core.stuffs.stuff_factory import StuffFactory+from pipelex.pipeline.input_normalizer import normalize_data_urls_to_storage+from pipelex.tools.storage.storage_provider_abstract import PIPELEX_STORAGE_SCHEME+from tests.cases import ImageTestCases++[email protected](loop_scope="class")[email protected]("storage_provider_patched")+class TestUserProvidedImageStorage:+ """Integration tests for user-provided images with storage providers.++ These tests verify the flow: User Input -> normalize_data_urls_to_storage -> Storage -> Retrieval++ Future storage implementations (S3, GCP) must pass these tests to ensure+ compatibility with user-provided image handling.+ """++ async def test_user_image_data_url_to_storage_roundtrip(self) -> None:+ """Test that user-provided data URL images are normalized to storage and can be retrieved.++ Flow:+ 1. User provides ImageContent with data:// URL+ 2. normalize_data_urls_to_storage converts to pipelex-storage:// URI+ 3. Image can be retrieved via PromptImageFactory -> PreparedImage+ """+ # Create ImageContent with data URL (simulating user input)+ image_content = ImageContent(url=ImageTestCases.MINIMAL_PNG_DATA_URL)+ stuff = StuffFactory.make_stuff(+ concept=ConceptFactory.make_native_concept(native_concept_code=NativeConceptCode.IMAGE),+ content=image_content,+ name="user_image",+ )+ working_memory = WorkingMemoryFactory.make_from_single_stuff(stuff=stuff)++ # Normalize to storage+ normalized_memory = await normalize_data_urls_to_storage(working_memory)++ # Verify the URL was converted to pipelex-storage://+ normalized_stuff = normalized_memory.get_stuff("user_image")+ assert isinstance(normalized_stuff.content, ImageContent)+ assert normalized_stuff.content.url.startswith(PIPELEX_STORAGE_SCHEME)++ # Verify roundtrip: storage -> PreparedImage+ prompt_image = PromptImageFactory.make_prompt_image(uri=normalized_stuff.content.url)+ assert isinstance(prompt_image, PromptImageUri)++ prepared = await prepare_prompt_image(+ prompt_image=prompt_image,+ is_http_url_enabled=False,+ )++ assert isinstance(prepared, PreparedImageBase64)+ # Verify the data matches the original+ assert prepared.base64_data == ImageTestCases.MINIMAL_PNG_BASE64++ async def test_user_image_local_file_to_prepared(self) -> None:+ """Test that user-provided local file paths work through the image flow.++ Local file paths are not normalized to storage but should still work+ when converted to PreparedImage for LLM consumption.+ """+ # Create ImageContent with local file path+ local_path = ImageTestCases.IMAGE_FILE_PATH_LOGO_TINY+ image_content = ImageContent(url=local_path)++ # Convert to PreparedImage (bypassing normalization since local paths aren't data URLs)+ prompt_image = PromptImageFactory.make_prompt_image(uri=image_content.url)+ prepared = await prepare_prompt_image(+ prompt_image=prompt_image,+ is_http_url_enabled=False,+ )++ assert isinstance(prepared, PreparedImageBase64)+ assert prepared.file_type.mime == "image/png"+ # Verify the data can be decoded+ decoded = base64.b64decode(prepared.base64_data)+ assert len(decoded) > 0++ @pytest.mark.usefixtures("mock_fetch_remote_content_disabled")+ async def test_user_image_http_url_passthrough_when_fetch_disabled(self) -> None:+ """Test that HTTP URLs are passed through when fetch is disabled.++ When is_fetch_remote_content_enabled=False, HTTP URLs should not be+ fetched and stored. They should be passed through unchanged.+ """+ # Create ImageContent with HTTP URL+ http_url = "https://example.com/test_image.png"+ image_content = ImageContent(url=http_url)+ stuff = StuffFactory.make_stuff(+ concept=ConceptFactory.make_native_concept(native_concept_code=NativeConceptCode.IMAGE),+ content=image_content,+ name="remote_image",+ )+ working_memory = WorkingMemoryFactory.make_from_single_stuff(stuff=stuff)++ # Normalize (should not change HTTP URLs when fetch is disabled)+ normalized_memory = await normalize_data_urls_to_storage(working_memory)++ # Verify URL was NOT changed+ normalized_stuff = normalized_memory.get_stuff("remote_image")+ assert isinstance(normalized_stuff.content, ImageContent)+ assert normalized_stuff.content.url == http_urldiff --git a/tests/pipelex_unit_test.toml b/tests/pipelex_unit_test.tomlindex 927ffbd21..08017527c 100644--- a/tests/pipelex_unit_test.toml+++ b/tests/pipelex_unit_test.toml@@ -1,4 +1,7 @@+[pipelex.storage_config]+method = "local"+[cogt.model_deck_config]# Allow tests to run with restrictive routing profiles that don't support all presetsmissing_presets_reaction = "none"diff --git a/tests/unit/pipelex/plugins/test_plugin_pipelex_storage_images.py b/tests/unit/pipelex/plugins/test_plugin_pipelex_storage_images.pyindex 7c0ee7849..dca346eeb 100644--- a/tests/unit/pipelex/plugins/test_plugin_pipelex_storage_images.py+++ b/tests/unit/pipelex/plugins/test_plugin_pipelex_storage_images.py@@ -4,6 +4,7 @@ These tests verify that all plugins can handle images stored via pipelex-storage"""import pytest+import pytest_asynciofrom google.genai import types as genai_typesfrom mistralai.models import ImageURLChunkfrom pytest_mock import MockerFixture@@ -89,12 +90,12 @@ MINIMAL_PNG_BYTES = bytes()-def storage_with_image() -> tuple[InMemoryStorageProvider, str]:+@pytest_asyncio.fixture+async def storage_with_image() -> tuple[InMemoryStorageProvider, str]:"""Create an in-memory storage with a test image and return (provider, uri)."""provider = InMemoryStorageProvider()key = "test_images/sample.png"- uri = provider.store(data=MINIMAL_PNG_BYTES, key=key)+ uri = await provider.store(data=MINIMAL_PNG_BYTES, key=key)return provider, uridiff --git a/tests/unit/pipelex/tools/storage/test_gcp_storage_provider.py b/tests/unit/pipelex/tools/storage/test_gcp_storage_provider.pynew file mode 100644index 000000000..449a61e61--- /dev/null+++ b/tests/unit/pipelex/tools/storage/test_gcp_storage_provider.py@@ -0,0 +1,270 @@+from datetime import timedelta+from pathlib import Path+from typing import Any+from unittest.mock import MagicMock++import pytest+from pytest_mock import MockerFixture++from pipelex.tools.storage.exceptions import (+ StorageFileNotFoundError,+ StorageGcpCredentialsError,+ StorageInvalidKeyError,+)+from pipelex.tools.storage.gcp_storage_provider import GCS_SIGNED_URL_VERSION, GcpStorageProvider+from pipelex.tools.storage.storage_provider_abstract import PIPELEX_STORAGE_SCHEME++[email protected](loop_scope="class")+class TestGcpStorageProvider:+ """Unit tests for GcpStorageProvider using mocks for google.cloud.storage."""++ @pytest.fixture+ def gcp_bucket_name(self) -> str:+ """Return the test bucket name."""+ return "test-pipelex-bucket"++ @pytest.fixture+ def gcp_project_id(self) -> str:+ """Return the test GCP project ID."""+ return "test-project-id"++ @pytest.fixture+ def gcp_credentials_file(self, tmp_path: Path) -> str:+ """Create a mock credentials file and return its path."""+ credentials_path = tmp_path / "credentials.json"+ credentials_path.write_text('{"type": "service_account"}')+ return str(credentials_path)++ @pytest.fixture+ def mock_gcp_storage(self, mocker: MockerFixture) -> dict[str, Any]:+ """Mock the google.cloud.storage module.++ Returns a dict containing the mocked client, bucket, and blob objects+ for test assertions.+ """+ # Import the module first so it can be patched+ from google.cloud import storage # type: ignore[import-untyped] # noqa: PLC0415++ # Create mock objects+ mock_blob = MagicMock()+ mock_blob.exists.return_value = True+ mock_blob.download_as_bytes.return_value = b""+ mock_blob.upload_from_string = MagicMock()+ mock_blob.generate_signed_url.return_value = "https://storage.googleapis.com/signed-url"++ mock_bucket = MagicMock()+ mock_bucket.blob.return_value = mock_blob++ mock_client = MagicMock()+ mock_client.bucket.return_value = mock_bucket++ # Patch the Client.from_service_account_json method directly+ mock_from_service_account = mocker.patch.object(+ storage.Client,+ "from_service_account_json",+ return_value=mock_client,+ )++ return {+ "client": mock_client,+ "bucket": mock_bucket,+ "blob": mock_blob,+ "from_service_account_json": mock_from_service_account,+ }++ @pytest.fixture+ def gcp_provider_no_signed_urls(+ self,+ gcp_bucket_name: str,+ gcp_project_id: str,+ gcp_credentials_file: str,+ mock_gcp_storage: dict[str, Any], # noqa: ARG002+ ) -> GcpStorageProvider:+ """Create a GcpStorageProvider with signed URLs disabled."""+ return GcpStorageProvider(+ bucket_name=gcp_bucket_name,+ project_id=gcp_project_id,+ credentials_file_path=gcp_credentials_file,+ signed_urls_lifespan=None,+ )++ @pytest.fixture+ def gcp_provider_with_signed_urls(+ self,+ gcp_bucket_name: str,+ gcp_project_id: str,+ gcp_credentials_file: str,+ mock_gcp_storage: dict[str, Any], # noqa: ARG002+ ) -> GcpStorageProvider:+ """Create a GcpStorageProvider with signed URLs enabled (1 hour lifespan)."""+ return GcpStorageProvider(+ bucket_name=gcp_bucket_name,+ project_id=gcp_project_id,+ credentials_file_path=gcp_credentials_file,+ signed_urls_lifespan=3600,+ )++ async def test_store_returns_uri_with_scheme(+ self,+ gcp_provider_no_signed_urls: GcpStorageProvider,+ ) -> None:+ """Test that store() returns a URI with the pipelex-storage:// scheme prefix."""+ test_data = b"Hello, World!"+ key = "test/file.bin"++ returned_uri = await gcp_provider_no_signed_urls.store(data=test_data, key=key)++ assert returned_uri == f"{PIPELEX_STORAGE_SCHEME}{key}"+ assert returned_uri.startswith(PIPELEX_STORAGE_SCHEME)++ async def test_store_calls_upload_from_string(+ self,+ gcp_provider_no_signed_urls: GcpStorageProvider,+ mock_gcp_storage: dict[str, Any],+ ) -> None:+ """Test that store() correctly calls the GCS upload method."""+ test_data = b"Test data"+ key = "test/file.bin"+ content_type = "application/octet-stream"++ await gcp_provider_no_signed_urls.store(data=test_data, key=key, content_type=content_type)++ mock_gcp_storage["bucket"].blob.assert_called_once_with(key)+ mock_gcp_storage["blob"].upload_from_string.assert_called_once_with(+ test_data,+ content_type=content_type,+ )++ async def test_load_returns_data(+ self,+ gcp_provider_no_signed_urls: GcpStorageProvider,+ mock_gcp_storage: dict[str, Any],+ ) -> None:+ """Test that load() returns the data from GCS."""+ expected_data = b"Test data \x00\x01\x02\xff"+ mock_gcp_storage["blob"].download_as_bytes.return_value = expected_data++ uri = f"{PIPELEX_STORAGE_SCHEME}test/file.bin"+ loaded_data = await gcp_provider_no_signed_urls.load(uri=uri)++ assert loaded_data == expected_data++ async def test_load_with_nonexistent_key_raises_error(+ self,+ gcp_provider_no_signed_urls: GcpStorageProvider,+ mock_gcp_storage: dict[str, Any],+ ) -> None:+ """Test that loading a non-existent object raises StorageFileNotFoundError."""+ from google.api_core.exceptions import NotFound # type: ignore[import-untyped] # noqa: PLC0415++ mock_gcp_storage["blob"].download_as_bytes.side_effect = NotFound("Object not found")+ nonexistent_uri = f"{PIPELEX_STORAGE_SCHEME}nonexistent/file.bin"++ with pytest.raises(StorageFileNotFoundError) as exc_info:+ await gcp_provider_no_signed_urls.load(uri=nonexistent_uri)++ assert "nonexistent/file.bin" in str(exc_info.value)++ async def test_store_raises_if_key_has_scheme_prefix(+ self,+ gcp_provider_no_signed_urls: GcpStorageProvider,+ ) -> None:+ """Test that passing a key with pipelex-storage:// prefix raises an error."""+ invalid_key = f"{PIPELEX_STORAGE_SCHEME}already/prefixed.bin"++ with pytest.raises(StorageInvalidKeyError) as exc_info:+ await gcp_provider_no_signed_urls.store(data=b"test", key=invalid_key)++ assert "should not include scheme prefix" in str(exc_info.value).lower()++ async def test_display_link_returns_public_url_when_signed_urls_disabled(+ self,+ gcp_provider_no_signed_urls: GcpStorageProvider,+ gcp_bucket_name: str,+ ) -> None:+ """Test that display_link() returns a public URL when signed URLs are disabled."""+ key = "display/test.bin"+ uri = f"{PIPELEX_STORAGE_SCHEME}{key}"++ display = await gcp_provider_no_signed_urls.display_link(uri=uri)++ expected_url = f"https://storage.googleapis.com/{gcp_bucket_name}/{key}"+ assert display == expected_url++ async def test_display_link_returns_signed_url_when_signed_urls_enabled(+ self,+ gcp_provider_with_signed_urls: GcpStorageProvider,+ mock_gcp_storage: dict[str, Any],+ ) -> None:+ """Test that display_link() returns a signed URL when signed URLs are enabled."""+ key = "presigned/test.bin"+ uri = f"{PIPELEX_STORAGE_SCHEME}{key}"+ expected_signed_url = "https://storage.googleapis.com/signed-url?signature=xyz"+ mock_gcp_storage["blob"].generate_signed_url.return_value = expected_signed_url++ display = await gcp_provider_with_signed_urls.display_link(uri=uri)++ assert display == expected_signed_url+ mock_gcp_storage["blob"].generate_signed_url.assert_called_once_with(+ version=GCS_SIGNED_URL_VERSION,+ expiration=timedelta(seconds=3600),+ method="GET",+ )++ async def test_credentials_file_not_found_raises_error(+ self,+ gcp_bucket_name: str,+ gcp_project_id: str,+ ) -> None:+ """Test that a missing credentials file raises StorageGcpCredentialsError."""+ provider = GcpStorageProvider(+ bucket_name=gcp_bucket_name,+ project_id=gcp_project_id,+ credentials_file_path="/nonexistent/credentials.json",+ signed_urls_lifespan=None,+ )++ with pytest.raises(StorageGcpCredentialsError) as exc_info:+ await provider.store(data=b"test", key="test.bin")++ assert "credentials file not found" in str(exc_info.value).lower()++ async def test_store_with_nested_path(+ self,+ gcp_provider_no_signed_urls: GcpStorageProvider,+ mock_gcp_storage: dict[str, Any],+ ) -> None:+ """Test storing data in a deeply nested path structure."""+ test_data = b"nested content"+ key = "level1/level2/level3/deep_file.bin"++ returned_uri = await gcp_provider_no_signed_urls.store(data=test_data, key=key)++ assert returned_uri == f"{PIPELEX_STORAGE_SCHEME}{key}"+ mock_gcp_storage["bucket"].blob.assert_called_with(key)++ async def test_store_empty_bytes(+ self,+ gcp_provider_no_signed_urls: GcpStorageProvider,+ mock_gcp_storage: dict[str, Any],+ ) -> None:+ """Test storing empty bytes."""+ key = "empty.bin"++ returned_uri = await gcp_provider_no_signed_urls.store(data=b"", key=key)++ assert returned_uri == f"{PIPELEX_STORAGE_SCHEME}{key}"+ mock_gcp_storage["blob"].upload_from_string.assert_called_once_with(b"", content_type=None)++ async def test_multiple_operations_use_same_bucket(+ self,+ gcp_provider_no_signed_urls: GcpStorageProvider,+ mock_gcp_storage: dict[str, Any],+ ) -> None:+ """Test that multiple operations reuse the same bucket (lazy initialization)."""+ await gcp_provider_no_signed_urls.store(data=b"data1", key="file1.bin")+ await gcp_provider_no_signed_urls.store(data=b"data2", key="file2.bin")++ # Client should only be created once (lazy initialization)+ mock_gcp_storage["from_service_account_json"].assert_called_once()diff --git a/tests/unit/pipelex/tools/storage/test_in_memory_storage_provider.py b/tests/unit/pipelex/tools/storage/test_in_memory_storage_provider.pyindex 8d32f60f0..bfe2e6914 100644--- a/tests/unit/pipelex/tools/storage/test_in_memory_storage_provider.py+++ b/tests/unit/pipelex/tools/storage/test_in_memory_storage_provider.py@@ -5,99 +5,100 @@ from pipelex.tools.storage.in_memory_storage_provider import InMemoryStorageProvfrom pipelex.tools.storage.storage_provider_abstract import PIPELEX_STORAGE_SCHEME[email protected](loop_scope="class")class TestInMemoryStorageProvider:"""Unit tests for InMemoryStorageProvider with pipelex-storage:// URI scheme."""- def test_store_returns_uri_with_scheme(self) -> None:+ async def test_store_returns_uri_with_scheme(self) -> None:"""Test that store() returns a URI with the pipelex-storage:// scheme prefix."""provider = InMemoryStorageProvider()test_data = b"Hello, World!"key = "test/file.bin"- returned_uri = provider.store(data=test_data, key=key)+ returned_uri = await provider.store(data=test_data, key=key)assert returned_uri == f"{PIPELEX_STORAGE_SCHEME}{key}"assert returned_uri.startswith(PIPELEX_STORAGE_SCHEME)- def test_load_with_valid_uri_returns_data(self) -> None:+ async def test_load_with_valid_uri_returns_data(self) -> None:"""Test roundtrip: store data with key, load with returned URI."""provider = InMemoryStorageProvider()test_data = b"Test data \x00\x01\x02\xff"key = "roundtrip/test.bin"- returned_uri = provider.store(data=test_data, key=key)- loaded_data = provider.load(uri=returned_uri)+ returned_uri = await provider.store(data=test_data, key=key)+ loaded_data = await provider.load(uri=returned_uri)assert loaded_data == test_data- def test_load_with_invalid_uri_raises_error(self) -> None:+ async def test_load_with_invalid_uri_raises_error(self) -> None:"""Test that loading a non-existent URI raises StorageFileNotFoundError."""provider = InMemoryStorageProvider()nonexistent_uri = f"{PIPELEX_STORAGE_SCHEME}nonexistent/file.bin"with pytest.raises(StorageFileNotFoundError) as exc_info:- provider.load(uri=nonexistent_uri)+ await provider.load(uri=nonexistent_uri)assert "nonexistent/file.bin" in str(exc_info.value)- def test_store_raises_if_key_has_scheme_prefix(self) -> None:+ async def test_store_raises_if_key_has_scheme_prefix(self) -> None:"""Test that passing a key with pipelex-storage:// prefix raises an error."""provider = InMemoryStorageProvider()invalid_key = f"{PIPELEX_STORAGE_SCHEME}already/prefixed.bin"with pytest.raises(StorageInvalidKeyError) as exc_info:- provider.store(data=b"test", key=invalid_key)+ await provider.store(data=b"test", key=invalid_key)assert "should not include scheme prefix" in str(exc_info.value).lower()- def test_display_link_returns_memory_reference(self) -> None:+ async def test_display_link_returns_memory_reference(self) -> None:"""Test that display_link() returns a human-readable memory reference."""provider = InMemoryStorageProvider()test_data = b"display test"key = "display/test.bin"- returned_uri = provider.store(data=test_data, key=key)- display = provider.display_link(uri=returned_uri)+ returned_uri = await provider.store(data=test_data, key=key)+ display = await provider.display_link(uri=returned_uri)# Display should indicate it's in-memory storageassert display is None- def test_store_overwrites_existing_data(self) -> None:+ async def test_store_overwrites_existing_data(self) -> None:"""Test that storing with the same key overwrites the previous data."""provider = InMemoryStorageProvider()key = "overwrite/test.bin"original_data = b"original"new_data = b"updated content"- uri1 = provider.store(data=original_data, key=key)- uri2 = provider.store(data=new_data, key=key)- loaded_data = provider.load(uri=uri2)+ uri1 = await provider.store(data=original_data, key=key)+ uri2 = await provider.store(data=new_data, key=key)+ loaded_data = await provider.load(uri=uri2)assert uri1 == uri2assert loaded_data == new_data- def test_store_empty_bytes(self) -> None:+ async def test_store_empty_bytes(self) -> None:"""Test storing and loading empty bytes."""provider = InMemoryStorageProvider()key = "empty.bin"- returned_uri = provider.store(data=b"", key=key)- loaded_data = provider.load(uri=returned_uri)+ returned_uri = await provider.store(data=b"", key=key)+ loaded_data = await provider.load(uri=returned_uri)assert loaded_data == b""- def test_store_large_binary_data(self) -> None:+ async def test_store_large_binary_data(self) -> None:"""Test storing and loading larger binary data."""provider = InMemoryStorageProvider()test_data = bytes(range(256)) * 1000 # ~256KB of datakey = "large_file.bin"- returned_uri = provider.store(data=test_data, key=key)- loaded_data = provider.load(uri=returned_uri)+ returned_uri = await provider.store(data=test_data, key=key)+ loaded_data = await provider.load(uri=returned_uri)assert loaded_data == test_data- def test_multiple_keys_isolated(self) -> None:+ async def test_multiple_keys_isolated(self) -> None:"""Test that different keys store different data independently."""provider = InMemoryStorageProvider()data1 = b"first"@@ -105,8 +106,8 @@ class TestInMemoryStorageProvider:key1 = "file1.bin"key2 = "file2.bin"- uri1 = provider.store(data=data1, key=key1)- uri2 = provider.store(data=data2, key=key2)+ uri1 = await provider.store(data=data1, key=key1)+ uri2 = await provider.store(data=data2, key=key2)- assert provider.load(uri=uri1) == data1- assert provider.load(uri=uri2) == data2+ assert await provider.load(uri=uri1) == data1+ assert await provider.load(uri=uri2) == data2diff --git a/tests/unit/pipelex/tools/storage/test_local_storage_provider.py b/tests/unit/pipelex/tools/storage/test_local_storage_provider.pyindex f039b4164..3f23086a7 100644--- a/tests/unit/pipelex/tools/storage/test_local_storage_provider.py+++ b/tests/unit/pipelex/tools/storage/test_local_storage_provider.py@@ -7,146 +7,147 @@ from pipelex.tools.storage.local_storage_provider import LocalStorageProviderfrom pipelex.tools.storage.storage_provider_abstract import PIPELEX_STORAGE_SCHEME[email protected](loop_scope="class")class TestLocalStorageProvider:"""Unit tests for LocalStorageProvider with pipelex-storage:// URI scheme."""- def test_store_returns_uri_with_scheme(self, tmp_path: Path) -> None:+ async def test_store_returns_uri_with_scheme(self, tmp_path: Path) -> None:"""Test that store() returns a URI with the pipelex-storage:// scheme prefix."""provider = LocalStorageProvider(root_path=tmp_path)test_data = b"Hello, World!"key = "test_file.bin"- returned_uri = provider.store(data=test_data, key=key)+ returned_uri = await provider.store(data=test_data, key=key)assert returned_uri == f"{PIPELEX_STORAGE_SCHEME}{key}"assert returned_uri.startswith(PIPELEX_STORAGE_SCHEME)- def test_load_with_valid_uri_returns_data(self, tmp_path: Path) -> None:+ async def test_load_with_valid_uri_returns_data(self, tmp_path: Path) -> None:"""Test roundtrip: store data with key, load with returned URI."""provider = LocalStorageProvider(root_path=tmp_path)test_data = b"Hello, World! \x00\x01\x02\xff"key = "test_file.bin"- returned_uri = provider.store(data=test_data, key=key)- loaded_data = provider.load(uri=returned_uri)+ returned_uri = await provider.store(data=test_data, key=key)+ loaded_data = await provider.load(uri=returned_uri)assert loaded_data == test_data- def test_load_with_invalid_uri_raises_error(self, tmp_path: Path) -> None:+ async def test_load_with_invalid_uri_raises_error(self, tmp_path: Path) -> None:"""Test that loading a non-existent URI raises StorageFileNotFoundError."""provider = LocalStorageProvider(root_path=tmp_path)nonexistent_uri = f"{PIPELEX_STORAGE_SCHEME}nonexistent.bin"with pytest.raises(StorageFileNotFoundError) as exc_info:- provider.load(uri=nonexistent_uri)+ await provider.load(uri=nonexistent_uri)assert "nonexistent.bin" in str(exc_info.value)- def test_store_raises_if_key_has_scheme_prefix(self, tmp_path: Path) -> None:+ async def test_store_raises_if_key_has_scheme_prefix(self, tmp_path: Path) -> None:"""Test that passing a key with pipelex-storage:// prefix raises an error."""provider = LocalStorageProvider(root_path=tmp_path)invalid_key = f"{PIPELEX_STORAGE_SCHEME}already/prefixed.bin"with pytest.raises(StorageInvalidKeyError) as exc_info:- provider.store(data=b"test", key=invalid_key)+ await provider.store(data=b"test", key=invalid_key)assert "should not include scheme prefix" in str(exc_info.value).lower()- def test_display_link_returns_file_uri(self, tmp_path: Path) -> None:+ async def test_display_link_returns_file_uri(self, tmp_path: Path) -> None:"""Test that display_link() returns a file:// URI for clickable terminal links."""provider = LocalStorageProvider(root_path=tmp_path)test_data = b"display test"key = "subdir/display_test.bin"- returned_uri = provider.store(data=test_data, key=key)- display = provider.display_link(uri=returned_uri)+ returned_uri = await provider.store(data=test_data, key=key)+ display = await provider.display_link(uri=returned_uri)expected_uri = (tmp_path / key).as_uri()assert display == expected_uriassert display.startswith("file://")- def test_store_creates_parent_directories(self, tmp_path: Path) -> None:+ async def test_store_creates_parent_directories(self, tmp_path: Path) -> None:"""Test that storing to a nested path creates intermediate directories."""provider = LocalStorageProvider(root_path=tmp_path)test_data = b"nested content"key = "subdir/nested/deep/file.bin"- returned_uri = provider.store(data=test_data, key=key)- loaded_data = provider.load(uri=returned_uri)+ returned_uri = await provider.store(data=test_data, key=key)+ loaded_data = await provider.load(uri=returned_uri)assert loaded_data == test_dataassert (tmp_path / "subdir" / "nested" / "deep" / "file.bin").exists()- def test_store_overwrites_existing_file(self, tmp_path: Path) -> None:+ async def test_store_overwrites_existing_file(self, tmp_path: Path) -> None:"""Test that storing with the same key overwrites the file."""provider = LocalStorageProvider(root_path=tmp_path)key = "overwrite_test.bin"original_data = b"original"new_data = b"updated content"- provider.store(data=original_data, key=key)- returned_uri = provider.store(data=new_data, key=key)- loaded_data = provider.load(uri=returned_uri)+ await provider.store(data=original_data, key=key)+ returned_uri = await provider.store(data=new_data, key=key)+ loaded_data = await provider.load(uri=returned_uri)assert loaded_data == new_data- def test_invalid_key_with_path_traversal_raises_error(self, tmp_path: Path) -> None:+ async def test_invalid_key_with_path_traversal_raises_error(self, tmp_path: Path) -> None:"""Test that keys with path traversal (../) raise StorageInvalidUriError."""provider = LocalStorageProvider(root_path=tmp_path)with pytest.raises(StorageInvalidUriError) as exc_info:- provider.store(data=b"malicious", key="../outside.bin")+ await provider.store(data=b"malicious", key="../outside.bin")assert "path traversal" in str(exc_info.value).lower()- def test_load_with_path_traversal_key_raises_error(self, tmp_path: Path) -> None:+ async def test_load_with_path_traversal_key_raises_error(self, tmp_path: Path) -> None:"""Test that loading with path traversal in URI raises StorageInvalidUriError."""provider = LocalStorageProvider(root_path=tmp_path)# Manually construct a URI with path traversal (bypassing store validation)malicious_uri = f"{PIPELEX_STORAGE_SCHEME}../../etc/passwd"with pytest.raises(StorageInvalidUriError) as exc_info:- provider.load(uri=malicious_uri)+ await provider.load(uri=malicious_uri)assert "path traversal" in str(exc_info.value).lower()- def test_store_empty_bytes(self, tmp_path: Path) -> None:+ async def test_store_empty_bytes(self, tmp_path: Path) -> None:"""Test storing and loading empty bytes."""provider = LocalStorageProvider(root_path=tmp_path)key = "empty.bin"- returned_uri = provider.store(data=b"", key=key)- loaded_data = provider.load(uri=returned_uri)+ returned_uri = await provider.store(data=b"", key=key)+ loaded_data = await provider.load(uri=returned_uri)assert loaded_data == b""- def test_store_large_binary_data(self, tmp_path: Path) -> None:+ async def test_store_large_binary_data(self, tmp_path: Path) -> None:"""Test storing and loading larger binary data."""provider = LocalStorageProvider(root_path=tmp_path)test_data = bytes(range(256)) * 1000 # ~256KB of datakey = "large_file.bin"- returned_uri = provider.store(data=test_data, key=key)- loaded_data = provider.load(uri=returned_uri)+ returned_uri = await provider.store(data=test_data, key=key)+ loaded_data = await provider.load(uri=returned_uri)assert loaded_data == test_data- def test_absolute_key_raises_error(self, tmp_path: Path) -> None:+ async def test_absolute_key_raises_error(self, tmp_path: Path) -> None:"""Test that absolute paths as keys are rejected."""provider = LocalStorageProvider(root_path=tmp_path)with pytest.raises(StorageInvalidUriError) as exc_info:- provider.store(data=b"test", key="/absolute/path.bin")+ await provider.store(data=b"test", key="/absolute/path.bin")assert "absolute" in str(exc_info.value).lower()- def test_file_actually_written_to_disk(self, tmp_path: Path) -> None:+ async def test_file_actually_written_to_disk(self, tmp_path: Path) -> None:"""Test that the file is actually written to the filesystem."""provider = LocalStorageProvider(root_path=tmp_path)test_data = b"disk content"key = "on_disk.bin"- provider.store(data=test_data, key=key)+ await provider.store(data=test_data, key=key)file_path = tmp_path / keyassert file_path.exists()diff --git a/tests/unit/pipelex/tools/storage/test_s3_storage_provider.py b/tests/unit/pipelex/tools/storage/test_s3_storage_provider.pynew file mode 100644index 000000000..dc688f8cd--- /dev/null+++ b/tests/unit/pipelex/tools/storage/test_s3_storage_provider.py@@ -0,0 +1,403 @@+from typing import Any+from unittest.mock import AsyncMock, MagicMock++import pytest+from botocore.exceptions import ClientError, EndpointConnectionError, NoCredentialsError+from pytest_mock import MockerFixture++from pipelex.tools.storage.exceptions import StorageFileNotFoundError, StorageInvalidKeyError, StorageS3Error+from pipelex.tools.storage.s3_storage_provider import S3StorageProvider+from pipelex.tools.storage.storage_provider_abstract import PIPELEX_STORAGE_SCHEME++# Test constants+S3_TEST_BUCKET = "test-pipelex-bucket"+S3_TEST_REGION = "us-east-1"++[email protected](loop_scope="class")+class TestS3StorageProvider:+ """Unit tests for S3StorageProvider using mocks for aioboto3."""++ @pytest.fixture+ def mock_aioboto3(self, mocker: MockerFixture) -> dict[str, Any]:+ """Mock aioboto3 session and client.++ Returns a dict containing the mocked session, client, and response objects+ for test assertions.+ """+ # Create mock stream for response body+ mock_stream = AsyncMock()+ mock_stream.read = AsyncMock(return_value=b"")+ mock_stream.__aenter__ = AsyncMock(return_value=mock_stream)+ mock_stream.__aexit__ = AsyncMock(return_value=None)++ # Create mock client+ mock_client = AsyncMock()+ mock_client.get_object = AsyncMock(return_value={"Body": mock_stream})+ mock_client.put_object = AsyncMock(return_value={})+ mock_client.generate_presigned_url = AsyncMock(+ return_value=f"https://{S3_TEST_BUCKET}.s3.{S3_TEST_REGION}.amazonaws.com/test?signature=abc123"+ )++ # Create mock exceptions (modeled exceptions on the client)+ mock_exceptions = MagicMock()+ mock_exceptions.NoSuchKey = type("NoSuchKey", (Exception,), {})+ mock_exceptions.NoSuchBucket = type("NoSuchBucket", (Exception,), {})+ mock_client.exceptions = mock_exceptions++ # Create async context manager for client+ mock_client_context = AsyncMock()+ mock_client_context.__aenter__ = AsyncMock(return_value=mock_client)+ mock_client_context.__aexit__ = AsyncMock(return_value=None)++ # Create mock session+ mock_session = MagicMock()+ mock_session.client = MagicMock(return_value=mock_client_context)++ # Patch aioboto3.Session+ mocker.patch("aioboto3.Session", return_value=mock_session)++ return {+ "session": mock_session,+ "client": mock_client,+ "client_context": mock_client_context,+ "stream": mock_stream,+ "exceptions": mock_exceptions,+ }++ @pytest.fixture+ def s3_provider_no_signed_urls(self) -> S3StorageProvider:+ """Create an S3StorageProvider with signed URLs disabled."""+ return S3StorageProvider(+ bucket_name=S3_TEST_BUCKET,+ region=S3_TEST_REGION,+ signed_urls_lifespan=None,+ )++ @pytest.fixture+ def s3_provider_with_signed_urls(self) -> S3StorageProvider:+ """Create an S3StorageProvider with signed URLs enabled (1 hour lifespan)."""+ return S3StorageProvider(+ bucket_name=S3_TEST_BUCKET,+ region=S3_TEST_REGION,+ signed_urls_lifespan=3600,+ )++ async def test_store_returns_uri_with_scheme(+ self,+ s3_provider_no_signed_urls: S3StorageProvider,+ mock_aioboto3: dict[str, Any], # noqa: ARG002+ ) -> None:+ """Test that store() returns a URI with the pipelex-storage:// scheme prefix."""+ test_data = b"Hello, World!"+ key = "test/file.bin"++ returned_uri = await s3_provider_no_signed_urls.store(data=test_data, key=key)++ assert returned_uri == f"{PIPELEX_STORAGE_SCHEME}{key}"+ assert returned_uri.startswith(PIPELEX_STORAGE_SCHEME)++ async def test_store_calls_put_object(+ self,+ s3_provider_no_signed_urls: S3StorageProvider,+ mock_aioboto3: dict[str, Any],+ ) -> None:+ """Test that store() correctly calls the S3 put_object method."""+ test_data = b"Test data"+ key = "test/file.bin"+ content_type = "application/octet-stream"++ await s3_provider_no_signed_urls.store(data=test_data, key=key, content_type=content_type)++ mock_aioboto3["client"].put_object.assert_called_once_with(+ Bucket=S3_TEST_BUCKET,+ Key=key,+ Body=test_data,+ ContentType=content_type,+ )++ async def test_load_returns_data(+ self,+ s3_provider_no_signed_urls: S3StorageProvider,+ mock_aioboto3: dict[str, Any],+ ) -> None:+ """Test that load() returns the data from S3."""+ expected_data = b"Test data \x00\x01\x02\xff"+ mock_aioboto3["stream"].read = AsyncMock(return_value=expected_data)++ uri = f"{PIPELEX_STORAGE_SCHEME}test/file.bin"+ loaded_data = await s3_provider_no_signed_urls.load(uri=uri)++ assert loaded_data == expected_data++ async def test_load_with_nonexistent_key_raises_error(+ self,+ s3_provider_no_signed_urls: S3StorageProvider,+ mock_aioboto3: dict[str, Any],+ ) -> None:+ """Test that loading a non-existent object raises StorageFileNotFoundError."""+ mock_aioboto3["client"].get_object = AsyncMock(side_effect=mock_aioboto3["exceptions"].NoSuchKey("Key not found"))+ nonexistent_uri = f"{PIPELEX_STORAGE_SCHEME}nonexistent/file.bin"++ with pytest.raises(StorageFileNotFoundError) as exc_info:+ await s3_provider_no_signed_urls.load(uri=nonexistent_uri)++ assert "nonexistent/file.bin" in str(exc_info.value)++ async def test_store_raises_if_key_has_scheme_prefix(+ self,+ s3_provider_no_signed_urls: S3StorageProvider,+ mock_aioboto3: dict[str, Any], # noqa: ARG002+ ) -> None:+ """Test that passing a key with pipelex-storage:// prefix raises an error."""+ invalid_key = f"{PIPELEX_STORAGE_SCHEME}already/prefixed.bin"++ with pytest.raises(StorageInvalidKeyError) as exc_info:+ await s3_provider_no_signed_urls.store(data=b"test", key=invalid_key)++ assert "should not include scheme prefix" in str(exc_info.value).lower()++ async def test_display_link_returns_public_url_when_signed_urls_disabled(+ self,+ s3_provider_no_signed_urls: S3StorageProvider,+ mock_aioboto3: dict[str, Any], # noqa: ARG002+ ) -> None:+ """Test that display_link() returns a public URL when signed URLs are disabled."""+ key = "display/test.bin"+ uri = f"{PIPELEX_STORAGE_SCHEME}{key}"++ display = await s3_provider_no_signed_urls.display_link(uri=uri)++ expected_url = f"https://{S3_TEST_BUCKET}.s3.{S3_TEST_REGION}.amazonaws.com/{key}"+ assert display == expected_url++ async def test_display_link_returns_presigned_url_when_signed_urls_enabled(+ self,+ s3_provider_with_signed_urls: S3StorageProvider,+ mock_aioboto3: dict[str, Any],+ ) -> None:+ """Test that display_link() returns a presigned URL when signed URLs are enabled."""+ key = "presigned/test.bin"+ uri = f"{PIPELEX_STORAGE_SCHEME}{key}"+ expected_presigned = "https://test-bucket.s3.amazonaws.com/presigned/test.bin?X-Amz-Signature=xyz"+ mock_aioboto3["client"].generate_presigned_url = AsyncMock(return_value=expected_presigned)++ display = await s3_provider_with_signed_urls.display_link(uri=uri)++ assert display == expected_presigned+ mock_aioboto3["client"].generate_presigned_url.assert_called_once_with(+ "get_object",+ Params={"Bucket": S3_TEST_BUCKET, "Key": key},+ ExpiresIn=3600,+ )++ async def test_display_link_handles_sync_presigned_url(+ self,+ s3_provider_with_signed_urls: S3StorageProvider,+ mock_aioboto3: dict[str, Any],+ ) -> None:+ """Test that display_link() handles sync presigned URL generation (non-awaitable)."""+ key = "sync-presign/test.bin"+ uri = f"{PIPELEX_STORAGE_SCHEME}{key}"+ expected_presigned = "https://test-bucket.s3.amazonaws.com/sync-presign/test.bin?X-Amz-Signature=sync"+ # Return a plain string (non-awaitable) to simulate sync behavior in some aioboto3 versions+ mock_aioboto3["client"].generate_presigned_url = MagicMock(return_value=expected_presigned)++ display = await s3_provider_with_signed_urls.display_link(uri=uri)++ assert display == expected_presigned++ async def test_store_with_nested_path(+ self,+ s3_provider_no_signed_urls: S3StorageProvider,+ mock_aioboto3: dict[str, Any],+ ) -> None:+ """Test storing data in a deeply nested path structure."""+ test_data = b"nested content"+ key = "level1/level2/level3/deep_file.bin"++ returned_uri = await s3_provider_no_signed_urls.store(data=test_data, key=key)++ assert returned_uri == f"{PIPELEX_STORAGE_SCHEME}{key}"+ mock_aioboto3["client"].put_object.assert_called_once()++ async def test_store_empty_bytes(+ self,+ s3_provider_no_signed_urls: S3StorageProvider,+ mock_aioboto3: dict[str, Any],+ ) -> None:+ """Test storing empty bytes."""+ key = "empty.bin"++ returned_uri = await s3_provider_no_signed_urls.store(data=b"", key=key)++ assert returned_uri == f"{PIPELEX_STORAGE_SCHEME}{key}"+ mock_aioboto3["client"].put_object.assert_called_once_with(+ Bucket=S3_TEST_BUCKET,+ Key=key,+ Body=b"",+ )++ async def test_store_large_binary_data(+ self,+ s3_provider_no_signed_urls: S3StorageProvider,+ mock_aioboto3: dict[str, Any],+ ) -> None:+ """Test storing larger binary data."""+ test_data = bytes(range(256)) * 1000 # ~256KB of data+ key = "large_file.bin"++ returned_uri = await s3_provider_no_signed_urls.store(data=test_data, key=key)++ assert returned_uri == f"{PIPELEX_STORAGE_SCHEME}{key}"+ mock_aioboto3["client"].put_object.assert_called_once()++ async def test_store_bucket_not_found_raises_error(+ self,+ s3_provider_no_signed_urls: S3StorageProvider,+ mock_aioboto3: dict[str, Any],+ ) -> None:+ """Test that storing to a non-existent bucket raises StorageS3Error."""+ mock_aioboto3["client"].put_object = AsyncMock(side_effect=mock_aioboto3["exceptions"].NoSuchBucket("Bucket not found"))++ with pytest.raises(StorageS3Error) as exc_info:+ await s3_provider_no_signed_urls.store(data=b"test", key="test.bin")++ assert "Bucket not found" in str(exc_info.value)++ async def test_load_bucket_not_found_raises_error(+ self,+ s3_provider_no_signed_urls: S3StorageProvider,+ mock_aioboto3: dict[str, Any],+ ) -> None:+ """Test that loading from a non-existent bucket raises StorageS3Error."""+ mock_aioboto3["client"].get_object = AsyncMock(side_effect=mock_aioboto3["exceptions"].NoSuchBucket("Bucket not found"))+ uri = f"{PIPELEX_STORAGE_SCHEME}test.bin"++ with pytest.raises(StorageS3Error) as exc_info:+ await s3_provider_no_signed_urls.load(uri=uri)++ assert "Bucket not found" in str(exc_info.value)++ async def test_load_client_error_with_no_such_key_code_raises_file_not_found(+ self,+ s3_provider_no_signed_urls: S3StorageProvider,+ mock_aioboto3: dict[str, Any],+ ) -> None:+ """Test that ClientError with NoSuchKey code raises StorageFileNotFoundError."""+ error_response: Any = {"Error": {"Code": "NoSuchKey", "Message": "The specified key does not exist."}}+ mock_aioboto3["client"].get_object = AsyncMock(side_effect=ClientError(error_response, "GetObject"))+ uri = f"{PIPELEX_STORAGE_SCHEME}missing/file.bin"++ with pytest.raises(StorageFileNotFoundError) as exc_info:+ await s3_provider_no_signed_urls.load(uri=uri)++ assert "missing/file.bin" in str(exc_info.value)++ async def test_load_client_error_with_access_denied_raises_s3_error(+ self,+ s3_provider_no_signed_urls: S3StorageProvider,+ mock_aioboto3: dict[str, Any],+ ) -> None:+ """Test that ClientError with AccessDenied code raises StorageS3Error."""+ error_response: Any = {"Error": {"Code": "AccessDenied", "Message": "Access Denied"}}+ mock_aioboto3["client"].get_object = AsyncMock(side_effect=ClientError(error_response, "GetObject"))+ uri = f"{PIPELEX_STORAGE_SCHEME}forbidden/file.bin"++ with pytest.raises(StorageS3Error) as exc_info:+ await s3_provider_no_signed_urls.load(uri=uri)++ assert "AccessDenied" in str(exc_info.value)++ async def test_store_client_error_raises_s3_error(+ self,+ s3_provider_no_signed_urls: S3StorageProvider,+ mock_aioboto3: dict[str, Any],+ ) -> None:+ """Test that ClientError during store raises StorageS3Error."""+ error_response: Any = {"Error": {"Code": "InvalidAccessKeyId", "Message": "Invalid access key"}}+ mock_aioboto3["client"].put_object = AsyncMock(side_effect=ClientError(error_response, "PutObject"))++ with pytest.raises(StorageS3Error) as exc_info:+ await s3_provider_no_signed_urls.store(data=b"test", key="test.bin")++ assert "InvalidAccessKeyId" in str(exc_info.value)++ async def test_load_no_credentials_error_raises_s3_error(+ self,+ s3_provider_no_signed_urls: S3StorageProvider,+ mock_aioboto3: dict[str, Any],+ ) -> None:+ """Test that NoCredentialsError raises StorageS3Error."""+ mock_aioboto3["client"].get_object = AsyncMock(side_effect=NoCredentialsError())+ uri = f"{PIPELEX_STORAGE_SCHEME}test.bin"++ with pytest.raises(StorageS3Error) as exc_info:+ await s3_provider_no_signed_urls.load(uri=uri)++ assert "connectivity/credentials error" in str(exc_info.value)++ async def test_store_no_credentials_error_raises_s3_error(+ self,+ s3_provider_no_signed_urls: S3StorageProvider,+ mock_aioboto3: dict[str, Any],+ ) -> None:+ """Test that NoCredentialsError during store raises StorageS3Error."""+ mock_aioboto3["client"].put_object = AsyncMock(side_effect=NoCredentialsError())++ with pytest.raises(StorageS3Error) as exc_info:+ await s3_provider_no_signed_urls.store(data=b"test", key="test.bin")++ assert "connectivity/credentials error" in str(exc_info.value)++ async def test_load_endpoint_connection_error_raises_s3_error(+ self,+ s3_provider_no_signed_urls: S3StorageProvider,+ mock_aioboto3: dict[str, Any],+ ) -> None:+ """Test that EndpointConnectionError raises StorageS3Error."""+ mock_aioboto3["client"].get_object = AsyncMock(side_effect=EndpointConnectionError(endpoint_url="https://s3.amazonaws.com"))+ uri = f"{PIPELEX_STORAGE_SCHEME}test.bin"++ with pytest.raises(StorageS3Error) as exc_info:+ await s3_provider_no_signed_urls.load(uri=uri)++ assert "connectivity/credentials error" in str(exc_info.value)++ async def test_display_link_falls_back_to_public_url_on_client_error(+ self,+ s3_provider_with_signed_urls: S3StorageProvider,+ mock_aioboto3: dict[str, Any],+ ) -> None:+ """Test that display_link() falls back to public URL on botocore ClientError."""+ key = "fallback/test.bin"+ uri = f"{PIPELEX_STORAGE_SCHEME}{key}"+ error_response: Any = {"Error": {"Code": "SignatureDoesNotMatch", "Message": "Signature error"}}+ mock_aioboto3["client"].generate_presigned_url = MagicMock(side_effect=ClientError(error_response, "GeneratePresignedUrl"))++ display = await s3_provider_with_signed_urls.display_link(uri=uri)++ expected_public_url = f"https://{S3_TEST_BUCKET}.s3.{S3_TEST_REGION}.amazonaws.com/{key}"+ assert display == expected_public_url++ @pytest.mark.usefixtures("mock_aioboto3")+ async def test_session_is_reused(self) -> None:+ """Test that the session is lazily initialized and reused."""+ import aioboto3 # noqa: PLC0415++ provider = S3StorageProvider(+ bucket_name=S3_TEST_BUCKET,+ region=S3_TEST_REGION,+ signed_urls_lifespan=None,+ )++ # First call initializes the session+ await provider.store(data=b"data1", key="file1.bin")+ first_call_count: int = aioboto3.Session.call_count # type: ignore[attr-defined] # pyright: ignore[reportUnknownMemberType,reportAttributeAccessIssue,reportUnknownVariableType]++ # Second call reuses the session+ await provider.store(data=b"data2", key="file2.bin")+ second_call_count: int = aioboto3.Session.call_count # type: ignore[attr-defined] # pyright: ignore[reportUnknownMemberType,reportAttributeAccessIssue,reportUnknownVariableType]++ assert first_call_count == 1+ assert second_call_count == 1 # Session should only be created oncediff --git a/uv.lock b/uv.lockindex 3ce94b489..564b2497b 100644--- a/uv.lock+++ b/uv.lock@@ -2,9 +2,12 @@ version = 1revision = 3requires-python = ">=3.10, <3.15"resolution-markers = [- "python_full_version >= '3.12' and sys_platform == 'darwin'",- "python_full_version >= '3.12' and platform_machine == 'aarch64' and sys_platform == 'linux'",- "(python_full_version >= '3.12' and platform_machine != 'aarch64' and sys_platform == 'linux') or (python_full_version >= '3.12' and sys_platform != 'darwin' and sys_platform != 'linux')",+ "python_full_version >= '3.13' and sys_platform == 'darwin'",+ "python_full_version == '3.12.*' and sys_platform == 'darwin'",+ "python_full_version >= '3.13' and platform_machine == 'aarch64' and sys_platform == 'linux'",+ "python_full_version == '3.12.*' and platform_machine == 'aarch64' and sys_platform == 'linux'",+ "(python_full_version >= '3.13' and platform_machine != 'aarch64' and sys_platform == 'linux') or (python_full_version >= '3.13' and sys_platform != 'darwin' and sys_platform != 'linux')",+ "(python_full_version == '3.12.*' and platform_machine != 'aarch64' and sys_platform == 'linux') or (python_full_version == '3.12.*' and sys_platform != 'darwin' and sys_platform != 'linux')","python_full_version == '3.11.*' and sys_platform == 'darwin'","python_full_version == '3.11.*' and platform_machine == 'aarch64' and sys_platform == 'linux'","(python_full_version == '3.11.*' and platform_machine != 'aarch64' and sys_platform == 'linux') or (python_full_version == '3.11.*' and sys_platform != 'darwin' and sys_platform != 'linux')",@@ -450,6 +453,88 @@ wheels = [{ url = "https://files.pythonhosted.org/packages/70/7d/9bc192684cea499815ff478dfcdc13835ddf401365057044fb721ec6bddb/certifi-2025.11.12-py3-none-any.whl", hash = "sha256:97de8790030bbd5c2d96b7ec782fc2f7820ef8dba6db909ccf95449f2d062d4b", size = 159438, upload-time = "2025-11-12T02:54:49.735Z" },]+[[package]]+name = "cffi"+version = "2.0.0"+source = { registry = "https://pypi.org/simple" }+dependencies = [+ { name = "pycparser", marker = "implementation_name != 'PyPy'" },+]+sdist = { url = "https://files.pythonhosted.org/packages/eb/56/b1ba7935a17738ae8453301356628e8147c79dbb825bcbc73dc7401f9846/cffi-2.0.0.tar.gz", hash = "sha256:44d1b5909021139fe36001ae048dbdde8214afa20200eda0f64c068cac5d5529", size = 523588, upload-time = "2025-09-08T23:24:04.541Z" }+wheels = [+ { url = "https://files.pythonhosted.org/packages/93/d7/516d984057745a6cd96575eea814fe1edd6646ee6efd552fb7b0921dec83/cffi-2.0.0-cp310-cp310-macosx_10_13_x86_64.whl", hash = "sha256:0cf2d91ecc3fcc0625c2c530fe004f82c110405f101548512cce44322fa8ac44", size = 184283, upload-time = "2025-09-08T23:22:08.01Z" },+ { url = "https://files.pythonhosted.org/packages/9e/84/ad6a0b408daa859246f57c03efd28e5dd1b33c21737c2db84cae8c237aa5/cffi-2.0.0-cp310-cp310-macosx_11_0_arm64.whl", hash = "sha256:f73b96c41e3b2adedc34a7356e64c8eb96e03a3782b535e043a986276ce12a49", size = 180504, upload-time = "2025-09-08T23:22:10.637Z" },+ { url = "https://files.pythonhosted.org/packages/50/bd/b1a6362b80628111e6653c961f987faa55262b4002fcec42308cad1db680/cffi-2.0.0-cp310-cp310-manylinux1_i686.manylinux2014_i686.manylinux_2_17_i686.manylinux_2_5_i686.whl", hash = "sha256:53f77cbe57044e88bbd5ed26ac1d0514d2acf0591dd6bb02a3ae37f76811b80c", size = 208811, upload-time = "2025-09-08T23:22:12.267Z" },+ { url = "https://files.pythonhosted.org/packages/4f/27/6933a8b2562d7bd1fb595074cf99cc81fc3789f6a6c05cdabb46284a3188/cffi-2.0.0-cp310-cp310-manylinux2014_aarch64.manylinux_2_17_aarch64.whl", hash = "sha256:3e837e369566884707ddaf85fc1744b47575005c0a229de3327f8f9a20f4efeb", size = 216402, upload-time = "2025-09-08T23:22:13.455Z" },+ { url = "https://files.pythonhosted.org/packages/05/eb/b86f2a2645b62adcfff53b0dd97e8dfafb5c8aa864bd0d9a2c2049a0d551/cffi-2.0.0-cp310-cp310-manylinux2014_ppc64le.manylinux_2_17_ppc64le.whl", hash = "sha256:5eda85d6d1879e692d546a078b44251cdd08dd1cfb98dfb77b670c97cee49ea0", size = 203217, upload-time = "2025-09-08T23:22:14.596Z" },+ { url = "https://files.pythonhosted.org/packages/9f/e0/6cbe77a53acf5acc7c08cc186c9928864bd7c005f9efd0d126884858a5fe/cffi-2.0.0-cp310-cp310-manylinux2014_s390x.manylinux_2_17_s390x.whl", hash = "sha256:9332088d75dc3241c702d852d4671613136d90fa6881da7d770a483fd05248b4", size = 203079, upload-time = "2025-09-08T23:22:15.769Z" },+ { url = "https://files.pythonhosted.org/packages/98/29/9b366e70e243eb3d14a5cb488dfd3a0b6b2f1fb001a203f653b93ccfac88/cffi-2.0.0-cp310-cp310-manylinux2014_x86_64.manylinux_2_17_x86_64.whl", hash = "sha256:fc7de24befaeae77ba923797c7c87834c73648a05a4bde34b3b7e5588973a453", size = 216475, upload-time = "2025-09-08T23:22:17.427Z" },+ { url = "https://files.pythonhosted.org/packages/21/7a/13b24e70d2f90a322f2900c5d8e1f14fa7e2a6b3332b7309ba7b2ba51a5a/cffi-2.0.0-cp310-cp310-musllinux_1_2_aarch64.whl", hash = "sha256:cf364028c016c03078a23b503f02058f1814320a56ad535686f90565636a9495", size = 218829, upload-time = "2025-09-08T23:22:19.069Z" },+ { url = "https://files.pythonhosted.org/packages/60/99/c9dc110974c59cc981b1f5b66e1d8af8af764e00f0293266824d9c4254bc/cffi-2.0.0-cp310-cp310-musllinux_1_2_i686.whl", hash = "sha256:e11e82b744887154b182fd3e7e8512418446501191994dbf9c9fc1f32cc8efd5", size = 211211, upload-time = "2025-09-08T23:22:20.588Z" },+ { url = "https://files.pythonhosted.org/packages/49/72/ff2d12dbf21aca1b32a40ed792ee6b40f6dc3a9cf1644bd7ef6e95e0ac5e/cffi-2.0.0-cp310-cp310-musllinux_1_2_x86_64.whl", hash = "sha256:8ea985900c5c95ce9db1745f7933eeef5d314f0565b27625d9a10ec9881e1bfb", size = 218036, upload-time = "2025-09-08T23:22:22.143Z" },+ { url = "https://files.pythonhosted.org/packages/e2/cc/027d7fb82e58c48ea717149b03bcadcbdc293553edb283af792bd4bcbb3f/cffi-2.0.0-cp310-cp310-win32.whl", hash = "sha256:1f72fb8906754ac8a2cc3f9f5aaa298070652a0ffae577e0ea9bd480dc3c931a", size = 172184, upload-time = "2025-09-08T23:22:23.328Z" },+ { url = "https://files.pythonhosted.org/packages/33/fa/072dd15ae27fbb4e06b437eb6e944e75b068deb09e2a2826039e49ee2045/cffi-2.0.0-cp310-cp310-win_amd64.whl", hash = "sha256:b18a3ed7d5b3bd8d9ef7a8cb226502c6bf8308df1525e1cc676c3680e7176739", size = 182790, upload-time = "2025-09-08T23:22:24.752Z" },+ { url = "https://files.pythonhosted.org/packages/12/4a/3dfd5f7850cbf0d06dc84ba9aa00db766b52ca38d8b86e3a38314d52498c/cffi-2.0.0-cp311-cp311-macosx_10_13_x86_64.whl", hash = "sha256:b4c854ef3adc177950a8dfc81a86f5115d2abd545751a304c5bcf2c2c7283cfe", size = 184344, upload-time = "2025-09-08T23:22:26.456Z" },+ { url = "https://files.pythonhosted.org/packages/4f/8b/f0e4c441227ba756aafbe78f117485b25bb26b1c059d01f137fa6d14896b/cffi-2.0.0-cp311-cp311-macosx_11_0_arm64.whl", hash = "sha256:2de9a304e27f7596cd03d16f1b7c72219bd944e99cc52b84d0145aefb07cbd3c", size = 180560, upload-time = "2025-09-08T23:22:28.197Z" },+ { url = "https://files.pythonhosted.org/packages/b1/b7/1200d354378ef52ec227395d95c2576330fd22a869f7a70e88e1447eb234/cffi-2.0.0-cp311-cp311-manylinux1_i686.manylinux2014_i686.manylinux_2_17_i686.manylinux_2_5_i686.whl", hash = "sha256:baf5215e0ab74c16e2dd324e8ec067ef59e41125d3eade2b863d294fd5035c92", size = 209613, upload-time = "2025-09-08T23:22:29.475Z" },+ { url = "https://files.pythonhosted.org/packages/b8/56/6033f5e86e8cc9bb629f0077ba71679508bdf54a9a5e112a3c0b91870332/cffi-2.0.0-cp311-cp311-manylinux2014_aarch64.manylinux_2_17_aarch64.whl", hash = "sha256:730cacb21e1bdff3ce90babf007d0a0917cc3e6492f336c2f0134101e0944f93", size = 216476, upload-time = "2025-09-08T23:22:31.063Z" },+ { url = "https://files.pythonhosted.org/packages/dc/7f/55fecd70f7ece178db2f26128ec41430d8720f2d12ca97bf8f0a628207d5/cffi-2.0.0-cp311-cp311-manylinux2014_ppc64le.manylinux_2_17_ppc64le.whl", hash = "sha256:6824f87845e3396029f3820c206e459ccc91760e8fa24422f8b0c3d1731cbec5", size = 203374, upload-time = "2025-09-08T23:22:32.507Z" },+ { url = "https://files.pythonhosted.org/packages/84/ef/a7b77c8bdc0f77adc3b46888f1ad54be8f3b7821697a7b89126e829e676a/cffi-2.0.0-cp311-cp311-manylinux2014_s390x.manylinux_2_17_s390x.whl", hash = "sha256:9de40a7b0323d889cf8d23d1ef214f565ab154443c42737dfe52ff82cf857664", size = 202597, upload-time = "2025-09-08T23:22:34.132Z" },+ { url = "https://files.pythonhosted.org/packages/d7/91/500d892b2bf36529a75b77958edfcd5ad8e2ce4064ce2ecfeab2125d72d1/cffi-2.0.0-cp311-cp311-manylinux2014_x86_64.manylinux_2_17_x86_64.whl", hash = "sha256:8941aaadaf67246224cee8c3803777eed332a19d909b47e29c9842ef1e79ac26", size = 215574, upload-time = "2025-09-08T23:22:35.443Z" },+ { url = "https://files.pythonhosted.org/packages/44/64/58f6255b62b101093d5df22dcb752596066c7e89dd725e0afaed242a61be/cffi-2.0.0-cp311-cp311-musllinux_1_2_aarch64.whl", hash = "sha256:a05d0c237b3349096d3981b727493e22147f934b20f6f125a3eba8f994bec4a9", size = 218971, upload-time = "2025-09-08T23:22:36.805Z" },+ { url = "https://files.pythonhosted.org/packages/ab/49/fa72cebe2fd8a55fbe14956f9970fe8eb1ac59e5df042f603ef7c8ba0adc/cffi-2.0.0-cp311-cp311-musllinux_1_2_i686.whl", hash = "sha256:94698a9c5f91f9d138526b48fe26a199609544591f859c870d477351dc7b2414", size = 211972, upload-time = "2025-09-08T23:22:38.436Z" },+ { url = "https://files.pythonhosted.org/packages/0b/28/dd0967a76aab36731b6ebfe64dec4e981aff7e0608f60c2d46b46982607d/cffi-2.0.0-cp311-cp311-musllinux_1_2_x86_64.whl", hash = "sha256:5fed36fccc0612a53f1d4d9a816b50a36702c28a2aa880cb8a122b3466638743", size = 217078, upload-time = "2025-09-08T23:22:39.776Z" },+ { url = "https://files.pythonhosted.org/packages/2b/c0/015b25184413d7ab0a410775fdb4a50fca20f5589b5dab1dbbfa3baad8ce/cffi-2.0.0-cp311-cp311-win32.whl", hash = "sha256:c649e3a33450ec82378822b3dad03cc228b8f5963c0c12fc3b1e0ab940f768a5", size = 172076, upload-time = "2025-09-08T23:22:40.95Z" },+ { url = "https://files.pythonhosted.org/packages/ae/8f/dc5531155e7070361eb1b7e4c1a9d896d0cb21c49f807a6c03fd63fc877e/cffi-2.0.0-cp311-cp311-win_amd64.whl", hash = "sha256:66f011380d0e49ed280c789fbd08ff0d40968ee7b665575489afa95c98196ab5", size = 182820, upload-time = "2025-09-08T23:22:42.463Z" },+ { url = "https://files.pythonhosted.org/packages/95/5c/1b493356429f9aecfd56bc171285a4c4ac8697f76e9bbbbb105e537853a1/cffi-2.0.0-cp311-cp311-win_arm64.whl", hash = "sha256:c6638687455baf640e37344fe26d37c404db8b80d037c3d29f58fe8d1c3b194d", size = 177635, upload-time = "2025-09-08T23:22:43.623Z" },+ { url = "https://files.pythonhosted.org/packages/ea/47/4f61023ea636104d4f16ab488e268b93008c3d0bb76893b1b31db1f96802/cffi-2.0.0-cp312-cp312-macosx_10_13_x86_64.whl", hash = "sha256:6d02d6655b0e54f54c4ef0b94eb6be0607b70853c45ce98bd278dc7de718be5d", size = 185271, upload-time = "2025-09-08T23:22:44.795Z" },+ { url = "https://files.pythonhosted.org/packages/df/a2/781b623f57358e360d62cdd7a8c681f074a71d445418a776eef0aadb4ab4/cffi-2.0.0-cp312-cp312-macosx_11_0_arm64.whl", hash = "sha256:8eca2a813c1cb7ad4fb74d368c2ffbbb4789d377ee5bb8df98373c2cc0dee76c", size = 181048, upload-time = "2025-09-08T23:22:45.938Z" },+ { url = "https://files.pythonhosted.org/packages/ff/df/a4f0fbd47331ceeba3d37c2e51e9dfc9722498becbeec2bd8bc856c9538a/cffi-2.0.0-cp312-cp312-manylinux1_i686.manylinux2014_i686.manylinux_2_17_i686.manylinux_2_5_i686.whl", hash = "sha256:21d1152871b019407d8ac3985f6775c079416c282e431a4da6afe7aefd2bccbe", size = 212529, upload-time = "2025-09-08T23:22:47.349Z" },+ { url = "https://files.pythonhosted.org/packages/d5/72/12b5f8d3865bf0f87cf1404d8c374e7487dcf097a1c91c436e72e6badd83/cffi-2.0.0-cp312-cp312-manylinux2014_aarch64.manylinux_2_17_aarch64.whl", hash = "sha256:b21e08af67b8a103c71a250401c78d5e0893beff75e28c53c98f4de42f774062", size = 220097, upload-time = "2025-09-08T23:22:48.677Z" },+ { url = "https://files.pythonhosted.org/packages/c2/95/7a135d52a50dfa7c882ab0ac17e8dc11cec9d55d2c18dda414c051c5e69e/cffi-2.0.0-cp312-cp312-manylinux2014_ppc64le.manylinux_2_17_ppc64le.whl", hash = "sha256:1e3a615586f05fc4065a8b22b8152f0c1b00cdbc60596d187c2a74f9e3036e4e", size = 207983, upload-time = "2025-09-08T23:22:50.06Z" },+ { url = "https://files.pythonhosted.org/packages/3a/c8/15cb9ada8895957ea171c62dc78ff3e99159ee7adb13c0123c001a2546c1/cffi-2.0.0-cp312-cp312-manylinux2014_s390x.manylinux_2_17_s390x.whl", hash = "sha256:81afed14892743bbe14dacb9e36d9e0e504cd204e0b165062c488942b9718037", size = 206519, upload-time = "2025-09-08T23:22:51.364Z" },+ { url = "https://files.pythonhosted.org/packages/78/2d/7fa73dfa841b5ac06c7b8855cfc18622132e365f5b81d02230333ff26e9e/cffi-2.0.0-cp312-cp312-manylinux2014_x86_64.manylinux_2_17_x86_64.whl", hash = "sha256:3e17ed538242334bf70832644a32a7aae3d83b57567f9fd60a26257e992b79ba", size = 219572, upload-time = "2025-09-08T23:22:52.902Z" },+ { url = "https://files.pythonhosted.org/packages/07/e0/267e57e387b4ca276b90f0434ff88b2c2241ad72b16d31836adddfd6031b/cffi-2.0.0-cp312-cp312-musllinux_1_2_aarch64.whl", hash = "sha256:3925dd22fa2b7699ed2617149842d2e6adde22b262fcbfada50e3d195e4b3a94", size = 222963, upload-time = "2025-09-08T23:22:54.518Z" },+ { url = "https://files.pythonhosted.org/packages/b6/75/1f2747525e06f53efbd878f4d03bac5b859cbc11c633d0fb81432d98a795/cffi-2.0.0-cp312-cp312-musllinux_1_2_x86_64.whl", hash = "sha256:2c8f814d84194c9ea681642fd164267891702542f028a15fc97d4674b6206187", size = 221361, upload-time = "2025-09-08T23:22:55.867Z" },+ { url = "https://files.pythonhosted.org/packages/7b/2b/2b6435f76bfeb6bbf055596976da087377ede68df465419d192acf00c437/cffi-2.0.0-cp312-cp312-win32.whl", hash = "sha256:da902562c3e9c550df360bfa53c035b2f241fed6d9aef119048073680ace4a18", size = 172932, upload-time = "2025-09-08T23:22:57.188Z" },+ { url = "https://files.pythonhosted.org/packages/f8/ed/13bd4418627013bec4ed6e54283b1959cf6db888048c7cf4b4c3b5b36002/cffi-2.0.0-cp312-cp312-win_amd64.whl", hash = "sha256:da68248800ad6320861f129cd9c1bf96ca849a2771a59e0344e88681905916f5", size = 183557, upload-time = "2025-09-08T23:22:58.351Z" },+ { url = "https://files.pythonhosted.org/packages/95/31/9f7f93ad2f8eff1dbc1c3656d7ca5bfd8fb52c9d786b4dcf19b2d02217fa/cffi-2.0.0-cp312-cp312-win_arm64.whl", hash = "sha256:4671d9dd5ec934cb9a73e7ee9676f9362aba54f7f34910956b84d727b0d73fb6", size = 177762, upload-time = "2025-09-08T23:22:59.668Z" },+ { url = "https://files.pythonhosted.org/packages/4b/8d/a0a47a0c9e413a658623d014e91e74a50cdd2c423f7ccfd44086ef767f90/cffi-2.0.0-cp313-cp313-macosx_10_13_x86_64.whl", hash = "sha256:00bdf7acc5f795150faa6957054fbbca2439db2f775ce831222b66f192f03beb", size = 185230, upload-time = "2025-09-08T23:23:00.879Z" },+ { url = "https://files.pythonhosted.org/packages/4a/d2/a6c0296814556c68ee32009d9c2ad4f85f2707cdecfd7727951ec228005d/cffi-2.0.0-cp313-cp313-macosx_11_0_arm64.whl", hash = "sha256:45d5e886156860dc35862657e1494b9bae8dfa63bf56796f2fb56e1679fc0bca", size = 181043, upload-time = "2025-09-08T23:23:02.231Z" },+ { url = "https://files.pythonhosted.org/packages/b0/1e/d22cc63332bd59b06481ceaac49d6c507598642e2230f201649058a7e704/cffi-2.0.0-cp313-cp313-manylinux1_i686.manylinux2014_i686.manylinux_2_17_i686.manylinux_2_5_i686.whl", hash = "sha256:07b271772c100085dd28b74fa0cd81c8fb1a3ba18b21e03d7c27f3436a10606b", size = 212446, upload-time = "2025-09-08T23:23:03.472Z" },+ { url = "https://files.pythonhosted.org/packages/a9/f5/a2c23eb03b61a0b8747f211eb716446c826ad66818ddc7810cc2cc19b3f2/cffi-2.0.0-cp313-cp313-manylinux2014_aarch64.manylinux_2_17_aarch64.whl", hash = "sha256:d48a880098c96020b02d5a1f7d9251308510ce8858940e6fa99ece33f610838b", size = 220101, upload-time = "2025-09-08T23:23:04.792Z" },+ { url = "https://files.pythonhosted.org/packages/f2/7f/e6647792fc5850d634695bc0e6ab4111ae88e89981d35ac269956605feba/cffi-2.0.0-cp313-cp313-manylinux2014_ppc64le.manylinux_2_17_ppc64le.whl", hash = "sha256:f93fd8e5c8c0a4aa1f424d6173f14a892044054871c771f8566e4008eaa359d2", size = 207948, upload-time = "2025-09-08T23:23:06.127Z" },+ { url = "https://files.pythonhosted.org/packages/cb/1e/a5a1bd6f1fb30f22573f76533de12a00bf274abcdc55c8edab639078abb6/cffi-2.0.0-cp313-cp313-manylinux2014_s390x.manylinux_2_17_s390x.whl", hash = "sha256:dd4f05f54a52fb558f1ba9f528228066954fee3ebe629fc1660d874d040ae5a3", size = 206422, upload-time = "2025-09-08T23:23:07.753Z" },+ { url = "https://files.pythonhosted.org/packages/98/df/0a1755e750013a2081e863e7cd37e0cdd02664372c754e5560099eb7aa44/cffi-2.0.0-cp313-cp313-manylinux2014_x86_64.manylinux_2_17_x86_64.whl", hash = "sha256:c8d3b5532fc71b7a77c09192b4a5a200ea992702734a2e9279a37f2478236f26", size = 219499, upload-time = "2025-09-08T23:23:09.648Z" },+ { url = "https://files.pythonhosted.org/packages/50/e1/a969e687fcf9ea58e6e2a928ad5e2dd88cc12f6f0ab477e9971f2309b57c/cffi-2.0.0-cp313-cp313-musllinux_1_2_aarch64.whl", hash = "sha256:d9b29c1f0ae438d5ee9acb31cadee00a58c46cc9c0b2f9038c6b0b3470877a8c", size = 222928, upload-time = "2025-09-08T23:23:10.928Z" },+ { url = "https://files.pythonhosted.org/packages/36/54/0362578dd2c9e557a28ac77698ed67323ed5b9775ca9d3fe73fe191bb5d8/cffi-2.0.0-cp313-cp313-musllinux_1_2_x86_64.whl", hash = "sha256:6d50360be4546678fc1b79ffe7a66265e28667840010348dd69a314145807a1b", size = 221302, upload-time = "2025-09-08T23:23:12.42Z" },+ { url = "https://files.pythonhosted.org/packages/eb/6d/bf9bda840d5f1dfdbf0feca87fbdb64a918a69bca42cfa0ba7b137c48cb8/cffi-2.0.0-cp313-cp313-win32.whl", hash = "sha256:74a03b9698e198d47562765773b4a8309919089150a0bb17d829ad7b44b60d27", size = 172909, upload-time = "2025-09-08T23:23:14.32Z" },+ { url = "https://files.pythonhosted.org/packages/37/18/6519e1ee6f5a1e579e04b9ddb6f1676c17368a7aba48299c3759bbc3c8b3/cffi-2.0.0-cp313-cp313-win_amd64.whl", hash = "sha256:19f705ada2530c1167abacb171925dd886168931e0a7b78f5bffcae5c6b5be75", size = 183402, upload-time = "2025-09-08T23:23:15.535Z" },+ { url = "https://files.pythonhosted.org/packages/cb/0e/02ceeec9a7d6ee63bb596121c2c8e9b3a9e150936f4fbef6ca1943e6137c/cffi-2.0.0-cp313-cp313-win_arm64.whl", hash = "sha256:256f80b80ca3853f90c21b23ee78cd008713787b1b1e93eae9f3d6a7134abd91", size = 177780, upload-time = "2025-09-08T23:23:16.761Z" },+ { url = "https://files.pythonhosted.org/packages/92/c4/3ce07396253a83250ee98564f8d7e9789fab8e58858f35d07a9a2c78de9f/cffi-2.0.0-cp314-cp314-macosx_10_13_x86_64.whl", hash = "sha256:fc33c5141b55ed366cfaad382df24fe7dcbc686de5be719b207bb248e3053dc5", size = 185320, upload-time = "2025-09-08T23:23:18.087Z" },+ { url = "https://files.pythonhosted.org/packages/59/dd/27e9fa567a23931c838c6b02d0764611c62290062a6d4e8ff7863daf9730/cffi-2.0.0-cp314-cp314-macosx_11_0_arm64.whl", hash = "sha256:c654de545946e0db659b3400168c9ad31b5d29593291482c43e3564effbcee13", size = 181487, upload-time = "2025-09-08T23:23:19.622Z" },+ { url = "https://files.pythonhosted.org/packages/d6/43/0e822876f87ea8a4ef95442c3d766a06a51fc5298823f884ef87aaad168c/cffi-2.0.0-cp314-cp314-manylinux2014_aarch64.manylinux_2_17_aarch64.whl", hash = "sha256:24b6f81f1983e6df8db3adc38562c83f7d4a0c36162885ec7f7b77c7dcbec97b", size = 220049, upload-time = "2025-09-08T23:23:20.853Z" },+ { url = "https://files.pythonhosted.org/packages/b4/89/76799151d9c2d2d1ead63c2429da9ea9d7aac304603de0c6e8764e6e8e70/cffi-2.0.0-cp314-cp314-manylinux2014_ppc64le.manylinux_2_17_ppc64le.whl", hash = "sha256:12873ca6cb9b0f0d3a0da705d6086fe911591737a59f28b7936bdfed27c0d47c", size = 207793, upload-time = "2025-09-08T23:23:22.08Z" },+ { url = "https://files.pythonhosted.org/packages/bb/dd/3465b14bb9e24ee24cb88c9e3730f6de63111fffe513492bf8c808a3547e/cffi-2.0.0-cp314-cp314-manylinux2014_s390x.manylinux_2_17_s390x.whl", hash = "sha256:d9b97165e8aed9272a6bb17c01e3cc5871a594a446ebedc996e2397a1c1ea8ef", size = 206300, upload-time = "2025-09-08T23:23:23.314Z" },+ { url = "https://files.pythonhosted.org/packages/47/d9/d83e293854571c877a92da46fdec39158f8d7e68da75bf73581225d28e90/cffi-2.0.0-cp314-cp314-manylinux2014_x86_64.manylinux_2_17_x86_64.whl", hash = "sha256:afb8db5439b81cf9c9d0c80404b60c3cc9c3add93e114dcae767f1477cb53775", size = 219244, upload-time = "2025-09-08T23:23:24.541Z" },+ { url = "https://files.pythonhosted.org/packages/2b/0f/1f177e3683aead2bb00f7679a16451d302c436b5cbf2505f0ea8146ef59e/cffi-2.0.0-cp314-cp314-musllinux_1_2_aarch64.whl", hash = "sha256:737fe7d37e1a1bffe70bd5754ea763a62a066dc5913ca57e957824b72a85e205", size = 222828, upload-time = "2025-09-08T23:23:26.143Z" },+ { url = "https://files.pythonhosted.org/packages/c6/0f/cafacebd4b040e3119dcb32fed8bdef8dfe94da653155f9d0b9dc660166e/cffi-2.0.0-cp314-cp314-musllinux_1_2_x86_64.whl", hash = "sha256:38100abb9d1b1435bc4cc340bb4489635dc2f0da7456590877030c9b3d40b0c1", size = 220926, upload-time = "2025-09-08T23:23:27.873Z" },+ { url = "https://files.pythonhosted.org/packages/3e/aa/df335faa45b395396fcbc03de2dfcab242cd61a9900e914fe682a59170b1/cffi-2.0.0-cp314-cp314-win32.whl", hash = "sha256:087067fa8953339c723661eda6b54bc98c5625757ea62e95eb4898ad5e776e9f", size = 175328, upload-time = "2025-09-08T23:23:44.61Z" },+ { url = "https://files.pythonhosted.org/packages/bb/92/882c2d30831744296ce713f0feb4c1cd30f346ef747b530b5318715cc367/cffi-2.0.0-cp314-cp314-win_amd64.whl", hash = "sha256:203a48d1fb583fc7d78a4c6655692963b860a417c0528492a6bc21f1aaefab25", size = 185650, upload-time = "2025-09-08T23:23:45.848Z" },+ { url = "https://files.pythonhosted.org/packages/9f/2c/98ece204b9d35a7366b5b2c6539c350313ca13932143e79dc133ba757104/cffi-2.0.0-cp314-cp314-win_arm64.whl", hash = "sha256:dbd5c7a25a7cb98f5ca55d258b103a2054f859a46ae11aaf23134f9cc0d356ad", size = 180687, upload-time = "2025-09-08T23:23:47.105Z" },+ { url = "https://files.pythonhosted.org/packages/3e/61/c768e4d548bfa607abcda77423448df8c471f25dbe64fb2ef6d555eae006/cffi-2.0.0-cp314-cp314t-macosx_10_13_x86_64.whl", hash = "sha256:9a67fc9e8eb39039280526379fb3a70023d77caec1852002b4da7e8b270c4dd9", size = 188773, upload-time = "2025-09-08T23:23:29.347Z" },+ { url = "https://files.pythonhosted.org/packages/2c/ea/5f76bce7cf6fcd0ab1a1058b5af899bfbef198bea4d5686da88471ea0336/cffi-2.0.0-cp314-cp314t-macosx_11_0_arm64.whl", hash = "sha256:7a66c7204d8869299919db4d5069a82f1561581af12b11b3c9f48c584eb8743d", size = 185013, upload-time = "2025-09-08T23:23:30.63Z" },+ { url = "https://files.pythonhosted.org/packages/be/b4/c56878d0d1755cf9caa54ba71e5d049479c52f9e4afc230f06822162ab2f/cffi-2.0.0-cp314-cp314t-manylinux2014_aarch64.manylinux_2_17_aarch64.whl", hash = "sha256:7cc09976e8b56f8cebd752f7113ad07752461f48a58cbba644139015ac24954c", size = 221593, upload-time = "2025-09-08T23:23:31.91Z" },+ { url = "https://files.pythonhosted.org/packages/e0/0d/eb704606dfe8033e7128df5e90fee946bbcb64a04fcdaa97321309004000/cffi-2.0.0-cp314-cp314t-manylinux2014_ppc64le.manylinux_2_17_ppc64le.whl", hash = "sha256:92b68146a71df78564e4ef48af17551a5ddd142e5190cdf2c5624d0c3ff5b2e8", size = 209354, upload-time = "2025-09-08T23:23:33.214Z" },+ { url = "https://files.pythonhosted.org/packages/d8/19/3c435d727b368ca475fb8742ab97c9cb13a0de600ce86f62eab7fa3eea60/cffi-2.0.0-cp314-cp314t-manylinux2014_s390x.manylinux_2_17_s390x.whl", hash = "sha256:b1e74d11748e7e98e2f426ab176d4ed720a64412b6a15054378afdb71e0f37dc", size = 208480, upload-time = "2025-09-08T23:23:34.495Z" },+ { url = "https://files.pythonhosted.org/packages/d0/44/681604464ed9541673e486521497406fadcc15b5217c3e326b061696899a/cffi-2.0.0-cp314-cp314t-manylinux2014_x86_64.manylinux_2_17_x86_64.whl", hash = "sha256:28a3a209b96630bca57cce802da70c266eb08c6e97e5afd61a75611ee6c64592", size = 221584, upload-time = "2025-09-08T23:23:36.096Z" },+ { url = "https://files.pythonhosted.org/packages/25/8e/342a504ff018a2825d395d44d63a767dd8ebc927ebda557fecdaca3ac33a/cffi-2.0.0-cp314-cp314t-musllinux_1_2_aarch64.whl", hash = "sha256:7553fb2090d71822f02c629afe6042c299edf91ba1bf94951165613553984512", size = 224443, upload-time = "2025-09-08T23:23:37.328Z" },+ { url = "https://files.pythonhosted.org/packages/e1/5e/b666bacbbc60fbf415ba9988324a132c9a7a0448a9a8f125074671c0f2c3/cffi-2.0.0-cp314-cp314t-musllinux_1_2_x86_64.whl", hash = "sha256:6c6c373cfc5c83a975506110d17457138c8c63016b563cc9ed6e056a82f13ce4", size = 223437, upload-time = "2025-09-08T23:23:38.945Z" },+ { url = "https://files.pythonhosted.org/packages/a0/1d/ec1a60bd1a10daa292d3cd6bb0b359a81607154fb8165f3ec95fe003b85c/cffi-2.0.0-cp314-cp314t-win32.whl", hash = "sha256:1fc9ea04857caf665289b7a75923f2c6ed559b8298a1b8c49e59f7dd95c8481e", size = 180487, upload-time = "2025-09-08T23:23:40.423Z" },+ { url = "https://files.pythonhosted.org/packages/bf/41/4c1168c74fac325c0c8156f04b6749c8b6a8f405bbf91413ba088359f60d/cffi-2.0.0-cp314-cp314t-win_amd64.whl", hash = "sha256:d68b6cef7827e8641e8ef16f4494edda8b36104d79773a334beaa1e3521430f6", size = 191726, upload-time = "2025-09-08T23:23:41.742Z" },+ { url = "https://files.pythonhosted.org/packages/ae/3a/dbeec9d1ee0844c679f6bb5d6ad4e9f198b1224f4e7a32825f47f6192b0c/cffi-2.0.0-cp314-cp314t-win_arm64.whl", hash = "sha256:0a1527a803f0a659de1af2e1fd700213caba79377e27e4693648c2923da066f9", size = 184195, upload-time = "2025-09-08T23:23:43.004Z" },+]+[[package]]name = "cfgv"version = "3.5.0"@@ -685,6 +770,71 @@ toml = [{ name = "tomli", marker = "python_full_version <= '3.11'" },]+[[package]]+name = "cryptography"+version = "46.0.3"+source = { registry = "https://pypi.org/simple" }+dependencies = [+ { name = "cffi", marker = "platform_python_implementation != 'PyPy'" },+ { name = "typing-extensions", marker = "python_full_version < '3.11'" },+]+sdist = { url = "https://files.pythonhosted.org/packages/9f/33/c00162f49c0e2fe8064a62cb92b93e50c74a72bc370ab92f86112b33ff62/cryptography-46.0.3.tar.gz", hash = "sha256:a8b17438104fed022ce745b362294d9ce35b4c2e45c1d958ad4a4b019285f4a1", size = 749258, upload-time = "2025-10-15T23:18:31.74Z" }+wheels = [+ { url = "https://files.pythonhosted.org/packages/1d/42/9c391dd801d6cf0d561b5890549d4b27bafcc53b39c31a817e69d87c625b/cryptography-46.0.3-cp311-abi3-macosx_10_9_universal2.whl", hash = "sha256:109d4ddfadf17e8e7779c39f9b18111a09efb969a301a31e987416a0191ed93a", size = 7225004, upload-time = "2025-10-15T23:16:52.239Z" },+ { url = "https://files.pythonhosted.org/packages/1c/67/38769ca6b65f07461eb200e85fc1639b438bdc667be02cf7f2cd6a64601c/cryptography-46.0.3-cp311-abi3-manylinux2014_aarch64.manylinux_2_17_aarch64.whl", hash = "sha256:09859af8466b69bc3c27bdf4f5d84a665e0f7ab5088412e9e2ec49758eca5cbc", size = 4296667, upload-time = "2025-10-15T23:16:54.369Z" },+ { url = "https://files.pythonhosted.org/packages/5c/49/498c86566a1d80e978b42f0d702795f69887005548c041636df6ae1ca64c/cryptography-46.0.3-cp311-abi3-manylinux2014_x86_64.manylinux_2_17_x86_64.whl", hash = "sha256:01ca9ff2885f3acc98c29f1860552e37f6d7c7d013d7334ff2a9de43a449315d", size = 4450807, upload-time = "2025-10-15T23:16:56.414Z" },+ { url = "https://files.pythonhosted.org/packages/4b/0a/863a3604112174c8624a2ac3c038662d9e59970c7f926acdcfaed8d61142/cryptography-46.0.3-cp311-abi3-manylinux_2_28_aarch64.whl", hash = "sha256:6eae65d4c3d33da080cff9c4ab1f711b15c1d9760809dad6ea763f3812d254cb", size = 4299615, upload-time = "2025-10-15T23:16:58.442Z" },+ { url = "https://files.pythonhosted.org/packages/64/02/b73a533f6b64a69f3cd3872acb6ebc12aef924d8d103133bb3ea750dc703/cryptography-46.0.3-cp311-abi3-manylinux_2_28_armv7l.manylinux_2_31_armv7l.whl", hash = "sha256:e5bf0ed4490068a2e72ac03d786693adeb909981cc596425d09032d372bcc849", size = 4016800, upload-time = "2025-10-15T23:17:00.378Z" },+ { url = "https://files.pythonhosted.org/packages/25/d5/16e41afbfa450cde85a3b7ec599bebefaef16b5c6ba4ec49a3532336ed72/cryptography-46.0.3-cp311-abi3-manylinux_2_28_ppc64le.whl", hash = "sha256:5ecfccd2329e37e9b7112a888e76d9feca2347f12f37918facbb893d7bb88ee8", size = 4984707, upload-time = "2025-10-15T23:17:01.98Z" },+ { url = "https://files.pythonhosted.org/packages/c9/56/e7e69b427c3878352c2fb9b450bd0e19ed552753491d39d7d0a2f5226d41/cryptography-46.0.3-cp311-abi3-manylinux_2_28_x86_64.whl", hash = "sha256:a2c0cd47381a3229c403062f764160d57d4d175e022c1df84e168c6251a22eec", size = 4482541, upload-time = "2025-10-15T23:17:04.078Z" },+ { url = "https://files.pythonhosted.org/packages/78/f6/50736d40d97e8483172f1bb6e698895b92a223dba513b0ca6f06b2365339/cryptography-46.0.3-cp311-abi3-manylinux_2_34_aarch64.whl", hash = "sha256:549e234ff32571b1f4076ac269fcce7a808d3bf98b76c8dd560e42dbc66d7d91", size = 4299464, upload-time = "2025-10-15T23:17:05.483Z" },+ { url = "https://files.pythonhosted.org/packages/00/de/d8e26b1a855f19d9994a19c702fa2e93b0456beccbcfe437eda00e0701f2/cryptography-46.0.3-cp311-abi3-manylinux_2_34_ppc64le.whl", hash = "sha256:c0a7bb1a68a5d3471880e264621346c48665b3bf1c3759d682fc0864c540bd9e", size = 4950838, upload-time = "2025-10-15T23:17:07.425Z" },+ { url = "https://files.pythonhosted.org/packages/8f/29/798fc4ec461a1c9e9f735f2fc58741b0daae30688f41b2497dcbc9ed1355/cryptography-46.0.3-cp311-abi3-manylinux_2_34_x86_64.whl", hash = "sha256:10b01676fc208c3e6feeb25a8b83d81767e8059e1fe86e1dc62d10a3018fa926", size = 4481596, upload-time = "2025-10-15T23:17:09.343Z" },+ { url = "https://files.pythonhosted.org/packages/15/8d/03cd48b20a573adfff7652b76271078e3045b9f49387920e7f1f631d125e/cryptography-46.0.3-cp311-abi3-musllinux_1_2_aarch64.whl", hash = "sha256:0abf1ffd6e57c67e92af68330d05760b7b7efb243aab8377e583284dbab72c71", size = 4426782, upload-time = "2025-10-15T23:17:11.22Z" },+ { url = "https://files.pythonhosted.org/packages/fa/b1/ebacbfe53317d55cf33165bda24c86523497a6881f339f9aae5c2e13e57b/cryptography-46.0.3-cp311-abi3-musllinux_1_2_x86_64.whl", hash = "sha256:a04bee9ab6a4da801eb9b51f1b708a1b5b5c9eb48c03f74198464c66f0d344ac", size = 4698381, upload-time = "2025-10-15T23:17:12.829Z" },+ { url = "https://files.pythonhosted.org/packages/96/92/8a6a9525893325fc057a01f654d7efc2c64b9de90413adcf605a85744ff4/cryptography-46.0.3-cp311-abi3-win32.whl", hash = "sha256:f260d0d41e9b4da1ed1e0f1ce571f97fe370b152ab18778e9e8f67d6af432018", size = 3055988, upload-time = "2025-10-15T23:17:14.65Z" },+ { url = "https://files.pythonhosted.org/packages/7e/bf/80fbf45253ea585a1e492a6a17efcb93467701fa79e71550a430c5e60df0/cryptography-46.0.3-cp311-abi3-win_amd64.whl", hash = "sha256:a9a3008438615669153eb86b26b61e09993921ebdd75385ddd748702c5adfddb", size = 3514451, upload-time = "2025-10-15T23:17:16.142Z" },+ { url = "https://files.pythonhosted.org/packages/2e/af/9b302da4c87b0beb9db4e756386a7c6c5b8003cd0e742277888d352ae91d/cryptography-46.0.3-cp311-abi3-win_arm64.whl", hash = "sha256:5d7f93296ee28f68447397bf5198428c9aeeab45705a55d53a6343455dcb2c3c", size = 2928007, upload-time = "2025-10-15T23:17:18.04Z" },+ { url = "https://files.pythonhosted.org/packages/f5/e2/a510aa736755bffa9d2f75029c229111a1d02f8ecd5de03078f4c18d91a3/cryptography-46.0.3-cp314-cp314t-macosx_10_9_universal2.whl", hash = "sha256:00a5e7e87938e5ff9ff5447ab086a5706a957137e6e433841e9d24f38a065217", size = 7158012, upload-time = "2025-10-15T23:17:19.982Z" },+ { url = "https://files.pythonhosted.org/packages/73/dc/9aa866fbdbb95b02e7f9d086f1fccfeebf8953509b87e3f28fff927ff8a0/cryptography-46.0.3-cp314-cp314t-manylinux2014_aarch64.manylinux_2_17_aarch64.whl", hash = "sha256:c8daeb2d2174beb4575b77482320303f3d39b8e81153da4f0fb08eb5fe86a6c5", size = 4288728, upload-time = "2025-10-15T23:17:21.527Z" },+ { url = "https://files.pythonhosted.org/packages/c5/fd/bc1daf8230eaa075184cbbf5f8cd00ba9db4fd32d63fb83da4671b72ed8a/cryptography-46.0.3-cp314-cp314t-manylinux2014_x86_64.manylinux_2_17_x86_64.whl", hash = "sha256:39b6755623145ad5eff1dab323f4eae2a32a77a7abef2c5089a04a3d04366715", size = 4435078, upload-time = "2025-10-15T23:17:23.042Z" },+ { url = "https://files.pythonhosted.org/packages/82/98/d3bd5407ce4c60017f8ff9e63ffee4200ab3e23fe05b765cab805a7db008/cryptography-46.0.3-cp314-cp314t-manylinux_2_28_aarch64.whl", hash = "sha256:db391fa7c66df6762ee3f00c95a89e6d428f4d60e7abc8328f4fe155b5ac6e54", size = 4293460, upload-time = "2025-10-15T23:17:24.885Z" },+ { url = "https://files.pythonhosted.org/packages/26/e9/e23e7900983c2b8af7a08098db406cf989d7f09caea7897e347598d4cd5b/cryptography-46.0.3-cp314-cp314t-manylinux_2_28_armv7l.manylinux_2_31_armv7l.whl", hash = "sha256:78a97cf6a8839a48c49271cdcbd5cf37ca2c1d6b7fdd86cc864f302b5e9bf459", size = 3995237, upload-time = "2025-10-15T23:17:26.449Z" },+ { url = "https://files.pythonhosted.org/packages/91/15/af68c509d4a138cfe299d0d7ddb14afba15233223ebd933b4bbdbc7155d3/cryptography-46.0.3-cp314-cp314t-manylinux_2_28_ppc64le.whl", hash = "sha256:dfb781ff7eaa91a6f7fd41776ec37c5853c795d3b358d4896fdbb5df168af422", size = 4967344, upload-time = "2025-10-15T23:17:28.06Z" },+ { url = "https://files.pythonhosted.org/packages/ca/e3/8643d077c53868b681af077edf6b3cb58288b5423610f21c62aadcbe99f4/cryptography-46.0.3-cp314-cp314t-manylinux_2_28_x86_64.whl", hash = "sha256:6f61efb26e76c45c4a227835ddeae96d83624fb0d29eb5df5b96e14ed1a0afb7", size = 4466564, upload-time = "2025-10-15T23:17:29.665Z" },+ { url = "https://files.pythonhosted.org/packages/0e/43/c1e8726fa59c236ff477ff2b5dc071e54b21e5a1e51aa2cee1676f1c986f/cryptography-46.0.3-cp314-cp314t-manylinux_2_34_aarch64.whl", hash = "sha256:23b1a8f26e43f47ceb6d6a43115f33a5a37d57df4ea0ca295b780ae8546e8044", size = 4292415, upload-time = "2025-10-15T23:17:31.686Z" },+ { url = "https://files.pythonhosted.org/packages/42/f9/2f8fefdb1aee8a8e3256a0568cffc4e6d517b256a2fe97a029b3f1b9fe7e/cryptography-46.0.3-cp314-cp314t-manylinux_2_34_ppc64le.whl", hash = "sha256:b419ae593c86b87014b9be7396b385491ad7f320bde96826d0dd174459e54665", size = 4931457, upload-time = "2025-10-15T23:17:33.478Z" },+ { url = "https://files.pythonhosted.org/packages/79/30/9b54127a9a778ccd6d27c3da7563e9f2d341826075ceab89ae3b41bf5be2/cryptography-46.0.3-cp314-cp314t-manylinux_2_34_x86_64.whl", hash = "sha256:50fc3343ac490c6b08c0cf0d704e881d0d660be923fd3076db3e932007e726e3", size = 4466074, upload-time = "2025-10-15T23:17:35.158Z" },+ { url = "https://files.pythonhosted.org/packages/ac/68/b4f4a10928e26c941b1b6a179143af9f4d27d88fe84a6a3c53592d2e76bf/cryptography-46.0.3-cp314-cp314t-musllinux_1_2_aarch64.whl", hash = "sha256:22d7e97932f511d6b0b04f2bfd818d73dcd5928db509460aaf48384778eb6d20", size = 4420569, upload-time = "2025-10-15T23:17:37.188Z" },+ { url = "https://files.pythonhosted.org/packages/a3/49/3746dab4c0d1979888f125226357d3262a6dd40e114ac29e3d2abdf1ec55/cryptography-46.0.3-cp314-cp314t-musllinux_1_2_x86_64.whl", hash = "sha256:d55f3dffadd674514ad19451161118fd010988540cee43d8bc20675e775925de", size = 4681941, upload-time = "2025-10-15T23:17:39.236Z" },+ { url = "https://files.pythonhosted.org/packages/fd/30/27654c1dbaf7e4a3531fa1fc77986d04aefa4d6d78259a62c9dc13d7ad36/cryptography-46.0.3-cp314-cp314t-win32.whl", hash = "sha256:8a6e050cb6164d3f830453754094c086ff2d0b2f3a897a1d9820f6139a1f0914", size = 3022339, upload-time = "2025-10-15T23:17:40.888Z" },+ { url = "https://files.pythonhosted.org/packages/f6/30/640f34ccd4d2a1bc88367b54b926b781b5a018d65f404d409aba76a84b1c/cryptography-46.0.3-cp314-cp314t-win_amd64.whl", hash = "sha256:760f83faa07f8b64e9c33fc963d790a2edb24efb479e3520c14a45741cd9b2db", size = 3494315, upload-time = "2025-10-15T23:17:42.769Z" },+ { url = "https://files.pythonhosted.org/packages/ba/8b/88cc7e3bd0a8e7b861f26981f7b820e1f46aa9d26cc482d0feba0ecb4919/cryptography-46.0.3-cp314-cp314t-win_arm64.whl", hash = "sha256:516ea134e703e9fe26bcd1277a4b59ad30586ea90c365a87781d7887a646fe21", size = 2919331, upload-time = "2025-10-15T23:17:44.468Z" },+ { url = "https://files.pythonhosted.org/packages/fd/23/45fe7f376a7df8daf6da3556603b36f53475a99ce4faacb6ba2cf3d82021/cryptography-46.0.3-cp38-abi3-macosx_10_9_universal2.whl", hash = "sha256:cb3d760a6117f621261d662bccc8ef5bc32ca673e037c83fbe565324f5c46936", size = 7218248, upload-time = "2025-10-15T23:17:46.294Z" },+ { url = "https://files.pythonhosted.org/packages/27/32/b68d27471372737054cbd34c84981f9edbc24fe67ca225d389799614e27f/cryptography-46.0.3-cp38-abi3-manylinux2014_aarch64.manylinux_2_17_aarch64.whl", hash = "sha256:4b7387121ac7d15e550f5cb4a43aef2559ed759c35df7336c402bb8275ac9683", size = 4294089, upload-time = "2025-10-15T23:17:48.269Z" },+ { url = "https://files.pythonhosted.org/packages/26/42/fa8389d4478368743e24e61eea78846a0006caffaf72ea24a15159215a14/cryptography-46.0.3-cp38-abi3-manylinux2014_x86_64.manylinux_2_17_x86_64.whl", hash = "sha256:15ab9b093e8f09daab0f2159bb7e47532596075139dd74365da52ecc9cb46c5d", size = 4440029, upload-time = "2025-10-15T23:17:49.837Z" },+ { url = "https://files.pythonhosted.org/packages/5f/eb/f483db0ec5ac040824f269e93dd2bd8a21ecd1027e77ad7bdf6914f2fd80/cryptography-46.0.3-cp38-abi3-manylinux_2_28_aarch64.whl", hash = "sha256:46acf53b40ea38f9c6c229599a4a13f0d46a6c3fa9ef19fc1a124d62e338dfa0", size = 4297222, upload-time = "2025-10-15T23:17:51.357Z" },+ { url = "https://files.pythonhosted.org/packages/fd/cf/da9502c4e1912cb1da3807ea3618a6829bee8207456fbbeebc361ec38ba3/cryptography-46.0.3-cp38-abi3-manylinux_2_28_armv7l.manylinux_2_31_armv7l.whl", hash = "sha256:10ca84c4668d066a9878890047f03546f3ae0a6b8b39b697457b7757aaf18dbc", size = 4012280, upload-time = "2025-10-15T23:17:52.964Z" },+ { url = "https://files.pythonhosted.org/packages/6b/8f/9adb86b93330e0df8b3dcf03eae67c33ba89958fc2e03862ef1ac2b42465/cryptography-46.0.3-cp38-abi3-manylinux_2_28_ppc64le.whl", hash = "sha256:36e627112085bb3b81b19fed209c05ce2a52ee8b15d161b7c643a7d5a88491f3", size = 4978958, upload-time = "2025-10-15T23:17:54.965Z" },+ { url = "https://files.pythonhosted.org/packages/d1/a0/5fa77988289c34bdb9f913f5606ecc9ada1adb5ae870bd0d1054a7021cc4/cryptography-46.0.3-cp38-abi3-manylinux_2_28_x86_64.whl", hash = "sha256:1000713389b75c449a6e979ffc7dcc8ac90b437048766cef052d4d30b8220971", size = 4473714, upload-time = "2025-10-15T23:17:56.754Z" },+ { url = "https://files.pythonhosted.org/packages/14/e5/fc82d72a58d41c393697aa18c9abe5ae1214ff6f2a5c18ac470f92777895/cryptography-46.0.3-cp38-abi3-manylinux_2_34_aarch64.whl", hash = "sha256:b02cf04496f6576afffef5ddd04a0cb7d49cf6be16a9059d793a30b035f6b6ac", size = 4296970, upload-time = "2025-10-15T23:17:58.588Z" },+ { url = "https://files.pythonhosted.org/packages/78/06/5663ed35438d0b09056973994f1aec467492b33bd31da36e468b01ec1097/cryptography-46.0.3-cp38-abi3-manylinux_2_34_ppc64le.whl", hash = "sha256:71e842ec9bc7abf543b47cf86b9a743baa95f4677d22baa4c7d5c69e49e9bc04", size = 4940236, upload-time = "2025-10-15T23:18:00.897Z" },+ { url = "https://files.pythonhosted.org/packages/fc/59/873633f3f2dcd8a053b8dd1d38f783043b5fce589c0f6988bf55ef57e43e/cryptography-46.0.3-cp38-abi3-manylinux_2_34_x86_64.whl", hash = "sha256:402b58fc32614f00980b66d6e56a5b4118e6cb362ae8f3fda141ba4689bd4506", size = 4472642, upload-time = "2025-10-15T23:18:02.749Z" },+ { url = "https://files.pythonhosted.org/packages/3d/39/8e71f3930e40f6877737d6f69248cf74d4e34b886a3967d32f919cc50d3b/cryptography-46.0.3-cp38-abi3-musllinux_1_2_aarch64.whl", hash = "sha256:ef639cb3372f69ec44915fafcd6698b6cc78fbe0c2ea41be867f6ed612811963", size = 4423126, upload-time = "2025-10-15T23:18:04.85Z" },+ { url = "https://files.pythonhosted.org/packages/cd/c7/f65027c2810e14c3e7268353b1681932b87e5a48e65505d8cc17c99e36ae/cryptography-46.0.3-cp38-abi3-musllinux_1_2_x86_64.whl", hash = "sha256:3b51b8ca4f1c6453d8829e1eb7299499ca7f313900dd4d89a24b8b87c0a780d4", size = 4686573, upload-time = "2025-10-15T23:18:06.908Z" },+ { url = "https://files.pythonhosted.org/packages/0a/6e/1c8331ddf91ca4730ab3086a0f1be19c65510a33b5a441cb334e7a2d2560/cryptography-46.0.3-cp38-abi3-win32.whl", hash = "sha256:6276eb85ef938dc035d59b87c8a7dc559a232f954962520137529d77b18ff1df", size = 3036695, upload-time = "2025-10-15T23:18:08.672Z" },+ { url = "https://files.pythonhosted.org/packages/90/45/b0d691df20633eff80955a0fc7695ff9051ffce8b69741444bd9ed7bd0db/cryptography-46.0.3-cp38-abi3-win_amd64.whl", hash = "sha256:416260257577718c05135c55958b674000baef9a1c7d9e8f306ec60d71db850f", size = 3501720, upload-time = "2025-10-15T23:18:10.632Z" },+ { url = "https://files.pythonhosted.org/packages/e8/cb/2da4cc83f5edb9c3257d09e1e7ab7b23f049c7962cae8d842bbef0a9cec9/cryptography-46.0.3-cp38-abi3-win_arm64.whl", hash = "sha256:d89c3468de4cdc4f08a57e214384d0471911a3830fcdaf7a8cc587e42a866372", size = 2918740, upload-time = "2025-10-15T23:18:12.277Z" },+ { url = "https://files.pythonhosted.org/packages/d9/cd/1a8633802d766a0fa46f382a77e096d7e209e0817892929655fe0586ae32/cryptography-46.0.3-pp310-pypy310_pp73-macosx_10_9_x86_64.whl", hash = "sha256:a23582810fedb8c0bc47524558fb6c56aac3fc252cb306072fd2815da2a47c32", size = 3689163, upload-time = "2025-10-15T23:18:13.821Z" },+ { url = "https://files.pythonhosted.org/packages/4c/59/6b26512964ace6480c3e54681a9859c974172fb141c38df11eadd8416947/cryptography-46.0.3-pp310-pypy310_pp73-win_amd64.whl", hash = "sha256:e7aec276d68421f9574040c26e2a7c3771060bc0cff408bae1dcb19d3ab1e63c", size = 3429474, upload-time = "2025-10-15T23:18:15.477Z" },+ { url = "https://files.pythonhosted.org/packages/06/8a/e60e46adab4362a682cf142c7dcb5bf79b782ab2199b0dcb81f55970807f/cryptography-46.0.3-pp311-pypy311_pp73-macosx_10_9_x86_64.whl", hash = "sha256:7ce938a99998ed3c8aa7e7272dca1a610401ede816d36d0693907d863b10d9ea", size = 3698132, upload-time = "2025-10-15T23:18:17.056Z" },+ { url = "https://files.pythonhosted.org/packages/da/38/f59940ec4ee91e93d3311f7532671a5cef5570eb04a144bf203b58552d11/cryptography-46.0.3-pp311-pypy311_pp73-manylinux_2_28_aarch64.whl", hash = "sha256:191bb60a7be5e6f54e30ba16fdfae78ad3a342a0599eb4193ba88e3f3d6e185b", size = 4243992, upload-time = "2025-10-15T23:18:18.695Z" },+ { url = "https://files.pythonhosted.org/packages/b0/0c/35b3d92ddebfdfda76bb485738306545817253d0a3ded0bfe80ef8e67aa5/cryptography-46.0.3-pp311-pypy311_pp73-manylinux_2_28_x86_64.whl", hash = "sha256:c70cc23f12726be8f8bc72e41d5065d77e4515efae3690326764ea1b07845cfb", size = 4409944, upload-time = "2025-10-15T23:18:20.597Z" },+ { url = "https://files.pythonhosted.org/packages/99/55/181022996c4063fc0e7666a47049a1ca705abb9c8a13830f074edb347495/cryptography-46.0.3-pp311-pypy311_pp73-manylinux_2_34_aarch64.whl", hash = "sha256:9394673a9f4de09e28b5356e7fff97d778f8abad85c9d5ac4a4b7e25a0de7717", size = 4242957, upload-time = "2025-10-15T23:18:22.18Z" },+ { url = "https://files.pythonhosted.org/packages/ba/af/72cd6ef29f9c5f731251acadaeb821559fe25f10852f44a63374c9ca08c1/cryptography-46.0.3-pp311-pypy311_pp73-manylinux_2_34_x86_64.whl", hash = "sha256:94cd0549accc38d1494e1f8de71eca837d0509d0d44bf11d158524b0e12cebf9", size = 4409447, upload-time = "2025-10-15T23:18:24.209Z" },+ { url = "https://files.pythonhosted.org/packages/0d/c3/e90f4a4feae6410f914f8ebac129b9ae7a8c92eb60a638012dde42030a9d/cryptography-46.0.3-pp311-pypy311_pp73-win_amd64.whl", hash = "sha256:6b5063083824e5509fdba180721d55909ffacccc8adbec85268b48439423d78c", size = 3438528, upload-time = "2025-10-15T23:18:26.227Z" },+]+[[package]]name = "dill"version = "0.4.0"@@ -1095,6 +1245,22 @@ wheels = [{ url = "https://files.pythonhosted.org/packages/f7/ec/67fbef5d497f86283db54c22eec6f6140243aae73265799baaaa19cd17fb/ghp_import-2.1.0-py3-none-any.whl", hash = "sha256:8337dd7b50877f163d4c0289bc1f1c7f127550241988d568c1db512c4324a619", size = 11034, upload-time = "2022-05-02T15:47:14.552Z" },]+[[package]]+name = "google-api-core"+version = "2.28.1"+source = { registry = "https://pypi.org/simple" }+dependencies = [+ { name = "google-auth" },+ { name = "googleapis-common-protos" },+ { name = "proto-plus" },+ { name = "protobuf" },+ { name = "requests" },+]+sdist = { url = "https://files.pythonhosted.org/packages/61/da/83d7043169ac2c8c7469f0e375610d78ae2160134bf1b80634c482fa079c/google_api_core-2.28.1.tar.gz", hash = "sha256:2b405df02d68e68ce0fbc138559e6036559e685159d148ae5861013dc201baf8", size = 176759, upload-time = "2025-10-28T21:34:51.529Z" }+wheels = [+ { url = "https://files.pythonhosted.org/packages/ed/d4/90197b416cb61cefd316964fd9e7bd8324bcbafabf40eef14a9f20b81974/google_api_core-2.28.1-py3-none-any.whl", hash = "sha256:4021b0f8ceb77a6fb4de6fde4502cecab45062e66ff4f2895169e0b35bc9466c", size = 173706, upload-time = "2025-10-28T21:34:50.151Z" },+]+[[package]]name = "google-auth"version = "2.41.1"@@ -1127,6 +1293,71 @@ wheels = [{ url = "https://files.pythonhosted.org/packages/38/07/a54c100da461ffc5968457823fcc665a48fb4b875c68bcfecbfe24a10dbe/google_auth_oauthlib-1.2.3-py3-none-any.whl", hash = "sha256:7c0940e037677f25e71999607493640d071212e7f3c15aa0febea4c47a5a0680", size = 19184, upload-time = "2025-10-30T21:28:17.88Z" },]+[[package]]+name = "google-cloud-core"+version = "2.5.0"+source = { registry = "https://pypi.org/simple" }+dependencies = [+ { name = "google-api-core" },+ { name = "google-auth" },+]+sdist = { url = "https://files.pythonhosted.org/packages/a6/03/ef0bc99d0e0faf4fdbe67ac445e18cdaa74824fd93cd069e7bb6548cb52d/google_cloud_core-2.5.0.tar.gz", hash = "sha256:7c1b7ef5c92311717bd05301aa1a91ffbc565673d3b0b4163a52d8413a186963", size = 36027, upload-time = "2025-10-29T23:17:39.513Z" }+wheels = [+ { url = "https://files.pythonhosted.org/packages/89/20/bfa472e327c8edee00f04beecc80baeddd2ab33ee0e86fd7654da49d45e9/google_cloud_core-2.5.0-py3-none-any.whl", hash = "sha256:67d977b41ae6c7211ee830c7912e41003ea8194bff15ae7d72fd6f51e57acabc", size = 29469, upload-time = "2025-10-29T23:17:38.548Z" },+]++[[package]]+name = "google-cloud-storage"+version = "3.7.0"+source = { registry = "https://pypi.org/simple" }+dependencies = [+ { name = "google-api-core" },+ { name = "google-auth" },+ { name = "google-cloud-core" },+ { name = "google-crc32c" },+ { name = "google-resumable-media" },+ { name = "requests" },+]+sdist = { url = "https://files.pythonhosted.org/packages/d2/8e/fab2de1a0ab7fdbd452eaae5a9a5c933d0911c26b04efa0c76ddfd921259/google_cloud_storage-3.7.0.tar.gz", hash = "sha256:9ce59c65f4d6e372effcecc0456680a8d73cef4f2dc9212a0704799cb3d69237", size = 17258914, upload-time = "2025-12-09T18:24:48.97Z" }+wheels = [+ { url = "https://files.pythonhosted.org/packages/2d/80/6e5c7c83cea15ed4dfc4843b9df9db0716bc551ac938f7b5dd18a72bd5e4/google_cloud_storage-3.7.0-py3-none-any.whl", hash = "sha256:469bc9540936e02f8a4bfd1619e9dca1e42dec48f95e4204d783b36476a15093", size = 303364, upload-time = "2025-12-09T18:24:47.343Z" },+]++[[package]]+name = "google-crc32c"+version = "1.8.0"+source = { registry = "https://pypi.org/simple" }+sdist = { url = "https://files.pythonhosted.org/packages/03/41/4b9c02f99e4c5fb477122cd5437403b552873f014616ac1d19ac8221a58d/google_crc32c-1.8.0.tar.gz", hash = "sha256:a428e25fb7691024de47fecfbff7ff957214da51eddded0da0ae0e0f03a2cf79", size = 14192, upload-time = "2025-12-16T00:35:25.142Z" }+wheels = [+ { url = "https://files.pythonhosted.org/packages/95/ac/6f7bc93886a823ab545948c2dd48143027b2355ad1944c7cf852b338dc91/google_crc32c-1.8.0-cp310-cp310-macosx_12_0_arm64.whl", hash = "sha256:0470b8c3d73b5f4e3300165498e4cf25221c7eb37f1159e221d1825b6df8a7ff", size = 31296, upload-time = "2025-12-16T00:19:07.261Z" },+ { url = "https://files.pythonhosted.org/packages/f7/97/a5accde175dee985311d949cfcb1249dcbb290f5ec83c994ea733311948f/google_crc32c-1.8.0-cp310-cp310-macosx_12_0_x86_64.whl", hash = "sha256:119fcd90c57c89f30040b47c211acee231b25a45d225e3225294386f5d258288", size = 30870, upload-time = "2025-12-16T00:29:17.669Z" },+ { url = "https://files.pythonhosted.org/packages/3d/63/bec827e70b7a0d4094e7476f863c0dbd6b5f0f1f91d9c9b32b76dcdfeb4e/google_crc32c-1.8.0-cp310-cp310-manylinux1_x86_64.manylinux2014_x86_64.manylinux_2_17_x86_64.manylinux_2_5_x86_64.whl", hash = "sha256:6f35aaffc8ccd81ba3162443fabb920e65b1f20ab1952a31b13173a67811467d", size = 33214, upload-time = "2025-12-16T00:40:19.618Z" },+ { url = "https://files.pythonhosted.org/packages/63/bc/11b70614df04c289128d782efc084b9035ef8466b3d0a8757c1b6f5cf7ac/google_crc32c-1.8.0-cp310-cp310-manylinux2014_aarch64.manylinux_2_17_aarch64.whl", hash = "sha256:864abafe7d6e2c4c66395c1eb0fe12dc891879769b52a3d56499612ca93b6092", size = 33589, upload-time = "2025-12-16T00:40:20.7Z" },+ { url = "https://files.pythonhosted.org/packages/3e/00/a08a4bc24f1261cc5b0f47312d8aebfbe4b53c2e6307f1b595605eed246b/google_crc32c-1.8.0-cp310-cp310-win_amd64.whl", hash = "sha256:db3fe8eaf0612fc8b20fa21a5f25bd785bc3cd5be69f8f3412b0ac2ffd49e733", size = 34437, upload-time = "2025-12-16T00:35:19.437Z" },+ { url = "https://files.pythonhosted.org/packages/5d/ef/21ccfaab3d5078d41efe8612e0ed0bfc9ce22475de074162a91a25f7980d/google_crc32c-1.8.0-cp311-cp311-macosx_12_0_arm64.whl", hash = "sha256:014a7e68d623e9a4222d663931febc3033c5c7c9730785727de2a81f87d5bab8", size = 31298, upload-time = "2025-12-16T00:20:32.241Z" },+ { url = "https://files.pythonhosted.org/packages/c5/b8/f8413d3f4b676136e965e764ceedec904fe38ae8de0cdc52a12d8eb1096e/google_crc32c-1.8.0-cp311-cp311-macosx_12_0_x86_64.whl", hash = "sha256:86cfc00fe45a0ac7359e5214a1704e51a99e757d0272554874f419f79838c5f7", size = 30872, upload-time = "2025-12-16T00:33:58.785Z" },+ { url = "https://files.pythonhosted.org/packages/f6/fd/33aa4ec62b290477181c55bb1c9302c9698c58c0ce9a6ab4874abc8b0d60/google_crc32c-1.8.0-cp311-cp311-manylinux1_x86_64.manylinux2014_x86_64.manylinux_2_17_x86_64.manylinux_2_5_x86_64.whl", hash = "sha256:19b40d637a54cb71e0829179f6cb41835f0fbd9e8eb60552152a8b52c36cbe15", size = 33243, upload-time = "2025-12-16T00:40:21.46Z" },+ { url = "https://files.pythonhosted.org/packages/71/03/4820b3bd99c9653d1a5210cb32f9ba4da9681619b4d35b6a052432df4773/google_crc32c-1.8.0-cp311-cp311-manylinux2014_aarch64.manylinux_2_17_aarch64.whl", hash = "sha256:17446feb05abddc187e5441a45971b8394ea4c1b6efd88ab0af393fd9e0a156a", size = 33608, upload-time = "2025-12-16T00:40:22.204Z" },+ { url = "https://files.pythonhosted.org/packages/7c/43/acf61476a11437bf9733fb2f70599b1ced11ec7ed9ea760fdd9a77d0c619/google_crc32c-1.8.0-cp311-cp311-win_amd64.whl", hash = "sha256:71734788a88f551fbd6a97be9668a0020698e07b2bf5b3aa26a36c10cdfb27b2", size = 34439, upload-time = "2025-12-16T00:35:20.458Z" },+ { url = "https://files.pythonhosted.org/packages/e9/5f/7307325b1198b59324c0fa9807cafb551afb65e831699f2ce211ad5c8240/google_crc32c-1.8.0-cp312-cp312-macosx_12_0_arm64.whl", hash = "sha256:4b8286b659c1335172e39563ab0a768b8015e88e08329fa5321f774275fc3113", size = 31300, upload-time = "2025-12-16T00:21:56.723Z" },+ { url = "https://files.pythonhosted.org/packages/21/8e/58c0d5d86e2220e6a37befe7e6a94dd2f6006044b1a33edf1ff6d9f7e319/google_crc32c-1.8.0-cp312-cp312-macosx_12_0_x86_64.whl", hash = "sha256:2a3dc3318507de089c5384cc74d54318401410f82aa65b2d9cdde9d297aca7cb", size = 30867, upload-time = "2025-12-16T00:38:31.302Z" },+ { url = "https://files.pythonhosted.org/packages/ce/a9/a780cc66f86335a6019f557a8aaca8fbb970728f0efd2430d15ff1beae0e/google_crc32c-1.8.0-cp312-cp312-manylinux1_x86_64.manylinux2014_x86_64.manylinux_2_17_x86_64.manylinux_2_5_x86_64.whl", hash = "sha256:14f87e04d613dfa218d6135e81b78272c3b904e2a7053b841481b38a7d901411", size = 33364, upload-time = "2025-12-16T00:40:22.96Z" },+ { url = "https://files.pythonhosted.org/packages/21/3f/3457ea803db0198c9aaca2dd373750972ce28a26f00544b6b85088811939/google_crc32c-1.8.0-cp312-cp312-manylinux2014_aarch64.manylinux_2_17_aarch64.whl", hash = "sha256:cb5c869c2923d56cb0c8e6bcdd73c009c36ae39b652dbe46a05eb4ef0ad01454", size = 33740, upload-time = "2025-12-16T00:40:23.96Z" },+ { url = "https://files.pythonhosted.org/packages/df/c0/87c2073e0c72515bb8733d4eef7b21548e8d189f094b5dad20b0ecaf64f6/google_crc32c-1.8.0-cp312-cp312-win_amd64.whl", hash = "sha256:3cc0c8912038065eafa603b238abf252e204accab2a704c63b9e14837a854962", size = 34437, upload-time = "2025-12-16T00:35:21.395Z" },+ { url = "https://files.pythonhosted.org/packages/d1/db/000f15b41724589b0e7bc24bc7a8967898d8d3bc8caf64c513d91ef1f6c0/google_crc32c-1.8.0-cp313-cp313-macosx_12_0_arm64.whl", hash = "sha256:3ebb04528e83b2634857f43f9bb8ef5b2bbe7f10f140daeb01b58f972d04736b", size = 31297, upload-time = "2025-12-16T00:23:20.709Z" },+ { url = "https://files.pythonhosted.org/packages/d7/0d/8ebed0c39c53a7e838e2a486da8abb0e52de135f1b376ae2f0b160eb4c1a/google_crc32c-1.8.0-cp313-cp313-macosx_12_0_x86_64.whl", hash = "sha256:450dc98429d3e33ed2926fc99ee81001928d63460f8538f21a5d6060912a8e27", size = 30867, upload-time = "2025-12-16T00:43:14.628Z" },+ { url = "https://files.pythonhosted.org/packages/ce/42/b468aec74a0354b34c8cbf748db20d6e350a68a2b0912e128cabee49806c/google_crc32c-1.8.0-cp313-cp313-manylinux1_x86_64.manylinux2014_x86_64.manylinux_2_17_x86_64.manylinux_2_5_x86_64.whl", hash = "sha256:3b9776774b24ba76831609ffbabce8cdf6fa2bd5e9df37b594221c7e333a81fa", size = 33344, upload-time = "2025-12-16T00:40:24.742Z" },+ { url = "https://files.pythonhosted.org/packages/1c/e8/b33784d6fc77fb5062a8a7854e43e1e618b87d5ddf610a88025e4de6226e/google_crc32c-1.8.0-cp313-cp313-manylinux2014_aarch64.manylinux_2_17_aarch64.whl", hash = "sha256:89c17d53d75562edfff86679244830599ee0a48efc216200691de8b02ab6b2b8", size = 33694, upload-time = "2025-12-16T00:40:25.505Z" },+ { url = "https://files.pythonhosted.org/packages/92/b1/d3cbd4d988afb3d8e4db94ca953df429ed6db7282ed0e700d25e6c7bfc8d/google_crc32c-1.8.0-cp313-cp313-win_amd64.whl", hash = "sha256:57a50a9035b75643996fbf224d6661e386c7162d1dfdab9bc4ca790947d1007f", size = 34435, upload-time = "2025-12-16T00:35:22.107Z" },+ { url = "https://files.pythonhosted.org/packages/21/88/8ecf3c2b864a490b9e7010c84fd203ec8cf3b280651106a3a74dd1b0ca72/google_crc32c-1.8.0-cp314-cp314-macosx_12_0_arm64.whl", hash = "sha256:e6584b12cb06796d285d09e33f63309a09368b9d806a551d8036a4207ea43697", size = 31301, upload-time = "2025-12-16T00:24:48.527Z" },+ { url = "https://files.pythonhosted.org/packages/36/c6/f7ff6c11f5ca215d9f43d3629163727a272eabc356e5c9b2853df2bfe965/google_crc32c-1.8.0-cp314-cp314-macosx_12_0_x86_64.whl", hash = "sha256:f4b51844ef67d6cf2e9425983274da75f18b1597bb2c998e1c0a0e8d46f8f651", size = 30868, upload-time = "2025-12-16T00:48:12.163Z" },+ { url = "https://files.pythonhosted.org/packages/56/15/c25671c7aad70f8179d858c55a6ae8404902abe0cdcf32a29d581792b491/google_crc32c-1.8.0-cp314-cp314-manylinux1_x86_64.manylinux2014_x86_64.manylinux_2_17_x86_64.manylinux_2_5_x86_64.whl", hash = "sha256:b0d1a7afc6e8e4635564ba8aa5c0548e3173e41b6384d7711a9123165f582de2", size = 33381, upload-time = "2025-12-16T00:40:26.268Z" },+ { url = "https://files.pythonhosted.org/packages/42/fa/f50f51260d7b0ef5d4898af122d8a7ec5a84e2984f676f746445f783705f/google_crc32c-1.8.0-cp314-cp314-manylinux2014_aarch64.manylinux_2_17_aarch64.whl", hash = "sha256:8b3f68782f3cbd1bce027e48768293072813469af6a61a86f6bb4977a4380f21", size = 33734, upload-time = "2025-12-16T00:40:27.028Z" },+ { url = "https://files.pythonhosted.org/packages/08/a5/7b059810934a09fb3ccb657e0843813c1fee1183d3bc2c8041800374aa2c/google_crc32c-1.8.0-cp314-cp314-win_amd64.whl", hash = "sha256:d511b3153e7011a27ab6ee6bb3a5404a55b994dc1a7322c0b87b29606d9790e2", size = 34878, upload-time = "2025-12-16T00:35:23.142Z" },+ { url = "https://files.pythonhosted.org/packages/52/c5/c171e4d8c44fec1422d801a6d2e5d7ddabd733eeda505c79730ee9607f07/google_crc32c-1.8.0-pp311-pypy311_pp73-manylinux1_x86_64.manylinux2014_x86_64.manylinux_2_17_x86_64.manylinux_2_5_x86_64.whl", hash = "sha256:87fa445064e7db928226b2e6f0d5304ab4cd0339e664a4e9a25029f384d9bb93", size = 28615, upload-time = "2025-12-16T00:40:29.298Z" },+ { url = "https://files.pythonhosted.org/packages/9c/97/7d75fe37a7a6ed171a2cf17117177e7aab7e6e0d115858741b41e9dd4254/google_crc32c-1.8.0-pp311-pypy311_pp73-manylinux2014_aarch64.manylinux_2_17_aarch64.whl", hash = "sha256:f639065ea2042d5c034bf258a9f085eaa7af0cd250667c0635a3118e8f92c69c", size = 28800, upload-time = "2025-12-16T00:40:30.322Z" },+]+[[package]]name = "google-genai"version = "1.55.0"@@ -1148,6 +1379,18 @@ wheels = [{ url = "https://files.pythonhosted.org/packages/3e/86/a5a8e32b2d40b30b5fb20e7b8113fafd1e38befa4d1801abd5ce6991065a/google_genai-1.55.0-py3-none-any.whl", hash = "sha256:98c422762b5ff6e16b8d9a1e4938e8e0ad910392a5422e47f5301498d7f373a1", size = 703389, upload-time = "2025-12-11T02:49:27.105Z" },]+[[package]]+name = "google-resumable-media"+version = "2.8.0"+source = { registry = "https://pypi.org/simple" }+dependencies = [+ { name = "google-crc32c" },+]+sdist = { url = "https://files.pythonhosted.org/packages/64/d7/520b62a35b23038ff005e334dba3ffc75fcf583bee26723f1fd8fd4b6919/google_resumable_media-2.8.0.tar.gz", hash = "sha256:f1157ed8b46994d60a1bc432544db62352043113684d4e030ee02e77ebe9a1ae", size = 2163265, upload-time = "2025-11-17T15:38:06.659Z" }+wheels = [+ { url = "https://files.pythonhosted.org/packages/1f/0b/93afde9cfe012260e9fe1522f35c9b72d6ee222f316586b1f23ecf44d518/google_resumable_media-2.8.0-py3-none-any.whl", hash = "sha256:dd14a116af303845a8d932ddae161a26e86cc229645bc98b39f026f9b1717582", size = 81340, upload-time = "2025-11-17T15:38:05.594Z" },+]+[[package]]name = "googleapis-common-protos"version = "1.72.0"@@ -1982,6 +2225,32 @@ wheels = [{ url = "https://files.pythonhosted.org/packages/fc/5b/0a90ae6576cd7fa2593e7bdde4fa52fe4e79f5812b8f1ada001414aaaa54/mkdocs_meta_manager-1.1.0-py3-none-any.whl", hash = "sha256:bdd1b5e9403bcfe222720dcd93102e0f1df5befef9b40a6ff91ad67926528658", size = 3186, upload-time = "2025-01-20T21:58:02.314Z" },]+[[package]]+name = "moto"+version = "5.1.19"+source = { registry = "https://pypi.org/simple" }+dependencies = [+ { name = "boto3" },+ { name = "botocore" },+ { name = "cryptography" },+ { name = "jinja2" },+ { name = "python-dateutil" },+ { name = "requests" },+ { name = "responses" },+ { name = "werkzeug" },+ { name = "xmltodict" },+]+sdist = { url = "https://files.pythonhosted.org/packages/45/eb/100a04d1b49859d05a9c701815117cd31bc436c3d9e959d399d9d2ff7e9c/moto-5.1.19.tar.gz", hash = "sha256:a13423e402366b6affab07ed28e1df5f3fcc54ef68fc8d83dc9f824da7a4024e", size = 8361592, upload-time = "2025-12-28T20:14:57.211Z" }+wheels = [+ { url = "https://files.pythonhosted.org/packages/89/07/5ca7ba79615b88ee2325224894667f263b992d266a52b83d215c4b3caa39/moto-5.1.19-py3-none-any.whl", hash = "sha256:7adb0caacf0e2d0dbb09550bcb49a7f158ee7c460a09cb54d4599a9a94cfef70", size = 6451569, upload-time = "2025-12-28T20:14:54.701Z" },+]++[package.optional-dependencies]+s3 = [+ { name = "py-partiql-parser" },+ { name = "pyyaml" },+]+[[package]]name = "mpire"version = "2.10.2"@@ -2306,9 +2575,12 @@ name = "networkx"version = "3.6.1"source = { registry = "https://pypi.org/simple" }resolution-markers = [- "python_full_version >= '3.12' and sys_platform == 'darwin'",- "python_full_version >= '3.12' and platform_machine == 'aarch64' and sys_platform == 'linux'",- "(python_full_version >= '3.12' and platform_machine != 'aarch64' and sys_platform == 'linux') or (python_full_version >= '3.12' and sys_platform != 'darwin' and sys_platform != 'linux')",+ "python_full_version >= '3.13' and sys_platform == 'darwin'",+ "python_full_version == '3.12.*' and sys_platform == 'darwin'",+ "python_full_version >= '3.13' and platform_machine == 'aarch64' and sys_platform == 'linux'",+ "python_full_version == '3.12.*' and platform_machine == 'aarch64' and sys_platform == 'linux'",+ "(python_full_version >= '3.13' and platform_machine != 'aarch64' and sys_platform == 'linux') or (python_full_version >= '3.13' and sys_platform != 'darwin' and sys_platform != 'linux')",+ "(python_full_version == '3.12.*' and platform_machine != 'aarch64' and sys_platform == 'linux') or (python_full_version == '3.12.*' and sys_platform != 'darwin' and sys_platform != 'linux')","python_full_version == '3.11.*' and sys_platform == 'darwin'","python_full_version == '3.11.*' and platform_machine == 'aarch64' and sys_platform == 'linux'","(python_full_version == '3.11.*' and platform_machine != 'aarch64' and sys_platform == 'linux') or (python_full_version == '3.11.*' and sys_platform != 'darwin' and sys_platform != 'linux')",@@ -2399,9 +2671,12 @@ name = "numpy"version = "2.4.0"source = { registry = "https://pypi.org/simple" }resolution-markers = [- "python_full_version >= '3.12' and sys_platform == 'darwin'",- "python_full_version >= '3.12' and platform_machine == 'aarch64' and sys_platform == 'linux'",- "(python_full_version >= '3.12' and platform_machine != 'aarch64' and sys_platform == 'linux') or (python_full_version >= '3.12' and sys_platform != 'darwin' and sys_platform != 'linux')",+ "python_full_version >= '3.13' and sys_platform == 'darwin'",+ "python_full_version == '3.12.*' and sys_platform == 'darwin'",+ "python_full_version >= '3.13' and platform_machine == 'aarch64' and sys_platform == 'linux'",+ "python_full_version == '3.12.*' and platform_machine == 'aarch64' and sys_platform == 'linux'",+ "(python_full_version >= '3.13' and platform_machine != 'aarch64' and sys_platform == 'linux') or (python_full_version >= '3.13' and sys_platform != 'darwin' and sys_platform != 'linux')",+ "(python_full_version == '3.12.*' and platform_machine != 'aarch64' and sys_platform == 'linux') or (python_full_version == '3.12.*' and sys_platform != 'darwin' and sys_platform != 'linux')","python_full_version == '3.11.*' and sys_platform == 'darwin'","python_full_version == '3.11.*' and platform_machine == 'aarch64' and sys_platform == 'linux'","(python_full_version == '3.11.*' and platform_machine != 'aarch64' and sys_platform == 'linux') or (python_full_version == '3.11.*' and sys_platform != 'darwin' and sys_platform != 'linux')",@@ -3020,6 +3295,7 @@ bedrock = []dev = [{ name = "boto3-stubs" },+ { name = "moto", extra = ["s3"] },{ name = "mypy" },{ name = "pylint" },{ name = "pyright" },@@ -3048,6 +3324,9 @@ docs = [fal = [{ name = "fal-client" },]+gcp-storage = [+ { name = "google-cloud-storage" },+]google = [{ name = "google-auth-oauthlib" },]@@ -3061,19 +3340,26 @@ huggingface = [mistralai = [{ name = "mistralai" },]+s3 = [+ { name = "aioboto3" },+ { name = "boto3" },+][package.metadata]requires-dist = [{ name = "aioboto3", marker = "extra == 'bedrock'", specifier = ">=13.4.0" },+ { name = "aioboto3", marker = "extra == 's3'", specifier = ">=13.4.0" },{ name = "aiofiles", specifier = ">=23.2.1" },{ name = "anthropic", marker = "extra == 'anthropic'", specifier = ">=0.60.0" },{ name = "backports-strenum", marker = "python_full_version < '3.11'", specifier = ">=1.3.0" },{ name = "boto3", marker = "extra == 'bedrock'", specifier = ">=1.34.131" },+ { name = "boto3", marker = "extra == 's3'", specifier = ">=1.34.131" },{ name = "boto3-stubs", marker = "extra == 'dev'", specifier = ">=1.35.24" },{ name = "docling", marker = "extra == 'docling'", specifier = ">=2.64.0" },{ name = "fal-client", marker = "extra == 'fal'", specifier = ">=0.4.1" },{ name = "filetype", specifier = ">=1.2.0" },{ name = "google-auth-oauthlib", marker = "extra == 'google'", specifier = ">=1.2.1" },+ { name = "google-cloud-storage", marker = "extra == 'gcp-storage'", specifier = ">=2.10.0" },{ name = "google-genai", marker = "extra == 'google-genai'" },{ name = "httpx", specifier = ">=0.23.0,<1.0.0" },{ name = "huggingface-hub", marker = "extra == 'huggingface'", specifier = ">=0.23,<1.0.0" },@@ -3088,6 +3374,7 @@ requires-dist = [{ name = "mkdocs-glightbox", marker = "extra == 'docs'", specifier = "==0.4.0" },{ name = "mkdocs-material", marker = "extra == 'docs'", specifier = "==9.6.14" },{ name = "mkdocs-meta-manager", marker = "extra == 'docs'", specifier = "==1.1.0" },+ { name = "moto", extras = ["s3"], marker = "extra == 'dev'", specifier = ">=5.0.0" },{ name = "mypy", marker = "extra == 'dev'", specifier = ">=1.11.2" },{ name = "networkx", specifier = ">=3.4.2" },{ name = "openai", specifier = ">=1.108.1" },@@ -3124,7 +3411,7 @@ requires-dist = [{ name = "types-pyyaml", marker = "extra == 'dev'", specifier = ">=6.0.12.20250326" },{ name = "typing-extensions", specifier = ">=4.13.2" },]-provides-extras = ["anthropic", "bedrock", "docling", "fal", "google", "google-genai", "huggingface", "mistralai", "docs", "dev"]+provides-extras = ["anthropic", "bedrock", "docling", "fal", "gcp-storage", "google", "google-genai", "huggingface", "mistralai", "s3", "docs", "dev"][[package]]name = "platformdirs"@@ -3325,6 +3612,18 @@ wheels = [{ url = "https://files.pythonhosted.org/packages/5b/5a/bc7b4a4ef808fa59a816c17b20c4bef6884daebbdf627ff2a161da67da19/propcache-0.4.1-py3-none-any.whl", hash = "sha256:af2a6052aeb6cf17d3e46ee169099044fd8224cbaf75c76a2ef596e8163e2237", size = 13305, upload-time = "2025-10-08T19:49:00.792Z" },]+[[package]]+name = "proto-plus"+version = "1.27.0"+source = { registry = "https://pypi.org/simple" }+dependencies = [+ { name = "protobuf" },+]+sdist = { url = "https://files.pythonhosted.org/packages/01/89/9cbe2f4bba860e149108b683bc2efec21f14d5f7ed6e25562ad86acbc373/proto_plus-1.27.0.tar.gz", hash = "sha256:873af56dd0d7e91836aee871e5799e1c6f1bda86ac9a983e0bb9f0c266a568c4", size = 56158, upload-time = "2025-12-16T13:46:25.729Z" }+wheels = [+ { url = "https://files.pythonhosted.org/packages/cd/24/3b7a0818484df9c28172857af32c2397b6d8fcd99d9468bd4684f98ebf0a/proto_plus-1.27.0-py3-none-any.whl", hash = "sha256:1baa7f81cf0f8acb8bc1f6d085008ba4171eaf669629d1b6d1673b21ed1c0a82", size = 50205, upload-time = "2025-12-16T13:46:24.76Z" },+]+[[package]]name = "protobuf"version = "6.33.2"@@ -3368,6 +3667,15 @@ wheels = [{ url = "https://files.pythonhosted.org/packages/3e/73/2ce007f4198c80fcf2cb24c169884f833fe93fbc03d55d302627b094ee91/psutil-7.2.1-cp37-abi3-win_arm64.whl", hash = "sha256:0d67c1822c355aa6f7314d92018fb4268a76668a536f133599b91edd48759442", size = 133836, upload-time = "2025-12-29T08:26:43.086Z" },]+[[package]]+name = "py-partiql-parser"+version = "0.6.3"+source = { registry = "https://pypi.org/simple" }+sdist = { url = "https://files.pythonhosted.org/packages/56/7a/a0f6bda783eb4df8e3dfd55973a1ac6d368a89178c300e1b5b91cd181e5e/py_partiql_parser-0.6.3.tar.gz", hash = "sha256:09cecf916ce6e3da2c050f0cb6106166de42c33d34a078ec2eb19377ea70389a", size = 17456, upload-time = "2025-10-18T13:56:13.441Z" }+wheels = [+ { url = "https://files.pythonhosted.org/packages/c9/33/a7cbfccc39056a5cf8126b7aab4c8bafbedd4f0ca68ae40ecb627a2d2cd3/py_partiql_parser-0.6.3-py2.py3-none-any.whl", hash = "sha256:deb0769c3346179d2f590dcbde556f708cdb929059fb654bad75f4cf6e07f582", size = 23752, upload-time = "2025-10-18T13:56:12.256Z" },+]+[[package]]name = "pyasn1"version = "0.6.1"@@ -3432,6 +3740,15 @@ wheels = [{ url = "https://files.pythonhosted.org/packages/18/59/81050abdc9e5b90ffc2c765738c5e40e9abd8e44864aaa737b600f16c562/pyclipper-1.4.0-pp311-pypy311_pp73-manylinux2014_x86_64.manylinux_2_17_x86_64.whl", hash = "sha256:98b2a40f98e1fc1b29e8a6094072e7e0c7dfe901e573bf6cfc6eb7ce84a7ae87", size = 126495, upload-time = "2025-12-01T13:15:33.743Z" },]+[[package]]+name = "pycparser"+version = "2.23"+source = { registry = "https://pypi.org/simple" }+sdist = { url = "https://files.pythonhosted.org/packages/fe/cf/d2d3b9f5699fb1e4615c8e32ff220203e43b248e1dfcc6736ad9057731ca/pycparser-2.23.tar.gz", hash = "sha256:78816d4f24add8f10a06d6f05b4d424ad9e96cfebf68a4ddc99c65c0720d00c2", size = 173734, upload-time = "2025-09-09T13:23:47.91Z" }+wheels = [+ { url = "https://files.pythonhosted.org/packages/a0/e3/59cd50310fc9b59512193629e1984c1f95e5c8ae6e5d8c69532ccc65a7fe/pycparser-2.23-py3-none-any.whl", hash = "sha256:e5c6e8d3fbad53479cab09ac03729e0a9faf2bee3db8208a550daf5af81a5934", size = 118140, upload-time = "2025-09-09T13:23:46.651Z" },+]+[[package]]name = "pydantic"version = "2.12.5"@@ -4162,6 +4479,20 @@ wheels = [{ url = "https://files.pythonhosted.org/packages/3b/5d/63d4ae3b9daea098d5d6f5da83984853c1bbacd5dc826764b249fe119d24/requests_oauthlib-2.0.0-py2.py3-none-any.whl", hash = "sha256:7dd8a5c40426b779b0868c404bdef9768deccf22749cde15852df527e6269b36", size = 24179, upload-time = "2024-03-22T20:32:28.055Z" },]+[[package]]+name = "responses"+version = "0.25.8"+source = { registry = "https://pypi.org/simple" }+dependencies = [+ { name = "pyyaml" },+ { name = "requests" },+ { name = "urllib3" },+]+sdist = { url = "https://files.pythonhosted.org/packages/0e/95/89c054ad70bfef6da605338b009b2e283485835351a9935c7bfbfaca7ffc/responses-0.25.8.tar.gz", hash = "sha256:9374d047a575c8f781b94454db5cab590b6029505f488d12899ddb10a4af1cf4", size = 79320, upload-time = "2025-08-08T19:01:46.709Z" }+wheels = [+ { url = "https://files.pythonhosted.org/packages/1c/4c/cc276ce57e572c102d9542d383b2cfd551276581dc60004cb94fe8774c11/responses-0.25.8-py3-none-any.whl", hash = "sha256:0c710af92def29c8352ceadff0c3fe340ace27cf5af1bbe46fb71275bcd2831c", size = 34769, upload-time = "2025-08-08T19:01:45.018Z" },+]+[[package]]name = "rich"version = "14.2.0"@@ -4463,9 +4794,12 @@ name = "scipy"version = "1.16.3"source = { registry = "https://pypi.org/simple" }resolution-markers = [- "python_full_version >= '3.12' and sys_platform == 'darwin'",- "python_full_version >= '3.12' and platform_machine == 'aarch64' and sys_platform == 'linux'",- "(python_full_version >= '3.12' and platform_machine != 'aarch64' and sys_platform == 'linux') or (python_full_version >= '3.12' and sys_platform != 'darwin' and sys_platform != 'linux')",+ "python_full_version >= '3.13' and sys_platform == 'darwin'",+ "python_full_version == '3.12.*' and sys_platform == 'darwin'",+ "python_full_version >= '3.13' and platform_machine == 'aarch64' and sys_platform == 'linux'",+ "python_full_version == '3.12.*' and platform_machine == 'aarch64' and sys_platform == 'linux'",+ "(python_full_version >= '3.13' and platform_machine != 'aarch64' and sys_platform == 'linux') or (python_full_version >= '3.13' and sys_platform != 'darwin' and sys_platform != 'linux')",+ "(python_full_version == '3.12.*' and platform_machine != 'aarch64' and sys_platform == 'linux') or (python_full_version == '3.12.*' and sys_platform != 'darwin' and sys_platform != 'linux')","python_full_version == '3.11.*' and sys_platform == 'darwin'","python_full_version == '3.11.*' and platform_machine == 'aarch64' and sys_platform == 'linux'","(python_full_version == '3.11.*' and platform_machine != 'aarch64' and sys_platform == 'linux') or (python_full_version == '3.11.*' and sys_platform != 'darwin' and sys_platform != 'linux')",@@ -5392,6 +5726,18 @@ wheels = [{ url = "https://files.pythonhosted.org/packages/fa/a8/5b41e0da817d64113292ab1f8247140aac61cbf6cfd085d6a0fa77f4984f/websockets-15.0.1-py3-none-any.whl", hash = "sha256:f7a866fbc1e97b5c617ee4116daaa09b722101d4a3c170c787450ba409f9736f", size = 169743, upload-time = "2025-03-05T20:03:39.41Z" },]+[[package]]+name = "werkzeug"+version = "3.1.4"+source = { registry = "https://pypi.org/simple" }+dependencies = [+ { name = "markupsafe" },+]+sdist = { url = "https://files.pythonhosted.org/packages/45/ea/b0f8eeb287f8df9066e56e831c7824ac6bab645dd6c7a8f4b2d767944f9b/werkzeug-3.1.4.tar.gz", hash = "sha256:cd3cd98b1b92dc3b7b3995038826c68097dcb16f9baa63abe35f20eafeb9fe5e", size = 864687, upload-time = "2025-11-29T02:15:22.841Z" }+wheels = [+ { url = "https://files.pythonhosted.org/packages/2f/f9/9e082990c2585c744734f85bec79b5dae5df9c974ffee58fe421652c8e91/werkzeug-3.1.4-py3-none-any.whl", hash = "sha256:2ad50fb9ed09cc3af22c54698351027ace879a0b60a3b5edf5730b2f7d876905", size = 224960, upload-time = "2025-11-29T02:15:21.13Z" },+]+[[package]]name = "wrapt"version = "1.17.3"@@ -5470,6 +5816,15 @@ wheels = [{ url = "https://files.pythonhosted.org/packages/3a/0c/3662f4a66880196a590b202f0db82d919dd2f89e99a27fadef91c4a33d41/xlsxwriter-3.2.9-py3-none-any.whl", hash = "sha256:9a5db42bc5dff014806c58a20b9eae7322a134abb6fce3c92c181bfb275ec5b3", size = 175315, upload-time = "2025-09-16T00:16:20.108Z" },]+[[package]]+name = "xmltodict"+version = "1.0.2"+source = { registry = "https://pypi.org/simple" }+sdist = { url = "https://files.pythonhosted.org/packages/6a/aa/917ceeed4dbb80d2f04dbd0c784b7ee7bba8ae5a54837ef0e5e062cd3cfb/xmltodict-1.0.2.tar.gz", hash = "sha256:54306780b7c2175a3967cad1db92f218207e5bc1aba697d887807c0fb68b7649", size = 25725, upload-time = "2025-09-17T21:59:26.459Z" }+wheels = [+ { url = "https://files.pythonhosted.org/packages/c0/20/69a0e6058bc5ea74892d089d64dfc3a62ba78917ec5e2cfa70f7c92ba3a5/xmltodict-1.0.2-py3-none-any.whl", hash = "sha256:62d0fddb0dcbc9f642745d8bbf4d81fd17d6dfaec5a15b5c1876300aad92af0d", size = 13893, upload-time = "2025-09-17T21:59:24.859Z" },+]+[[package]]name = "yarl"version = "1.22.0"