Skip to content

Build AI methods with your coding agent, then run them anywhere: as an MCP for chatbots, as a webapp for people, or via API for your software.

Feature/merge improve builder flow (#382)

Louis Choquel70e675caba4c
diff --git a/pipelex/builder/builder.plx b/pipelex/builder/builder.plx
index b2bece9a0..9b342ab4e 100644
--- a/pipelex/builder/builder.plx
+++ b/pipelex/builder/builder.plx
@@ -7,8 +7,6 @@ PlanDraft = "Natural-language pipeline plan text describing sequences, inputs, o
ConceptDrafts = "Textual draft of the concepts to create."
PipelexBundleSpec = "A Pipelex bundle spec."
ValidationResult = "Status (success or failure) and details of the validation failure if applicable."
-# PipeFailure = "Details of a single pipe failure during dry run."
-# DryRunResult = "A result of a dry run of a pipelex bundle spec."
BundleHeaderSpec = "A domain information object."
# ────────────────────────────────────────────────────────────────────────────────
@@ -65,6 +63,7 @@ Return a draft of a plan that narrates the pipeline as pseudo-steps (no code):
- For each pipe: state the pipe's description, inputs (by name using snake_case), and the output (by name using snake_case),
DO NOT indicate the inputs or output type. Just name them.
- Be aware of the steps where you will want structured outputs or inputs. Make sense of it but be concise.
+- Do not bother with planning a final step that gathers all the elements unless it's clear from the brief that the user wants the pipe to do that.
Available pipe controllers:
- PipeSequence: A pipe that executes a sequence of pipes: it needs to reference the pipes it will execute.
@@ -83,9 +82,8 @@ Available pipe operators:
- Then set output = "Article[]" to get a variable list, or output = "Article[5]" for exactly 5 items
- Examples: output = "Text[]" (multiple texts), output = "Image[3]" (exactly 3 images), output = "Employee[]" (list of employees)
- PipeImgGen: A pipe that uses an AI model to generate images.
- - Use bracket notation for multiple images: output = "Image[3]" generates exactly 3 images
- VERY IMPORTANT: IF YOU DECIDE TO CREATE A PipeImgGen, YOU ALSO HAVE TO CREATE A PIPELLM THAT WILL WRITE THE PROMPT, AND THAT NEEDS TO PRECEED THE PIPEIMGEN, based on the necessary elements.
- That means that in the MAIN pipeline, the prompt MUST NOT be considered as an input. It should be the output of a step that generates the prompt.
+ - VERY IMPORTANT: IF YOU DECIDE TO CREATE A PipeImgGen to generate an image, YOU ALSO HAVE TO CREATE A PIPELLM THAT WILL WRITE THE PROMPT
+ AND THAT NEEDS TO PRECEED THE PIPEIMGEN, based on the necessary elements, unless it's clear from the brief that the prompt should be part of the main pipe's original inputs.
- PipeExtract: A pipe that uses OCR technology to extract text from an image or a pdf.
- Always outputs a list of pages: output = "Page[]"
VERY IMPORTANT: THE INPUT OF THE PIPEEXTRACT MUST BE either an image or a pdf or a concept which refines one of them.
@@ -135,7 +133,7 @@ For instance:
- Concepts are always expressed as singular nouns, even if we're to use them as a list:
for instance, define the concept as "Article" not "Articles", "Employee" not "Employees".
If we need multiple items, we'll indicate it elsewhere so you don't bother with it here.
-- Provide a short description concise description for each concept
+- Provide a concise description for each concept
If the concept can be expressed as a text, image, pdf, number, or page:
- Name the concept, define it and just write "refines: Text", "refines: PDF", or "refines: Image" etc.
@@ -203,6 +201,7 @@ Define the contracts of the pipes to build:
- For each pipe: give a unique snake_case pipe_code, a type and description, specify inputs (one or more) and output (one)
- Add as much details as possible for the description.
- Be clear which is the main pipe of the pipeline, don't write "main" in its pipe_code, but make it clear in its description.
+- Do not bother with planning a final step that gathers all the elements unless it's clear from the brief that the user wants the pipe to do that.
Available pipe controllers:
- PipeSequence: A pipe that executes a sequence of pipes: it needs to reference the pipes it will execute.
@@ -236,7 +235,7 @@ Be smart about splitting the workflow into steps (sequence or parallel):
- But don't ask the LLM for many things which are unrelated, it would lose reliability.
- Apply the DRY principle: don't repeat yourself. if you have a task to apply several times, make it a dedicated pipe.
- If you're in a sequence and you are to apply that pipe to a previous output which is multiple, use batch_over/batch_as attributes in that step.
-
+- The output concept of a pipe sequence must always be the same as the output concept of the last pipe in the sequence.
You must never include nore than one batch step in the same pipe sequence.
Instead, you must create a pipe sequence specifically for the process to apply to each batched element
diff --git a/pipelex/builder/builder.py b/pipelex/builder/builder.py
index 09f82b133..7d4412a18 100644
--- a/pipelex/builder/builder.py
+++ b/pipelex/builder/builder.py
@@ -2,14 +2,13 @@ from typing import TYPE_CHECKING, cast
from pydantic import ValidationError
+from pipelex import pretty_print
from pipelex.builder.builder_errors import (
PipeBuilderError,
PipelexBundleError,
PipelexBundleUnexpectedError,
)
-from pipelex.builder.builder_validation import (
- validate_dry_run_bundle_blueprint,
-)
+from pipelex.builder.builder_validation import validate_dry_run_bundle_blueprint
from pipelex.builder.bundle_header_spec import BundleHeaderSpec
from pipelex.builder.bundle_spec import PipelexBundleSpec
from pipelex.builder.concept.concept_spec import ConceptSpec
@@ -30,6 +29,7 @@ from pipelex.core.interpreter import PipelexInterpreter
from pipelex.core.memory.working_memory import WorkingMemory
from pipelex.core.stuffs.list_content import ListContent
from pipelex.core.stuffs.structured_content import StructuredContent
+from pipelex.exceptions import StuffContentTypeError
from pipelex.system.registries.func_registry import pipe_func
from pipelex.tools.typing.pydantic_utils import format_pydantic_validation_error
@@ -63,13 +63,21 @@ async def assemble_pipelex_bundle_spec(working_memory: WorkingMemory) -> Pipelex
"""
# The working memory actually contains ConceptSpec objects (not ConceptSpecDraft)
# but they may have been deserialized incorrectly
- concept_specs = working_memory.get_stuff_as_list(
- name="concept_specs",
- item_type=ConceptSpec,
- )
-
+ try:
+ concept_specs = working_memory.get_stuff_as_list(
+ name="concept_specs",
+ item_type=ConceptSpec,
+ )
+ except StuffContentTypeError as exc:
+ msg = f"assemble_pipelex_bundle_spec: Failed to get concept specs: {exc}."
+ raise PipeBuilderError(message=msg, working_memory=working_memory) from exc
# pipe_specs: list[PipeSpecUnion] = cast("ListContent[PipeSpecUnion]", working_memory.get_stuff(name="pipe_specs").content).items
- pipe_specs_list: ListContent[StuffContent] = working_memory.get_stuff_as_list(name="pipe_specs", item_type=StructuredContent)
+ try:
+ pipe_specs_list: ListContent[StuffContent] = working_memory.get_stuff_as_list(name="pipe_specs", item_type=StructuredContent)
+ except StuffContentTypeError as exc:
+ msg = f"assemble_pipelex_bundle_spec: Failed to get pipe specs: {exc}."
+ raise PipeBuilderError(message=msg, working_memory=working_memory) from exc
+
pipe_specs_list_items: list[StuffContent] = pipe_specs_list.items
pipe_specs: list[PipeSpecUnion] = []
for pipe_spec_item in pipe_specs_list_items:
@@ -115,12 +123,6 @@ async def assemble_pipelex_bundle_spec(working_memory: WorkingMemory) -> Pipelex
)
-async def reconstruct_bundle_with_pipe_fixes_from_memory(working_memory: WorkingMemory) -> PipelexBundleSpec:
- pipelex_bundle_spec = working_memory.get_stuff_as(name="pipelex_bundle_spec", content_type=PipelexBundleSpec)
- fixed_pipes_list = cast("ListContent[PipeSpecUnion]", working_memory.get_stuff(name="fixed_pipes").content)
- return reconstruct_bundle_with_pipe_fixes(pipelex_bundle_spec=pipelex_bundle_spec, fixed_pipes=fixed_pipes_list.items)
-
-
def reconstruct_bundle_with_pipe_fixes(pipelex_bundle_spec: PipelexBundleSpec, fixed_pipes: list[PipeSpecUnion]) -> PipelexBundleSpec:
if not pipelex_bundle_spec.pipe:
msg = "No pipes section found in bundle spec"
@@ -178,6 +180,7 @@ async def load_and_validate_bundle(bundle_path: str) -> PipelexBundleBlueprint:
await validate_dry_run_bundle_blueprint(bundle_blueprint=bundle_blueprint)
except PipelexBundleError as exc:
msg = f"Bundle at '{bundle_path}' failed to validate"
+ pretty_print(exc, title="Validation Error Details")
raise PipelexBundleError(message=msg, pipe_failures=exc.pipe_failures) from exc
return bundle_blueprint
diff --git a/pipelex/builder/builder_errors.py b/pipelex/builder/builder_errors.py
index 5dd74dbf0..86c1b8b64 100644
--- a/pipelex/builder/builder_errors.py
+++ b/pipelex/builder/builder_errors.py
@@ -1,8 +1,5 @@
-"""Error classes and their structured content representations for the builder domain."""
+from typing_extensions import override
-from pydantic import Field
-
-from pipelex.builder.concept.concept_spec import ConceptSpec
from pipelex.builder.pipe.pipe_batch_spec import PipeBatchSpec
from pipelex.builder.pipe.pipe_compose_spec import PipeComposeSpec
from pipelex.builder.pipe.pipe_condition_spec import PipeConditionSpec
@@ -12,11 +9,19 @@ from pipelex.builder.pipe.pipe_img_gen_spec import PipeImgGenSpec
from pipelex.builder.pipe.pipe_llm_spec import PipeLLMSpec
from pipelex.builder.pipe.pipe_parallel_spec import PipeParallelSpec
from pipelex.builder.pipe.pipe_sequence_spec import PipeSequenceSpec
-from pipelex.core.stuffs.structured_content import StructuredContent
-from pipelex.exceptions import (
- PipelexException,
- StaticValidationErrorType,
+from pipelex.builder.validation_error_data import (
+ ConceptDefinitionErrorData,
+ ConceptFailure,
+ DomainFailure,
+ PipeDefinitionErrorData,
+ PipeFailure,
+ PipeInputErrorData,
+ PipelexBundleErrorData,
+ StaticValidationErrorData,
)
+from pipelex.core.memory.working_memory import WorkingMemory
+from pipelex.core.validation_errors import ValidationErrorDetailsProtocol
+from pipelex.exceptions import PipelexException
from pipelex.types import Self
# Type alias for pipe spec union
@@ -33,105 +38,10 @@ PipeSpecUnion = (
)
-# ============================================================================
-# BaseModel (StructuredContent) versions of error information
-# ============================================================================
-
-
-class StaticValidationErrorData(StructuredContent):
- """Structured data for StaticValidationError."""
-
- error_type: StaticValidationErrorType = Field(description="The type of static validation error")
- domain: str = Field(description="The domain where the error occurred")
- pipe_code: str | None = Field(None, description="The pipe code if applicable")
- variable_names: list[str] | None = Field(None, description="Variable names involved in the error")
- required_concept_codes: list[str] | None = Field(None, description="Required concept codes")
- provided_concept_code: str | None = Field(None, description="The provided concept code")
- file_path: str | None = Field(None, description="The file path where the error occurred")
- explanation: str | None = Field(None, description="Additional explanation of the error")
-
-
-class ConceptDefinitionErrorData(StructuredContent):
- """Structured data for ConceptDefinitionError."""
-
- message: str = Field(description="The error message")
- domain_code: str = Field(description="The domain code")
- concept_code: str = Field(description="The concept code")
- description: str = Field(description="Description of the concept")
- structure_class_python_code: str | None = Field(None, description="Python code for the structure class if available")
- source: str | None = Field(None, description="Source of the error")
-
-
-class PipeDefinitionErrorData(StructuredContent):
- """Structured data for PipeDefinitionError."""
-
- message: str = Field(description="The error message")
- domain_code: str | None = Field(None, description="The domain code")
- pipe_code: str | None = Field(None, description="The pipe code")
- description: str | None = Field(None, description="Description of the pipe")
- source: str | None = Field(None, description="Source of the error")
-
-
-class PipeInputErrorData(StructuredContent):
- """Structured data for PipeInputError."""
-
- message: str = Field(description="The error message")
- pipe_code: str | None = Field(None, description="The pipe code")
- variable_name: str | None = Field(None, description="The variable name")
- concept_code: str | None = Field(None, description="The concept code")
-
-
-class DomainFailure(StructuredContent):
- """Details of a single domain failure during dry run."""
-
- domain_code: str = Field(description="The code of the domain that failed")
- error_message: str = Field(description="The error message for this domain")
-
-
-class ConceptFailure(StructuredContent):
- """Details of a single concept failure during dry run."""
-
- concept_spec: ConceptSpec = Field(description="The failing concept spec with concept code")
- error_message: str = Field(description="The error message for this concept")
-
-
-class PipeFailure(StructuredContent):
- """Details of a single pipe failure during dry run."""
-
- pipe_spec: (
- PipeFuncSpec
- | PipeImgGenSpec
- | PipeComposeSpec
- | PipeLLMSpec
- | PipeExtractSpec
- | PipeBatchSpec
- | PipeConditionSpec
- | PipeParallelSpec
- | PipeSequenceSpec
- ) = Field(description="The failing pipe spec with pipe code")
- error_message: str = Field(description="The error message for this pipe")
-
-
-class PipelexBundleErrorData(StructuredContent):
- """Structured data for PipelexBundleError."""
-
- message: str = Field(description="The main error message")
- static_validation_error: StaticValidationErrorData | None = Field(None, description="Static validation error if present")
- domain_failures: list[DomainFailure] | None = Field(None, description="List of domain failures")
- pipe_input_errors: list[PipeInputErrorData] | None = Field(None, description="List of pipe input errors")
- pipe_failures: list[PipeFailure] | None = Field(None, description="List of pipe failures")
- concept_failures: list[ConceptFailure] | None = Field(None, description="List of concept failures")
- concept_definition_errors: list[ConceptDefinitionErrorData] | None = Field(None, description="List of concept definition errors")
- pipe_definition_errors: list[PipeDefinitionErrorData] | None = Field(None, description="List of pipe definition errors")
-
-
-# ============================================================================
-# Exception classes with as_structured_content() methods
-# ============================================================================
-
-
class PipeBuilderError(Exception):
- """Base exception for pipe builder errors."""
+ def __init__(self: Self, message: str, working_memory: WorkingMemory | None = None) -> None:
+ self.working_memory = working_memory
+ super().__init__(message)
class ConceptSpecError(PipelexException):
@@ -162,7 +72,7 @@ class ValidateDryRunError(Exception):
"""Raised when validating the dry run of a pipe."""
-class PipelexBundleError(PipelexException):
+class PipelexBundleError(PipelexException, ValidationErrorDetailsProtocol):
"""Main bundle error that aggregates multiple types of errors."""
def __init__(
@@ -185,6 +95,11 @@ class PipelexBundleError(PipelexException):
self.pipe_definition_errors = pipe_definition_errors
super().__init__(message)
+ @override
+ def get_concept_definition_errors(self: Self) -> list[ConceptDefinitionErrorData]:
+ """Get concept definition errors."""
+ return self.concept_definition_errors or []
+
def as_structured_content(self: Self) -> PipelexBundleErrorData:
"""Convert the error to structured content."""
return PipelexBundleErrorData(
diff --git a/pipelex/builder/builder_loop.py b/pipelex/builder/builder_loop.py
index a44998714..3ecaa7b7a 100644
--- a/pipelex/builder/builder_loop.py
+++ b/pipelex/builder/builder_loop.py
@@ -5,6 +5,7 @@ from pipelex.builder.builder import (
reconstruct_bundle_with_pipe_fixes,
)
from pipelex.builder.builder_errors import (
+ PipeBuilderError,
PipelexBundleError,
PipelexBundleNoFixForError,
PipelexBundleUnexpectedError,
@@ -12,7 +13,7 @@ from pipelex.builder.builder_errors import (
from pipelex.builder.builder_validation import validate_bundle_spec
from pipelex.client.protocol import PipelineInputs
from pipelex.core.pipes.pipe_blueprint import AllowedPipeCategories
-from pipelex.exceptions import StaticValidationErrorType
+from pipelex.exceptions import StaticValidationErrorType, WorkingMemoryStuffNotFoundError
from pipelex.hub import get_required_pipe
from pipelex.language.plx_factory import PlxFactory
from pipelex.pipeline.execute import execute_pipeline
@@ -34,7 +35,11 @@ class BuilderLoop:
)
pretty_print(pipe_output, title="Pipe Output")
- pipelex_bundle_spec = pipe_output.working_memory.get_stuff_as(name="pipelex_bundle_spec", content_type=PipelexBundleSpec)
+ try:
+ pipelex_bundle_spec = pipe_output.working_memory.get_stuff_as(name="pipelex_bundle_spec", content_type=PipelexBundleSpec)
+ except WorkingMemoryStuffNotFoundError as exc:
+ msg = f"Builder loop: Failed to get pipelex bundle spec: {exc}."
+ raise PipeBuilderError(message=msg, working_memory=pipe_output.working_memory) from exc
pretty_print(pipelex_bundle_spec, title="Pipelex Bundle Spec • 1st iteration")
plx_content = PlxFactory.make_plx_content(blueprint=pipelex_bundle_spec.to_blueprint())
diff --git a/pipelex/builder/builder_validation.py b/pipelex/builder/builder_validation.py
index 2b7405120..3ff9ef8a5 100644
--- a/pipelex/builder/builder_validation.py
+++ b/pipelex/builder/builder_validation.py
@@ -132,6 +132,7 @@ async def dry_run_bundle_blueprint(bundle_blueprint: PipelexBundleBlueprint) ->
concept_code=concept_def_error.concept_code,
description=concept_def_error.description,
structure_class_python_code=concept_def_error.structure_class_python_code,
+ structure_class_syntax_error_data=concept_def_error.structure_class_syntax_error_data,
source=concept_def_error.source,
)
raise PipelexBundleError(
diff --git a/pipelex/builder/concept/concept_spec.py b/pipelex/builder/concept/concept_spec.py
index c94e101c6..4704ec5b2 100644
--- a/pipelex/builder/concept/concept_spec.py
+++ b/pipelex/builder/concept/concept_spec.py
@@ -15,9 +15,7 @@ from pipelex.core.concepts.concept_blueprint import (
from pipelex.core.concepts.concept_native import NativeConceptCode
from pipelex.core.concepts.exceptions import ConceptCodeError, ConceptStringOrConceptCodeError
from pipelex.core.domains.domain_blueprint import DomainBlueprint
-from pipelex.core.memory.working_memory import WorkingMemory
from pipelex.core.stuffs.structured_content import StructuredContent
-from pipelex.system.registries.func_registry import pipe_func
from pipelex.tools.misc.string_utils import is_pascal_case, normalize_to_ascii, snake_to_pascal_case
from pipelex.types import Self, StrEnum
@@ -260,27 +258,3 @@ class ConceptSpec(StructuredContent):
converted_structure[field_name] = field_spec.to_blueprint()
return ConceptBlueprint(description=self.description, structure=converted_structure, refines=self.refines)
-
-
-@pipe_func()
-async def create_concept_spec(working_memory: WorkingMemory) -> ConceptSpec:
- concept_spec_draft = working_memory.get_stuff_as(name="concept_spec_draft", content_type=ConceptSpecDraft)
- concept_spec_structures_stuff = working_memory.get_stuff_as_list(name="concept_spec_structures", item_type=ConceptStructureSpec)
-
- structure_dict: dict[str, ConceptStructureSpec] = {}
- for structure_item in concept_spec_structures_stuff.items:
- structure_spec = ConceptStructureSpec(
- the_field_name=structure_item.the_field_name,
- description=structure_item.description,
- type=structure_item.type,
- required=structure_item.required,
- default_value=structure_item.default_value,
- )
- structure_dict[structure_item.the_field_name] = structure_spec
-
- return ConceptSpec(
- the_concept_code=concept_spec_draft.the_concept_code,
- description=concept_spec_draft.description,
- structure=structure_dict,
- refines=concept_spec_draft.refines,
- )
diff --git a/pipelex/builder/concept/concept_spec_factory.py b/pipelex/builder/concept/concept_spec_factory.py
new file mode 100644
index 000000000..0ff8c8e37
--- /dev/null
+++ b/pipelex/builder/concept/concept_spec_factory.py
@@ -0,0 +1,27 @@
+from pipelex.builder.concept.concept_spec import ConceptSpec, ConceptSpecDraft, ConceptStructureSpec
+from pipelex.core.memory.working_memory import WorkingMemory
+from pipelex.system.registries.func_registry import pipe_func
+
+
+@pipe_func()
+async def create_concept_spec(working_memory: WorkingMemory) -> ConceptSpec:
+ concept_spec_draft = working_memory.get_stuff_as(name="concept_spec_draft", content_type=ConceptSpecDraft)
+ concept_spec_structures_stuff = working_memory.get_stuff_as_list(name="concept_spec_structures", item_type=ConceptStructureSpec)
+
+ structure_dict: dict[str, ConceptStructureSpec] = {}
+ for structure_item in concept_spec_structures_stuff.items:
+ structure_spec = ConceptStructureSpec(
+ the_field_name=structure_item.the_field_name,
+ description=structure_item.description,
+ type=structure_item.type,
+ required=structure_item.required,
+ default_value=structure_item.default_value,
+ )
+ structure_dict[structure_item.the_field_name] = structure_spec
+
+ return ConceptSpec(
+ the_concept_code=concept_spec_draft.the_concept_code,
+ description=concept_spec_draft.description,
+ structure=structure_dict,
+ refines=concept_spec_draft.refines,
+ )
diff --git a/pipelex/builder/pipe/pipe_design.plx b/pipelex/builder/pipe/pipe_design.plx
index 103740422..03bb869f7 100644
--- a/pipelex/builder/pipe/pipe_design.plx
+++ b/pipelex/builder/pipe/pipe_design.plx
@@ -58,6 +58,7 @@ Here are the concepts you can use for inputs/outputs:
@concept_specs
- If you are to apply a pipe step to a previous output which is multiple, use batch_over/batch_as attributes in that step.
+- The output concept of a pipe sequence must always be the same as the output concept of the last pipe in the sequence.
"""
[pipe.detail_pipe_parallel]
diff --git a/pipelex/builder/pipe/pipe_extract_spec.py b/pipelex/builder/pipe/pipe_extract_spec.py
index a8919da2e..4e3e36f31 100644
--- a/pipelex/builder/pipe/pipe_extract_spec.py
+++ b/pipelex/builder/pipe/pipe_extract_spec.py
@@ -5,7 +5,7 @@ from pydantic.json_schema import SkipJsonSchema
from typing_extensions import override
from pipelex.builder.pipe.pipe_spec import PipeSpec
-from pipelex.exceptions import PipeDefinitionError
+from pipelex.builder.pipe.pipe_spec_exceptions import PipeExtractSpecError
from pipelex.pipe_operators.extract.pipe_extract_blueprint import PipeExtractBlueprint
from pipelex.types import StrEnum
@@ -53,10 +53,10 @@ class PipeExtractSpec(PipeSpec):
def validate_extract_inputs(cls, inputs_value: dict[str, str] | None) -> dict[str, str] | None:
if inputs_value is None:
msg = "PipeExtract must have exactly one input which must be either `Image` or `PDF`."
- raise PipeDefinitionError(msg)
+ raise PipeExtractSpecError(msg)
if len(inputs_value) != 1:
msg = "PipeExtract must have exactly one input which must be either `Image` or `PDF`."
- raise PipeDefinitionError(msg)
+ raise PipeExtractSpecError(msg)
return inputs_value
@override
diff --git a/pipelex/builder/pipe/pipe_parallel_spec.py b/pipelex/builder/pipe/pipe_parallel_spec.py
index 629651ff3..a70d0e632 100644
--- a/pipelex/builder/pipe/pipe_parallel_spec.py
+++ b/pipelex/builder/pipe/pipe_parallel_spec.py
@@ -5,8 +5,8 @@ from typing_extensions import override
from pipelex.builder.concept.concept_spec import ConceptSpec
from pipelex.builder.pipe.pipe_spec import PipeSpec
+from pipelex.builder.pipe.pipe_spec_exceptions import PipeParallelSpecError
from pipelex.builder.pipe.sub_pipe_spec import SubPipeSpec
-from pipelex.exceptions import PipeDefinitionError
from pipelex.pipe_controllers.parallel.pipe_parallel_blueprint import PipeParallelBlueprint
from pipelex.types import Self
@@ -46,7 +46,7 @@ class PipeParallelSpec(PipeSpec):
"PipeParallel requires either add_each_output to be True or combined_output to be set, "
"or both, otherwise the pipe won't output anything"
)
- raise PipeDefinitionError(
+ raise PipeParallelSpecError(
message=msg,
pipe_code=self.pipe_code,
description=self.description,
diff --git a/pipelex/builder/pipe/pipe_signature.py b/pipelex/builder/pipe/pipe_signature.py
index 39dfecc6f..45c080d0e 100644
--- a/pipelex/builder/pipe/pipe_signature.py
+++ b/pipelex/builder/pipe/pipe_signature.py
@@ -29,12 +29,19 @@ class PipeSignature(StructuredContent):
pipe_category: SkipJsonSchema[AllowedPipeCategories] = Field(description="Pipe category set according to its type.")
description: str = Field(description="What the pipe does")
inputs: dict[str, str] = Field(
- description="Pipe inputs: keys are the input variable_names in snake_case, values are the ConceptCodes in PascalCase."
+ description=(
+ "Input specifications mapping variable names to concept codes. "
+ "Keys: input variable names in snake_case. "
+ "Values: ConceptCodes in PascalCase. Don't use multiplicity brackets. "
+ )
)
- result: str = Field(
- description="variable_name for the result of the pipe. Must be snake_case. It could be referenced as input in a following pipe."
+ result: str = Field(description="Variable name for the pipe's result in snake_case. This name can be referenced as input in subsequent pipes.")
+ output: str = Field(
+ description=(
+ "Output concept code in PascalCase with optional multiplicity brackets. "
+ "Examples: 'Text' (single text), 'Article[]' (list of articles), 'Image[5]' (exactly 5 images)."
+ )
)
- output: str = Field(description="The output ConceptCode in PascalCase, possibly with multiplicity brackets")
pipe_dependencies: list[str] = Field(description="List of pipe codes that this pipe depends on. This is for the PipeControllers")
@model_validator(mode="before")
diff --git a/pipelex/builder/pipe/pipe_spec_exceptions.py b/pipelex/builder/pipe/pipe_spec_exceptions.py
new file mode 100644
index 000000000..177bb5d65
--- /dev/null
+++ b/pipelex/builder/pipe/pipe_spec_exceptions.py
@@ -0,0 +1,33 @@
+from pipelex.core.pipe_errors import PipeDefinitionError
+
+
+class PipeSpecException(PipeDefinitionError):
+ pass
+
+
+class PipeExtractSpecError(PipeSpecException):
+ pass
+
+
+class PipeParallelSpecError(PipeSpecException):
+ pass
+
+
+class PipeSequenceSpecError(PipeSpecException):
+ pass
+
+
+class PipeFuncSpecError(PipeSpecException):
+ pass
+
+
+class PipeImgGenSpecError(PipeSpecException):
+ pass
+
+
+class PipeComposeSpecError(PipeSpecException):
+ pass
+
+
+class PipeLLMSpecError(PipeSpecException):
+ pass
diff --git a/pipelex/builder/pipe/sub_pipe_spec.py b/pipelex/builder/pipe/sub_pipe_spec.py
index 1fde3b673..209633623 100644
--- a/pipelex/builder/pipe/sub_pipe_spec.py
+++ b/pipelex/builder/pipe/sub_pipe_spec.py
@@ -1,7 +1,7 @@
from pydantic import ConfigDict, model_validator
+from pipelex.core.pipe_errors import PipeDefinitionError
from pipelex.core.stuffs.structured_content import StructuredContent
-from pipelex.exceptions import PipeDefinitionError
from pipelex.pipe_controllers.sub_pipe_blueprint import SubPipeBlueprint
from pipelex.types import Self
diff --git a/pipelex/builder/validation_error_data.py b/pipelex/builder/validation_error_data.py
new file mode 100644
index 000000000..01d17d2d8
--- /dev/null
+++ b/pipelex/builder/validation_error_data.py
@@ -0,0 +1,142 @@
+from pydantic import Field
+
+from pipelex.builder.concept.concept_spec import ConceptSpec
+from pipelex.builder.pipe.pipe_batch_spec import PipeBatchSpec
+from pipelex.builder.pipe.pipe_compose_spec import PipeComposeSpec
+from pipelex.builder.pipe.pipe_condition_spec import PipeConditionSpec
+from pipelex.builder.pipe.pipe_extract_spec import PipeExtractSpec
+from pipelex.builder.pipe.pipe_func_spec import PipeFuncSpec
+from pipelex.builder.pipe.pipe_img_gen_spec import PipeImgGenSpec
+from pipelex.builder.pipe.pipe_llm_spec import PipeLLMSpec
+from pipelex.builder.pipe.pipe_parallel_spec import PipeParallelSpec
+from pipelex.builder.pipe.pipe_sequence_spec import PipeSequenceSpec
+from pipelex.core.stuffs.structured_content import StructuredContent
+from pipelex.types import StrEnum
+
+
+class StaticValidationErrorType(StrEnum):
+ MISSING_INPUT_VARIABLE = "missing_input_variable"
+ EXTRANEOUS_INPUT_VARIABLE = "extraneous_input_variable"
+ INADEQUATE_INPUT_CONCEPT = "inadequate_input_concept"
+ TOO_MANY_CANDIDATE_INPUTS = "too_many_candidate_inputs"
+
+
+class SyntaxErrorData(StructuredContent):
+ message: str
+ lineno: int | None = None
+ offset: int | None = None
+ text: str | None = None
+ end_lineno: int | None = None
+ end_offset: int | None = None
+
+ @classmethod
+ def from_syntax_error(cls, syntax_error: SyntaxError) -> "SyntaxErrorData":
+ return cls(
+ message=syntax_error.msg,
+ lineno=syntax_error.lineno,
+ offset=syntax_error.offset,
+ text=syntax_error.text,
+ end_lineno=syntax_error.end_lineno,
+ end_offset=syntax_error.end_offset,
+ )
+
+
+# ============================================================================
+# BaseModel (StructuredContent) versions of error information
+# ============================================================================
+
+
+class StaticValidationErrorData(StructuredContent):
+ """Structured data for StaticValidationError."""
+
+ error_type: StaticValidationErrorType = Field(description="The type of static validation error")
+ domain: str = Field(description="The domain where the error occurred")
+ pipe_code: str | None = Field(None, description="The pipe code if applicable")
+ variable_names: list[str] | None = Field(None, description="Variable names involved in the error")
+ required_concept_codes: list[str] | None = Field(None, description="Required concept codes")
+ provided_concept_code: str | None = Field(None, description="The provided concept code")
+ file_path: str | None = Field(None, description="The file path where the error occurred")
+ explanation: str | None = Field(None, description="Additional explanation of the error")
+
+
+class ConceptDefinitionErrorData(StructuredContent):
+ """Structured data for ConceptDefinitionError."""
+
+ message: str = Field(description="The error message")
+ domain_code: str = Field(description="The domain code")
+ concept_code: str = Field(description="The concept code")
+ description: str = Field(description="Description of the concept")
+ structure_class_python_code: str | None = Field(None, description="Python code for the structure class if available")
+ structure_class_syntax_error_data: SyntaxErrorData | None = Field(None, description="Syntax error data for the structure class if available")
+ source: str | None = Field(None, description="Source of the error")
+
+
+class PipeDefinitionErrorData(StructuredContent):
+ """Structured data for PipeDefinitionError."""
+
+ message: str = Field(description="The error message")
+ domain_code: str | None = Field(None, description="The domain code")
+ pipe_code: str | None = Field(None, description="The pipe code")
+ description: str | None = Field(None, description="Description of the pipe")
+ source: str | None = Field(None, description="Source of the error")
+
+
+class PipeInputErrorData(StructuredContent):
+ """Structured data for PipeInputError."""
+
+ message: str = Field(description="The error message")
+ pipe_code: str | None = Field(None, description="The pipe code")
+ variable_name: str | None = Field(None, description="The variable name")
+ concept_code: str | None = Field(None, description="The concept code")
+
+
+class DomainFailure(StructuredContent):
+ """Details of a single domain failure during dry run."""
+
+ domain_code: str = Field(description="The code of the domain that failed")
+ error_message: str = Field(description="The error message for this domain")
+
+
+class ConceptFailure(StructuredContent):
+ """Details of a single concept failure during dry run."""
+
+ concept_spec: ConceptSpec = Field(description="The failing concept spec with concept code")
+ error_message: str = Field(description="The error message for this concept")
+
+
+class PipeFailure(StructuredContent):
+ """Details of a single pipe failure during dry run."""
+
+ pipe_spec: (
+ PipeFuncSpec
+ | PipeImgGenSpec
+ | PipeComposeSpec
+ | PipeLLMSpec
+ | PipeExtractSpec
+ | PipeBatchSpec
+ | PipeConditionSpec
+ | PipeParallelSpec
+ | PipeSequenceSpec
+ ) = Field(description="The failing pipe spec with pipe code")
+ error_message: str = Field(description="The error message for this pipe")
+
+
+class LibraryLoadingErrorData(StructuredContent):
+ """Structured data for LibraryLoadingError."""
+
+ message: str = Field(description="The main error message")
+ concept_definition_errors: list[ConceptDefinitionErrorData] | None = Field(None, description="List of concept definition errors")
+ pipe_definition_errors: list[PipeDefinitionErrorData] | None = Field(None, description="List of pipe definition errors")
+
+
+class PipelexBundleErrorData(StructuredContent):
+ """Structured data for PipelexBundleError."""
+
+ message: str = Field(description="The main error message")
+ static_validation_error: StaticValidationErrorData | None = Field(None, description="Static validation error if present")
+ domain_failures: list[DomainFailure] | None = Field(None, description="List of domain failures")
+ pipe_input_errors: list[PipeInputErrorData] | None = Field(None, description="List of pipe input errors")
+ pipe_failures: list[PipeFailure] | None = Field(None, description="List of pipe failures")
+ concept_failures: list[ConceptFailure] | None = Field(None, description="List of concept failures")
+ concept_definition_errors: list[ConceptDefinitionErrorData] | None = Field(None, description="List of concept definition errors")
+ pipe_definition_errors: list[PipeDefinitionErrorData] | None = Field(None, description="List of pipe definition errors")
diff --git a/pipelex/cli/commands/build_cmd.py b/pipelex/cli/commands/build_cmd.py
index 12720b340..cf712f777 100644
--- a/pipelex/cli/commands/build_cmd.py
+++ b/pipelex/cli/commands/build_cmd.py
@@ -1,13 +1,13 @@
import asyncio
import time
-from typing import Annotated
+from typing import TYPE_CHECKING, Annotated
import click
import typer
from pipelex import pretty_print
from pipelex.builder.builder import PipelexBundleSpec, load_and_validate_bundle
-from pipelex.builder.builder_errors import PipelexBundleError
+from pipelex.builder.builder_errors import PipeBuilderError, PipelexBundleError
from pipelex.builder.builder_loop import BuilderLoop
from pipelex.builder.runner_code import generate_runner_code
from pipelex.exceptions import PipeInputError
@@ -16,7 +16,10 @@ from pipelex.language.plx_factory import PlxFactory
from pipelex.pipelex import Pipelex
from pipelex.pipeline.execute import execute_pipeline
from pipelex.tools.misc.file_utils import ensure_directory_for_file_path, get_incremental_file_path, save_text_to_path
-from pipelex.tools.misc.json_utils import save_as_json_to_path
+from pipelex.tools.misc.json_utils import load_json_dict_from_path, save_as_json_to_path
+
+if TYPE_CHECKING:
+ from pipelex.client.protocol import PipelineInputs
build_app = typer.Typer(help="Build working pipelines from natural language requirements", no_args_is_help=True)
@@ -37,7 +40,7 @@ Other ideas:
pipelex build pipe "Take a photo as input, and render the opposite of the photo, don't structure anything, use only text content, be super concise"
pipelex build pipe "Take a photo as input, and render the opposite of the photo"
pipelex build pipe "Given an RDFP PDF, build a compliance matrix"
-pipelex build pipe "Given an theme, write a Haiku"
+pipelex build pipe "Given a theme, write a Haiku"
"""
@@ -47,6 +50,10 @@ def build_pipe_cmd(
str,
typer.Argument(help="Prompt describing what the pipeline should do"),
],
+ builder_pipe: Annotated[
+ str,
+ typer.Option("--builder-pipe", help="Builder pipe to use for generating the pipeline"),
+ ] = "pipe_builder",
output_path: Annotated[
str,
typer.Option("--output", "-o", help="Path to save the generated PLX file"),
@@ -76,7 +83,23 @@ def build_pipe_cmd(
typer.secho("\n⚠️ Pipeline not saved to file (--no-output specified)", fg=typer.colors.YELLOW)
return
- pipelex_bundle_spec = await builder_loop.build_and_fix(pipe_code="pipe_builder", input_memory={"brief": prompt})
+ try:
+ pipelex_bundle_spec = await builder_loop.build_and_fix(pipe_code=builder_pipe, input_memory={"brief": prompt})
+ except PipeBuilderError as exc:
+ msg = f"Builder loop: Failed to execute pipeline: {exc}."
+ if exc.working_memory:
+ failure_memory_path = get_incremental_file_path(
+ base_path="results",
+ base_name="failure_memory",
+ extension="json",
+ )
+ save_as_json_to_path(object_to_save=exc.working_memory.smart_dump(), path=failure_memory_path)
+ typer.secho(f"❌ {msg}", fg=typer.colors.RED)
+ typer.secho(f"❌ Failure memory saved to: {failure_memory_path}", fg=typer.colors.RED)
+ else:
+ typer.secho(f"❌ {msg}", fg=typer.colors.RED)
+ typer.secho("❌ No failure memory available", fg=typer.colors.RED)
+ raise typer.Exit(1) from exc
plx_content = PlxFactory.make_plx_content(blueprint=pipelex_bundle_spec.to_blueprint())
save_text_to_path(text=plx_content, path=output_path)
typer.secho(f"\n✅ Pipeline saved to: {output_path}", fg=typer.colors.GREEN)
@@ -236,6 +259,10 @@ def build_one_shot_cmd(
str,
typer.Argument(help="Brief description of what the pipeline should do"),
],
+ builder_pipe: Annotated[
+ str,
+ typer.Option("--builder-pipe", help="Builder pipe to use for generating the pipeline"),
+ ] = "pipe_builder",
output_path: Annotated[
str,
typer.Option("--output", "-o", help="Path to save the generated PLX file"),
@@ -260,7 +287,7 @@ def build_one_shot_cmd(
ensure_directory_for_file_path(file_path=output_path)
pipe_output = await execute_pipeline(
- pipe_code="pipe_builder",
+ pipe_code=builder_pipe,
inputs={"brief": brief},
)
pretty_print(pipe_output, title="Pipe Output")
@@ -287,14 +314,26 @@ def build_one_shot_cmd(
"partial-pipe", help="Developer utility for contributors: deliver a partial pipeline specification (not an actual bundle) and save it as JSON"
)
def build_partial_cmd(
- brief: Annotated[
+ inputs: Annotated[
str,
- typer.Argument(help="Brief description of what the pipeline should do"),
+ typer.Argument(help="Inline brief or path to JSON file with input_memory"),
],
- output_path: Annotated[
+ builder_pipe: Annotated[
+ str,
+ typer.Option("--builder-pipe", help="Builder pipe to use for generating the pipeline"),
+ ] = "pipe_builder",
+ output_dir_path: Annotated[
str,
typer.Option("--output", "-o", help="Path to save the generated PLX file"),
- ] = "./results/generated_pipeline.plx",
+ ] = "./results",
+ output_base_name: Annotated[
+ str,
+ typer.Option("--output-file-name", "-b", help="Name of the generated JSON file"),
+ ] = "partial_pipe",
+ extension: Annotated[
+ str,
+ typer.Option("--extension", "-e", help="Extension of the generated file"),
+ ] = "json",
no_output: Annotated[
bool,
typer.Option("--no-output", help="Skip saving the pipeline to file"),
@@ -306,25 +345,47 @@ def build_partial_cmd(
typer.echo("")
async def run_pipeline():
+ output_path: str | None = None
if no_output:
typer.secho("\n⚠️ Pipeline will not be saved to file (--no-output specified)", fg=typer.colors.YELLOW)
- elif not output_path:
- typer.secho("\n🛑 Cannot save a pipeline to an empty file name", fg=typer.colors.RED)
- raise typer.Exit(1)
else:
+ output_path = get_incremental_file_path(
+ base_path=output_dir_path,
+ base_name=output_base_name,
+ extension=extension,
+ )
ensure_directory_for_file_path(file_path=output_path)
+ input_memory: PipelineInputs | None = None
+ if inputs.endswith(".json"):
+ input_memory = load_json_dict_from_path(inputs)
+ else:
+ input_memory = {"brief": inputs}
pipe_output = await execute_pipeline(
- pipe_code="pipe_builder",
- inputs={"brief": brief},
+ pipe_code=builder_pipe,
+ inputs=input_memory,
)
# Save to file unless explicitly disabled with --no-output
- if no_output:
+ if output_path:
+ match extension:
+ case "md":
+ markdown_output = pipe_output.main_stuff.content.rendered_markdown()
+ save_text_to_path(text=markdown_output, path=output_path)
+ case "txt":
+ text_output = pipe_output.main_stuff.content.rendered_plain()
+ save_text_to_path(text=text_output, path=output_path)
+ case "html":
+ html_output = pipe_output.main_stuff.content.rendered_html()
+ save_text_to_path(text=html_output, path=output_path)
+ case "json":
+ json_output = pipe_output.main_stuff.content.smart_dump()
+ save_as_json_to_path(object_to_save=json_output, path=output_path)
+ case _:
+ json_output = pipe_output.main_stuff.content.smart_dump()
+ save_as_json_to_path(object_to_save=json_output, path=output_path)
+ typer.secho(f"\n✅ Pipeline saved to: {output_path}", fg=typer.colors.GREEN)
+ else:
typer.secho("\n⚠️ Pipeline not saved to file (--no-output specified)", fg=typer.colors.YELLOW)
- return
- json_output = pipe_output.main_stuff.content.smart_dump()
- save_as_json_to_path(object_to_save=json_output, path=output_path)
- typer.secho(f"\n✅ Pipeline saved to: {output_path}", fg=typer.colors.GREEN)
start_time = time.time()
asyncio.run(run_pipeline())
diff --git a/pipelex/cli/commands/validate_cmd.py b/pipelex/cli/commands/validate_cmd.py
index a786113c2..472277dcf 100644
--- a/pipelex/cli/commands/validate_cmd.py
+++ b/pipelex/cli/commands/validate_cmd.py
@@ -1,19 +1,26 @@
from __future__ import annotations
import asyncio
-from typing import Annotated
+from typing import TYPE_CHECKING, Annotated
import click
import typer
+from rich.console import Console
+from rich.syntax import Syntax
-from pipelex import log
-from pipelex.builder.builder import load_and_validate_bundle
+from pipelex import log, pretty_print
from pipelex.builder.builder_errors import PipelexBundleError
-from pipelex.exceptions import PipeInputError
+from pipelex.builder.builder_validation import validate_dry_run_bundle_blueprint
+from pipelex.core.interpreter import PipelexInterpreter
+from pipelex.exceptions import LibraryLoadingError, PipeInputError
from pipelex.hub import get_pipes, get_required_pipe
from pipelex.pipe_run.dry_run import dry_run_pipe, dry_run_pipes
from pipelex.pipelex import Pipelex
+if TYPE_CHECKING:
+ from pipelex.core.validation_errors import ValidationErrorDetailsProtocol
+console = Console()
+
def do_validate_all_libraries_and_dry_run() -> None:
"""Validate libraries and dry-run all pipes."""
@@ -97,24 +104,33 @@ def validate_cmd(
async def validate_pipe(pipe_code: str | None = None, bundle_path: str | None = None):
# Initialize Pipelex
- pipelex_instance = Pipelex.make()
+ try:
+ pipelex_instance = Pipelex.make()
+ except LibraryLoadingError as library_loading_error:
+ typer.secho(f"Failed to validate: {library_loading_error}", fg=typer.colors.RED, err=True)
+ present_validation_error(details_provider=library_loading_error)
+ raise typer.Exit(1) from library_loading_error
if bundle_path:
# When validating a bundle, load_pipe_from_bundle validates ALL pipes in the bundle
try:
- await load_and_validate_bundle(bundle_path)
+ bundle_blueprint = PipelexInterpreter.load_bundle_blueprint(bundle_path=bundle_path)
+ await validate_dry_run_bundle_blueprint(bundle_blueprint=bundle_blueprint)
if not pipe_code:
typer.secho(f"✅ Successfully validated all pipes in bundle '{bundle_path}'", fg=typer.colors.GREEN)
else:
typer.secho(f"✅ Successfully validated all pipes in bundle '{bundle_path}' (including '{pipe_code}')", fg=typer.colors.GREEN)
except FileNotFoundError as exc:
typer.secho(f"Failed to load bundle '{bundle_path}': {exc}", fg=typer.colors.RED, err=True)
+ console.print(exc)
raise typer.Exit(1) from exc
- except PipelexBundleError as exc:
- typer.secho(f"Failed to validate bundle '{bundle_path}': {exc}", fg=typer.colors.RED, err=True)
- raise typer.Exit(1) from exc
+ except PipelexBundleError as bundle_error:
+ typer.secho(f"\n❌ Failed to validate bundle '{bundle_path}':", fg=typer.colors.RED, err=True)
+ present_validation_error(details_provider=bundle_error)
+ raise typer.Exit(1) from bundle_error
except PipeInputError as exc:
- typer.secho(f"Failed to validate bundle '{bundle_path}': {exc}", fg=typer.colors.RED, err=True)
+ typer.secho(f"\n❌ Failed to validate bundle '{bundle_path}': {exc}", fg=typer.colors.RED, err=True)
+ console.print(exc)
raise typer.Exit(1) from exc
elif pipe_code:
# Validate a single pipe by code
@@ -130,3 +146,42 @@ def validate_cmd(
raise typer.Exit(1)
asyncio.run(validate_pipe(pipe_code=pipe_code, bundle_path=bundle_path))
+
+
+def present_validation_error(details_provider: ValidationErrorDetailsProtocol):
+ console.print(details_provider)
+ concept_definition_errors = details_provider.get_concept_definition_errors()
+ if not concept_definition_errors:
+ return
+ for concept_definition_error in concept_definition_errors:
+ syntax_error_data = concept_definition_error.structure_class_syntax_error_data
+ if not syntax_error_data:
+ continue
+ message = concept_definition_error.message
+ code = concept_definition_error.structure_class_python_code or ""
+ highlight_lines: set[int] | None = None
+ if syntax_error_data.lineno:
+ highlight_lines = {syntax_error_data.lineno}
+ syntax = Syntax(
+ code=code,
+ lexer="python",
+ line_numbers=True,
+ word_wrap=False,
+ # theme="monokai", # pick any theme rich knows; omit to use default
+ theme="ansi_dark", # pick any theme rich knows; omit to use default
+ line_range=None,
+ highlight_lines=highlight_lines,
+ )
+ pretty_range = ""
+ if syntax_error_data.lineno and syntax_error_data.end_lineno:
+ pretty_range = f"lines {syntax_error_data.lineno} to {syntax_error_data.end_lineno}"
+ elif syntax_error_data.lineno:
+ pretty_range = f"line {syntax_error_data.lineno}"
+ if syntax_error_data.offset and syntax_error_data.end_offset:
+ pretty_range += f", column {syntax_error_data.offset} to {syntax_error_data.end_offset}"
+ elif syntax_error_data.offset:
+ pretty_range += f", column {syntax_error_data.offset}"
+ console.print(message)
+ if pretty_range:
+ pretty_print(f"Generated code error at {pretty_range}")
+ console.print(syntax)
diff --git a/pipelex/core/bundles/pipe_sorter.py b/pipelex/core/bundles/pipe_sorter.py
index 594ed3372..6691be067 100644
--- a/pipelex/core/bundles/pipe_sorter.py
+++ b/pipelex/core/bundles/pipe_sorter.py
@@ -1,7 +1,7 @@
"""Topological sorting utilities for pipe dependencies."""
from pipelex.core.bundles.pipelex_bundle_blueprint import PipeBlueprintUnion
-from pipelex.exceptions import PipeDefinitionError
+from pipelex.core.pipe_errors import PipeDefinitionError
def sort_pipes_by_dependencies(
diff --git a/pipelex/core/concepts/concept_factory.py b/pipelex/core/concepts/concept_factory.py
index eb08432e6..4f402a594 100644
--- a/pipelex/core/concepts/concept_factory.py
+++ b/pipelex/core/concepts/concept_factory.py
@@ -261,6 +261,7 @@ class ConceptFactory:
concept_code=concept_code,
description=blueprint.description,
structure_class_python_code=exc.structure_class_python_code,
+ structure_class_syntax_error_data=exc.syntax_error_data,
source=blueprint.source,
) from exc
diff --git a/pipelex/core/concepts/structure_generator.py b/pipelex/core/concepts/structure_generator.py
index 5d5aace3b..bcec78202 100644
--- a/pipelex/core/concepts/structure_generator.py
+++ b/pipelex/core/concepts/structure_generator.py
@@ -5,6 +5,7 @@ from typing import Any, Literal, Optional
from pydantic import Field
+from pipelex.builder.validation_error_data import SyntaxErrorData
from pipelex.core.concepts.concept_blueprint import ConceptStructureBlueprint, ConceptStructureBlueprintFieldType
from pipelex.core.stuffs.structured_content import StructuredContent
from pipelex.exceptions import ConceptStructureGeneratorError, PipelexException
@@ -54,8 +55,14 @@ class StructureGenerator:
the_class = self.validate_generated_code(
python_code=generated_code, expected_class_name=class_name, required_base_class=StructuredContent
)
- except (ConceptStructureValidationError, SyntaxError, ValueError, ImportError, Exception) as exc:
- msg = f"Error validating generated code: {exc}"
+ except SyntaxError as syntax_error:
+ msg = f"Error validating generated code: {syntax_error}"
+ syntax_error_data = SyntaxErrorData.from_syntax_error(syntax_error)
+ raise ConceptStructureGeneratorError(
+ msg, structure_class_python_code=generated_code, syntax_error_data=syntax_error_data
+ ) from syntax_error
+ except (ConceptStructureValidationError, ValueError, ImportError, Exception) as exc:
+ msg = f"Error validating generated code: {exc}\nGenerated code:\n```python\n{generated_code}\n```"
raise ConceptStructureGeneratorError(msg, structure_class_python_code=generated_code) from exc
return generated_code, the_class
@@ -83,10 +90,42 @@ class StructureGenerator:
# Private methods
############################################################
+ def _escape_string_for_python(self, value: str) -> str:
+ """Escape a string value for safe inclusion in generated Python code.
+
+ This method ensures that strings containing special characters like quotes,
+ backslashes, newlines, etc. are properly escaped when inserted into generated
+ Python source code.
+
+ Args:
+ value: The string to escape
+
+ Returns:
+ A properly escaped string representation suitable for Python code with double quotes
+ """
+ # Use repr() which handles all escaping correctly (backslashes, newlines, quotes, etc.)
+ # repr() returns a string with surrounding quotes included
+ escaped = repr(value)
+
+ # If repr used single quotes, convert to double quotes
+ # This is for consistency with existing code expectations
+ if escaped.startswith("'") and escaped.endswith("'") and not escaped.startswith("'''"):
+ # Single-line string with single quotes
+ # Remove the single quotes and re-add as double quotes
+ inner = escaped[1:-1]
+ # Unescape single quotes that repr escaped (since we're switching to double quotes)
+ inner = inner.replace("\\'", "'")
+ # Escape any double quotes
+ inner = inner.replace('"', '\\"')
+ return f'"{inner}"'
+
+ # If repr already used double quotes (because the string contains single quotes), return as is
+ return escaped
+
def _format_default_value(self, value: Any) -> str:
"""Format default value for Python code, ensuring strings use double quotes."""
if isinstance(value, str):
- return f'"{value}"'
+ return self._escape_string_for_python(value)
return repr(value)
def _generate_class_source_code_from_blueprint(self, class_name: str, structure_blueprint: dict[str, ConceptStructureBlueprint]) -> str:
@@ -140,7 +179,7 @@ class StructureGenerator:
python_type = f"Optional[{python_type}]"
# Generate Field parameters
- field_params = [f'description="{field_blueprint.description}"']
+ field_params = [f"description={self._escape_string_for_python(field_blueprint.description)}"]
if field_blueprint.required:
if field_blueprint.default_value is not None:
@@ -244,7 +283,7 @@ class StructureGenerator:
python_type = f"Optional[{python_type}]"
# Generate Field parameters
- field_params = [f'description="{description}"']
+ field_params = [f"description={self._escape_string_for_python(description)}"]
if required:
if default_value is not None:
diff --git a/pipelex/core/pipe_errors.py b/pipelex/core/pipe_errors.py
new file mode 100644
index 000000000..f94c6be8b
--- /dev/null
+++ b/pipelex/core/pipe_errors.py
@@ -0,0 +1,37 @@
+from pipelex.system.exceptions import RootException
+
+
+# TODO: add details from all cases raising this error
+class PipeDefinitionError(RootException):
+ def __init__(
+ self,
+ message: str,
+ domain_code: str | None = None,
+ pipe_code: str | None = None,
+ description: str | None = None,
+ source: str | None = None,
+ ):
+ self.domain_code = domain_code
+ self.pipe_code = pipe_code
+ self.description = description
+ self.source = source
+ message = message + " • " + self.pipe_details()
+ super().__init__(message)
+
+ def pipe_details(self) -> str:
+ if not self.domain_code and not self.pipe_code and not self.description and not self.source:
+ return "No pipe details provided"
+ details = "Pipe details:"
+ if self.domain_code:
+ details += f" • domain='{self.domain_code}'"
+ if self.pipe_code:
+ details += f" • pipe='{self.pipe_code}'"
+ if self.description:
+ details += f" • description='{self.description}'"
+ if self.source:
+ details += f" • source='{self.source}'"
+ return details
+
+
+class UnexpectedPipeDefinitionError(PipeDefinitionError):
+ pass
diff --git a/pipelex/core/validation_errors.py b/pipelex/core/validation_errors.py
new file mode 100644
index 000000000..50a9ce948
--- /dev/null
+++ b/pipelex/core/validation_errors.py
@@ -0,0 +1,7 @@
+from typing_extensions import Protocol
+
+from pipelex.builder.validation_error_data import ConceptDefinitionErrorData
+
+
+class ValidationErrorDetailsProtocol(Protocol):
+ def get_concept_definition_errors(self) -> list[ConceptDefinitionErrorData]: ...
diff --git a/pipelex/exceptions.py b/pipelex/exceptions.py
index 7f66817e6..c584a3ece 100644
--- a/pipelex/exceptions.py
+++ b/pipelex/exceptions.py
@@ -1,9 +1,18 @@
+from __future__ import annotations
+
from click import ClickException
from typing_extensions import override
+from pipelex.builder.validation_error_data import (
+ ConceptDefinitionErrorData,
+ LibraryLoadingErrorData,
+ PipeDefinitionErrorData,
+ StaticValidationErrorType,
+ SyntaxErrorData,
+)
+from pipelex.core.validation_errors import ValidationErrorDetailsProtocol
from pipelex.system.exceptions import RootException
from pipelex.tools.misc.context_provider_abstract import ContextProviderException
-from pipelex.types import StrEnum
class PipelexException(RootException):
@@ -14,13 +23,6 @@ class PipelexUnexpectedError(PipelexException):
pass
-class StaticValidationErrorType(StrEnum):
- MISSING_INPUT_VARIABLE = "missing_input_variable"
- EXTRANEOUS_INPUT_VARIABLE = "extraneous_input_variable"
- INADEQUATE_INPUT_CONCEPT = "inadequate_input_concept"
- TOO_MANY_CANDIDATE_INPUTS = "too_many_candidate_inputs"
-
-
class StaticValidationError(Exception):
def __init__(
self,
@@ -116,8 +118,29 @@ class LibraryError(PipelexException):
pass
-class LibraryLoadingError(LibraryError):
- pass
+class LibraryLoadingError(LibraryError, ValidationErrorDetailsProtocol):
+ """Error raised when loading library components fails."""
+
+ def __init__(
+ self,
+ message: str,
+ concept_definition_errors: list[ConceptDefinitionErrorData] | None = None,
+ pipe_definition_errors: list[PipeDefinitionErrorData] | None = None,
+ ):
+ self.concept_definition_errors = concept_definition_errors
+ self.pipe_definition_errors = pipe_definition_errors
+ super().__init__(message)
+
+ @override
+ def get_concept_definition_errors(self) -> list[ConceptDefinitionErrorData]:
+ return self.concept_definition_errors or []
+
+ def as_structured_content(self) -> LibraryLoadingErrorData:
+ return LibraryLoadingErrorData(
+ message=str(self),
+ concept_definition_errors=self.concept_definition_errors,
+ pipe_definition_errors=self.pipe_definition_errors,
+ )
class DomainLibraryError(LibraryError):
@@ -160,54 +183,36 @@ class ConceptDefinitionError(PipelexException):
concept_code: str,
description: str,
structure_class_python_code: str | None = None,
+ structure_class_syntax_error_data: SyntaxErrorData | None = None,
source: str | None = None,
):
self.domain_code = domain_code
self.concept_code = concept_code
self.description = description
self.structure_class_python_code = structure_class_python_code
+ self.structure_class_syntax_error_data = structure_class_syntax_error_data
self.source = source
super().__init__(message)
+ def as_structured_content(self) -> ConceptDefinitionErrorData:
+ return ConceptDefinitionErrorData(
+ message=str(self),
+ domain_code=self.domain_code,
+ concept_code=self.concept_code,
+ description=self.description,
+ structure_class_python_code=self.structure_class_python_code,
+ structure_class_syntax_error_data=self.structure_class_syntax_error_data,
+ source=self.source,
+ )
+
class ConceptStructureGeneratorError(PipelexException):
- def __init__(self, message: str, structure_class_python_code: str | None = None):
+ def __init__(self, message: str, structure_class_python_code: str | None = None, syntax_error_data: SyntaxErrorData | None = None):
self.structure_class_python_code = structure_class_python_code
+ self.syntax_error_data = syntax_error_data
super().__init__(message)
-# TODO: add details from all cases raising this error
-class PipeDefinitionError(PipelexException):
- def __init__(
- self,
- message: str,
- domain_code: str | None = None,
- pipe_code: str | None = None,
- description: str | None = None,
- source: str | None = None,
- ):
- self.domain_code = domain_code
- self.pipe_code = pipe_code
- self.description = description
- self.source = source
- message = message + " • " + self.pipe_details()
- super().__init__(message)
-
- def pipe_details(self) -> str:
- if not self.domain_code and not self.pipe_code and not self.description and not self.source:
- return "No pipe details provided"
- details = "Pipe details:"
- if self.domain_code:
- details += f" • domain='{self.domain_code}'"
- if self.pipe_code:
- details += f" • pipe='{self.pipe_code}'"
- if self.description:
- details += f" • description='{self.description}'"
- if self.source:
- details += f" • source='{self.source}'"
- return details
-
-
class PipeInputError(PipelexException):
def __init__(self, message: str, pipe_code: str, variable_name: str, concept_code: str | None = None):
self.pipe_code = pipe_code
@@ -236,7 +241,7 @@ class ConceptLoadingError(LibraryLoadingError):
class PipeLoadingError(LibraryLoadingError):
- def __init__(self, message: str, pipe_definition_error: PipeDefinitionError, pipe_code: str, description: str, source: str | None = None):
+ def __init__(self, message: str, pipe_definition_error: PipeDefinitionErrorData, pipe_code: str, description: str, source: str | None = None):
self.pipe_definition_error = pipe_definition_error
self.pipe_code = pipe_code
self.description = description
@@ -244,10 +249,6 @@ class PipeLoadingError(LibraryLoadingError):
super().__init__(message)
-class UnexpectedPipeDefinitionError(PipeDefinitionError):
- pass
-
-
class StuffError(PipelexException):
pass
diff --git a/pipelex/libraries/library_manager.py b/pipelex/libraries/library_manager.py
index cb0761519..dc877a1db 100644
--- a/pipelex/libraries/library_manager.py
+++ b/pipelex/libraries/library_manager.py
@@ -5,6 +5,7 @@ from pydantic import ValidationError
from typing_extensions import override
from pipelex import log
+from pipelex.builder.validation_error_data import PipeDefinitionErrorData
from pipelex.core.bundles.pipelex_bundle_blueprint import PipelexBundleBlueprint
from pipelex.core.concepts.concept import Concept
from pipelex.core.concepts.concept_factory import ConceptFactory
@@ -14,6 +15,7 @@ from pipelex.core.domains.domain_blueprint import DomainBlueprint
from pipelex.core.domains.domain_factory import DomainFactory
from pipelex.core.domains.domain_library import DomainLibrary
from pipelex.core.interpreter import PipelexInterpreter
+from pipelex.core.pipe_errors import PipeDefinitionError
from pipelex.core.pipes.pipe_abstract import PipeAbstract
from pipelex.core.pipes.pipe_factory import PipeFactory
from pipelex.core.pipes.pipe_library import PipeLibrary
@@ -27,7 +29,6 @@ from pipelex.exceptions import (
DomainLoadingError,
LibraryError,
LibraryLoadingError,
- PipeDefinitionError,
PipeLibraryError,
PipeLoadingError,
)
@@ -136,29 +137,45 @@ class LibraryManager(LibraryManagerAbstract):
# Create and load domain
try:
domain = self._load_domain_from_blueprint(blueprint)
- except DomainDefinitionError as exc:
- msg = f"Could not load domain from PLX blueprint at '{blueprint.source}', domain code: '{blueprint.domain}': {exc}"
- raise DomainLoadingError(message=msg, domain_code=exc.domain_code, description=exc.description, source=exc.source) from exc
+ except DomainDefinitionError as pipe_def_error:
+ msg = f"Could not load domain from PLX blueprint at '{blueprint.source}', domain code: '{blueprint.domain}': {pipe_def_error}"
+ raise DomainLoadingError(
+ message=msg, domain_code=pipe_def_error.domain_code, description=pipe_def_error.description, source=pipe_def_error.source
+ ) from pipe_def_error
self.domain_library.add_domain(domain=domain)
# Create and load concepts
try:
concepts = self._load_concepts_from_blueprint(blueprint)
- except ConceptDefinitionError as exc:
- msg = f"Could not load concepts from PLX blueprint at '{blueprint.source}', domain code: '{blueprint.domain}': {exc}"
+ except ConceptDefinitionError as pipe_def_error:
+ msg = f"Could not load concepts from PLX blueprint at '{blueprint.source}', domain code: '{blueprint.domain}': {pipe_def_error}"
raise ConceptLoadingError(
- message=msg, concept_definition_error=exc, concept_code=exc.concept_code, description=exc.description, source=exc.source
- ) from exc
+ message=msg,
+ concept_definition_error=pipe_def_error,
+ concept_code=pipe_def_error.concept_code,
+ description=pipe_def_error.description,
+ source=pipe_def_error.source,
+ ) from pipe_def_error
self.concept_library.add_concepts(concepts=concepts)
# Create and load pipes
try:
pipes = self._load_pipes_from_blueprint(blueprint)
- except PipeDefinitionError as exc:
- msg = f"Could not load pipes from PLX blueprint at '{blueprint.source}', domain code: '{blueprint.domain}': {exc}"
+ except PipeDefinitionError as pipe_def_error:
+ msg = f"Could not load pipes from PLX blueprint at '{blueprint.source}', domain code: '{blueprint.domain}': {pipe_def_error}"
raise PipeLoadingError(
- message=msg, pipe_definition_error=exc, pipe_code=exc.pipe_code or "", description=exc.description or "", source=exc.source
- ) from exc
+ message=msg,
+ pipe_definition_error=PipeDefinitionErrorData(
+ message=pipe_def_error.message,
+ domain_code=pipe_def_error.domain_code,
+ pipe_code=pipe_def_error.pipe_code,
+ description=pipe_def_error.description,
+ source=pipe_def_error.source,
+ ),
+ pipe_code=pipe_def_error.pipe_code or "",
+ description=pipe_def_error.description or "",
+ source=pipe_def_error.source,
+ ) from pipe_def_error
self.pipe_library.add_pipes(pipes=pipes)
return pipes
@@ -352,7 +369,10 @@ class LibraryManager(LibraryManagerAbstract):
concepts = self._load_concepts_from_blueprint(blueprint)
except ConceptDefinitionError as concept_def_error:
msg = f"Could not load concepts from PLX blueprint at '{blueprint.source}', domain code: '{blueprint.domain}': {concept_def_error}"
- raise LibraryLoadingError(msg) from concept_def_error
+ raise LibraryLoadingError(
+ msg,
+ concept_definition_errors=[concept_def_error.as_structured_content()],
+ ) from concept_def_error
except ValidationError as validation_error:
validation_error_msg = report_validation_error(category="plx", validation_error=validation_error)
msg = f"Could not load concepts from PLX blueprint at '{blueprint.source}', domain code: '{blueprint.domain}': {validation_error_msg}"
@@ -367,7 +387,18 @@ class LibraryManager(LibraryManagerAbstract):
pipes = self._load_pipes_from_blueprint(blueprint)
except PipeDefinitionError as pipe_def_error:
msg = f"Could not load pipes from PLX blueprint at '{blueprint.source}', domain code: '{blueprint.domain}': {pipe_def_error}"
- raise LibraryLoadingError(msg) from pipe_def_error
+ raise LibraryLoadingError(
+ msg,
+ pipe_definition_errors=[
+ PipeDefinitionErrorData(
+ message=pipe_def_error.message,
+ domain_code=pipe_def_error.domain_code,
+ pipe_code=pipe_def_error.pipe_code,
+ description=pipe_def_error.description,
+ source=pipe_def_error.source,
+ )
+ ],
+ ) from pipe_def_error
except ValidationError as validation_error:
validation_error_msg = report_validation_error(category="plx", validation_error=validation_error)
msg = f"Could not load pipes from PLX blueprint at '{blueprint.source}', domain code: '{blueprint.domain}': {validation_error_msg}"
diff --git a/pipelex/pipe_controllers/condition/pipe_condition.py b/pipelex/pipe_controllers/condition/pipe_condition.py
index 50d4ba84a..df95af863 100644
--- a/pipelex/pipe_controllers/condition/pipe_condition.py
+++ b/pipelex/pipe_controllers/condition/pipe_condition.py
@@ -10,13 +10,13 @@ from pipelex.config import StaticValidationReaction, get_config
from pipelex.core.concepts.concept_factory import ConceptFactory
from pipelex.core.concepts.concept_native import NativeConceptCode
from pipelex.core.memory.working_memory import WorkingMemory
+from pipelex.core.pipe_errors import PipeDefinitionError
from pipelex.core.pipes.input_requirements import InputRequirements
from pipelex.core.pipes.input_requirements_factory import InputRequirementsFactory
from pipelex.core.pipes.pipe_output import PipeOutput
from pipelex.exceptions import (
DryRunError,
PipeConditionError,
- PipeDefinitionError,
PipeExecutionError,
PipeInputError,
StaticValidationError,
diff --git a/pipelex/pipe_controllers/parallel/pipe_parallel.py b/pipelex/pipe_controllers/parallel/pipe_parallel.py
index c6173a0b1..7be52e71e 100644
--- a/pipelex/pipe_controllers/parallel/pipe_parallel.py
+++ b/pipelex/pipe_controllers/parallel/pipe_parallel.py
@@ -8,11 +8,12 @@ from pipelex import log
from pipelex.config import StaticValidationReaction, get_config
from pipelex.core.concepts.concept import Concept
from pipelex.core.memory.working_memory import WorkingMemory
+from pipelex.core.pipe_errors import PipeDefinitionError
from pipelex.core.pipes.input_requirements import InputRequirements
from pipelex.core.pipes.input_requirements_factory import InputRequirementsFactory
from pipelex.core.pipes.pipe_output import PipeOutput
from pipelex.core.stuffs.stuff_factory import StuffFactory
-from pipelex.exceptions import DryRunError, PipeDefinitionError, PipeRunParamsError, StaticValidationError, StaticValidationErrorType
+from pipelex.exceptions import DryRunError, PipeRunParamsError, StaticValidationError, StaticValidationErrorType
from pipelex.hub import get_pipeline_tracker, get_required_pipe
from pipelex.pipe_controllers.pipe_controller import PipeController
from pipelex.pipe_controllers.sub_pipe import SubPipe
diff --git a/pipelex/pipe_controllers/parallel/pipe_parallel_blueprint.py b/pipelex/pipe_controllers/parallel/pipe_parallel_blueprint.py
index 5c498cf0a..9a3e3828c 100644
--- a/pipelex/pipe_controllers/parallel/pipe_parallel_blueprint.py
+++ b/pipelex/pipe_controllers/parallel/pipe_parallel_blueprint.py
@@ -4,8 +4,8 @@ from pydantic import field_validator, model_validator
from typing_extensions import override
from pipelex.core.concepts.concept_blueprint import ConceptBlueprint
+from pipelex.core.pipe_errors import PipeDefinitionError
from pipelex.core.pipes.pipe_blueprint import PipeBlueprint
-from pipelex.exceptions import PipeDefinitionError
from pipelex.pipe_controllers.sub_pipe_blueprint import SubPipeBlueprint
from pipelex.types import Self
diff --git a/pipelex/pipe_controllers/parallel/pipe_parallel_factory.py b/pipelex/pipe_controllers/parallel/pipe_parallel_factory.py
index ba48e68c1..fdfffe546 100644
--- a/pipelex/pipe_controllers/parallel/pipe_parallel_factory.py
+++ b/pipelex/pipe_controllers/parallel/pipe_parallel_factory.py
@@ -3,10 +3,10 @@ from typing import TYPE_CHECKING
from typing_extensions import override
from pipelex.core.concepts.concept_factory import ConceptFactory
+from pipelex.core.pipe_errors import PipeDefinitionError
from pipelex.core.pipes.input_requirements_factory import InputRequirementsFactory
from pipelex.core.pipes.pipe_factory import PipeFactoryProtocol
from pipelex.core.pipes.variable_multiplicity import parse_concept_with_multiplicity
-from pipelex.exceptions import PipeDefinitionError
from pipelex.hub import get_required_concept
from pipelex.pipe_controllers.parallel.pipe_parallel import PipeParallel
from pipelex.pipe_controllers.parallel.pipe_parallel_blueprint import PipeParallelBlueprint
diff --git a/pipelex/pipe_controllers/sub_pipe_blueprint.py b/pipelex/pipe_controllers/sub_pipe_blueprint.py
index eb69fbff6..e26e6d342 100644
--- a/pipelex/pipe_controllers/sub_pipe_blueprint.py
+++ b/pipelex/pipe_controllers/sub_pipe_blueprint.py
@@ -1,6 +1,6 @@
from pydantic import BaseModel, ConfigDict, model_validator
-from pipelex.exceptions import PipeDefinitionError
+from pipelex.core.pipe_errors import PipeDefinitionError
from pipelex.tools.typing.validation_utils import has_more_than_one_among_attributes_from_list
from pipelex.types import Self
diff --git a/pipelex/pipe_operators/compose/pipe_compose.py b/pipelex/pipe_operators/compose/pipe_compose.py
index 4d5553fee..cdc7871ad 100644
--- a/pipelex/pipe_operators/compose/pipe_compose.py
+++ b/pipelex/pipe_operators/compose/pipe_compose.py
@@ -13,12 +13,13 @@ from pipelex.core.concepts.concept import Concept
from pipelex.core.concepts.concept_factory import ConceptFactory
from pipelex.core.concepts.concept_native import NativeConceptCode
from pipelex.core.memory.working_memory import WorkingMemory
+from pipelex.core.pipe_errors import PipeDefinitionError
from pipelex.core.pipes.input_requirements import InputRequirements
from pipelex.core.pipes.input_requirements_factory import InputRequirementsFactory
from pipelex.core.pipes.pipe_output import PipeOutput
from pipelex.core.stuffs.stuff_factory import StuffFactory
from pipelex.core.stuffs.text_content import TextContent
-from pipelex.exceptions import PipeDefinitionError, PipeRunParamsError
+from pipelex.exceptions import PipeRunParamsError
from pipelex.hub import get_content_generator
from pipelex.pipe_operators.pipe_operator import PipeOperator
from pipelex.pipe_run.pipe_run_params import PipeRunMode, PipeRunParams
diff --git a/pipelex/pipe_operators/extract/pipe_extract.py b/pipelex/pipe_operators/extract/pipe_extract.py
index 19dcc454a..c56eb9993 100644
--- a/pipelex/pipe_operators/extract/pipe_extract.py
+++ b/pipelex/pipe_operators/extract/pipe_extract.py
@@ -13,6 +13,7 @@ from pipelex.cogt.models.model_deck_check import check_extract_choice_with_deck
from pipelex.config import StaticValidationReaction, get_config
from pipelex.core.concepts.concept_native import NativeConceptCode
from pipelex.core.memory.working_memory import WorkingMemory
+from pipelex.core.pipe_errors import PipeDefinitionError
from pipelex.core.pipes.input_requirements import InputRequirements
from pipelex.core.pipes.pipe_output import PipeOutput
from pipelex.core.stuffs.image_content import ImageContent
@@ -22,7 +23,6 @@ from pipelex.core.stuffs.stuff_factory import StuffFactory
from pipelex.core.stuffs.text_and_images_content import TextAndImagesContent
from pipelex.core.stuffs.text_content import TextContent
from pipelex.exceptions import (
- PipeDefinitionError,
StaticValidationError,
StaticValidationErrorType,
)
diff --git a/pipelex/pipe_operators/img_gen/pipe_img_gen.py b/pipelex/pipe_operators/img_gen/pipe_img_gen.py
index f94d693c3..c32fcd38d 100644
--- a/pipelex/pipe_operators/img_gen/pipe_img_gen.py
+++ b/pipelex/pipe_operators/img_gen/pipe_img_gen.py
@@ -14,6 +14,7 @@ from pipelex.config import StaticValidationReaction, get_config
from pipelex.core.concepts.concept_factory import ConceptFactory
from pipelex.core.concepts.concept_native import NativeConceptCode
from pipelex.core.memory.working_memory import WorkingMemory
+from pipelex.core.pipe_errors import PipeDefinitionError, UnexpectedPipeDefinitionError
from pipelex.core.pipes.input_requirements import InputRequirements
from pipelex.core.pipes.input_requirements_factory import InputRequirementsFactory
from pipelex.core.pipes.pipe_output import PipeOutput
@@ -22,13 +23,11 @@ from pipelex.core.stuffs.image_content import ImageContent
from pipelex.core.stuffs.list_content import ListContent
from pipelex.core.stuffs.stuff_factory import StuffFactory
from pipelex.exceptions import (
- PipeDefinitionError,
PipeInputError,
PipeRunParamsError,
StaticValidationError,
StaticValidationErrorType,
StuffContentTypeError,
- UnexpectedPipeDefinitionError,
WorkingMemoryStuffNotFoundError,
)
from pipelex.hub import get_concept_library, get_content_generator, get_model_deck, get_native_concept
diff --git a/pipelex/pipe_operators/llm/pipe_llm.py b/pipelex/pipe_operators/llm/pipe_llm.py
index 1b2f93a75..c9d771730 100644
--- a/pipelex/pipe_operators/llm/pipe_llm.py
+++ b/pipelex/pipe_operators/llm/pipe_llm.py
@@ -17,6 +17,7 @@ from pipelex.core.concepts.concept_factory import ConceptFactory
from pipelex.core.concepts.concept_native import NativeConceptCode
from pipelex.core.domains.domain import SpecialDomain
from pipelex.core.memory.working_memory import WorkingMemory
+from pipelex.core.pipe_errors import PipeDefinitionError
from pipelex.core.pipes.input_requirements import InputRequirements
from pipelex.core.pipes.input_requirements_factory import InputRequirementsFactory
from pipelex.core.pipes.pipe_output import PipeOutput
@@ -26,7 +27,6 @@ from pipelex.core.stuffs.stuff_content import StuffContent
from pipelex.core.stuffs.stuff_factory import StuffFactory
from pipelex.core.stuffs.text_content import TextContent
from pipelex.exceptions import (
- PipeDefinitionError,
StaticValidationError,
StaticValidationErrorType,
)
diff --git a/pipelex/pipe_operators/llm/pipe_llm_factory.py b/pipelex/pipe_operators/llm/pipe_llm_factory.py
index c97d82970..6a39ae224 100644
--- a/pipelex/pipe_operators/llm/pipe_llm_factory.py
+++ b/pipelex/pipe_operators/llm/pipe_llm_factory.py
@@ -6,10 +6,10 @@ from pipelex.cogt.templating.template_category import TemplateCategory
from pipelex.core.concepts.concept import Concept
from pipelex.core.concepts.concept_factory import ConceptFactory
from pipelex.core.concepts.concept_native import NativeConceptCode
+from pipelex.core.pipe_errors import PipeDefinitionError
from pipelex.core.pipes.input_requirements_factory import InputRequirementsFactory
from pipelex.core.pipes.pipe_factory import PipeFactoryProtocol
from pipelex.core.pipes.variable_multiplicity import make_variable_multiplicity, parse_concept_with_multiplicity
-from pipelex.exceptions import PipeDefinitionError
from pipelex.hub import get_native_concept, get_optional_domain, get_required_concept
from pipelex.pipe_operators.llm.llm_prompt_blueprint import LLMPromptBlueprint
from pipelex.pipe_operators.llm.pipe_llm import PipeLLM
diff --git a/pipelex/pipelex.py b/pipelex/pipelex.py
index 2c048c758..a42a24607 100644
--- a/pipelex/pipelex.py
+++ b/pipelex/pipelex.py
@@ -1,5 +1,5 @@
from importlib.metadata import metadata
-from typing import cast
+from typing import Any, cast
from kajson.class_registry import ClassRegistry
from kajson.class_registry_abstract import ClassRegistryAbstract
@@ -150,7 +150,11 @@ If you need help, drop by our Discord: we're happy to assist: {URLs.discord}.
pipe_router: PipeRouterProtocol | None = None,
reporting_delegate: ReportingProtocol | None = None,
observers: dict[str, ObserverProtocol] | None = None,
+ **kwargs: Any,
):
+ if kwargs:
+ msg = f"The base setup method does not support any additional arguments: {kwargs}"
+ raise PipelexSetupError(msg)
# tools
self.class_registry = class_registry or ClassRegistry()
self.pipelex_hub.set_class_registry(self.class_registry)
@@ -286,16 +290,47 @@ If you need help, drop by our Discord: we're happy to assist: {URLs.discord}.
if self.__class__ in MetaSingleton.instances:
del MetaSingleton.instances[self.__class__]
- # TODO: add kwargs to make() so that subclasses can employ specific parameters
@classmethod
- def make(cls) -> Self:
- """Create and initialize a Pipelex instance.
+ def make(
+ cls,
+ class_registry: ClassRegistryAbstract | None = None,
+ secrets_provider: SecretsProviderAbstract | None = None,
+ storage_provider: StorageProviderAbstract | None = None,
+ models_manager: ModelManagerAbstract | None = None,
+ inference_manager: InferenceManager | None = None,
+ content_generator: ContentGeneratorProtocol | None = None,
+ pipeline_manager: PipelineManager | None = None,
+ pipeline_tracker: PipelineTracker | None = None,
+ pipe_router: PipeRouterProtocol | None = None,
+ reporting_delegate: ReportingProtocol | None = None,
+ observers: dict[str, ObserverProtocol] | None = None,
+ **kwargs: Any,
+ ) -> Self:
+ """Create and initialize a Pipelex singleton instance.
+
+ All parameters are optional dependency injections. If None, default implementations
+ are used during setup. This enables customization of core components like secrets
+ management, storage, model routing, and pipeline execution.
+
+ Args:
+ class_registry: Custom class registry for dynamic loading
+ secrets_provider: Custom secrets/credentials provider
+ storage_provider: Custom storage backend
+ models_manager: Custom model configuration manager
+ inference_manager: Custom inference routing manager
+ content_generator: Custom content generation implementation
+ pipeline_manager: Custom pipeline management
+ pipeline_tracker: Custom pipeline tracking/logging
+ pipe_router: Custom pipe routing logic
+ reporting_delegate: Custom reporting handler
+ observers: Custom observers for pipeline events
+ **kwargs: Additional configuration options, only supported by your own subclass of Pipelex if you really need one
Returns:
Initialized Pipelex instance.
Raises:
- if setup fails
+ PipelexSetupError: If Pipelex is already initialized or setup fails
"""
if cls.get_optional_instance() is not None:
@@ -303,7 +338,20 @@ If you need help, drop by our Discord: we're happy to assist: {URLs.discord}.
raise PipelexSetupError(msg)
pipelex_instance = cls()
- pipelex_instance.setup()
+ pipelex_instance.setup(
+ class_registry=class_registry,
+ secrets_provider=secrets_provider,
+ storage_provider=storage_provider,
+ models_manager=models_manager,
+ inference_manager=inference_manager,
+ content_generator=content_generator,
+ pipeline_manager=pipeline_manager,
+ pipeline_tracker=pipeline_tracker,
+ pipe_router=pipe_router,
+ reporting_delegate=reporting_delegate,
+ observers=observers,
+ **kwargs,
+ )
pipelex_instance.setup_libraries()
log.info(f"{PACKAGE_NAME} version {PACKAGE_VERSION} ready")
return pipelex_instance
diff --git a/tests/integration/pipelex/test_inline_structure_concepts.py b/tests/integration/pipelex/test_inline_structure_concepts.py
index 45fc6c8d0..1652ea960 100644
--- a/tests/integration/pipelex/test_inline_structure_concepts.py
+++ b/tests/integration/pipelex/test_inline_structure_concepts.py
@@ -239,3 +239,55 @@ not a registered subclass of StuffContent",
assert product.name == "Widget"
assert person.age == 30
assert product.age == 365
+
+ def test_inline_structure_with_quoted_descriptions(self):
+ """Test that inline structures with quotes in descriptions work end-to-end."""
+ # Real-world example: field descriptions containing quoted examples
+ inline_structure: dict[str, str | ConceptStructureBlueprint] = {
+ "animal_type": ConceptStructureBlueprint(
+ type=ConceptStructureBlueprintFieldType.TEXT,
+ description='The species or type of animal (e.g., "fox", "penguin", "octopus")',
+ required=True,
+ ),
+ "owner_name": ConceptStructureBlueprint(
+ type=ConceptStructureBlueprintFieldType.TEXT,
+ description="The owner's full name with apostrophe",
+ required=False,
+ ),
+ "habitat": ConceptStructureBlueprint(
+ type=ConceptStructureBlueprintFieldType.TEXT,
+ description='Natural habitat like "forest", "ocean", or "desert"',
+ required=False,
+ ),
+ }
+
+ # Create concept blueprint with inline structure
+ blueprint = ConceptBlueprint(description="Information about an animal", structure=inline_structure)
+
+ # Create concept from blueprint
+ concept = ConceptFactory.make_from_blueprint(
+ domain="test_domain",
+ concept_code="AnimalInfo",
+ blueprint=blueprint,
+ concept_codes_from_the_same_domain=["AnimalInfo"],
+ )
+
+ # Verify concept properties
+ assert concept.domain == "test_domain"
+ assert concept.code == "AnimalInfo"
+ assert concept.structure_class_name == "AnimalInfo"
+
+ # Verify the generated class is registered and accessible
+ assert get_class_registry().has_class("AnimalInfo")
+ generated_class = get_class_registry().get_required_subclass("AnimalInfo", StructuredContent)
+
+ # Test instantiation with quoted data
+ instance = generated_class(
+ animal_type="fox", # pyright: ignore[reportCallIssue]
+ owner_name="John's Pet Store", # pyright: ignore[reportCallIssue]
+ habitat="forest", # pyright: ignore[reportCallIssue]
+ )
+
+ assert instance.animal_type == "fox" # type: ignore[attr-defined] # pyright: ignore[reportUnknownMemberType,reportAttributeAccessIssue]
+ assert instance.owner_name == "John's Pet Store" # type: ignore[attr-defined] # pyright: ignore[reportUnknownMemberType,reportAttributeAccessIssue]
+ assert instance.habitat == "forest" # type: ignore[attr-defined] # pyright: ignore[reportUnknownMemberType,reportAttributeAccessIssue]
diff --git a/tests/unit/pipelex/core/bundles/test_data_pipe_sorter.py b/tests/unit/pipelex/core/bundles/test_data_pipe_sorter.py
index 3adeea351..6a33e8d26 100644
--- a/tests/unit/pipelex/core/bundles/test_data_pipe_sorter.py
+++ b/tests/unit/pipelex/core/bundles/test_data_pipe_sorter.py
@@ -3,7 +3,7 @@
from typing import ClassVar
from pipelex.core.bundles.pipelex_bundle_blueprint import PipeBlueprintUnion
-from pipelex.exceptions import PipeDefinitionError
+from pipelex.core.pipe_errors import PipeDefinitionError
from pipelex.pipe_controllers.batch.pipe_batch_blueprint import PipeBatchBlueprint
from pipelex.pipe_controllers.condition.pipe_condition_blueprint import PipeConditionBlueprint
from pipelex.pipe_controllers.parallel.pipe_parallel_blueprint import PipeParallelBlueprint
diff --git a/tests/unit/pipelex/core/concepts/test_structure_generator_escaping.py b/tests/unit/pipelex/core/concepts/test_structure_generator_escaping.py
new file mode 100644
index 000000000..49b4e1f2b
--- /dev/null
+++ b/tests/unit/pipelex/core/concepts/test_structure_generator_escaping.py
@@ -0,0 +1,385 @@
+"""Unit tests for string escaping in StructureGenerator.
+
+This module tests that field descriptions and default values containing special
+characters (quotes, backslashes, newlines, etc.) are properly escaped in the
+generated Python code.
+"""
+
+import ast
+
+from pipelex.core.concepts.concept_blueprint import ConceptStructureBlueprint, ConceptStructureBlueprintFieldType
+from pipelex.core.concepts.structure_generator import StructureGenerator
+from pipelex.core.stuffs.structured_content import StructuredContent
+
+
+class TestStructureGeneratorEscaping:
+ """Test string escaping in structure generation."""
+
+ def test_escape_double_quotes_in_description(self):
+ """Test that double quotes in field descriptions are properly escaped."""
+ # This is the user's reported bug case
+ description = 'The species or type of animal (e.g., "fox", "penguin", "octopus")'
+
+ blueprint = {
+ "animal_type": ConceptStructureBlueprint(
+ type=ConceptStructureBlueprintFieldType.TEXT,
+ description=description,
+ required=True,
+ )
+ }
+
+ generator = StructureGenerator()
+ generated_code, generated_class = generator.generate_from_structure_blueprint("Animal", blueprint)
+
+ # Verify the code compiles without SyntaxError
+ ast.parse(generated_code)
+ compile(generated_code, "<generated>", "exec")
+
+ # Verify the generated class can be instantiated
+ assert issubclass(generated_class, StructuredContent)
+ instance = generated_class(animal_type="fox") # pyright: ignore[reportCallIssue]
+ assert instance.animal_type == "fox" # type: ignore[attr-defined] # pyright: ignore[reportAttributeAccessIssue]
+
+ # Verify the description in the generated code matches
+ # The description should be preserved in Field(description=...)
+ assert "animal_type" in generated_code
+ assert "description=" in generated_code
+
+ def test_escape_single_quotes_in_description(self):
+ """Test that single quotes in field descriptions are properly escaped."""
+ description = "This is John's description with a single quote"
+
+ blueprint = {
+ "field_name": ConceptStructureBlueprint(
+ type=ConceptStructureBlueprintFieldType.TEXT,
+ description=description,
+ required=True,
+ )
+ }
+
+ generator = StructureGenerator()
+ generated_code, generated_class = generator.generate_from_structure_blueprint("TestClass", blueprint)
+
+ # Verify the code compiles
+ ast.parse(generated_code)
+ compile(generated_code, "<generated>", "exec")
+
+ # Verify instantiation works
+ instance = generated_class(field_name="test") # pyright: ignore[reportCallIssue]
+ assert instance.field_name == "test" # type: ignore[attr-defined] # pyright: ignore[reportAttributeAccessIssue]
+
+ def test_escape_mixed_quotes_in_description(self):
+ """Test that mixed single and double quotes are properly escaped."""
+ description = """He said "it's working!" with excitement"""
+
+ blueprint = {
+ "message": ConceptStructureBlueprint(
+ type=ConceptStructureBlueprintFieldType.TEXT,
+ description=description,
+ required=True,
+ )
+ }
+
+ generator = StructureGenerator()
+ generated_code, generated_class = generator.generate_from_structure_blueprint("Message", blueprint)
+
+ # Verify the code compiles
+ ast.parse(generated_code)
+ compile(generated_code, "<generated>", "exec")
+
+ # Verify instantiation works
+ instance = generated_class(message="Hello world") # pyright: ignore[reportCallIssue]
+ assert instance.message == "Hello world" # type: ignore[attr-defined] # pyright: ignore[reportAttributeAccessIssue]
+
+ def test_escape_backslashes_in_description(self):
+ """Test that backslashes in field descriptions are properly escaped."""
+ description = r"Path separator: C:\Users\Documents\file.txt"
+
+ blueprint = {
+ "path": ConceptStructureBlueprint(
+ type=ConceptStructureBlueprintFieldType.TEXT,
+ description=description,
+ required=True,
+ )
+ }
+
+ generator = StructureGenerator()
+ generated_code, generated_class = generator.generate_from_structure_blueprint("FilePath", blueprint)
+
+ # Verify the code compiles
+ ast.parse(generated_code)
+ compile(generated_code, "<generated>", "exec")
+
+ # Verify instantiation works
+ instance = generated_class(path="/usr/local/bin") # pyright: ignore[reportCallIssue]
+ assert instance.path == "/usr/local/bin" # type: ignore[attr-defined] # pyright: ignore[reportAttributeAccessIssue]
+
+ def test_escape_newlines_in_description(self):
+ """Test that newlines in field descriptions are properly escaped."""
+ description = "Line 1\nLine 2\nLine 3"
+
+ blueprint = {
+ "multiline": ConceptStructureBlueprint(
+ type=ConceptStructureBlueprintFieldType.TEXT,
+ description=description,
+ required=True,
+ )
+ }
+
+ generator = StructureGenerator()
+ generated_code, generated_class = generator.generate_from_structure_blueprint("MultiLine", blueprint)
+
+ # Verify the code compiles
+ ast.parse(generated_code)
+ compile(generated_code, "<generated>", "exec")
+
+ # Verify instantiation works
+ instance = generated_class(multiline="test") # pyright: ignore[reportCallIssue]
+ assert instance.multiline == "test" # type: ignore[attr-defined] # pyright: ignore[reportAttributeAccessIssue]
+
+ def test_escape_tabs_in_description(self):
+ """Test that tabs in field descriptions are properly escaped."""
+ description = "Column1\tColumn2\tColumn3"
+
+ blueprint = {
+ "columns": ConceptStructureBlueprint(
+ type=ConceptStructureBlueprintFieldType.TEXT,
+ description=description,
+ required=True,
+ )
+ }
+
+ generator = StructureGenerator()
+ generated_code, generated_class = generator.generate_from_structure_blueprint("Columns", blueprint)
+
+ # Verify the code compiles
+ ast.parse(generated_code)
+ compile(generated_code, "<generated>", "exec")
+
+ # Verify instantiation works
+ instance = generated_class(columns="A,B,C") # pyright: ignore[reportCallIssue]
+ assert instance.columns == "A,B,C" # type: ignore[attr-defined] # pyright: ignore[reportAttributeAccessIssue]
+
+ def test_escape_multiple_special_characters_combined(self):
+ """Test combination of multiple special characters."""
+ description = r"""Complex: "quoted", 'single', backslash\, tab , newline
+and more!"""
+
+ blueprint = {
+ "complex_field": ConceptStructureBlueprint(
+ type=ConceptStructureBlueprintFieldType.TEXT,
+ description=description,
+ required=True,
+ )
+ }
+
+ generator = StructureGenerator()
+ generated_code, generated_class = generator.generate_from_structure_blueprint("ComplexType", blueprint)
+
+ # Verify the code compiles
+ ast.parse(generated_code)
+ compile(generated_code, "<generated>", "exec")
+
+ # Verify instantiation works
+ instance = generated_class(complex_field="value") # pyright: ignore[reportCallIssue]
+ assert instance.complex_field == "value" # type: ignore[attr-defined] # pyright: ignore[reportAttributeAccessIssue]
+
+ def test_escape_default_value_with_quotes(self):
+ """Test that default values with quotes are properly escaped."""
+ default_value = 'Example: "quoted text"'
+
+ blueprint = {
+ "field_with_default": ConceptStructureBlueprint(
+ type=ConceptStructureBlueprintFieldType.TEXT,
+ description="A field with quoted default",
+ required=False,
+ default_value=default_value,
+ )
+ }
+
+ generator = StructureGenerator()
+ generated_code, generated_class = generator.generate_from_structure_blueprint("WithDefault", blueprint)
+
+ # Verify the code compiles
+ ast.parse(generated_code)
+ compile(generated_code, "<generated>", "exec")
+
+ # Verify instantiation works with default
+ instance = generated_class() # pyright: ignore[reportCallIssue]
+ assert instance.field_with_default == default_value # type: ignore[attr-defined] # pyright: ignore[reportAttributeAccessIssue]
+
+ # Verify instantiation works with explicit value
+ instance2 = generated_class(field_with_default="custom") # pyright: ignore[reportCallIssue]
+ assert instance2.field_with_default == "custom" # type: ignore[attr-defined] # pyright: ignore[reportAttributeAccessIssue]
+
+ def test_escape_default_value_with_backslashes(self):
+ """Test that default values with backslashes are properly escaped."""
+ default_value = r"C:\Program Files\App"
+
+ blueprint = {
+ "path": ConceptStructureBlueprint(
+ type=ConceptStructureBlueprintFieldType.TEXT,
+ description="Default path",
+ required=False,
+ default_value=default_value,
+ )
+ }
+
+ generator = StructureGenerator()
+ generated_code, generated_class = generator.generate_from_structure_blueprint("DefaultPath", blueprint)
+
+ # Verify the code compiles
+ ast.parse(generated_code)
+ compile(generated_code, "<generated>", "exec")
+
+ # Verify instantiation works with default
+ instance = generated_class() # pyright: ignore[reportCallIssue]
+ assert instance.path == default_value # type: ignore[attr-defined] # pyright: ignore[reportAttributeAccessIssue]
+
+ def test_empty_string_description(self):
+ """Test that empty string descriptions are handled correctly."""
+ description = ""
+
+ blueprint = {
+ "empty_desc": ConceptStructureBlueprint(
+ type=ConceptStructureBlueprintFieldType.TEXT,
+ description=description,
+ required=True,
+ )
+ }
+
+ generator = StructureGenerator()
+ generated_code, generated_class = generator.generate_from_structure_blueprint("EmptyDesc", blueprint)
+
+ # Verify the code compiles
+ ast.parse(generated_code)
+ compile(generated_code, "<generated>", "exec")
+
+ # Verify instantiation works
+ instance = generated_class(empty_desc="value") # pyright: ignore[reportCallIssue]
+ assert instance.empty_desc == "value" # type: ignore[attr-defined] # pyright: ignore[reportAttributeAccessIssue]
+
+ def test_very_long_description_with_quotes(self):
+ """Test that very long descriptions with quotes are handled correctly."""
+ description = (
+ "This is a very long description that contains many words and also has "
+ 'some "quoted sections" that need to be properly escaped. It goes on and on '
+ "with more details about the field, including examples like "
+ '"example1", "example2", and "example3". The description continues with '
+ "even more information to test the robustness of the escaping mechanism."
+ )
+
+ blueprint = {
+ "long_field": ConceptStructureBlueprint(
+ type=ConceptStructureBlueprintFieldType.TEXT,
+ description=description,
+ required=True,
+ )
+ }
+
+ generator = StructureGenerator()
+ generated_code, generated_class = generator.generate_from_structure_blueprint("LongDesc", blueprint)
+
+ # Verify the code compiles
+ ast.parse(generated_code)
+ compile(generated_code, "<generated>", "exec")
+
+ # Verify instantiation works
+ instance = generated_class(long_field="test") # pyright: ignore[reportCallIssue]
+ assert instance.long_field == "test" # type: ignore[attr-defined] # pyright: ignore[reportAttributeAccessIssue]
+
+ def test_unicode_characters_in_description(self):
+ """Test that unicode characters in descriptions are handled correctly."""
+ description = "Unicode characters: café, naïve, 日本語, emoji 🎉"
+
+ blueprint = {
+ "unicode_field": ConceptStructureBlueprint(
+ type=ConceptStructureBlueprintFieldType.TEXT,
+ description=description,
+ required=True,
+ )
+ }
+
+ generator = StructureGenerator()
+ generated_code, generated_class = generator.generate_from_structure_blueprint("UnicodeTest", blueprint)
+
+ # Verify the code compiles
+ ast.parse(generated_code)
+ compile(generated_code, "<generated>", "exec")
+
+ # Verify instantiation works
+ instance = generated_class(unicode_field="test") # pyright: ignore[reportCallIssue]
+ assert instance.unicode_field == "test" # type: ignore[attr-defined] # pyright: ignore[reportAttributeAccessIssue]
+
+ def test_carriage_return_in_description(self):
+ """Test that carriage returns in descriptions are handled correctly."""
+ description = "Line 1\r\nLine 2\r\nLine 3"
+
+ blueprint = {
+ "crlf_field": ConceptStructureBlueprint(
+ type=ConceptStructureBlueprintFieldType.TEXT,
+ description=description,
+ required=True,
+ )
+ }
+
+ generator = StructureGenerator()
+ generated_code, generated_class = generator.generate_from_structure_blueprint("CRLF", blueprint)
+
+ # Verify the code compiles
+ ast.parse(generated_code)
+ compile(generated_code, "<generated>", "exec")
+
+ # Verify instantiation works
+ instance = generated_class(crlf_field="test") # pyright: ignore[reportCallIssue]
+ assert instance.crlf_field == "test" # type: ignore[attr-defined] # pyright: ignore[reportAttributeAccessIssue]
+
+ def test_multiple_fields_with_various_escaping_needs(self):
+ """Test a realistic scenario with multiple fields needing different escaping."""
+ blueprint = {
+ "animal_type": ConceptStructureBlueprint(
+ type=ConceptStructureBlueprintFieldType.TEXT,
+ description='The species or type of animal (e.g., "fox", "penguin", "octopus")',
+ required=True,
+ ),
+ "owner_name": ConceptStructureBlueprint(
+ type=ConceptStructureBlueprintFieldType.TEXT,
+ description="The owner's full name",
+ required=False,
+ ),
+ "file_path": ConceptStructureBlueprint(
+ type=ConceptStructureBlueprintFieldType.TEXT,
+ description=r"Location on disk (e.g., C:\Animals\data.txt)",
+ required=False,
+ default_value=r"C:\default\path",
+ ),
+ "notes": ConceptStructureBlueprint(
+ type=ConceptStructureBlueprintFieldType.TEXT,
+ description='Notes with "quotes" and\nnewlines',
+ required=False,
+ ),
+ }
+
+ generator = StructureGenerator()
+ generated_code, generated_class = generator.generate_from_structure_blueprint("AnimalRecord", blueprint)
+
+ # Verify the code compiles
+ ast.parse(generated_code)
+ compile(generated_code, "<generated>", "exec")
+
+ # Verify instantiation works with all fields
+ instance = generated_class( # pyright: ignore[reportCallIssue]
+ animal_type="dog",
+ owner_name="John Smith",
+ file_path=r"C:\custom\location",
+ notes="Some notes here",
+ )
+ assert instance.animal_type == "dog" # type: ignore[attr-defined] # pyright: ignore[reportAttributeAccessIssue]
+ assert instance.owner_name == "John Smith" # type: ignore[attr-defined] # pyright: ignore[reportAttributeAccessIssue]
+ assert instance.file_path == r"C:\custom\location" # type: ignore[attr-defined] # pyright: ignore[reportAttributeAccessIssue]
+ assert instance.notes == "Some notes here" # type: ignore[attr-defined] # pyright: ignore[reportAttributeAccessIssue]
+
+ # Verify instantiation with only required fields
+ instance2 = generated_class(animal_type="cat") # pyright: ignore[reportCallIssue]
+ assert instance2.animal_type == "cat" # type: ignore[attr-defined] # pyright: ignore[reportAttributeAccessIssue]
+ assert instance2.file_path == r"C:\default\path" # type: ignore[attr-defined] # pyright: ignore[reportAttributeAccessIssue]
diff --git a/tests/unit/pipelex/pipe_controllers/condition/test_pipe_condition_input.py b/tests/unit/pipelex/pipe_controllers/condition/test_pipe_condition_input.py
index 4f3b2857f..5f009620d 100644
--- a/tests/unit/pipelex/pipe_controllers/condition/test_pipe_condition_input.py
+++ b/tests/unit/pipelex/pipe_controllers/condition/test_pipe_condition_input.py
@@ -3,7 +3,8 @@ from typing import Any
import pytest
from pipelex import log
-from pipelex.exceptions import PipeDefinitionError, StaticValidationError
+from pipelex.core.pipe_errors import PipeDefinitionError
+from pipelex.exceptions import StaticValidationError
from pipelex.pipe_controllers.condition.pipe_condition_blueprint import PipeConditionBlueprint
from pipelex.pipe_controllers.condition.pipe_condition_factory import PipeConditionFactory
from tests.unit.pipelex.pipe_controllers.condition.data import PipeConditionInputTestCases
diff --git a/tests/unit/pipelex/pipe_controllers/parallel/test_pipe_parallel_input.py b/tests/unit/pipelex/pipe_controllers/parallel/test_pipe_parallel_input.py
index 83e54b6ee..1ee423b11 100644
--- a/tests/unit/pipelex/pipe_controllers/parallel/test_pipe_parallel_input.py
+++ b/tests/unit/pipelex/pipe_controllers/parallel/test_pipe_parallel_input.py
@@ -3,7 +3,8 @@ from typing import Any
import pytest
from pipelex import log
-from pipelex.exceptions import PipeDefinitionError, StaticValidationError
+from pipelex.core.pipe_errors import PipeDefinitionError
+from pipelex.exceptions import StaticValidationError
from pipelex.pipe_controllers.parallel.pipe_parallel_blueprint import PipeParallelBlueprint
from pipelex.pipe_controllers.parallel.pipe_parallel_factory import PipeParallelFactory
from tests.unit.pipelex.pipe_controllers.parallel.data import PipeParallelInputTestCases
diff --git a/tests/unit/pipelex/pipe_controllers/sequence/test_pipe_sequence_input.py b/tests/unit/pipelex/pipe_controllers/sequence/test_pipe_sequence_input.py
index f577ab588..99e29ce00 100644
--- a/tests/unit/pipelex/pipe_controllers/sequence/test_pipe_sequence_input.py
+++ b/tests/unit/pipelex/pipe_controllers/sequence/test_pipe_sequence_input.py
@@ -3,7 +3,8 @@ from typing import Any
import pytest
from pipelex import log
-from pipelex.exceptions import PipeDefinitionError, StaticValidationError
+from pipelex.core.pipe_errors import PipeDefinitionError
+from pipelex.exceptions import StaticValidationError
from pipelex.pipe_controllers.sequence.pipe_sequence_blueprint import PipeSequenceBlueprint
from pipelex.pipe_controllers.sequence.pipe_sequence_factory import PipeSequenceFactory
from tests.unit.pipelex.pipe_controllers.sequence.data import PipeSequenceInputTestCases
diff --git a/tests/unit/pipelex/pipe_operators/pipe_img_gen/test_pipe_img_gen_input.py b/tests/unit/pipelex/pipe_operators/pipe_img_gen/test_pipe_img_gen_input.py
index c35cc719e..6ccd8da83 100644
--- a/tests/unit/pipelex/pipe_operators/pipe_img_gen/test_pipe_img_gen_input.py
+++ b/tests/unit/pipelex/pipe_operators/pipe_img_gen/test_pipe_img_gen_input.py
@@ -1,7 +1,8 @@
import pytest
from pipelex import log
-from pipelex.exceptions import PipeDefinitionError, StaticValidationError
+from pipelex.core.pipe_errors import PipeDefinitionError
+from pipelex.exceptions import StaticValidationError
from pipelex.pipe_operators.img_gen.pipe_img_gen_blueprint import PipeImgGenBlueprint
from pipelex.pipe_operators.img_gen.pipe_img_gen_factory import PipeImgGenFactory
from tests.unit.pipelex.pipe_operators.pipe_img_gen.data import PipeImgGenInputTestCases