Build AI methods with your coding agent, then run them anywhere: as an MCP for chatbots, as a webapp for people, or via API for your software.
Features and Docs and tests to generate structure from Pipeline's toml, inc. inline in Concept def
Louis Choquel00e0477e67fa
diff --git a/.pipelex/create/design_pipelines.md b/.pipelex/create/design_pipelines.mddeleted file mode 100644index 2a3fcfb80..000000000--- a/.pipelex/create/design_pipelines.md+++ /dev/null@@ -1,270 +0,0 @@-## Designing Pipelex pipelines--This section explains how to design pipelines for Pipelex.--### Workflow for a designer agent--- Plan the pipeline in natural language (goal, inputs, steps, outputs).--- Translate the plan into a `PipelineBlueprint` object graph: domain, concepts, pipes.--- Define controllers first (`PipeSequence`, `PipeParallel`, `PipeCondition`), then operators (`PipeLLM`, `PipeOcr`, etc.).--- For each new pipe, detail its `type` and `definition`.--- Ensure each pipe’s `inputs` shows all required variables, including those used by nested steps or conditional branches. `inputs` is a dict mapping variable names to concept codes.--- Choose appropriate native or structured output concepts.--### Root blueprint object: PipelineBlueprint--`PipelineBlueprint` fields:--- `domain: str` — required. Single word or snake_case code for the domain.--- `definition: Optional[str]` — optional textual description of the domain.--- `system_prompt: Optional[str]` — default system prompt applied by LLM pipes in this domain if they don’t provide one.--- `system_prompt_to_structure: Optional[str]` — domain-level instructions to structure outputs.--- `prompt_template_to_structure: Optional[str]` — domain-level instructions to structure prompts.--- `concept: Dict[str, Union[str, ConceptBlueprint]]` — concept section. Keys are concept names (PascalCase). Values are either:-- - a string (quick form): the concept definition; or-- - a `ConceptBlueprint` object (full form).--- `pipe: Dict[str, Dict[str, Any]]` — pipes section. Keys are pipe codes (snake_case). Values are per-pipe details dicts that will be validated into concrete `PipeBlueprint` subclasses based on their `type`.--### Concepts--- Conceiving and naming concepts:- - Use PascalCase for concept names.-- - DO NOT use plurals in concept names (use `Item` not `Items`).-- - DO NOT use adjectives in concept names (use `Text`, not `LargeText`).-- - When a concept or concept reference is unqualified (no `domain.` prefix), it will be prefixed with the current blueprint `domain` during validation/expansion, unless its name corresponds to one of the native concepts.-- - Do not redefine native concepts (use them directly): `Text`, `Image`, `PDF`, `TextAndImages`, `Number`, `Page`, `LLMPrompt`.-- - Textual concepts refining `native.Text` are acceptable too, and you don't need to specify their structure: by default it'll be text.-- - Reuse existing concepts whenever possible. For instance if you take a `Poem` and make it dark, the result is a `Poem` too: you MUST NOT create a new `DarkPoem` concept in this kind of situation.-- - Concepts from the base library are also provided and must be used whenever possible:-- - `documents.TextAndImagesContent` = "A content that comprises text and images where the text can include local links to the images"-- - `images.VisualDescription` = "Visual description of something"-- - `images.ImgGenPrompt`:- - definition = "Prompt to generate an image"- - refines = "Text"-- - `images.Photo`:- - definition = "Photo"- - structure = "ImageContent"- - refines = "Image"-- - DO NOT create a concept qualified with circumstances like `InputPhoto`: it's just a `Photo`, it so happens that we're using it as input, OK, but that will be convened by the rest of the structure.--- Quick form (string):-- - `concept["ConceptName"] = "Definition of the concept"`-- - If no `structure` is specified and the name isn’t a known structure class, it implicitly refines `native.Text`.--- Full form (`ConceptBlueprint` from `pipelex/core/concept_factory.py`):-- - `definition: str` — required.-- - `structure: Optional[str]` — name of a registered `StructuredContent` subclass (typically defined in `pipelex_libraries/pipelines/<domain>.py`). If provided, it must exist in the class registry.-- - `refines: Union[str, List[str]]` — concept codes or names it refines. May be a single string or a list. Unqualified names are prefixed with the current `domain` automatically.-- - `domain: Optional[str]` — not required when provided via the library loader.--### Pipes: overall rules--Each entry under `pipe` is a dict that will be converted into a typed `PipeBlueprint` (see `pipelex/core/pipe_blueprint.py` for the base and factories in `pipe_controllers/` and `pipe_operators/`). Use the new format:--- `type: str` — the pipe class name, e.g., `PipeLLM`, `PipeOcr`, `PipeSequence`, `PipeParallel`, `PipeCondition`, `PipeFunc`, `PipeImgGen`.--- `definition: str` — human-readable description of the pipe’s responsibility.--- `inputs: Optional[Dict[str, str]]` — map of variable name -> concept code or name. Unqualified names are prefixed with `domain`.--- `output: str` — target concept code or name. Unqualified names are prefixed with `domain`.--Important:--- For controller pipes (`PipeSequence`, `PipeParallel`, `PipeCondition`), the parent pipe’s `inputs` must include all variables required by its sub-pipes or conditional branches that come from the initial working memory (i.e., not produced by previous steps).--- Multiplicity: some pipes and sub-steps can produce multiple outputs (see below). Plan result names accordingly.--### Pipe-specific blueprint schemas--Below are the additional fields expected per type, on top of the base fields above (`type`, `definition`, `inputs`, `output`). All fields are validated by their blueprint classes.--#### PipeLLM (operators/pipe_llm_factory.py)--- Prompt sources (choose at most one per group):-- - System prompt: one of `system_prompt`, `system_prompt_name`, `system_prompt_template`, `system_prompt_template_name`.-- - User prompt: one of `prompt`, `prompt_name`, `prompt_template`, `template_name`.--- LLM selection:-- - `llm: Optional[LLMSettingOrPresetId]` — model for text.-- - `llm_to_structure: Optional[LLMSettingOrPresetId]` — model for object structuring.--- Structuring:-- - `structuring_method: Optional[StructuringMethod]`-- - `prompt_template_to_structure: Optional[str]`-- - `system_prompt_to_structure: Optional[str]`--- Multiplicity:-- - `nb_output: Optional[int]` — exact number of outputs; mutually exclusive with `multiple_output`.-- - `multiple_output: Optional[bool]` — allow variable number of outputs.--Behavioral notes:--- If no system prompt is provided, the domain’s `system_prompt` (from the root blueprint) is used if available.--- Inputs that are `Image` concepts are automatically exposed as user images to the LLM; other inputs become text blocks.--- Prompt template insertion style for the agent:-- - Use block insertion (equivalent to `@name`) for long, multi-line stuff.-- - Use inline insertion (equivalent to `$name`) for short tokens embedded in sentences.-- - Do not synthesize literal markers; provide structured prompt fields and let the runtime render appropriately.--#### PipeOcr (operators/pipe_ocr_factory.py)--- Required input variable name: `ocr_input`.--- The `ocr_input` concept must be `Image` or `PDF`.--- Additional options:-- - `ocr_platform: Optional[OcrPlatform]` — defaults to Mistral.-- - `page_images: bool` — include detected images.-- - `page_image_captions: bool` — caption images.-- - `page_views: bool` — include rendered page views.-- - `page_views_dpi: Optional[int]` — DPI for page views; defaults to config.--- Output concept should be `Page`. The operational result is a list of `Page` items.--#### PipeSequence (controllers/pipe_sequence_factory.py)--- `steps: List[SubPipeBlueprint]`--- `SubPipeBlueprint` fields (controllers/sub_pipe_factory.py):-- - `pipe: str` — referenced pipe code to execute.-- - `result: Optional[str]` — name to assign to this step’s output in working memory.-- - Multiplicity (choose at most one): `nb_output: Optional[int]`, `multiple_output: Optional[bool]`.-- - Batch (both or none): `batch_over: Union[bool, str]` and `batch_as: Optional[str]`.--Rules:--- If `batch_over` is provided, `batch_as` is required; if `batch_as` is provided, `batch_over` is required.--- When batching, the step runs in parallel over each element of the specified list, and the `result` becomes a list.--#### PipeParallel (controllers/pipe_parallel_factory.py)--- `parallels: List[SubPipeBlueprint]` — each must specify `result`.--- `add_each_output: bool = True` — whether to add each parallel result separately to memory.--- `combined_output: Optional[str]` — optional combined output concept code; if provided without a domain prefix, it’s prefixed automatically.--Rules:--- At least one of `add_each_output` or `combined_output` must be set appropriately (cannot disable both behaviors).--#### PipeCondition (controllers/pipe_condition_factory.py)--- Choose exactly one of:-- - `expression: Optional[str]` — direct expression.-- - `expression_template: Optional[str]` — Jinja2 template expression.--- `pipe_map: Dict[str, str]` — map of evaluated expression values to pipe codes to execute.--- `default_pipe_code: Optional[str]` — optional fallback pipe code if no key matches.--- `add_alias_from_expression_to: Optional[str]` — if set, store the evaluated expression under this variable name in working memory.--### Input and output concepts resolution--- Unqualified concept names in `inputs` and `output` are automatically prefixed with the current `domain` during blueprint validation (see `PipeBlueprint.add_domain_prefix`).--- For `PipeParallel.combined_output`, the same domain-prefixing rule applies.--### Checklist for the agent--- Define `domain` and optional domain-level prompts (`system_prompt`, `system_prompt_to_structure`, `prompt_template_to_structure`).--- Create `concept` entries using either quick string form or full `ConceptBlueprint` with `definition`, optional `structure`, and `refines`.--- For each pipe under `pipe`:-- - Provide `type`, `definition`, `inputs`, `output`.-- - Add type-specific fields (see sections above).-- - Ensure controller pipes include all upstream inputs needed by nested steps/branches.-- - Set multiplicity (`nb_output`/`multiple_output`) and batch settings where needed.-- - For OCR, use `ocr_input` with `Image` or `PDF`, and set `output` to `Page`.--- Validate mentally that every referenced sub-pipe exists and every input it requires is either provided by the parent `inputs` or produced by a previous step.--### Minimal skeleton (object form)--The agent should generate a structure similar to the following (illustrative, not exhaustive):--```json-{- "domain": "my_domain",- "definition": "What this domain covers",- "system_prompt": "Global system prompt for LLM pipes (optional)",- "concept": {- "UserQuery": "Natural language question from a user",- "Answer": { "definition": "Answer text" }- },- "pipe": {- "answer_question": {- "type": "PipeLLM",- "definition": "Produce an answer from a question and context",- "inputs": { "question": "UserQuery", "context": "Text" },- "output": "Answer",- "prompt_template": "Analyze the question and context, then answer concisely.\n\n@question\n\n@context"- }- }-}-```diff --git a/.pipelex/create/structures.md b/.pipelex/create/structures.mdnew file mode 100644index 000000000..bd7df9127--- /dev/null+++ b/.pipelex/create/structures.md@@ -0,0 +1,396 @@+# Defining Structures for Pipelex Pipelines++This guide explains how to define schemas and enums for structured data generation in multi-step workflows. This step occurs between drafting pipelines (see `draft_pipelines.md`) and building them (see `build_pipelines.md`).++## When to Define Structures++After drafting your pipeline but before building it, review your draft for:++1. **Complex data that needs validation**: When your pipeline generates structured information that must follow specific formats+2. **Multi-field outputs**: When a single "result" actually contains multiple related pieces of information+3. **Constrained values**: When fields can only have specific allowed values+4. **Reusable data types**: When the same structure appears in multiple places++## Structure Definition Workflow++### Step 1: Identify Structure Needs from Draft++Review your pipeline draft and look for:++```markdown+# Example draft excerpt+LLM[analyze]: (document) → analysis_report+LLM[extract]: (page) → contact_info+LLM[classify]: (query) → support_category+```++Ask yourself:+- What specific fields does `analysis_report` contain?+- What format should `contact_info` follow?+- What are the valid values for `support_category`?++### Step 2: Design Schema Definitions++Create a TOML file defining your structures using the syntax from `structured_output_generator.py`.++## Structure Definition Syntax++### Basic Structure Definition++```toml+[structure.ModelName]+definition = "Clear description of what this represents"++[structure.ModelName.fields]+field_name = "Field description"+another_field = { type = "text", definition = "More detailed field", required = true }+```++### Field Types++| Type | Description | Example |+|------|-------------|---------|+| `text` | String content | `name = "Person's full name"` |+| `number` | Floating point | `score = { type = "number", definition = "Confidence score" }` |+| `integer` | Whole numbers | `count = { type = "integer", definition = "Number of items" }` |+| `boolean` | True/false | `active = { type = "boolean", definition = "Is active" }` |+| `list` | Array of items | `tags = { type = "list", item_type = "text", definition = "List of tags" }` |+| `dict` | Key-value pairs | `metadata = { type = "dict", key_type = "text", value_type = "text" }` |++### Field Configuration Options++```toml+# Required field (must be provided)+title = { type = "text", definition = "Document title", required = true }++# Optional field with default value+priority = { type = "integer", definition = "Priority level", default = 1 }++# List with specific item type+categories = { type = "list", item_type = "text", definition = "Document categories" }+```++## Enum Definition Syntax++### Simple Enum (List of Values)++```toml+[enum.Priority]+definition = "Task priority levels"+values = ["low", "medium", "high", "urgent"]+```++### Descriptive Enum (Key-Value Pairs)++```toml+[enum.DocumentType]+definition = "Types of documents we can process"++[enum.DocumentType.values]+technical = "Technical documentation and manuals"+legal = "Legal contracts and agreements"+financial = "Financial reports and statements"+marketing = "Marketing materials and content"+```++### Using Enums in Structures++```toml+# Reference defined enum+status = { type = "Priority", definition = "Current priority level" }++# Inline choices for simple cases+size = { choices = ["S", "M", "L", "XL"], definition = "Available sizes" }+```++## Complete Example: Document Analysis Pipeline++### From Draft to Structures++**Original Draft Step:**+```markdown+### Step 2: Extract structured information from each page+LLM[extract]: (page) → page_info++### Step 3: Classify document type and priority+LLM[classify]: (document_content) → document_metadata++### Step 4: Generate comprehensive analysis+LLM[analyze]: (page_info_list, document_metadata) → final_analysis+```++**Structure Definition (document_analysis_structures.toml):**++```toml+# Enums for constrained values+[enum.DocumentType]+definition = "Types of documents we can process"+values = ["technical", "legal", "financial", "marketing", "general"]++[enum.Priority]+definition = "Document processing priority"+values = ["low", "medium", "high", "urgent"]++[enum.ConfidenceLevel]+definition = "AI confidence in extraction"++[enum.ConfidenceLevel.values]+low = "Low confidence - manual review recommended"+medium = "Medium confidence - spot check suggested"+high = "High confidence - likely accurate"++# Structures for structured data+[structure.PageInfo]+definition = "Structured information extracted from a document page"++[structure.PageInfo.fields]+page_number = { type = "integer", definition = "Page number in document", required = true }+main_content = "Primary text content of the page"+key_points = { type = "list", item_type = "text", definition = "Important points found on page" }+entities = { type = "list", item_type = "text", definition = "Named entities (people, organizations, etc.)" }+confidence = { type = "ConfidenceLevel", definition = "Confidence in extraction quality" }++[structure.DocumentMetadata]+definition = "High-level document classification and properties"++[structure.DocumentMetadata.fields]+document_type = { type = "DocumentType", definition = "Classified document type", required = true }+priority = { type = "Priority", definition = "Processing priority", required = true }+title = "Inferred document title"+summary = "Brief document summary"+page_count = { type = "integer", definition = "Total number of pages" }+language = { choices = ["en", "fr", "es", "de"], definition = "Document language" }++[structure.DocumentAnalysis]+definition = "Comprehensive analysis combining all extracted information"++[structure.DocumentAnalysis.fields]+document_metadata = { type = "DocumentMetadata", definition = "Document classification info", required = true }+page_analyses = { type = "list", item_type = "PageInfo", definition = "Analysis for each page", required = true }+overall_insights = { type = "list", item_type = "text", definition = "Key insights across entire document" }+recommendations = { type = "list", item_type = "text", definition = "Recommended actions based on analysis" }+confidence_score = { type = "number", definition = "Overall confidence in analysis (0.0-1.0)" }+```++## Best Practices++### 1. Semantic Naming+- Use **PascalCase** for schema and enum names: `DocumentAnalysis`, `Priority`+- Use **snake_case** for field names: `page_number`, `confidence_score`+- Avoid plurals in schema names: `PageInfo` not `PageInfos`+- Avoid circumstantial names: `Analysis` not `FinalAnalysis`++### 2. Clear Definitions+- Every schema and enum should have a `definition`+- Field descriptions should explain what the field contains, not how it's used+- Use complete sentences for definitions++### 3. Appropriate Constraints+- Mark fields as `required = true` only when they're always expected+- Use enums for fields with limited, known values+- Use inline `choices` for simple, one-off constraints+- Set reasonable defaults where applicable++### 4. Logical Grouping+- Group related fields into the same schema+- Don't create overly granular schemas for simple data+- Consider if a field should be separate or part of a larger structure++### 5. Reusability+- Design schemas to be reusable across different pipes+- Create enums for values that appear in multiple contexts+- Avoid pipeline-specific naming in reusable structures++## Integration with Pipeline Building++### Updating Concept Definitions++After defining structures, update your pipeline blueprint concepts to reference the structure definitions:++**Simple approach (structure name matches concept name):**+```json+"concept": {+ "PageInfo": "Information extracted from a page",+ "DocumentMetadata": "Document classification data",+ "DocumentAnalysis": "Complete document analysis"+}+```++**Structure reference approaches:**+```json+"concept": {+ "PageInfo": {+ "definition": "Structured information extracted from a document page",+ "structure": "PageInfo"+ },+ "DocumentMetadata": {+ "definition": "High-level document classification and properties",+ "structure": {+ "document_type": {+ "type": "DocumentType",+ "definition": "Classified document type",+ "required": true+ },+ "priority": {+ "type": "Priority",+ "definition": "Processing priority",+ "required": true+ },+ "title": "Inferred document title",+ "summary": "Brief document summary",+ "page_count": {+ "type": "integer",+ "definition": "Total number of pages"+ },+ "language": {+ "choices": ["en", "fr", "es", "de"],+ "definition": "Document language"+ }+ }+ },+ "DocumentAnalysis": {+ "definition": "Comprehensive analysis combining all extracted information",+ "structure": "DocumentAnalysis"+ }+}+```++**Structure attribute options:**+- **String reference**: `"structure": "ClassName"` - References a predefined structure class+- **Inline definition**: `"structure": { ... }` - Defines fields directly within the concept+- **Auto-detection**: Omit `structure` entirely - System looks for StructuredContent subclass with matching concept name or falls back on `Text` (i.e. not structured)++### When to Use Each Approach++**Use string reference when:**+- Structure is reusable across multiple concepts or pipelines+- Structure is complex and benefits from being defined in a separate TOML file+- You want to maintain separation between concept definitions and structure details++**Use inline definition when:**+- Structure is specific to this one concept+- Structure is simple and doesn't warrant a separate definition+- You want to keep concept and structure together for clarity+- Rapid prototyping where you don't want to manage separate files++**Use auto-detection when:**+- Concept name exactly matches an existing StructuredContent class+- Working with simple, standard structures that are already registered++### Generating Python Classes++**From TOML files:**+```python+from pipelex.create.structured_output_generator import generate_structured_outputs_from_toml_file++# Generate Python classes from TOML definitions+generate_structured_outputs_from_toml_file(+ "document_analysis_structures.toml",+ "document_analysis.py"+)+```++**From inline definitions (for concept integration):**+```python+from pipelex.create.structured_output_generator import generate_structured_output_from_inline_definition++# Generate from inline field definitions (as used in concept.structure)+fields_def = {+ "document_type": {+ "type": "DocumentType",+ "definition": "Classified document type",+ "required": True+ },+ "priority": {+ "type": "Priority",+ "definition": "Processing priority",+ "required": True+ },+ "title": "Inferred document title",+ "page_count": {+ "type": "integer",+ "definition": "Total number of pages"+ },+ "language": {+ "choices": ["en", "fr", "es", "de"],+ "definition": "Document language"+ }+}++enums = {+ "DocumentType": {+ "definition": "Types of documents we can process",+ "values": ["technical", "legal", "financial", "marketing", "general"]+ },+ "Priority": {+ "definition": "Document processing priority",+ "values": ["low", "medium", "high", "urgent"]+ }+}++python_code = generate_structured_output_from_inline_definition(+ "DocumentMetadata",+ fields_def,+ enums+)+```++## Validation Checklist++Before finalizing your structure definitions:++- [ ] All structure names use PascalCase and are singular+- [ ] All field names use snake_case+- [ ] Every structure and enum has a clear `definition`+- [ ] Required fields are marked appropriately+- [ ] Enums are used for constrained values with known options+- [ ] Field types match the expected data format+- [ ] Structures group related fields logically+- [ ] Structure names are semantic (describe what they are, not how they're used)+- [ ] No native concepts (Text, Image, PDF) are redefined+- [ ] List item types are specified correctly+- [ ] Dictionary key and value types are appropriate++## Common Patterns++### Contact Information+```toml+[structure.ContactInfo]+definition = "Contact information extracted from document"++[structure.ContactInfo.fields]+name = "Full name of contact"+email = "Email address"+phone = "Phone number"+company = "Company or organization name"+address = "Physical address"+```++### Analysis Results+```toml+[enum.Sentiment]+definition = "Sentiment analysis results"+values = ["positive", "negative", "neutral"]++[structure.TextAnalysis]+definition = "Results of text analysis"++[structure.TextAnalysis.fields]+sentiment = { type = "Sentiment", definition = "Overall sentiment", required = true }+key_themes = { type = "list", item_type = "text", definition = "Main themes identified" }+confidence = { type = "number", definition = "Analysis confidence (0.0-1.0)" }+word_count = { type = "integer", definition = "Total word count" }+```++### Classification Results+```toml+[structure.Classification]+definition = "Multi-class classification result"++[structure.Classification.fields]+predicted_class = { type = "text", definition = "Most likely class", required = true }+confidence_scores = { type = "dict", key_type = "text", value_type = "number", definition = "Confidence for each class" }+threshold_met = { type = "boolean", definition = "Whether prediction meets confidence threshold" }+```++This structured approach ensures your pipeline generates well-validated, type-safe data that can be reliably processed by downstream systems.diff --git a/pipelex/core/concept_factory.py b/pipelex/core/concept_factory.pyindex 1f5cf7183..19d1e6a53 100644--- a/pipelex/core/concept_factory.py+++ b/pipelex/core/concept_factory.py@@ -8,6 +8,7 @@ from pipelex.core.concept_code_factory import ConceptCodeFactoryfrom pipelex.core.concept_native import NativeConcept, NativeConceptClassfrom pipelex.core.domain import SpecialDomainfrom pipelex.core.stuff_content import TextContent+from pipelex.create.structured_output_generator import generate_structured_output_from_inline_definitionfrom pipelex.exceptions import ConceptFactoryError, StructureClassErrorfrom pipelex.hub import get_class_registry@@ -16,7 +17,7 @@ class ConceptBlueprint(BaseModel):model_config = ConfigDict(extra="forbid")definition: str- structure: Optional[str] = None+ structure: Optional[Union[str, Dict[str, Any]]] = Nonerefines: Union[str, List[str]] = Field(default_factory=list)domain: Optional[str] = None@@ -123,12 +124,34 @@ class ConceptFactory:) -> Concept:structure_class_name: strif structure := concept_blueprint.structure:- # structure is set explicitly- if not Concept.is_valid_structure_class(structure_class_name=structure):- raise StructureClassError(- f"Structure class '{structure}' set for concept '{code}' in domain '{domain}' is not a registered subclass of StuffContent"- )- structure_class_name = structure+ if isinstance(structure, str):+ # structure is set explicitly as a class name reference+ if not Concept.is_valid_structure_class(structure_class_name=structure):+ raise StructureClassError(+ f"Structure class '{structure}' set for concept '{code}' in domain '{domain}' is not a registered subclass of StuffContent"+ )+ structure_class_name = structure+ else:+ # structure is defined inline - generate Python class dynamically+ structure_class_name = code+ try:+ # Generate Python class from inline definition+ python_code = generate_structured_output_from_inline_definition(+ class_name=code,+ fields_def=structure, # type: ignore[arg-type]+ enums=None, # TODO: Handle enums if needed in the future+ )++ # Execute the generated Python code to register the class+ exec_globals: Dict[str, Any] = {}+ exec(python_code, exec_globals)++ # Get the generated class and register it+ generated_class = exec_globals[code]+ get_class_registry().register_class(generated_class)++ except Exception as exc:+ raise ConceptFactoryError(f"Error generating structure class for concept '{code}' in domain '{domain}': {exc}") from excelif Concept.is_valid_structure_class(structure_class_name=code):# structure is set implicitly, by the concept's codestructure_class_name = codediff --git a/pipelex/create/structured_output_generator.py b/pipelex/create/structured_output_generator.pyindex cff3738af..f4a75d771 100644--- a/pipelex/create/structured_output_generator.py+++ b/pipelex/create/structured_output_generator.py@@ -46,10 +46,10 @@ class StructuredOutputGenerator:self.enum_definitions: Dict[str, Dict[str, Any]] = {} # Store enum definitionsdef generate_from_toml(self, toml_content: str) -> str:- """Generate Python module content from TOML structured output definitions.+ """Generate Python module content from TOML structure and enum definitions.Args:- toml_content: TOML content containing schema and enum definitions+ toml_content: TOML content containing structure and enum definitionsReturns:Generated Python module content@@ -62,18 +62,18 @@ class StructuredOutputGenerator:enum_defs = data["enum"]for enum_name, enum_def in enum_defs.items(): # type: ignore[attr-defined,union-attr]self.enum_definitions[str(enum_name)] = dict(enum_def) # type: ignore[arg-type]- enum_code = self._generate_enum(str(enum_name), dict(enum_def)) # type: ignore[arg-type]+ enum_code = self.generate_enum(str(enum_name), dict(enum_def)) # type: ignore[arg-type]enums.append(enum_code)- # Process schemas- if "schema" not in data:- raise ValueError("TOML must contain a 'schema' section")+ # Process structures+ if "structure" not in data:+ raise ValueError("TOML must contain a 'structure' section")- schemas = data["schema"]+ structures = data["structure"]classes: List[str] = []- for class_name, schema_def in schemas.items(): # type: ignore[attr-defined,union-attr]- class_code = self._generate_class(str(class_name), dict(schema_def)) # type: ignore[arg-type]+ for class_name, structure_def in structures.items(): # type: ignore[attr-defined,union-attr]+ class_code = self.generate_class(str(class_name), dict(structure_def)) # type: ignore[arg-type]classes.append(class_code)# Generate the complete module@@ -90,7 +90,7 @@ class StructuredOutputGenerator:return f"{imports_section}\n\n\n{definitions_section}\n"- def _generate_enum(self, enum_name: str, enum_def: Dict[str, Any]) -> str:+ def generate_enum(self, enum_name: str, enum_def: Dict[str, Any]) -> str:"""Generate an enum class definition.Args:@@ -131,18 +131,18 @@ class StructuredOutputGenerator:values_code = "\n".join(value_definitions)return enum_header + "\n" + values_code- def _generate_class(self, class_name: str, schema_def: Dict[str, Any]) -> str:+ def generate_class(self, class_name: str, structure_def: Dict[str, Any]) -> str:"""Generate a single class definition.Args:class_name: Name of the class- schema_def: Schema definition from TOML+ structure_def: Structure definition from TOMLReturns:Generated class code"""- definition = schema_def.get("definition", f"Generated {class_name} class")- fields = schema_def.get("fields", {})+ definition = structure_def.get("definition", f"Generated {class_name} class")+ fields = structure_def.get("fields", {})# Generate class headerclass_header = f'class {class_name}(StructuredContent):\n """{definition}"""\n'@@ -283,7 +283,7 @@ def generate_structured_outputs_from_toml_file(toml_file_path: str, output_file_"""Generate structured output Python module from TOML file.Args:- toml_file_path: Path to input TOML file+ toml_file_path: Path to input TOML file containing structure definitionsoutput_file_path: Path to output Python file"""with open(toml_file_path, "r", encoding="utf-8") as f:@@ -300,10 +300,51 @@ def generate_structured_outputs_from_toml_string(toml_content: str) -> str:"""Generate structured output Python code from TOML string.Args:- toml_content: TOML content as string+ toml_content: TOML content as string containing structure definitionsReturns:Generated Python module content"""generator = StructuredOutputGenerator()return generator.generate_from_toml(toml_content)+++def generate_structured_output_from_inline_definition(+ class_name: str, fields_def: Dict[str, Any], enums: Optional[Dict[str, Dict[str, Any]]] = None+) -> str:+ """Generate structured output Python code from inline field definitions.++ Args:+ class_name: Name of the class to generate+ fields_def: Dictionary of field definitions (same format as TOML structure.fields)+ enums: Optional dictionary of enum definitions to include++ Returns:+ Generated Python module content+ """+ generator = StructuredOutputGenerator()++ # Add any provided enums+ if enums:+ for enum_name, enum_def in enums.items():+ generator.enum_definitions[enum_name] = enum_def++ # Create a structure definition from the inline fields+ structure_def = {"definition": f"Generated {class_name} structure", "fields": fields_def}++ # Generate the class+ class_code = generator.generate_class(class_name, structure_def)++ # Generate enums if any+ enum_codes: List[str] = []+ if enums:+ for enum_name, enum_def in enums.items():+ enum_code = generator.generate_enum(enum_name, enum_def)+ enum_codes.append(enum_code)++ # Combine everything+ imports_section = "\n".join(sorted(generator.imports))+ all_definitions: List[str] = enum_codes + [class_code] if enum_codes else [class_code]+ definitions_section = "\n\n\n".join(all_definitions)++ return f"{imports_section}\n\n\n{definitions_section}\n"diff --git a/tests/integration/pipelex/test_inline_structure_concepts.py b/tests/integration/pipelex/test_inline_structure_concepts.pynew file mode 100644index 000000000..ee6c326fd--- /dev/null+++ b/tests/integration/pipelex/test_inline_structure_concepts.py@@ -0,0 +1,178 @@+"""Integration tests for inline structure definitions in concepts."""++import pytest++from pipelex.core.concept_factory import ConceptBlueprint, ConceptFactory+from pipelex.core.stuff_content import StructuredContent+from pipelex.exceptions import ConceptFactoryError, StructureClassError+from pipelex.hub import get_class_registry+++class TestInlineStructureConcepts:+ """Test inline structure definitions in concept blueprints."""++ def test_inline_structure_definition_creation(self):+ """Test that concepts with inline structure definitions are created correctly."""+ # Define inline structure with mixed syntax+ inline_structure = {+ "dominant_feature": {"type": "text", "definition": "The most important feature", "required": True},+ "visual_elements": {"type": "text", "definition": "Key visual elements", "required": True},+ "composition": "Analysis of the image composition",+ "color_palette": "Description of the main colors",+ "mood_atmosphere": "The overall mood or atmosphere",+ }++ # Create concept blueprint with inline structure+ blueprint = ConceptBlueprint(definition="Analysis of a photo's visual content", structure=inline_structure)++ # Create concept from blueprint+ concept = ConceptFactory.make_concept_from_blueprint(domain="test_domain", code="TestFeatureAnalysis", concept_blueprint=blueprint)++ # Verify concept properties+ assert concept.code == "test_domain.TestFeatureAnalysis"+ assert concept.definition == "Analysis of a photo's visual content"+ assert concept.structure_class_name == "TestFeatureAnalysis"+ assert concept.domain == "test_domain"++ # Verify the generated class is registered and accessible+ assert get_class_registry().has_class("TestFeatureAnalysis")+ generated_class = get_class_registry().get_required_subclass("TestFeatureAnalysis", StructuredContent)++ # Verify class structure+ assert issubclass(generated_class, StructuredContent)+ assert generated_class.__name__ == "TestFeatureAnalysis"++ # Test instantiation of generated class+ instance = generated_class(+ dominant_feature="A bright red car", # pyright: ignore[reportCallIssue]+ visual_elements="Car, road, trees, sky", # pyright: ignore[reportCallIssue]+ composition="Central composition with car in focus", # pyright: ignore[reportCallIssue]+ color_palette="Red, green, blue, white", # pyright: ignore[reportCallIssue]+ )++ assert instance.dominant_feature == "A bright red car" # type: ignore+ assert instance.visual_elements == "Car, road, trees, sky" # type: ignore+ assert instance.composition == "Central composition with car in focus" # type: ignore+ assert instance.color_palette == "Red, green, blue, white" # type: ignore+ assert instance.mood_atmosphere is None # type: ignore++ def test_string_reference_structure_definition(self):+ """Test that string reference structure definitions still work."""+ # Create blueprint with string reference+ blueprint = ConceptBlueprint(definition="Test with string reference", structure="TextContent")++ # Create concept from blueprint+ concept = ConceptFactory.make_concept_from_blueprint(domain="test_domain", code="TestStringRef", concept_blueprint=blueprint)++ # Verify concept properties+ assert concept.code == "test_domain.TestStringRef"+ assert concept.definition == "Test with string reference"+ assert concept.structure_class_name == "TextContent"++ def test_auto_detection_structure(self):+ """Test auto-detection when no structure is specified."""+ # Create blueprint without structure+ blueprint = ConceptBlueprint(definition="Test auto-detection")++ # Create concept from blueprint+ concept = ConceptFactory.make_concept_from_blueprint(domain="test_domain", code="TestAutoDetect", concept_blueprint=blueprint)++ # Should default to TextContent since TestAutoDetect is not a registered class+ assert concept.structure_class_name == "TextContent"++ def test_inline_structure_with_complex_types(self):+ """Test inline structure with complex field types."""+ inline_structure = {+ "title": {"type": "text", "definition": "Document title", "required": True},+ "tags": {"type": "list", "item_type": "text", "definition": "List of tags"},+ "metadata": {"type": "dict", "key_type": "text", "value_type": "text", "definition": "Metadata dictionary"},+ "priority": {"choices": ["low", "medium", "high"], "definition": "Priority level"},+ "page_count": {"type": "integer", "definition": "Number of pages"},+ "is_active": {"type": "boolean", "definition": "Whether document is active"},+ }++ blueprint = ConceptBlueprint(definition="Complex document structure", structure=inline_structure)++ concept = ConceptFactory.make_concept_from_blueprint(domain="test_domain", code="ComplexDocument", concept_blueprint=blueprint)++ # Verify concept creation+ assert concept.code == "test_domain.ComplexDocument"+ assert concept.structure_class_name == "ComplexDocument"++ # Verify the generated class works+ generated_class = get_class_registry().get_required_subclass("ComplexDocument", StructuredContent)++ # Test instantiation with complex types+ instance = generated_class(+ title="Test Document",+ tags=["test", "document"],+ metadata={"author": "Test Author", "version": "1.0"},+ priority="high",+ page_count=42,+ is_active=True,+ )++ assert instance.title == "Test Document"+ assert instance.tags == ["test", "document"]+ assert instance.metadata == {"author": "Test Author", "version": "1.0"}+ assert instance.priority == "high"+ assert instance.page_count == 42+ assert instance.is_active is True++ def test_invalid_string_reference_raises_error(self):+ """Test that invalid string references raise appropriate errors."""+ blueprint = ConceptBlueprint(definition="Test invalid reference", structure="NonExistentClass")++ with pytest.raises(StructureClassError, match="Structure class 'NonExistentClass'"):+ ConceptFactory.make_concept_from_blueprint(domain="test_domain", code="TestInvalid", concept_blueprint=blueprint)++ def test_inline_structure_generation_error_handling(self):+ """Test error handling when inline structure generation fails."""+ # Create invalid structure definition that should cause generation to fail+ invalid_structure = {+ "invalid_field": {"type": "nonexistent_type", "definition": "This should fail"},+ }++ blueprint = ConceptBlueprint(definition="Test error handling", structure=invalid_structure)++ with pytest.raises(ConceptFactoryError, match="Error generating structure class"):+ ConceptFactory.make_concept_from_blueprint(domain="test_domain", code="TestError", concept_blueprint=blueprint)++ def test_multiple_inline_structures_do_not_conflict(self):+ """Test that multiple inline structures with same field names don't conflict."""+ # First structure+ structure1 = {+ "name": {"type": "text", "definition": "Person name", "required": True},+ "age": {"type": "integer", "definition": "Person age"},+ }++ blueprint1 = ConceptBlueprint(definition="Person information", structure=structure1)+ concept1 = ConceptFactory.make_concept_from_blueprint(domain="test_domain", code="Person", concept_blueprint=blueprint1)++ # Second structure with same field names but different context+ structure2 = {+ "name": {"type": "text", "definition": "Product name", "required": True},+ "age": {"type": "integer", "definition": "Product age in days"},+ }++ blueprint2 = ConceptBlueprint(definition="Product information", structure=structure2)+ concept2 = ConceptFactory.make_concept_from_blueprint(domain="test_domain", code="Product", concept_blueprint=blueprint2)++ # Both should be created successfully+ assert concept1.structure_class_name == "Person"+ assert concept2.structure_class_name == "Product"++ # Both classes should be registered and independent+ person_class = get_class_registry().get_required_subclass("Person", StructuredContent)+ product_class = get_class_registry().get_required_subclass("Product", StructuredContent)++ assert person_class != product_class++ # Test instances are independent+ person = person_class(name="John Doe", age=30)+ product = product_class(name="Widget", age=365)++ assert person.name == "John Doe"+ assert product.name == "Widget"+ assert person.age == 30+ assert product.age == 365diff --git a/tests/unit/pipelex/create/test_structured_output_generator.py b/tests/unit/pipelex/create/test_structured_output_generator.pyindex 6cdbc7678..193e85fce 100644--- a/tests/unit/pipelex/create/test_structured_output_generator.py+++ b/tests/unit/pipelex/create/test_structured_output_generator.py@@ -6,18 +6,19 @@ from pipelex import pretty_printfrom pipelex.create.structured_output_generator import (FieldType,StructuredOutputGenerator,+ generate_structured_output_from_inline_definition,generate_structured_outputs_from_toml_string,)class TestStructuredOutputGenerator:- def test_simple_schema_generation(self):- """Test generation of a simple schema with basic fields."""+ def test_simple_structure_generation(self):+ """Test generation of a simple structure with basic fields."""toml_content = """-[schema.TestModel]+[structure.TestModel]definition = "A test model"-[schema.TestModel.fields]+[structure.TestModel.fields]name = { type = "text", definition = "Name field", required = true }age = { type = "integer", definition = "Age field" }active = { type = "boolean", definition = "Active status", default = true }@@ -42,10 +43,10 @@ active = { type = "boolean", definition = "Active status", default = true }def test_simplified_field_syntax(self):"""Test the simplified field syntax where a field is just a string definition."""toml_content = """-[schema.SimpleModel]+[structure.SimpleModel]definition = "A model with simplified field syntax"-[schema.SimpleModel.fields]+[structure.SimpleModel.fields]dominant_feature = "The most important or dominant feature in the image"simple_text = "Just some text""""@@ -62,10 +63,10 @@ simple_text = "Just some text"def test_complex_types_generation(self):"""Test generation with complex types like lists and dicts."""toml_content = """-[schema.ComplexModel]+[structure.ComplexModel]definition = "A model with complex types"-[schema.ComplexModel.fields]+[structure.ComplexModel.fields]tags = { type = "list", item_type = "text", definition = "List of tags" }metadata = { type = "dict", key_type = "text", value_type = "text", definition = "Metadata dictionary" }scores = { type = "list", item_type = "number", definition = "List of scores", required = true }@@ -81,19 +82,19 @@ scores = { type = "list", item_type = "number", definition = "List of scores", rassert 'metadata: Optional[Dict[str, str]] = Field(default=None, description="Metadata dictionary")' in resultassert 'scores: List[float] = Field(..., description="List of scores")' in result- def test_multiple_schemas(self):- """Test generation of multiple schemas in one TOML."""+ def test_multiple_structures(self):+ """Test generation of multiple structures in one TOML."""toml_content = """-[schema.FirstModel]+[structure.FirstModel]definition = "First model"-[schema.FirstModel.fields]+[structure.FirstModel.fields]title = { type = "text", definition = "Title", required = true }-[schema.SecondModel]+[structure.SecondModel]definition = "Second model"-[schema.SecondModel.fields]+[structure.SecondModel.fields]content = { type = "text", definition = "Content" }"""@@ -108,10 +109,10 @@ content = { type = "text", definition = "Content" }assert '"""First model"""' in resultassert '"""Second model"""' in result- def test_empty_schema(self):- """Test generation of schema with no fields."""+ def test_empty_structure(self):+ """Test generation of structure with no fields."""toml_content = """-[schema.EmptyModel]+[structure.EmptyModel]definition = "An empty model""""@@ -128,10 +129,10 @@ definition = "An empty model"def test_required_field_syntax(self):"""Test that fields can be marked as required."""toml_content = """-[schema.RequiredFieldsModel]+[structure.RequiredFieldsModel]definition = "Model with required fields"-[schema.RequiredFieldsModel.fields]+[structure.RequiredFieldsModel.fields]title = { type = "text", definition = "Required title", required = true }optional_field = { type = "text", definition = "Optional field" }"""@@ -148,10 +149,10 @@ optional_field = { type = "text", definition = "Optional field" }def test_custom_types(self):"""Test handling of custom/unknown types."""toml_content = """-[schema.CustomTypeModel]+[structure.CustomTypeModel]definition = "Model with custom types"-[schema.CustomTypeModel.fields]+[structure.CustomTypeModel.fields]custom_field = { type = "CustomClass", definition = "Custom class field" }"""@@ -163,8 +164,8 @@ custom_field = { type = "CustomClass", definition = "Custom class field" }assert 'custom_field: Optional[CustomClass] = Field(default=None, description="Custom class field")' in result- def test_missing_schema_section_raises_error(self):- """Test that missing schema section raises appropriate error."""+ def test_missing_structure_section_raises_error(self):+ """Test that missing structure section raises appropriate error."""toml_content = """[other_section]value = "test"@@ -174,16 +175,16 @@ value = "test"pretty_print(toml_content, title="Source TOML (should raise error)")- with pytest.raises(ValueError, match="TOML must contain a 'schema' section"):+ with pytest.raises(ValueError, match="TOML must contain a 'structure' section"):generator.generate_from_toml(toml_content)def test_generate_from_string_convenience_function(self):"""Test the convenience function for generating from string."""toml_content = """-[schema.ConvenienceTest]+[structure.ConvenienceTest]definition = "Test convenience function"-[schema.ConvenienceTest.fields]+[structure.ConvenienceTest.fields]value = { type = "text", definition = "Test value", required = true }"""@@ -198,10 +199,10 @@ value = { type = "text", definition = "Test value", required = true }def test_field_type_enum_usage(self):"""Test that FieldType enum values are properly handled."""toml_content = """-[schema.TypeMappingTest]+[structure.TypeMappingTest]definition = "Test all type mappings"-[schema.TypeMappingTest.fields]+[structure.TypeMappingTest.fields]text_field = { type = "text", definition = "Text field" }number_field = { type = "number", definition = "Number field" }integer_field = { type = "integer", definition = "Integer field" }@@ -226,10 +227,10 @@ dict_field = { type = "dict", key_type = "text", value_type = "integer", definitdef test_mixed_field_syntax(self):"""Test mixing simplified syntax with full field definitions."""toml_content = """-[schema.MixedSyntaxModel]+[structure.MixedSyntaxModel]definition = "Model with mixed field syntax"-[schema.MixedSyntaxModel.fields]+[structure.MixedSyntaxModel.fields]# Simplified syntaxsimple_description = "A simple text description"# Full syntax@@ -254,10 +255,10 @@ another_simple = "Another simple field"definition = "Possible states of an order"values = ["pending", "processing", "shipped", "delivered", "cancelled"]-[schema.Order]+[structure.Order]definition = "An order in the system"-[schema.Order.fields]+[structure.Order.fields]order_id = "Unique identifier for the order"status = { type = "OrderStatus", definition = "Current status of the order", required = true }customer_name = "Name of the customer"@@ -295,10 +296,10 @@ medium = "Medium Priority"high = "High Priority"urgent = "Urgent - Handle Immediately"-[schema.Task]+[structure.Task]definition = "A task to be completed"-[schema.Task.fields]+[structure.Task.fields]title = "Task title"priority = { type = "Priority", definition = "How urgent is this task", required = true }assigned_to = "Person responsible for the task"@@ -324,10 +325,10 @@ assigned_to = "Person responsible for the task"def test_inline_choices(self):"""Test generation with inline choices (Literal type)."""toml_content = """-[schema.Product]+[structure.Product]definition = "A product in our catalog"-[schema.Product.fields]+[structure.Product.fields]name = "Product name"category = { choices = ["electronics", "clothing", "food", "books"], definition = "Product category", required = true }size = { choices = ["XS", "S", "M", "L", "XL"], definition = "Size of the product" }@@ -355,10 +356,10 @@ values = ["red", "blue", "green", "yellow", "black", "white"]definition = "How the product will be shipped"values = ["standard", "express", "overnight", "pickup"]-[schema.ProductVariant]+[structure.ProductVariant]definition = "A specific variant of a product"-[schema.ProductVariant.fields]+[structure.ProductVariant.fields]sku = "Stock keeping unit"color = { type = "Color", definition = "Color of this variant" }size = { choices = ["S", "M", "L"], definition = "Size options" }@@ -392,10 +393,10 @@ availability = { choices = ["in_stock", "out_of_stock", "pre_order"], definitiondefinition = "Dietary restrictions and preferences"values = ["vegetarian", "vegan", "gluten_free", "dairy_free", "nut_free"]-[schema.MenuItem]+[structure.MenuItem]definition = "An item on the restaurant menu"-[schema.MenuItem.fields]+[structure.MenuItem.fields]name = "Name of the dish"dietary_info = { type = "list", item_type = "DietaryRestriction", definition = "Dietary accommodations" }tags = { type = "list", item_type = "text", definition = "General tags" }@@ -415,3 +416,41 @@ tags = { type = "list", item_type = "text", definition = "General tags" }# Check list with enum typeassert 'dietary_info: Optional[List[DietaryRestriction]] = Field(default=None, description="Dietary accommodations")' in resultassert 'tags: Optional[List[str]] = Field(default=None, description="General tags")' in result++ def test_inline_structure_definition(self):+ """Test generating structure from inline field definitions (for concept integration)."""+ fields_def = {+ "document_type": {"type": "DocumentType", "definition": "Classified document type", "required": True},+ "priority": {"type": "Priority", "definition": "Processing priority", "required": True},+ "title": "Inferred document title",+ "summary": "Brief document summary",+ "page_count": {"type": "integer", "definition": "Total number of pages"},+ "language": {"choices": ["en", "fr", "es", "de"], "definition": "Document language"},+ }++ enums = {+ "DocumentType": {+ "definition": "Types of documents we can process",+ "values": ["technical", "legal", "financial", "marketing", "general"],+ },+ "Priority": {"definition": "Document processing priority", "values": ["low", "medium", "high", "urgent"]},+ }++ result = generate_structured_output_from_inline_definition("DocumentMetadata", fields_def, enums)++ pretty_print(fields_def, title="Inline Field Definition")+ pretty_print(result, title="Generated Result")++ # Check enum generation+ assert "class DocumentType(str, Enum):" in result+ assert "class Priority(str, Enum):" in result+ assert 'TECHNICAL = "technical"' in result+ assert 'LOW = "low"' in result++ # Check structure generation+ assert "class DocumentMetadata(StructuredContent):" in result+ assert 'document_type: DocumentType = Field(..., description="Classified document type")' in result+ assert 'priority: Priority = Field(..., description="Processing priority")' in result+ assert 'title: Optional[str] = Field(default=None, description="Inferred document title")' in result+ assert 'page_count: Optional[int] = Field(default=None, description="Total number of pages")' in result+ assert "language: Optional[Literal['en', 'fr', 'es', 'de']] = Field(default=None, description=\"Document language\")" in result