Build AI methods with your coding agent, then run them anywhere: as an MCP for chatbots, as a webapp for people, or via API for your software.
feat: CSV input & output for pipelines (#955)
Louis Choquelfa15d15c3585
diff --git a/CHANGELOG.md b/CHANGELOG.mdindex d5cdab3f8..127c04d51 100644--- a/CHANGELOG.md+++ b/CHANGELOG.md@@ -4,6 +4,7 @@### Added+- **CSV input & output for pipelines.** A `.csv` file can now be read directly into a typed `ListContent[YourConcept]` and a list output can be written back out as CSV. On input, an `inputs.json` reference whose `url` ends in a tabular suffix under a flat structured concept (e.g. `{ "people": { "concept": "demo.Person", "content": { "url": "people.csv" } } }`) loads as one list whose items are instances of that concept — column headers must match the concept's field names, cells are coerced via Pydantic (`birth_year` → `int`, ISO dates → `date`), and an empty cell becomes `None` (so the field must be optional). On output, the new `--save-csv <path>` flag on `pipelex run pipe|bundle|method` writes the main stuff to a literal, cwd-relative CSV path when that output is a flat list. The row concept must be flat (scalar fields only — `text`/`integer`/`number`/`boolean`/`date`, plus optionals and choice-constrained scalars); a nested/list/dict/concept-typed field is rejected with a clear error naming the offending field. CSV reading uses the Python standard library; `.xlsx` is recognized but routed to a clear "needs `pipelex[tabular]`" message (the Excel backend is not built yet). v1 reads local file paths only — a remote `url` (`http(s)`/`s3`/`gs`/…) with a tabular suffix is actively rejected. See the [CSV Input & Output guide](https://docs.pipelex.com/latest/building-methods/pipes/csv-input-and-output/).- **`PipeSignature` — contract-only pipes for top-down pipeline design.** A new pipe type (`type = "PipeSignature"`) declares a pipe's `inputs`, `output`, and `description` without an implementation, so an author or agent can sketch a complete pipeline before committing to the operator that will eventually do the work. Signatures slot into `PipeBlueprintUnion` and `PipeSpecUnion`, are produced by the new `PipeSignatureFactory`, and ship with their own runtime (`PipeSignature` in `pipelex/pipe_signature/`). At dry-run time a signature mints a mock `Stuff` matching its declared output (multiplicity included); at live-run time it raises `PipeSignatureNotExecutableError`. Multiplicity is supported on inputs (`Doc[]`, `Image[3]`) just like every other pipe. Optional `signature_for` field hints the downstream pipe type the signature stands in for (tooling-only; cannot itself be `PipeSignature`). New docs page: [Signature Pipes](https://docs.pipelex.com/latest/building-methods/pipes/signature-pipes/). Cross-link: `wip/signature-based-validation/signature-based-validation.md`.- **`--allow-signatures` flag on `validate`.** `pipelex validate pipe`, `pipelex validate bundle`, and every `pipelex-agent validate` subcommand (`pipe`, `bundle`, `method`) accept `--allow-signatures` (default off) to dry-run pipelines that still contain `PipeSignature` placeholders. Without the flag, strict validation refuses any pipeline whose dependency graph reaches a signature, raising `SignaturesNotAllowedError` and reporting every reachable signature plus the controller chain (`signature_refs` and `dep_paths`) so the user knows exactly which pipes are still stubs. The main CLI and the agent CLI are both strict by default.- **Graph walk for signature detection.** `collect_signature_refs()` and `collect_signature_paths()` (in `pipelex/pipe_signature/signature_walk.py`) walk a pipe's dependency graph (via the new default `pipe_dependencies()` method on `PipeAbstract` and per-controller overrides), aggregating signature pipe_refs and the controller chains that reach them. Used by the strict-mode pre-check in `BundleValidator` (`_signature_pre_pass`, run from `validate_pipes`) so `validate_bundle` surfaces a single `SignaturesNotAllowedError` listing every offender across the bundle.diff --git a/docs/building-methods/pipes/csv-input-and-output.md b/docs/building-methods/pipes/csv-input-and-output.mdnew file mode 100644index 000000000..50b79179a--- /dev/null+++ b/docs/building-methods/pipes/csv-input-and-output.md@@ -0,0 +1,144 @@+---+description: "Read CSV files into typed lists and write list outputs back to CSV. Covers the inputs.json convention, the --save-csv flag, the flat-concept rule, type coercion, and v1 limitations."+---++# CSV Input & Output++Pipelex can read a CSV file directly into a typed list and write a list output back out as CSV. A CSV maps to a `ListContent` of a flat concept: each row becomes one instance of the concept, and each column maps to one of its fields. No new language construct is involved — you describe the row concept exactly as you would any structured concept.++This covers reading a `.csv` referenced from your `inputs.json`, writing a result with `--save-csv`, and the rules and limitations of the v1 codec.++## The flat-concept rule++A concept used with CSV must be **flat**: every field is a scalar. The accepted types are:++- `text` → `str`+- `integer` → `int`+- `number` → `float`+- `boolean` → `bool`+- `date` → `datetime.datetime` (an ISO `YYYY-MM-DD` value is accepted and parsed as midnight)++Optional fields are allowed, and so are choice-constrained scalars — but only **string-valued** ones (a `text`-typed `choices` field, a string `Literal`, or a `StrEnum`). A non-string choice such as `Literal[1, 2, 3]` or an `IntEnum` is rejected: it would serialize to a CSV string but cannot coerce back from one, so accepting it would break the round trip. Any nested concept, list, dict, union, or `Any`-typed field is likewise rejected with a clear error that names the offending field — a CSV cell has no room for nested structure, so you must project to a flat concept first.++```toml+[concept.Person]+description = "A person record read from a CSV row"+[concept.Person.structure]+name = { type = "text", required = true, description = "Full name" }+job = { type = "text", required = true, description = "Occupation" }+country = { type = "text", required = true, description = "Country of origin" }+birth_year = { type = "integer", required = true, description = "Year of birth" }+death_year = { type = "integer", required = false, description = "Year of death, if deceased" }+```++## Reading a CSV as input++Reference the file from `inputs.json` with a `url` whose path ends in a tabular suffix (`.csv`, or `.xlsx` once the Excel backend is available), under a flat structured concept. Pipelex detects the tabular suffix and reads the file into a `ListContent` of that concept — one CSV file becomes one list, and the concept names the **row** type:++```json+{+ "people": {+ "concept": "demo.Person",+ "content": { "url": "people.csv" }+ }+}+```++Given a `people.csv`:++```csv+name,job,country,birth_year,death_year+Ada Lovelace,Mathematician,United Kingdom,1815,1852+Grace Hopper,Computer Scientist,United States,1906,1992+Vint Cerf,Computer Scientist,United States,1943,+```++`people` arrives as a `ListContent[Person]` with three items. `birth_year` is coerced to `int`, and Vint Cerf's blank `death_year` cell becomes `None` (which is why `death_year` is declared optional).++### Header and column rules++- The header row is **required**, and headers must match the concept's field names exactly — there is no implicit renaming.++- An **extra** column (one with no matching field) is an error.++- A missing **required** column is an error.++- A missing **optional** column is fine — that field is `None` for every row. (Pipelex's own writer always emits every column, so this only matters for hand-authored partial CSVs.)++- An **empty cell** becomes `None` before validation, so the field must be optional to accept it.++### Type coercion++Cell strings are coerced through Pydantic's lax validation. The contract is pinned by tests, so you can rely on it:++| Field type | Accepted | Rejected |+| --- | --- | --- |+| `boolean` | `true`, `false`, `1`, `0`, `yes`, `no`, `on`, `off` | anything else |+| `integer` | `42`, `-3` | `1.5`, `1,000` |+| `number` | `1.5`, `3`, `-0.2` | `1,5` (comma decimal), `abc` |+| `date` | ISO `2020-12-31` | `31/12/2020`, `12/31/2020` |++A cell that cannot be coerced raises a clear error naming the 1-based row, the concept, and the field, so you can find the offending value.++## Writing a result as CSV++Pass `--save-csv <path>` to any run command to write the pipeline's main stuff to a CSV file:++```bash+pipelex run pipe summarize_people --inputs inputs.json --save-csv summaries.csv+pipelex run bundle demo.mthds --save-csv out.csv+pipelex run method my_method --save-csv out.csv+```++The path is **literal and cwd-relative** — it is written exactly where you point it, not under `--output-dir`. Parent directories are created as needed. The output's main stuff must be a flat list; if it is not a list, or its row concept is not flat, you get a clear error rather than a malformed file. The header is derived from the **declared** row concept (not from the first item), so an empty list still writes a correct header-only file. On write, `None` is rendered as an empty cell and booleans as lowercase `true`/`false`, so a value round-trips back through a read unchanged.++## Round-trip example++The pieces compose into a full CSV-in → process → CSV-out round trip. Here a `PipeBatch` runs once per person, a `PipeLLM` writes a one-sentence persona, and a `PipeCompose` assembles the output row from selected input fields plus that sentence. Both the input concept `Person` and the output concept `PersonSummary` are flat, so both sides satisfy the flat-concept rule:++```toml+[concept.PersonSummary]+description = "A person row reduced to name + country plus a one-sentence persona summary"+[concept.PersonSummary.structure]+name = { type = "text", required = true, description = "Full name" }+country = { type = "text", required = true, description = "Country of origin" }+summary = { type = "text", required = true, description = "One-sentence persona summary" }++[pipe.summarize_people]+type = "PipeBatch"+description = "Summarize each person in the list"+inputs = { people = "Person[]" }+output = "PersonSummary[]"+branch_pipe_code = "summarize_person"+input_list_name = "people"+input_item_name = "person"+```++```bash+pipelex run pipe summarize_people --inputs inputs.json --save-csv summaries.csv+```++This reads `people.csv` into `ListContent[Person]`, summarizes each person, and writes a `summaries.csv` with header `name,country,summary` and one row per input person.++## Configuring delimiter and encoding++The codec defaults to a comma delimiter and UTF-8 encoding. Reads use `utf-8-sig`, which transparently strips a leading byte-order mark (BOM) — Excel's "CSV UTF-8" export always prepends one — and reads plain UTF-8 unchanged when there is no BOM. Writes use plain `utf-8` (no BOM). Both delimiter and encoding are parameters of the codec functions (`list_content_from_csv` / `csv_from_list_content`), but there is no user-facing config surface (TOML default or CLI flag) in v1 — they are never guessed from the file. A user-facing knob for European semicolon-delimited CSVs is a planned follow-up.++## Limitations in v1++- **Local file paths only.** A remote `url` with a tabular suffix — `http(s)://`, `s3://`, `gs://`, a signed URL — is actively rejected with a clear error, never opened. Reading remote tabular files is a deferred follow-up.++- **`.xlsx` is recognized but not yet implemented.** An `.xlsx` path is detected as tabular and routed to a clear "needs `pipelex[tabular]`" message; the Excel backend itself is not built in v1.++- **Empty string is indistinguishable from `None`.** An empty cell reads as `None`, and on write `None` is rendered as an empty cell, so a genuinely empty string `""` cannot be told apart from a missing value through a CSV round trip.++- **A bare `{ "url": "...csv" }` reference reads as a table.** Detection triggers only on the exact single-key wrapper shape — a content reference that is precisely `{ "url": <tabular path> }`. So a concept whose only field is `url`, given `{ "url": "report.csv" }`, is read as a CSV **table** rather than as a single record holding `url = "report.csv"`; use a non-tabular value if you need the single-record reading. A record with any sibling key (e.g. `{ "label": "Home", "url": "report.csv" }`) is never affected — it stays an ordinary record, with its fields preserved.++- **No formula-injection escaping.** Cells are written verbatim to preserve data fidelity, so a value beginning with `=`, `+`, `-`, or `@` is not escaped. If you hand a generated CSV to a spreadsheet application, treat untrusted content accordingly. An opt-in escape flag is a deferred follow-up.++## Related Documentation++- [Providing Inputs to Pipelines](provide-inputs.md)+- [Executing Pipelines](executing-pipelines.md)+- [Pipe Output](pipe-output.md)+- [Run Commands](../../tools/cli/run.md)diff --git a/docs/tools/cli/run.md b/docs/tools/cli/run.mdindex b02333adc..32e126eb6 100644--- a/docs/tools/cli/run.md+++ b/docs/tools/cli/run.md@@ -33,6 +33,7 @@ Runs a pipe by code from your project's pipe library.- `--save-main-stuff` / `--no-save-main-stuff` - Whether to save the main output to a file- `--save-working-memory` / `--no-save-working-memory` - Whether to save the full working memory- `--working-memory-path` - Custom path for the working memory output file+- `--save-csv` - Write the main stuff to this literal CSV path (**not** under `--output-dir`; absolute or `~`/relative paths all work). Requires a flat list output. See [CSV Input & Output](../../building-methods/pipes/csv-input-and-output.md)- `--no-pretty-print` - Skip pretty printing the main output- `--graph` / `--no-graph` - Enable/disable execution graph visualization- `--graph-full-data` / `--graph-no-data` - Include full data in the graph visualization@@ -55,6 +56,9 @@ pipelex run pipe hello_world --output-dir my_output/# Run without saving or pretty printingpipelex run pipe my_pipe --no-save-main-stuff --no-pretty-print+# Write a flat list output to CSV (literal path, not under --output-dir)+pipelex run pipe summarize_people --inputs people.json --save-csv summaries.csv+# Dry-run (no AI calls)pipelex run pipe my_pipe --dry-run@@ -82,6 +86,7 @@ Runs a pipeline from a bundle file (`.mthds`) or a pipeline directory. When a di- `--save-main-stuff` / `--no-save-main-stuff` - Whether to save the main output- `--save-working-memory` / `--no-save-working-memory` - Whether to save the full working memory- `--working-memory-path` - Custom path for the working memory output file+- `--save-csv` - Write the main stuff to this literal CSV path (**not** under `--output-dir`; absolute or `~`/relative paths all work). Requires a flat list output. See [CSV Input & Output](../../building-methods/pipes/csv-input-and-output.md)- `--no-pretty-print` - Skip pretty printing the main output- `--graph` / `--no-graph` - Enable/disable execution graph visualization- `--graph-full-data` / `--graph-no-data` - Include full data in the graph visualization@@ -128,6 +133,7 @@ Runs a pipeline from an installed method package.- `--save-main-stuff` / `--no-save-main-stuff` - Whether to save the main output- `--save-working-memory` / `--no-save-working-memory` - Whether to save the full working memory- `--working-memory-path` - Custom path for the working memory output file+- `--save-csv` - Write the main stuff to this literal CSV path (**not** under `--output-dir`; absolute or `~`/relative paths all work). Requires a flat list output. See [CSV Input & Output](../../building-methods/pipes/csv-input-and-output.md)- `--no-pretty-print` - Skip pretty printing the main output- `--graph` / `--no-graph` - Enable/disable execution graph visualization- `--graph-full-data` / `--graph-no-data` - Include full data in the graph visualization@@ -170,4 +176,5 @@ The output JSON contains the complete working memory after pipeline execution, i- [Executing Pipelines](../../building-methods/pipes/executing-pipelines.md)- [Providing Inputs to Pipelines](../../building-methods/pipes/provide-inputs.md)+- [CSV Input & Output](../../building-methods/pipes/csv-input-and-output.md)- [Design and Run Pipelines](../../building-methods/pipes/index.md)diff --git a/mkdocs.yml b/mkdocs.ymlindex d288bc662..bca7a271b 100644--- a/mkdocs.yml+++ b/mkdocs.yml@@ -449,6 +449,7 @@ nav:- Providing Inputs to Pipelines: building-methods/pipes/provide-inputs.md- Working Memory: building-methods/pipes/working-memory.md- Pipe Output: building-methods/pipes/pipe-output.md+ - CSV Input & Output: building-methods/pipes/csv-input-and-output.md- Understanding Multiplicity: building-methods/pipes/understanding-multiplicity.md- Pipe Operators:- Overview: building-methods/pipes/pipe-operators/index.mddiff --git a/pipelex/cli/commands/run/_inputs_path_resolver.py b/pipelex/cli/commands/run/_inputs_path_resolver.pyindex 6fee8c3dc..38c0e3c85 100644--- a/pipelex/cli/commands/run/_inputs_path_resolver.py+++ b/pipelex/cli/commands/run/_inputs_path_resolver.py@@ -32,6 +32,13 @@ def is_relative_local_path(uri: str) -> bool:resolved = resolve_uri(uri)if not isinstance(resolved, ResolvedLocalPath):return False+ if "://" in resolved.path:+ # An unrecognized scheme (``s3://``, ``gs://``, …) slips through ``resolve_uri`` as a+ # ResolvedLocalPath but is NOT a local file path; do not rewrite it relative to base_dir+ # (that would mangle ``s3://bucket/x`` into ``<base_dir>/s3:/bucket/x`` and defeat the+ # downstream remote-url guards). Mirrors the same ``"://"`` stopgap in the CSV input hook;+ # both go away once ``resolve_uri`` classifies schemes (tools/uri follow-up).+ return Falsereturn not Path(resolved.path).is_absolute()diff --git a/pipelex/cli/commands/run/_run_core.py b/pipelex/cli/commands/run/_run_core.pyindex daa17076d..e9b00e02e 100644--- a/pipelex/cli/commands/run/_run_core.py+++ b/pipelex/cli/commands/run/_run_core.py@@ -3,6 +3,7 @@ from __future__ import annotationsimport asyncioimport jsonfrom pathlib import Path+from typing import TYPE_CHECKING, castimport typerfrom posthog import tag@@ -18,9 +19,11 @@ from pipelex.cli.error_handlers import (print_traceback_if_requested,)from pipelex.config import get_config+from pipelex.core.concepts.exceptions import ConceptValueErrorfrom pipelex.core.interpreter.exceptions import PipelexInterpreterErrorfrom pipelex.core.interpreter.interpreter import PipelexInterpreterfrom pipelex.core.pipes.exceptions import PipeOperatorModelChoiceError+from pipelex.core.stuffs.list_content import ListContentfrom pipelex.core.stuffs.stuff_viewer import render_stuff_viewerfrom pipelex.graph.graph_factory import generate_graph_outputs, save_graph_outputs_to_dirfrom pipelex.hub import get_console, get_report_delegate, get_telemetry_manager@@ -35,6 +38,11 @@ from pipelex.tools.misc.exceptions import JsonTypeErrorfrom pipelex.tools.misc.file_utils import get_incremental_directory_pathfrom pipelex.tools.misc.json_utils import load_json_dict_from_path, save_as_json_to_pathfrom pipelex.tools.misc.package_utils import get_package_version+from pipelex.tools.tabular.csv_codec import assert_supported_table_suffix, csv_from_list_content, flat_field_names+from pipelex.tools.tabular.exceptions import CsvError++if TYPE_CHECKING:+ from pipelex.core.stuffs.stuff_content import StuffContentCOMMAND = "run"@@ -55,11 +63,26 @@ async def _execute_run(library_dir: list[str] | None,cost_report: bool | None,dynamic_output_concept_ref: str | None = None,+ *,+ save_csv: str | None = None,) -> None:"""Core async execution logic for running a pipe.Shared between the ``method`` and ``pipe`` subcommands."""+ # Fail fast on an unusable --save-csv target BEFORE running the pipeline, so a bad path (empty,+ # or an unsupported suffix like .xlsx) doesn't waste a full inference run. The output-shape and+ # row-flatness checks need the result, so they stay in the save block after execution.+ if save_csv is not None:+ if not save_csv.strip():+ typer.secho("Failed to --save-csv: an empty output path was given.", fg=typer.colors.RED, err=True)+ raise typer.Exit(1)+ try:+ assert_supported_table_suffix(Path(save_csv))+ except CsvError as exc:+ typer.secho(f"Failed to --save-csv: {exc}", fg=typer.colors.RED, err=True)+ raise typer.Exit(1) from exc+mthds_content: str | None = Noneif bundle_path:try:@@ -222,6 +245,47 @@ async def _execute_run(save_as_json_to_path(object_to_save=working_memory_dict, path=Path(working_memory_output_path))log.verbose(f"Working memory saved to: {working_memory_output_path}")+ # Save main_stuff as CSV if requested. CQ1: write to the literal <path> (cwd-relative),+ # NOT resolved under --output-dir. Requires the main stuff to be a flat list (e.g. PersonSummary[]);+ # the codec rejects a non-flat row concept with a clear CsvFlatnessError.+ if save_csv is not None:+ # The path string and its suffix were already validated up front (fail-fast, before the run).+ csv_main_stuff = pipe_output.working_memory.get_optional_main_stuff()+ if csv_main_stuff is None:+ typer.secho("Failed to --save-csv: the pipeline produced no main stuff to write.", fg=typer.colors.RED, err=True)+ raise typer.Exit(1)+ csv_content = csv_main_stuff.content+ if not isinstance(csv_content, ListContent):+ typer.secho(+ f"Failed to --save-csv: the main stuff is a {type(csv_content).__name__}, not a list. "+ "CSV output requires the pipe to produce a flat list (e.g. 'PersonSummary[]').",+ fg=typer.colors.RED,+ err=True,+ )+ raise typer.Exit(1)+ csv_list_content = cast("ListContent[StuffContent]", csv_content)+ # expanduser so a quoted/`=`-form `~/out.csv` writes to the home dir, not a literal `./~` dir.+ csv_path = Path(save_csv).expanduser()+ # A CSV-save failure (output concept with no structure class, non-flat output, write error) is+ # framed as a --save-csv failure, not a pipeline failure — the pipeline already succeeded.+ # ConceptValueError (a ValueError) and CsvError would otherwise escape execute_run's generic+ # `except PipelexError` and read as "Failed to execute pipeline" (or as a raw traceback).+ try:+ # get_structure_class() can raise ConceptValueError if the output concept has no registered+ # structure class; keep it inside the guard so that, too, is framed as a save-csv failure.+ csv_row_model = csv_main_stuff.concept.get_structure_class()+ # Validate the row flatness BEFORE touching the filesystem, so a non-flat output leaves no+ # directory behind (the suffix was already validated up front).+ flat_field_names(csv_row_model)+ # CQ1: write to the literal cwd-relative path, but still create its parent dirs so a+ # nested target (e.g. reports/2026/out.csv) doesn't fail late after the run completed.+ csv_path.parent.mkdir(parents=True, exist_ok=True)+ csv_from_list_content(csv_list_content, row_model=csv_row_model, path=csv_path)+ except (CsvError, ConceptValueError, OSError) as csv_exc:+ typer.secho(f"Failed to --save-csv to '{save_csv}': {csv_exc}", fg=typer.colors.RED, err=True)+ raise typer.Exit(1) from csv_exc+ log.verbose(f"Main stuff CSV saved to: {save_csv}")+reporting_config = get_config().pipelex.reporting_config# --no-cost-report (cost_report is False) skips the report entirely: no table, no CSV.# Otherwise: console follows the flag (if given) or config; CSV follows config.@@ -253,6 +317,9 @@ async def _execute_run(console.print(" [green]✓[/green] working_memory.json")else:console.print(f" [green]✓[/green] working_memory: {working_memory_output_path}")+ # CSV output is written to a literal cwd-relative path (CQ1), independent of --output-dir.+ if save_csv is not None:+ console.print(f" [green]✓[/green] CSV saved to [bold magenta]{save_csv}[/bold magenta]")def execute_run(@@ -273,6 +340,8 @@ def execute_run(telemetry_command_label: str = COMMAND,temporal: bool | None = None,dynamic_output_concept_ref: str | None = None,+ *,+ save_csv: str | None = None,) -> None:"""Synchronous entry point that wraps the async execution with Pipelex setup/teardown.@@ -302,6 +371,7 @@ def execute_run(library_dir=library_dir,cost_report=cost_report,dynamic_output_concept_ref=dynamic_output_concept_ref,+ save_csv=save_csv,))diff --git a/pipelex/cli/commands/run/bundle_cmd.py b/pipelex/cli/commands/run/bundle_cmd.pyindex 280c1d176..96b30aca6 100644--- a/pipelex/cli/commands/run/bundle_cmd.py+++ b/pipelex/cli/commands/run/bundle_cmd.py@@ -88,6 +88,13 @@ def run_bundle_cmd(help="Override config: --cost-report forces the cost table on; --no-cost-report skips reporting entirely (no table and no CSV file).",),] = None,+ save_csv: Annotated[+ str | None,+ typer.Option(+ "--save-csv",+ help="Write the main stuff to this literal CSV path (not under --output-dir; absolute/~/relative ok). Requires a flat list output.",+ ),+ ] = None,) -> None:"""Run a pipeline from a bundle file (.mthds) or pipeline directory.@@ -182,4 +189,5 @@ def run_bundle_cmd(telemetry_command_label=f"{COMMAND} bundle",temporal=temporal,dynamic_output_concept_ref=dynamic_output_concept_ref,+ save_csv=save_csv,)diff --git a/pipelex/cli/commands/run/method_cmd.py b/pipelex/cli/commands/run/method_cmd.pyindex 17cf0e662..715673658 100644--- a/pipelex/cli/commands/run/method_cmd.py+++ b/pipelex/cli/commands/run/method_cmd.py@@ -87,6 +87,13 @@ def run_method_cmd(help="Override config: --cost-report forces the cost table on; --no-cost-report skips reporting entirely (no table and no CSV file).",),] = None,+ save_csv: Annotated[+ str | None,+ typer.Option(+ "--save-csv",+ help="Write the main stuff to this literal CSV path (not under --output-dir; absolute/~/relative ok). Requires a flat list output.",+ ),+ ] = None,) -> None:"""Run an installed method by name.@@ -152,4 +159,5 @@ def run_method_cmd(telemetry_command_label=f"{COMMAND} method",temporal=temporal,dynamic_output_concept_ref=dynamic_output_concept_ref,+ save_csv=save_csv,)diff --git a/pipelex/cli/commands/run/pipe_cmd.py b/pipelex/cli/commands/run/pipe_cmd.pyindex 055c390c7..fd95ae73b 100644--- a/pipelex/cli/commands/run/pipe_cmd.py+++ b/pipelex/cli/commands/run/pipe_cmd.py@@ -84,6 +84,13 @@ def run_pipe_cmd(help="Override config: --cost-report forces the cost table on; --no-cost-report skips reporting entirely (no table and no CSV file).",),] = None,+ save_csv: Annotated[+ str | None,+ typer.Option(+ "--save-csv",+ help="Write the main stuff to this literal CSV path (not under --output-dir; absolute/~/relative ok). Requires a flat list output.",+ ),+ ] = None,) -> None:"""Run a pipe by code.@@ -147,4 +154,5 @@ def run_pipe_cmd(telemetry_command_label=f"{COMMAND} pipe",temporal=temporal,dynamic_output_concept_ref=dynamic_output_concept_ref,+ save_csv=save_csv,)diff --git a/pipelex/core/stuffs/stuff_factory.py b/pipelex/core/stuffs/stuff_factory.pyindex 2fe782082..61802d82c 100644--- a/pipelex/core/stuffs/stuff_factory.py+++ b/pipelex/core/stuffs/stuff_factory.py@@ -1,4 +1,6 @@+from pathlib import Pathfrom typing import Any, cast+from urllib.parse import urlsplit, urlunsplitimport shortuuidfrom mthds.models.pipeline_inputs import StuffContentOrData@@ -6,6 +8,7 @@ from pydantic import BaseModel, ValidationError, field_validatorfrom pipelex.core.concepts.concept import Conceptfrom pipelex.core.concepts.concept_factory import ConceptFactory+from pipelex.core.concepts.exceptions import ConceptValueErrorfrom pipelex.core.concepts.native.concept_native import NativeConceptCodefrom pipelex.core.concepts.validation import validate_concept_reffrom pipelex.core.stuffs.exceptions import StuffFactoryError@@ -16,7 +19,11 @@ from pipelex.core.stuffs.stuff_content_factory import StuffContentFactoryfrom pipelex.core.stuffs.text_content import TextContentfrom pipelex.hub import get_class_registry, get_concept_library, get_native_concept, get_required_conceptfrom pipelex.libraries.concept.concept_library import ConceptLibraryConceptNotFoundError+from pipelex.tools.tabular.csv_codec import is_tabular_path, list_content_from_csv+from pipelex.tools.tabular.exceptions import CsvErrorfrom pipelex.tools.typing.pydantic_utils import format_pydantic_validation_error+from pipelex.tools.uri.resolved_uri import ResolvedLocalPath+from pipelex.tools.uri.uri_resolver import resolve_uriclass StuffBlueprint(BaseModel):@@ -123,6 +130,96 @@ class StuffFactory:name=name,)+ @classmethod+ def _try_make_csv_list_stuff(+ cls,+ concept: Concept,+ content: dict[str, Any],+ name: str | None,+ code: str | None,+ ) -> Stuff | None:+ """Build a ``ListContent[row-concept]`` from a ``{"url": "...csv"}`` input reference.++ Detection is gated to the explicit wrapper shape — ``content`` must be *exactly*+ ``{"url": <tabular path>}`` — under a non-native structured concept. Each data row then+ becomes one instance of the concept's structure class, so one CSV yields one+ ``ListContent`` (the concept names the *row* type). Returns ``None`` for an ordinary+ record dict — no ``url`` key, sibling keys alongside ``url``, a non-tabular suffix, or a+ native concept — so the caller falls through to normal Case 2.5 dict handling. The+ single-key gate keeps a real record that merely *has* a ``url`` field (e.g.+ ``{"label": "Home", "url": "report.csv"}``) from being silently reduced to a table with+ its sibling keys dropped.++ v1 reads LOCAL paths only: a tabular-suffixed remote ``url`` (``http(s)``/``s3``/``gs``/+ ``pipelex-storage``) is rejected with a clear ``CsvError`` rather than opened as a local path.+ (A base64 data URL carries no file suffix, so it is never detected as tabular and simply+ falls through to ordinary record handling.)+ """+ url = content.get("url")+ if not isinstance(url, str):+ return None+ if set(content) != {"url"}:+ # Only the bare {"url": ...} wrapper is a table reference; a record with other keys+ # alongside `url` stays a record (its siblings must not be dropped).+ return None+ if Concept.is_native_concept(concept=concept):+ # Native file concepts (Image, PDF, ...) own their own url handling; never hijack them.+ # Checked BEFORE url parsing so a native concept never raises a CSV-flavored error.+ return None+ # Parse the URL once, up front. A malformed url (e.g. a bad IPv6 bracket like ``https://[``)+ # makes ``urlsplit`` itself raise ``ValueError`` — convert that to a redacted CsvError rather+ # than let it escape into a traceback that would surface the raw (possibly token-bearing) url.+ try:+ url_parts = urlsplit(url)+ except ValueError as exc:+ msg = (+ f"CSV input supports local file paths only in v1, but stuff '{name}' for concept "+ f"'{concept.concept_ref}' has a url that could not be parsed. "+ "Download the file locally and reference it by path."+ )+ raise CsvError(msg) from exc+ # Detect the tabular suffix on the URL's PATH component only. A raw URL fed to ``Path`` keeps+ # any ``?query``/``#fragment`` inside ``.suffix`` (e.g. an S3 presigned ``...csv?X-Amz-...``),+ # which would hide the ``.csv`` and let a remote ref slip past the local-only guard below.+ if not is_tabular_path(Path(url_parts.path)):+ return None++ resolved = resolve_uri(url)+ # Accept only genuine local paths. `file://` resolves to a scheme-free local path; an http(s)/+ # base64/pipelex-storage url is a non-local ResolvedUri; an `s3://`/`gs://`-style scheme slips+ # through as a ResolvedLocalPath but keeps `://` in its path, so reject those too.+ if not isinstance(resolved, ResolvedLocalPath) or "://" in resolved.path:+ # Strip query/fragment/userinfo before echoing the url: CsvError is caller-facing and+ # survives STRICT disclosure, so a signed/token-bearing url (e.g. an S3 presigned link)+ # must not leak its credentials into the message. Keep scheme/host/path to identify it.+ safe_netloc = url_parts.hostname or ""+ try:+ safe_port = url_parts.port+ except ValueError:+ # A malformed/out-of-range port makes the `.port` property raise; don't let that+ # ValueError escape (it would bypass this redaction and surface the raw url in a+ # traceback). Drop the port from the sanitized display instead.+ safe_port = None+ if safe_port is not None:+ safe_netloc = f"{safe_netloc}:{safe_port}"+ safe_url = urlunsplit((url_parts.scheme, safe_netloc, url_parts.path, "", ""))+ msg = (+ f"CSV input supports local file paths only in v1, but stuff '{name}' for concept "+ f"'{concept.concept_ref}' points at a remote/non-local url: {safe_url!r}. "+ "Download the file locally and reference it by path."+ )+ raise CsvError(msg)++ try:+ row_model = concept.get_structure_class()+ except ConceptValueError as exc:+ # Keep the codec's typed-error boundary intact: an unregistered structure class is a+ # caller-fixable input problem, not a raw ValueError that escapes into core/runner.+ msg = f"CSV input for stuff '{name}': concept '{concept.concept_ref}' has no registered structure class to read CSV rows into."+ raise CsvError(msg) from exc+ list_content = list_content_from_csv(Path(resolved.path), row_model)+ return cls.make_stuff(concept=concept, content=list_content, name=name, code=code)+@classmethoddef make_stuff_from_stuff_content_or_data(cls,@@ -397,6 +494,12 @@ class StuffFactory:# Case 2.5: content is a dictif isinstance(content, dict):+ content_dict = cast("dict[str, Any]", content)+ # CSV input: a {"url": "...csv"} under a structured row concept loads as ListContent[row-concept].+ csv_stuff = cls._try_make_csv_list_stuff(concept=concept, content=content_dict, name=name, code=code)+ if csv_stuff is not None:+ return csv_stuff+the_class = get_class_registry().get_class(name=concept.structure_class_name)if the_class is None:msg = (diff --git a/pipelex/tools/tabular/__init__.py b/pipelex/tools/tabular/__init__.pynew file mode 100644index 000000000..e69de29bbdiff --git a/pipelex/tools/tabular/csv_codec.py b/pipelex/tools/tabular/csv_codec.pynew file mode 100644index 000000000..48af67131--- /dev/null+++ b/pipelex/tools/tabular/csv_codec.py@@ -0,0 +1,429 @@+"""CSV codec — the I/O bridge between CSV files and ``ListContent[row-concept]``.++CSV is an I/O codec, not a prompt-render format: it does NOT join the+``rendered_plain/markdown/html/json`` ``TextFormat`` router. This single module owns+row read/write (``read_rows``/``write_rows``), the shared flatness/field-order helper+(``flat_field_names``, used by BOTH read and write), the concept binding+(``list_content_from_csv``/``csv_from_list_content``), and a thin format seam+(``assert_supported_table_suffix``) so an ``openpyxl``-backed ``.xlsx`` codec can be+added later under ``pipelex[tabular]`` without touching callers.++Dialect contract: the stdlib default ``QUOTE_MINIMAL`` quoting is the codec's wire+format — a value is quoted only when it contains the delimiter, the quote char, or a+newline. The exact-line write assertions in the test suite rely on this.+"""++import csv+import types+from datetime import date, datetime+from enum import Enum+from pathlib import Path+from typing import TYPE_CHECKING, Any, Literal, Union, get_args, get_origin++from pydantic import ValidationError++from pipelex.core.stuffs.list_content import ListContent+from pipelex.core.stuffs.stuff_content import StuffContent, StuffContentType+from pipelex.tools.tabular.exceptions import (+ CsvCoercionError,+ CsvColumnError,+ CsvError,+ CsvFlatnessError,+ CsvReadError,+)++if TYPE_CHECKING:+ from pydantic_core import ErrorDetails++# v1 dialect defaults. Configurable via codec params, never auto-guessed (design §9).+# The user-facing config surface (toml default / --delimiter / --encoding flags) is a+# deferred follow-up — v1 exposes delimiter/encoding only as function params.+DEFAULT_DELIMITER = ","+DEFAULT_ENCODING = "utf-8"+# Read with utf-8-sig so a UTF-8 BOM is transparently stripped — Excel's "CSV UTF-8" export always+# prepends one, and the headline use case is round-tripping Excel files. utf-8-sig reads plain UTF-8+# unchanged when no BOM is present. Writes stay on plain utf-8 (utf-8-sig would prepend a BOM).+DEFAULT_READ_ENCODING = "utf-8-sig"++_NONE_TYPE = type(None)+_UNION_TYPE = getattr(types, "UnionType", None) # Py3.10+: types.UnionType (PEP 604 `X | None`)+# Scalar python types a CSV cell can round-trip. `bool` precedes `int` semantically+# (bool is an int subclass) but membership here is by identity, so both are accepted.+_FLAT_SCALAR_TYPES: frozenset[type] = frozenset({str, int, float, bool, date, datetime})+++# ---------------------------------------------------------------------------------------+# Flatness classifier (shared by read AND write)+# ---------------------------------------------------------------------------------------+++def _is_flat_annotation(annotation: Any) -> bool:+ """Return whether a pydantic field annotation is a CSV-flat scalar.++ Unwraps ``Optional`` (a single non-``None`` arm); ACCEPTS the scalar python types,+ ``Literal[...]`` and ``Enum`` (choice-constrained) scalars; REJECTS genuine unions,+ containers, nested models, and ``Any``.+ """+ origin = get_origin(annotation)+ if origin in {Union, _UNION_TYPE}:+ non_none_args = [arg for arg in get_args(annotation) if arg is not _NONE_TYPE]+ if len(non_none_args) != 1:+ # A genuine multi-type union (e.g. int | str) is not flat.+ return False+ return _is_flat_annotation(non_none_args[0])+ if origin is Literal:+ # Only string-valued choices round-trip through a CSV cell: the codec writes the value as a+ # bare string and pydantic does not coerce that string back into a non-string Literal member.+ return all(isinstance(arg, str) for arg in get_args(annotation))+ if isinstance(annotation, type):+ if issubclass(annotation, Enum):+ # Same reason: only a str-valued Enum (e.g. StrEnum) round-trips; an IntEnum would not.+ return all(isinstance(member.value, str) for member in annotation)+ return annotation in _FLAT_SCALAR_TYPES+ return False+++def _annotation_allows_none(annotation: Any) -> bool:+ """Whether a field annotation accepts ``None`` (an ``Optional`` / ``... | None`` field)."""+ if get_origin(annotation) in {Union, _UNION_TYPE}:+ return any(arg is _NONE_TYPE for arg in get_args(annotation))+ return annotation is _NONE_TYPE+++def flat_field_names(row_model: type[StuffContent]) -> list[str]:+ """Validate that ``row_model`` is CSV-flat and return its field names in declared order.++ The single shared flatness classifier used by BOTH read and write. Unwraps+ ``Optional``; ACCEPTS ``str``/``int``/``float``/``bool``/``datetime.date`` and+ ``Literal``/choice-constrained scalars; REJECTS list/dict/nested/concept-typed/+ ``Union``/``Any`` fields with a ``CsvFlatnessError`` naming the offending field.+ """+ field_names: list[str] = []+ for field_name, field_info in row_model.model_fields.items():+ if not _is_flat_annotation(field_info.annotation):+ msg = (+ f"Concept {row_model.__name__!r} is not CSV-flat: field {field_name!r} has type "+ f"{field_info.annotation!r}, but CSV rows accept scalar fields only "+ f"(text/integer/number/boolean/date, optionals, and Literal/choice-constrained scalars). "+ f"Project to a flat concept first."+ )+ raise CsvFlatnessError(msg)+ field_names.append(field_name)+ return field_names+++# ---------------------------------------------------------------------------------------+# read_rows / write_rows primitives+# ---------------------------------------------------------------------------------------+++def _assert_single_char_delimiter(delimiter: str) -> None:+ """Reject a delimiter the stdlib ``csv`` module cannot use.++ ``csv`` requires the delimiter to be exactly one character and additionally rejects the+ newline characters (LF/CR) and the quote character ``"`` (it collides with the default+ quotechar) — each passes the length check but makes ``csv.reader``/``csv.writer`` raise a raw+ ``ValueError``/``TypeError`` at construction. Validating them up front keeps the codec's+ typed-error boundary intact so no raw exception escapes for the exposed ``delimiter`` parameter.+ """+ if len(delimiter) != 1:+ msg = f"CSV delimiter must be a single character, got {delimiter!r}."+ raise CsvError(msg)+ if delimiter in {"\n", "\r"}:+ msg = f"CSV delimiter cannot be a newline character, got {delimiter!r}."+ raise CsvError(msg)+ if delimiter == '"':+ # The default quote character; csv raises "bad delimiter or quotechar value" if they collide.+ msg = f"CSV delimiter cannot be the quote character, got {delimiter!r}."+ raise CsvError(msg)+++def _validate_header(header: list[str], path: Path) -> None:+ """Reject an empty header, or a header with a blank or duplicate column."""+ if not header:+ # An empty first row (a leading blank line, or a file that is just a newline) names no+ # columns; treat it as a malformed header rather than letting it surface as a confusing+ # "row wider than header" / missing-required error downstream.+ msg = f"CSV file {path} has an empty header row; a header naming the columns is required."+ raise CsvColumnError(msg)+ seen: set[str] = set()+ for column in header:+ if not column.strip():+ msg = f"CSV file {path} has a blank header cell; every column needs a non-empty name."+ raise CsvColumnError(msg)+ if column in seen:+ msg = f"CSV file {path} has a duplicate header column {column!r}; column names must be unique."+ raise CsvColumnError(msg)+ seen.add(column)+++def _read_table(path: Path, *, delimiter: str, encoding: str) -> tuple[list[str], list[tuple[int, list[str]]]]:+ """Read a CSV file into a validated header row + the (1-based-line-numbered) data rows.++ The shared reader behind ``read_rows`` and ``list_content_from_csv`` — keeps the+ header explicit so column validation works even for a header-only file. Blank physical+ lines (which ``csv`` yields as ``[]``) are skipped, matching ``csv.DictReader``; a data+ row wider than the header is rejected (its surplus cells map to no column). Each kept row+ carries its 1-based physical CSV line number (the header is line 1, so the first data row+ is line 2, and skipped blank lines still advance the count) so error messages point at the+ line the author sees in their editor. The reader runs in ``strict=True`` mode so malformed+ quoting (e.g. an unterminated quoted field) raises ``csv.Error`` at the boundary rather than+ silently merging the following lines into one cell. Wraps every raw+ ``OSError``/``UnicodeDecodeError``/``LookupError``/``csv.Error`` as ``CsvReadError``.+ """+ _assert_single_char_delimiter(delimiter)+ # Capture each record's physical line number from ``reader.line_num`` (the count of physical lines+ # consumed so far) rather than enumerating, so a quoted cell that spans multiple physical lines+ # doesn't skew the line numbers of the records that follow it.+ numbered_rows: list[tuple[int, list[str]]] = []+ try:+ with path.open("r", encoding=encoding, newline="") as csv_file:+ reader = csv.reader(csv_file, delimiter=delimiter, strict=True)+ for row in reader:+ numbered_rows.append((reader.line_num, row))+ except UnicodeDecodeError as exc:+ msg = f"Could not decode CSV file {path} as {encoding}: {exc}"+ raise CsvReadError(msg) from exc+ except LookupError as exc:+ msg = f"Unknown encoding {encoding!r} for CSV file {path}: {exc}"+ raise CsvReadError(msg) from exc+ except csv.Error as exc:+ msg = f"Malformed CSV file {path}: {exc}"+ raise CsvReadError(msg) from exc+ except OSError as exc:+ msg = f"Could not read CSV file {path}: {exc}"+ raise CsvReadError(msg) from exc++ if not numbered_rows:+ msg = f"CSV file {path} is empty; a header row is required."+ raise CsvReadError(msg)++ header = numbered_rows[0][1]+ _validate_header(header, path)++ data_rows: list[tuple[int, list[str]]] = []+ for line_number, row in numbered_rows[1:]:+ if not row:+ # Blank physical line (csv yields []). Skip it; a single empty cell is [''] and is kept.+ continue+ if len(row) > len(header):+ msg = (+ f"CSV file {path} row {line_number} has {len(row)} fields but the header declares "+ f"{len(header)} column(s); the surplus cells map to no column."+ )+ raise CsvColumnError(msg)+ data_rows.append((line_number, row))+ return header, data_rows+++def _row_to_dict(header: list[str], data_row: list[str]) -> dict[str, str]:+ """Map a raw data row onto the header, padding a short row with empty cells."""+ return {column: (data_row[index] if index < len(data_row) else "") for index, column in enumerate(header)}+++def read_rows(path: Path, *, delimiter: str = DEFAULT_DELIMITER, encoding: str = DEFAULT_READ_ENCODING) -> list[dict[str, str]]:+ """Read a CSV file into a list of header-keyed string rows.++ A header row is required; blank lines are skipped. Raises ``CsvReadError`` for a+ missing/unreadable file, an unknown/wrong encoding, or malformed quoting; raises+ ``CsvColumnError`` for a duplicate/blank header cell or a data row wider than the header.+ """+ header, data_rows = _read_table(path, delimiter=delimiter, encoding=encoding)+ return [_row_to_dict(header, row) for _, row in data_rows]+++def write_rows(+ path: Path,+ headers: list[str],+ rows: list[dict[str, str]],+ *,+ delimiter: str = DEFAULT_DELIMITER,+ encoding: str = DEFAULT_ENCODING,+) -> None:+ """Write header-keyed string rows to a CSV file, emitting ``headers`` in order.++ An empty ``rows`` still writes the header line. Raises ``CsvError`` if the delimiter+ is invalid, the encoding is unknown, a cell is not encodable in ``encoding``, or the+ file cannot be written.+ """+ _assert_single_char_delimiter(delimiter)+ try:+ with path.open("w", encoding=encoding, newline="") as csv_file:+ writer = csv.writer(csv_file, delimiter=delimiter)+ writer.writerow(headers)+ for row in rows:+ writer.writerow([row.get(column, "") for column in headers])+ except LookupError as exc:+ msg = f"Unknown encoding {encoding!r} for CSV file {path}: {exc}"+ raise CsvError(msg) from exc+ except UnicodeEncodeError as exc:+ msg = f"Could not encode CSV file {path} as {encoding}: {exc}"+ raise CsvError(msg) from exc+ except OSError as exc:+ msg = f"Could not write CSV file {path}: {exc}"+ raise CsvError(msg) from exc+++# ---------------------------------------------------------------------------------------+# Format seam+# ---------------------------------------------------------------------------------------++# Suffixes the tabular codec claims: the .csv built-in plus the deferred .xlsx seam.+_TABULAR_SUFFIXES: frozenset[str] = frozenset({".csv", ".xlsx"})+++def is_tabular_path(path: Path) -> bool:+ """Whether ``path``'s suffix is one the tabular codec claims.++ ``True`` for ``.csv`` (the v1 built-in) and ``.xlsx`` (the deferred seam). The input+ hook uses this to decide whether a ``{"url": ...}`` reference is a table to read rather+ than an ordinary record dict. ``.xlsx`` is claimed here so it reaches the codec's clear+ "needs pipelex[tabular]" seam error (via ``assert_supported_table_suffix``) instead of+ being misread as a record.+ """+ return path.suffix.lower() in _TABULAR_SUFFIXES+++def assert_supported_table_suffix(path: Path) -> None:+ """Format seam: accept ``.csv`` (the only v1 built-in); reject everything else.++ A ``.xlsx`` path raises ``CsvError`` pointing at the optional ``pipelex[tabular]``+ extra; any other suffix raises a generic unsupported-format ``CsvError``. This is+ the seam an ``openpyxl``-backed codec slots into later without touching callers.+ """+ suffix = path.suffix.lower()+ if suffix == ".csv":+ return+ if suffix == ".xlsx":+ msg = (+ f"Reading/writing Excel files ({path.name}) requires the optional 'pipelex[tabular]' extra, "+ f"which is not part of v1. Convert to .csv, or install 'pipelex[tabular]' once it ships."+ )+ raise CsvError(msg)+ msg = f"Unsupported table file format for {path.name!r}: only .csv is supported (got suffix {path.suffix!r})."+ raise CsvError(msg)+++# ---------------------------------------------------------------------------------------+# Concept binding+# ---------------------------------------------------------------------------------------+++def list_content_from_csv(+ path: Path,+ row_model: type[StuffContentType],+ *,+ delimiter: str = DEFAULT_DELIMITER,+ encoding: str = DEFAULT_READ_ENCODING,+) -> ListContent[StuffContentType]:+ """Read a CSV file into a ``ListContent`` of ``row_model`` instances (one row → one item).++ Columns must match ``row_model``'s field names (no implicit remap): an extra column+ or a missing *required* column raises ``CsvColumnError``; a missing *optional* column+ sets that field to ``None`` for all rows. Empty cells map to ``None`` BEFORE pydantic+ validation (so they must target optional fields); remaining strings are coerced via+ pydantic's lax validation. A coercion failure raises ``CsvCoercionError`` naming the+ 1-based row/column and field. ``row_model`` must be CSV-flat (see ``flat_field_names``).+ """+ assert_supported_table_suffix(path)+ field_names = flat_field_names(row_model)+ field_name_set = set(field_names)+ required_fields = {name for name, field_info in row_model.model_fields.items() if field_info.is_required()}++ header, data_rows = _read_table(path, delimiter=delimiter, encoding=encoding)+ header_set = set(header)++ extra_columns = header_set - field_name_set+ if extra_columns:+ msg = (+ f"CSV file {path} has column(s) {sorted(extra_columns)} not declared on concept {row_model.__name__!r}. "+ f"Remove them or project to a matching flat concept."+ )+ raise CsvColumnError(msg)++ missing_required = required_fields - header_set+ if missing_required:+ msg = f"CSV file {path} is missing required column(s) {sorted(missing_required)} for concept {row_model.__name__!r}."+ raise CsvColumnError(msg)++ # A CSV that omits a *nullable* column means "no value" for every row → None (CT2), regardless of+ # the row model's own default. Carrying these as explicit None keeps the contract from silently+ # depending on a non-None field default (e.g. ``nickname: str | None = "anon"`` → None, not "anon").+ # A non-nullable defaulted field (e.g. ``count: int = 0``) is left absent so its own default+ # applies — forcing None there would fail validation, not honor the column's absence.+ omitted_nullable_fields = {name for name in (field_name_set - header_set) if _annotation_allows_none(row_model.model_fields[name].annotation)}++ items: list[StuffContentType] = []+ for row_number, data_row in data_rows:+ cell_map = _row_to_dict(header, data_row)+ # Empty cell -> None BEFORE validation (so it targets an optional field, or fails required).+ row_data: dict[str, str | None] = {column: (value or None) for column, value in cell_map.items()}+ for omitted_field in omitted_nullable_fields:+ row_data[omitted_field] = None+ try:+ item = row_model.model_validate(row_data)+ except ValidationError as exc:+ error_fields = sorted({_validation_error_label(error) for error in exc.errors()})+ fields_label = ", ".join(error_fields) or "?"+ # Use pydantic's per-error `msg` (e.g. "Input should be a valid integer"), NOT str(exc):+ # str(exc) appends `input_value=<raw cell>`, which would leak the cell content into this+ # caller-facing message (CsvError survives STRICT disclosure). Row/field identifiers suffice.+ reasons = "; ".join(sorted({str(error.get("msg", "")) for error in exc.errors()})) or "invalid value"+ msg = f"Could not coerce CSV row {row_number} of {path} for concept {row_model.__name__!r}: field(s) {fields_label} — {reasons}"+ raise CsvCoercionError(msg) from exc+ items.append(item)++ return ListContent(items=items)+++def csv_from_list_content(+ list_content: ListContent[StuffContentType],+ row_model: type[StuffContentType],+ path: Path,+ *,+ delimiter: str = DEFAULT_DELIMITER,+ encoding: str = DEFAULT_ENCODING,+) -> None:+ """Write a flat ``ListContent`` to a CSV file, one row per item.++ Columns are ``row_model``'s scalar fields in declared order — derived from the+ DECLARED model, never from ``list_content.items[0]`` — so an empty list still writes+ a correct header-only file. Each item is serialized via ``model_dump(mode="json")``;+ ``None`` maps to an empty cell (never the string ``"None"``). ``row_model`` must be+ CSV-flat (see ``flat_field_names``).+ """+ assert_supported_table_suffix(path)+ headers = flat_field_names(row_model)+ rows: list[dict[str, str]] = []+ for item_index, item in enumerate(list_content.items):+ if not isinstance(item, row_model):+ msg = f"Cannot write CSV: item {item_index} is a {type(item).__name__}, not the declared row model {row_model.__name__!r}."+ raise CsvError(msg)+ dumped = item.model_dump(mode="json")+ rows.append({field_name: _to_cell(dumped.get(field_name)) for field_name in headers})+ write_rows(path, headers, rows, delimiter=delimiter, encoding=encoding)+++def _validation_error_label(error: "ErrorDetails") -> str:+ """Best field label for one pydantic error: the dotted ``loc``, else the error type.++ A model-level failure (e.g. a ``model_validator``) has an empty ``loc``; falling back to+ the error type keeps the coercion message from naming an unhelpful ``field(s) ?``.+ """+ loc = error.get("loc") or ()+ if loc:+ return ".".join(str(part) for part in loc)+ return str(error.get("type", "model"))+++def _to_cell(value: Any) -> str:+ """Serialize one ``model_dump(mode="json")`` scalar to a CSV cell.++ ``None`` becomes an empty cell (never ``"None"``); booleans become lowercase+ ``true``/``false`` (round-trip-safe under pydantic's lax bool coercion).+ """+ if value is None:+ return ""+ if isinstance(value, bool):+ return "true" if value else "false"+ return str(value)diff --git a/pipelex/tools/tabular/exceptions.py b/pipelex/tools/tabular/exceptions.pynew file mode 100644index 000000000..0838c9969--- /dev/null+++ b/pipelex/tools/tabular/exceptions.py@@ -0,0 +1,63 @@+from pipelex.base_exceptions import ErrorDomain+from pipelex.system.exceptions import ToolError+++class CsvError(ToolError):+ """Base for CSV codec errors.++ CSV problems are caller-fixable input issues (a missing/unreadable file, a header+ that does not match the row concept, a cell that cannot be coerced), so the whole+ family is classified under the ``INPUT`` error domain.+ """++ error_domain = ErrorDomain.INPUT+ _declared_title = "CSV error"+ # The whole family describes faults in the caller's OWN CSV input (a missing file, a header that+ # does not match the concept, a cell that will not coerce), so the message is safe and useful to+ # expose. Opt in so STRICT disclosure keeps it instead of redacting it to a generic placeholder.+ _authors_caller_facing_message = True+++class CsvReadError(CsvError):+ """Raised when a CSV file cannot be opened, decoded, or parsed.++ Wraps the raw ``OSError`` (missing file, bad path), ``UnicodeDecodeError`` (wrong+ encoding), and ``csv.Error`` (malformed quoting) at the codec boundary so no raw+ third-party exception escapes into core/runner.+ """++ _declared_title = "CSV read error"+++class CsvFlatnessError(CsvError):+ """Raised when a concept used with CSV is not flat.++ A CSV row concept must have scalar fields only (``text``/``integer``/``number``/+ ``boolean``/``date`` plus optionals and ``Literal``/choice-constrained scalars).+ A nested/list/dict/concept-typed/``Union``/``Any`` field is rejected, naming the+ offending field and telling the author to project to a flat concept first.+ """++ _declared_title = "CSV flatness error"+++class CsvColumnError(CsvError):+ """Raised when CSV headers do not line up with the row concept's fields.++ Covers an unexpected extra column, a missing required column, and duplicate or+ blank header cells. A missing *optional* column is NOT an error: a nullable field+ is set to ``None`` for all rows, while a non-nullable field with a default keeps+ that default.+ """++ _declared_title = "CSV column error"+++class CsvCoercionError(CsvError):+ """Raised when a CSV cell value cannot be coerced to its declared field type.++ Carries the 1-based row/column, the concept, and the field so the author can find+ the offending cell.+ """++ _declared_title = "CSV coercion error"diff --git a/tests/e2e/pipelex/cli/test_csv_run.py b/tests/e2e/pipelex/cli/test_csv_run.pynew file mode 100644index 000000000..3a91be0fc--- /dev/null+++ b/tests/e2e/pipelex/cli/test_csv_run.py@@ -0,0 +1,175 @@+"""E2E + wiring tests for ``--save-csv`` and CSV input via the ``pipelex run`` CLI (Phase 1 — RED).++Two cheap, no-subprocess wiring checks pin that the run subcommands declare ``--save-csv``+and forward it to ``execute_run``. Two ``gha_disabled`` subprocess checks exercise the real+binary: a happy ``--save-csv`` dry-run that must drop ``summaries.csv`` at the literal cwd+path (CQ1), and a missing-input-``.csv`` run that must fail cleanly (A2 — typed error naming+the file, no Python traceback).++The subprocess tests inherit the ambient environment (they need a working ``.pipelex``+config to boot) and run with ``cwd`` set to a tmp dir so ``--save-csv summaries.csv`` and any+on-disk side effects stay out of the source tree.+"""++from __future__ import annotations++import csv+import os+import shutil+import subprocess # noqa: S404 — invokes the real pipelex binary for E2E coverage+from inspect import signature+from pathlib import Path+from typing import TYPE_CHECKING++import pytest++from pipelex.cli.commands.run.bundle_cmd import run_bundle_cmd+from pipelex.cli.commands.run.method_cmd import run_method_cmd+from pipelex.cli.commands.run.pipe_cmd import run_pipe_cmd++if TYPE_CHECKING:+ from collections.abc import Callable++ from pytest_mock import MockerFixture++REPO_ROOT = Path(__file__).resolve().parents[4]+PIPELEX_BIN = REPO_ROOT / ".venv" / "bin" / "pipelex"+BUNDLE_SRC = REPO_ROOT / "tests" / "integration" / "pipelex" / "csv" / "csv_demo"+++def _stage_bundle(dest_root: Path) -> Path:+ """Copy the csv_demo bundle into ``dest_root`` so CLI side effects don't touch the source tree."""+ staged = dest_root / "csv_demo"+ shutil.copytree(BUNDLE_SRC, staged)+ return staged+++class TestCsvRun:+ # ----------------------------------------------------------------------------------+ # Cheap wiring checks (no subprocess) — T2+ # ----------------------------------------------------------------------------------++ @pytest.mark.parametrize("command", [run_pipe_cmd, run_bundle_cmd, run_method_cmd])+ def test_run_command_declares_save_csv(self, command: Callable[..., None]) -> None:+ assert "save_csv" in signature(command).parameters, f"{command.__name__} must declare a --save-csv option"++ def test_bundle_cmd_forwards_save_csv(self, mocker: MockerFixture) -> None:+ mock_execute = mocker.patch("pipelex.cli.commands.run.bundle_cmd.execute_run")+ # Build the kwargs dynamically so this stays type-clean before `save_csv` exists+ # (Phase 4 adds it); the RED failure is a runtime TypeError, not a type-check error.+ call_kwargs: dict[str, object] = {"path": str(BUNDLE_SRC), "save_csv": "out.csv", "dry_run": True}+ run_bundle_cmd(**call_kwargs) # type: ignore[arg-type]+ assert mock_execute.call_args is not None+ assert mock_execute.call_args.kwargs.get("save_csv") == "out.csv"++ # ----------------------------------------------------------------------------------+ # Subprocess E2E — gha_disabled (slow; needs a working .pipelex config)+ # ----------------------------------------------------------------------------------++ @pytest.mark.gha_disabled+ def test_save_csv_writes_file_at_literal_path(self, tmp_path: Path) -> None:+ staged = _stage_bundle(tmp_path)+ result = subprocess.run( # noqa: S603+ [+ str(PIPELEX_BIN),+ "run",+ "pipe",+ "summarize_people",+ "-L",+ str(staged),+ "--inputs",+ str(staged / "inputs.json"),+ "--save-csv",+ "summaries.csv",+ "--dry-run",+ "--no-save-working-memory",+ "--no-save-main-stuff",+ "--no-graph",+ ],+ cwd=str(tmp_path),+ env=os.environ.copy(),+ capture_output=True,+ text=True,+ check=False,+ timeout=180,+ )+ assert result.returncode == 0, f"run must succeed.\nstdout={result.stdout!r}\nstderr={result.stderr!r}"++ out_csv = tmp_path / "summaries.csv"+ assert out_csv.exists(), f"--save-csv must write summaries.csv at the literal cwd path; cwd contents: {list(tmp_path.iterdir())}"+ with out_csv.open(encoding="utf-8", newline="") as csv_file:+ reader = csv.DictReader(csv_file)+ assert reader.fieldnames == ["name", "country", "summary"]+ records = list(reader)+ # Count parsed records, not physical lines: the dry-mode summary cell is multi-line,+ # so the codec RFC-quotes it and one logical row spans several physical lines.+ assert len(records) == 3 # one record per input person++ @pytest.mark.gha_disabled+ def test_save_csv_non_list_output_fails_cleanly(self, tmp_path: Path) -> None:+ # --save-csv requires a flat LIST output. `summarize_person` produces a single PersonSummary+ # (not a list), so the run must fail cleanly with the "not a list" guard, framed as a+ # --save-csv failure (not a pipeline failure) and with no raw traceback.+ staged = _stage_bundle(tmp_path)+ result = subprocess.run( # noqa: S603+ [+ str(PIPELEX_BIN),+ "run",+ "pipe",+ "summarize_person",+ "-L",+ str(staged),+ "--mock-inputs",+ "--dry-run",+ "--save-csv",+ "out.csv",+ "--no-save-working-memory",+ "--no-save-main-stuff",+ "--no-graph",+ ],+ cwd=str(tmp_path),+ env=os.environ.copy(),+ capture_output=True,+ text=True,+ check=False,+ timeout=180,+ )+ combined = result.stdout + result.stderr+ assert result.returncode != 0, f"a non-list --save-csv output must fail.\n{combined!r}"+ assert "Traceback (most recent call last)" not in combined, f"error must be clean: {combined!r}"+ assert "not a list" in combined, f"the error should explain the output is not a list: {combined!r}"+ assert not (tmp_path / "out.csv").exists(), "no CSV should be written when the output isn't a list"++ @pytest.mark.gha_disabled+ def test_missing_csv_input_fails_cleanly(self, tmp_path: Path) -> None:+ staged = _stage_bundle(tmp_path)+ (staged / "inputs_missing.json").write_text(+ '{ "people": { "concept": "csv_demo.Person", "content": { "url": "nope.csv" } } }',+ encoding="utf-8",+ )+ result = subprocess.run( # noqa: S603+ [+ str(PIPELEX_BIN),+ "run",+ "pipe",+ "summarize_people",+ "-L",+ str(staged),+ "--inputs",+ str(staged / "inputs_missing.json"),+ "--dry-run",+ "--no-save-working-memory",+ "--no-save-main-stuff",+ "--no-graph",+ ],+ cwd=str(tmp_path),+ env=os.environ.copy(),+ capture_output=True,+ text=True,+ check=False,+ timeout=180,+ )+ combined = result.stdout + result.stderr+ assert result.returncode != 0, f"a missing input .csv must fail.\n{combined!r}"+ assert "Traceback (most recent call last)" not in combined, f"error must be clean, not a raw traceback: {combined!r}"+ assert "nope.csv" in combined, f"the error should name the offending CSV file: {combined!r}"diff --git a/tests/integration/pipelex/csv/csv_demo/csv_demo.mthds b/tests/integration/pipelex/csv/csv_demo/csv_demo.mthdsnew file mode 100644index 000000000..869cc3593--- /dev/null+++ b/tests/integration/pipelex/csv/csv_demo/csv_demo.mthds@@ -0,0 +1,62 @@+domain = "csv_demo"+description = "CSV round-trip demo: summarize people, one persona sentence each"++[concept.Person]+description = "A person record read from a CSV row"+[concept.Person.structure]+name = { type = "text", required = true, description = "Full name" }+job = { type = "text", required = true, description = "Occupation" }+country = { type = "text", required = true, description = "Country of origin" }+birth_year = { type = "integer", required = true, description = "Year of birth" }+death_year = { type = "integer", required = false, description = "Year of death, if deceased" }++[concept.PersonSummary]+description = "A person row reduced to name + country plus a one-sentence persona summary"+[concept.PersonSummary.structure]+name = { type = "text", required = true, description = "Full name" }+country = { type = "text", required = true, description = "Country of origin" }+summary = { type = "text", required = true, description = "One-sentence persona summary" }++# Batch over the list — branch runs once per person+[pipe.summarize_people]+type = "PipeBatch"+description = "Summarize each person in the list"+inputs = { people = "Person[]" }+output = "PersonSummary[]"+branch_pipe_code = "summarize_person"+input_list_name = "people"+input_item_name = "person"++# Per-person branch: describe (LLM) then compose the row+[pipe.summarize_person]+type = "PipeSequence"+description = "Describe one person, then compose their summary row"+inputs = { person = "Person" }+output = "PersonSummary"+steps = [+ { pipe = "describe_person", result = "description" },+ { pipe = "compose_person_summary", result = "person_summary" },+]++# LLM produces ONLY the description text+[pipe.describe_person]+type = "PipeLLM"+description = "Write a one-sentence persona summary for a person"+inputs = { person = "Person" }+output = "Text"+prompt = """+Write a single vivid sentence summarizing this person as a persona, drawing on their job, country, and lifespan.+@person+"""++# Compose the output row from SELECTED original fields + the description (drops job, birth_year, death_year)+[pipe.compose_person_summary]+type = "PipeCompose"+description = "Build the summary row from selected Person fields plus the generated description"+inputs = { person = "Person", description = "Text" }+output = "PersonSummary"++[pipe.compose_person_summary.construct]+name = { from = "person.name" }+country = { from = "person.country" }+summary = { from = "description" }diff --git a/tests/integration/pipelex/csv/csv_demo/inputs.json b/tests/integration/pipelex/csv/csv_demo/inputs.jsonnew file mode 100644index 000000000..4ee66cb7a--- /dev/null+++ b/tests/integration/pipelex/csv/csv_demo/inputs.json@@ -0,0 +1 @@+{ "people": { "concept": "csv_demo.Person", "content": { "url": "people.csv" } } }diff --git a/tests/integration/pipelex/csv/csv_demo/people.csv b/tests/integration/pipelex/csv/csv_demo/people.csvnew file mode 100644index 000000000..448e1c782--- /dev/null+++ b/tests/integration/pipelex/csv/csv_demo/people.csv@@ -0,0 +1,4 @@+name,job,country,birth_year,death_year+Ada Lovelace,Mathematician,United Kingdom,1815,1852+Grace Hopper,Computer Scientist,United States,1906,1992+Vint Cerf,Computer Scientist,United States,1943,diff --git a/tests/integration/pipelex/csv/test_csv_input_detection.py b/tests/integration/pipelex/csv/test_csv_input_detection.pynew file mode 100644index 000000000..e4ba97690--- /dev/null+++ b/tests/integration/pipelex/csv/test_csv_input_detection.py@@ -0,0 +1,162 @@+"""Integration tests pinning CSV *input detection* in ``StuffFactory`` (Phase 3).++These fence the A3/CT1 design decision — CSV input is triggered by a ``{"url": "...csv"}``+reference under a non-native structured row concept — and its documented residual:++- a non-``.csv`` ``url`` under a concept that has a ``url`` field stays a single record;+- a ``.csv`` whose rows include a ``url`` column reads correctly (the file-level ``url`` is the+ table to read; the inner ``url`` column is ordinary cell data);+- a remote ``url`` (``http(s)``/``s3``/``gs``/…) with a ``.csv`` suffix is rejected with a clear+ local-paths-only error (A1), never opened;+- the read is eager core I/O: rebuilding the working memory after the file changes on disk+ reflects the new content.+"""++from collections.abc import Callable+from pathlib import Path+from typing import TYPE_CHECKING, Any, cast++import pytest++from pipelex.cli.commands.run._inputs_path_resolver import resolve_inputs_paths # noqa: PLC2701+from pipelex.core.memory.working_memory_factory import WorkingMemoryFactory+from pipelex.core.stuffs.list_content import ListContent+from pipelex.tools.tabular.exceptions import CsvError++if TYPE_CHECKING:+ from mthds.models.pipeline_inputs import PipelineInputs++ from pipelex.core.stuffs.stuff_content import StuffContent++BUNDLE_DIR = Path(__file__).parent / "csv_demo"+PEOPLE_CSV = BUNDLE_DIR / "people.csv"++# A separate fixture bundle whose `Link` concept declares its own `url` field — used to fence the+# url-key ambiguity in input detection (concept-with-a-url-field vs. the {"url": ...} table wrapper).+URL_FIELD_DIR = Path(__file__).parent / "url_field_concept"+LINKS_CSV = URL_FIELD_DIR / "links.csv"+++class TestCsvInputDetection:+ @pytest.mark.parametrize(+ "remote_url",+ [+ "https://example.com/people.csv",+ "s3://bucket/people.csv",+ "gs://bucket/people.csv",+ # Query-string / fragment forms must still be detected as tabular and rejected: pathlib's+ # `.suffix` keeps the `?…`/`#…`, so detection must look at the URL path component, not the+ # raw string (S3 presigned URLs always carry a query string).+ "https://example.com/people.csv?token=abc",+ "s3://bucket/people.csv?X-Amz-Signature=deadbeef",+ "https://example.com/people.csv#frag",+ # A malformed port must not crash URL sanitization (the `.port` property raises): the+ # redaction must still produce a clean CsvError without leaking the token via a traceback.+ "https://example.com:bad/people.csv?token=abc",+ ],+ )+ def test_remote_csv_url_rejected(self, remote_url: str, load_test_library: Callable[[list[Path]], None]) -> None:+ load_test_library([BUNDLE_DIR])+ inputs: PipelineInputs = {"people": {"concept": "csv_demo.Person", "content": {"url": remote_url}}}+ with pytest.raises(CsvError) as exc_info:+ WorkingMemoryFactory.make_from_pipeline_inputs(inputs)+ message = str(exc_info.value)+ assert "local" in message.lower()+ # The path is preserved so the user can identify the offending input...+ assert "people.csv" in message+ # ...but the signed query string / fragment (which can carry credentials) must be stripped:+ # CsvError is caller-facing and survives STRICT disclosure, so it must not leak the token.+ for secret in ("token=abc", "X-Amz-Signature=deadbeef", "frag"):+ assert secret not in message++ def test_remote_csv_url_rejected_through_cli_inputs_resolution(self, load_test_library: Callable[[list[Path]], None]) -> None:+ load_test_library([BUNDLE_DIR])+ # Simulate the CLI JSON-file path: resolve_inputs_paths runs first. An s3:// url must NOT be+ # rewritten into a local-looking path (which would defeat the remote guard) — the rejection+ # must still fire end to end.+ raw_inputs: dict[str, Any] = {"people": {"concept": "csv_demo.Person", "content": {"url": "s3://bucket/people.csv"}}}+ resolved = resolve_inputs_paths(raw_inputs, Path("/some/base/dir"))+ assert resolved["people"]["content"]["url"] == "s3://bucket/people.csv" # not rewritten+ inputs = cast("PipelineInputs", resolved)+ with pytest.raises(CsvError) as exc_info:+ WorkingMemoryFactory.make_from_pipeline_inputs(inputs)+ assert "local" in str(exc_info.value).lower()++ def test_xlsx_input_routes_to_tabular_seam(self, load_test_library: Callable[[list[Path]], None]) -> None:+ load_test_library([BUNDLE_DIR])+ # A .xlsx url is detected as tabular and routed through the hook to the format seam, which+ # raises a clear "needs pipelex[tabular]" CsvError (the Excel backend isn't built in v1).+ inputs: PipelineInputs = {"people": {"concept": "csv_demo.Person", "content": {"url": "people.xlsx"}}}+ with pytest.raises(CsvError) as exc_info:+ WorkingMemoryFactory.make_from_pipeline_inputs(inputs)+ assert "pipelex[tabular]" in str(exc_info.value)++ def test_unparseable_url_rejected_without_leak(self, load_test_library: Callable[[list[Path]], None]) -> None:+ load_test_library([BUNDLE_DIR])+ # A malformed url (a bad IPv6 bracket) makes `urlsplit` itself raise; it must become a clean+ # CsvError, never a raw ValueError that could surface the token-bearing url in a traceback.+ bad_url = "https://[bad/people.csv?token=abc"+ inputs: PipelineInputs = {"people": {"concept": "csv_demo.Person", "content": {"url": bad_url}}}+ with pytest.raises(CsvError) as exc_info:+ WorkingMemoryFactory.make_from_pipeline_inputs(inputs)+ message = str(exc_info.value)+ assert "local" in message.lower()+ # The unparseable url can't be sanitized, so it is not echoed at all — the token must not leak.+ assert "token=abc" not in message++ def test_non_csv_url_stays_record(self, load_test_library: Callable[[list[Path]], None]) -> None:+ load_test_library([URL_FIELD_DIR])+ # url field value that is NOT a tabular suffix → ordinary single record, not a CSV table.+ inputs: PipelineInputs = {"link": {"concept": "url_field_concept.Link", "content": {"label": "Home", "url": "https://example.com"}}}+ working_memory = WorkingMemoryFactory.make_from_pipeline_inputs(inputs)++ content = working_memory.get_stuff("link").content+ assert not isinstance(content, ListContent)+ record = content.model_dump()+ assert record["label"] == "Home"+ assert record["url"] == "https://example.com"++ def test_record_with_csv_url_field_and_siblings_stays_record(self, load_test_library: Callable[[list[Path]], None]) -> None:+ load_test_library([URL_FIELD_DIR])+ # A record that has sibling keys alongside a .csv-suffixed `url` is NOT a table reference:+ # detection is gated to the bare {"url": ...} wrapper, so the siblings are never dropped.+ inputs: PipelineInputs = {"link": {"concept": "url_field_concept.Link", "content": {"label": "Home", "url": "report.csv"}}}+ working_memory = WorkingMemoryFactory.make_from_pipeline_inputs(inputs)++ content = working_memory.get_stuff("link").content+ assert not isinstance(content, ListContent)+ record = content.model_dump()+ assert record["label"] == "Home"+ assert record["url"] == "report.csv"++ def test_csv_with_url_column_reads(self, load_test_library: Callable[[list[Path]], None]) -> None:+ load_test_library([URL_FIELD_DIR])+ # The file-level url ends .csv → read as a table; the inner `url` column is plain data.+ inputs: PipelineInputs = {"links": {"concept": "url_field_concept.Link", "content": {"url": str(LINKS_CSV)}}}+ working_memory = WorkingMemoryFactory.make_from_pipeline_inputs(inputs)++ content = working_memory.get_stuff("links").content+ assert isinstance(content, ListContent)+ links = cast("ListContent[StuffContent]", content)+ rows = [item.model_dump() for item in links.items]+ assert [row["label"] for row in rows] == ["Home", "Docs"]+ assert [row["url"] for row in rows] == ["https://example.com", "https://example.com/docs"]++ def test_eager_read_reflects_file_mutation(self, tmp_path: Path, load_test_library: Callable[[list[Path]], None]) -> None:+ load_test_library([BUNDLE_DIR])+ people_csv = tmp_path / "people.csv"+ people_csv.write_text(PEOPLE_CSV.read_text(encoding="utf-8"), encoding="utf-8")+ inputs: PipelineInputs = {"people": {"concept": "csv_demo.Person", "content": {"url": str(people_csv)}}}++ first = WorkingMemoryFactory.make_from_pipeline_inputs(inputs)+ first_people = cast("ListContent[StuffContent]", first.get_stuff("people").content)+ assert len(first_people.items) == 3++ # Mutate the file on disk; a fresh build must re-read it (eager core I/O, not cached).+ people_csv.write_text(+ "name,job,country,birth_year,death_year\nAda Lovelace,Mathematician,United Kingdom,1815,1852\n",+ encoding="utf-8",+ )+ second = WorkingMemoryFactory.make_from_pipeline_inputs(inputs)+ second_people = cast("ListContent[StuffContent]", second.get_stuff("people").content)+ assert len(second_people.items) == 1diff --git a/tests/integration/pipelex/csv/test_csv_roundtrip.py b/tests/integration/pipelex/csv/test_csv_roundtrip.pynew file mode 100644index 000000000..152888d49--- /dev/null+++ b/tests/integration/pipelex/csv/test_csv_roundtrip.py@@ -0,0 +1,116 @@+"""Integration tests for the CSV round-trip (Phase 1 — RED).++Split per the plan (T1) into three independent assertions so each fails for a single,+nameable reason:++- (a) input-codec: a ``.csv`` ``url`` under a flat row concept builds a+ ``ListContent[Person]`` (no pipeline run). RED today because ``stuff_factory`` Case 2.5+ treats ``{"url": ...}`` as a record dict and pydantic rejects it.+- (b) output-codec: a hand-built ``ListContent[PersonSummary]`` writes to CSV with the+ declared header + one row per item. RED today because the codec is unimplemented.+- (c) full pipeline: ``PipeBatch -> PipeSequence(PipeLLM + PipeCompose)`` over the parsed+ rows yields one ``PersonSummary`` per input row, with ``name``/``country`` carried from+ the real CSV rows (verified to hold in dry mode). RED today because the input ``.csv``+ is not parsed yet.++``death_year`` is asserted on the input list only — ``PersonSummary`` drops it.+"""++from collections.abc import Callable+from pathlib import Path+from typing import TYPE_CHECKING, cast++import pytest+from mthds.models.pipeline_inputs import PipelineInputs++from pipelex.core.memory.working_memory_factory import WorkingMemoryFactory+from pipelex.core.stuffs.list_content import ListContent+from pipelex.hub import get_library_manager+from pipelex.pipe_run.pipe_run_mode import PipeRunMode+from pipelex.pipeline.runner import PipelexRunner+from pipelex.tools.tabular.csv_codec import csv_from_list_content++if TYPE_CHECKING:+ from pipelex.core.stuffs.stuff_content import StuffContent++BUNDLE_DIR = Path(__file__).parent / "csv_demo"+PEOPLE_CSV = BUNDLE_DIR / "people.csv"++EXPECTED_NAMES = {"Ada Lovelace", "Grace Hopper", "Vint Cerf"}+EXPECTED_COUNTRIES = {"United Kingdom", "United States"}+++def people_inputs() -> PipelineInputs:+ """Pipeline inputs pointing the row concept at the local ``people.csv`` (absolute path)."""+ return {"people": {"concept": "csv_demo.Person", "content": {"url": str(PEOPLE_CSV)}}}+++class TestCsvRoundtrip:+ # ----------------------------------------------------------------------------------+ # (a) input-codec — a .csv url under a flat concept becomes ListContent[Person]+ # ----------------------------------------------------------------------------------++ def test_csv_input_builds_typed_list(self, load_test_library: Callable[[list[Path]], None]) -> None:+ load_test_library([BUNDLE_DIR])+ working_memory = WorkingMemoryFactory.make_from_pipeline_inputs(people_inputs())++ people = working_memory.get_stuff("people").content+ assert isinstance(people, ListContent)+ people_list = cast("ListContent[StuffContent]", people)+ assert len(people_list.items) == 3++ rows = [item.model_dump() for item in people_list.items]+ assert [row["name"] for row in rows] == ["Ada Lovelace", "Grace Hopper", "Vint Cerf"]+ # CSV strings coerced to declared field types.+ assert rows[0]["birth_year"] == 1815+ assert isinstance(rows[0]["birth_year"], int)+ assert rows[0]["death_year"] == 1852+ # Trailing blank death_year cell → None on the optional field (Vint Cerf, still living).+ assert rows[2]["death_year"] is None++ # ----------------------------------------------------------------------------------+ # (b) output-codec — a hand-built ListContent[PersonSummary] writes to CSV+ # ----------------------------------------------------------------------------------++ def test_list_content_writes_to_csv(self, tmp_path: Path, load_test_library: Callable[[list[Path]], None]) -> None:+ load_test_library([BUNDLE_DIR])+ summary_concept = get_library_manager().get_current_library().concept_library.get_required_concept("csv_demo.PersonSummary")+ summary_class = summary_concept.get_structure_class()++ items = [+ summary_class.model_validate({"name": "Ada Lovelace", "country": "United Kingdom", "summary": "A visionary mathematician."}),+ summary_class.model_validate({"name": "Grace Hopper", "country": "United States", "summary": "A pioneering computer scientist."}),+ ]+ list_content: ListContent[StuffContent] = ListContent(items=items)++ out_path = tmp_path / "summaries.csv"+ csv_from_list_content(list_content, row_model=summary_class, path=out_path)++ lines = out_path.read_text(encoding="utf-8").splitlines()+ assert lines[0] == "name,country,summary"+ assert lines[1] == "Ada Lovelace,United Kingdom,A visionary mathematician."+ assert lines[2] == "Grace Hopper,United States,A pioneering computer scientist."++ # ----------------------------------------------------------------------------------+ # (c) full pipeline — CSV in → batch+compose → PersonSummary[] (dry mode)+ # ----------------------------------------------------------------------------------++ @pytest.mark.dry_runnable+ @pytest.mark.llm+ @pytest.mark.inference+ @pytest.mark.asyncio+ async def test_full_pipeline_round_trip(self, pipe_run_mode: PipeRunMode) -> None:+ runner = PipelexRunner(library_dirs=[str(BUNDLE_DIR)], pipe_run_mode=pipe_run_mode)+ response = await runner.execute_pipeline(pipe_code="summarize_people", inputs=people_inputs())++ content = response.pipe_output.main_stuff.content+ assert isinstance(content, ListContent)+ summaries = cast("ListContent[StuffContent]", content)+ assert len(summaries.items) == 3++ rows = [item.model_dump() for item in summaries.items]+ # PipeCompose carries the real CSV name/country through even in dry mode.+ assert {row["name"] for row in rows} == EXPECTED_NAMES+ assert {row["country"] for row in rows} <= EXPECTED_COUNTRIES+ # The summary is mock text in dry mode — assert it is present, not its content.+ assert all(row["summary"] for row in rows)diff --git a/tests/integration/pipelex/csv/url_field_concept/links.csv b/tests/integration/pipelex/csv/url_field_concept/links.csvnew file mode 100644index 000000000..075b2dcd9--- /dev/null+++ b/tests/integration/pipelex/csv/url_field_concept/links.csv@@ -0,0 +1,3 @@+label,url+Home,https://example.com+Docs,https://example.com/docsdiff --git a/tests/integration/pipelex/csv/url_field_concept/url_field_concept.mthds b/tests/integration/pipelex/csv/url_field_concept/url_field_concept.mthdsnew file mode 100644index 000000000..7572cd816--- /dev/null+++ b/tests/integration/pipelex/csv/url_field_concept/url_field_concept.mthds@@ -0,0 +1,22 @@+domain = "url_field_concept"+description = "Detection-test fixture: a real concept whose own structure declares a `url` field"++# Why this bundle exists (it is NOT a runnable demo — there are no pipes on purpose):+#+# Pipelex reads a CSV file when a pipeline input is the bare wrapper `{ "url": "<...>.csv" }`+# under a row concept. But `url` is also a perfectly ordinary field name a concept may declare+# for its own data. `Link` is exactly such a concept — a real-world shape that legitimately+# carries a `url` field — so it lets the input-detection tests fence the ambiguous corners:+#+# { "label": ..., "url": "https://example.com" } -> stays ONE record (url value is not a tabular path)+# { "label": ..., "url": "report.csv" } -> stays ONE record (sibling `label` => not a table ref)+# { "url": "links.csv" } (and links.csv has a `url` column) -> reads as a TABLE; the inner+# `url` column is ordinary cell data+#+# The detection tests live in tests/integration/pipelex/csv/test_csv_input_detection.py.++[concept.Link]+description = "A labelled hyperlink — a real concept that naturally has a `url` field"+[concept.Link.structure]+label = { type = "text", required = true, description = "Human-readable label" }+url = { type = "text", required = true, description = "Target URL — ordinary cell data, not a table reference" }diff --git a/tests/unit/pipelex/cli/test_inputs_path_resolver.py b/tests/unit/pipelex/cli/test_inputs_path_resolver.pyindex 1e87edaf6..6d62d84c0 100644--- a/tests/unit/pipelex/cli/test_inputs_path_resolver.py+++ b/tests/unit/pipelex/cli/test_inputs_path_resolver.py@@ -20,6 +20,10 @@ class _IsRelativeLocalPathCases:DATA_URL = ("data:application/pdf;base64,AAAA", False)PIPELEX_STORAGE = ("pipelex-storage://bucket/file.pdf", False)FILE_URI = ("file:///home/user/file.pdf", False)+ # Unrecognized schemes must NOT be rewritten as relative local paths: resolve_uri returns them as+ # a ResolvedLocalPath but their `://` shows they are non-local, so the downstream remote guards work.+ S3_URL = ("s3://bucket/people.csv", False)+ GS_URL = ("gs://bucket/people.csv", False)class _ResolveInputsPathsCases:@@ -211,6 +215,8 @@ class TestInputsPathResolver:_IsRelativeLocalPathCases.DATA_URL,_IsRelativeLocalPathCases.PIPELEX_STORAGE,_IsRelativeLocalPathCases.FILE_URI,+ _IsRelativeLocalPathCases.S3_URL,+ _IsRelativeLocalPathCases.GS_URL,],)def test_is_relative_local_path(self, uri: str, expected: bool) -> None:diff --git a/tests/unit/pipelex/exceptions/test_class_level_metadata.py b/tests/unit/pipelex/exceptions/test_class_level_metadata.pyindex 961805870..e05a5e0bd 100644--- a/tests/unit/pipelex/exceptions/test_class_level_metadata.py+++ b/tests/unit/pipelex/exceptions/test_class_level_metadata.py@@ -27,6 +27,7 @@ from pipelex.system.pipelex_service.exceptions import (PipelexServiceError,RemoteConfigFetchError,)+from pipelex.tools.tabular.exceptions import CsvError_PIPELINE_EXEC_ERROR = PipelineExecutionError(message="boom",@@ -102,6 +103,7 @@ class TestClassLevelMetadata:[("interpreter", PipelexInterpreterError("boom"), True),("validate_bundle", ValidateBundleError("boom"), True),+ ("csv", CsvError("boom"), True),("signatures_not_allowed", SignaturesNotAllowedError(offending_pipe_refs=set(), signature_refs=set(), dep_paths={}), True),("config", PipelexConfigError("boom"), False),("pipe_execution", PipeExecutionError("boom"), False),diff --git a/tests/unit/pipelex/tools/tabular/test_csv_codec.py b/tests/unit/pipelex/tools/tabular/test_csv_codec.pynew file mode 100644index 000000000..f80376292--- /dev/null+++ b/tests/unit/pipelex/tools/tabular/test_csv_codec.py@@ -0,0 +1,604 @@+"""Unit tests locking the CSV codec contract (Phase 1 — RED).++These exercise the codec in isolation (no library load, no pipe runtime): read/write row+primitives, the shared flatness classifier, concept-bound read/write with pydantic+coercion, the strict-column rules, the ``None`` <-> empty-cell mapping on both sides, the+empty-list header-only contract, the format seam, and the accept/reject coercion table.++All assertions are RED until Phase 2 fills in ``csv_codec`` — the skeleton currently+raises ``NotImplementedError`` from every function.+"""++import csv+from datetime import date+from enum import IntEnum+from pathlib import Path+from typing import Any, Literal++import pytest+from pydantic import model_validator++from pipelex.core.stuffs.list_content import ListContent+from pipelex.core.stuffs.structured_content import StructuredContent+from pipelex.tools.tabular.csv_codec import (+ assert_supported_table_suffix,+ csv_from_list_content,+ flat_field_names,+ list_content_from_csv,+ read_rows,+ write_rows,+)+from pipelex.tools.tabular.exceptions import (+ CsvCoercionError,+ CsvColumnError,+ CsvError,+ CsvFlatnessError,+ CsvReadError,+)+from pipelex.types import Self, StrEnum++# ---------------------------------------------------------------------------------------+# Flat row models (CSV-legal: scalar fields only)+# ---------------------------------------------------------------------------------------+++class FlatRow(StructuredContent):+ name: str+ age: int+ height: float+ active: bool+ born: date+ nickname: str | None = None+++class IntRow(StructuredContent):+ value: int+++class FloatRow(StructuredContent):+ value: float+++class BoolRow(StructuredContent):+ value: bool+++class DateRow(StructuredContent):+ value: date+++class OptionalTextRow(StructuredContent):+ text: str | None = None+++class RequiredTextRow(StructuredContent):+ text: str+++class OptPairRow(StructuredContent):+ a: str | None = None+ b: str | None = None+++class OrderedPairRow(StructuredContent):+ a: int+ b: int++ @model_validator(mode="after")+ def _check_order(self) -> Self:+ if self.a > self.b:+ msg = "a must be <= b"+ raise ValueError(msg)+ return self+++class LiteralRow(StructuredContent):+ name: str+ rating: Literal["low", "high"]+++class Grade(StrEnum):+ PASS = "pass"+ FAIL = "fail"+++class StrEnumRow(StructuredContent):+ name: str+ grade: Grade+++class OptDefaultRow(StructuredContent):+ name: str+ nickname: str | None = "anon"+++class IntDefaultRow(StructuredContent):+ name: str+ count: int = 0+++class LabeledIntRow(StructuredContent):+ label: str+ number: int+++# ---------------------------------------------------------------------------------------+# Non-flat row models (must be rejected by the flatness classifier)+# ---------------------------------------------------------------------------------------+++class Priority(IntEnum):+ LOW = 1+ HIGH = 2+++class IntLiteralRow(StructuredContent):+ value: Literal[1, 2, 3]+++class IntEnumRow(StructuredContent):+ priority: Priority+++class Inner(StructuredContent):+ x: int+++class WithList(StructuredContent):+ tags: list[str]+++class WithDict(StructuredContent):+ meta: dict[str, str]+++class WithNested(StructuredContent):+ inner: Inner+++class WithUnion(StructuredContent):+ value: int | str+++class WithAny(StructuredContent):+ value: Any+++def write_csv_file(directory: Path, content: str, name: str = "data.csv", encoding: str = "utf-8") -> Path:+ """Write *content* to ``directory/name`` and return the path."""+ path = directory / name+ path.write_text(content, encoding=encoding)+ return path+++def write_value_csv(directory: Path, cell: str, name: str = "data.csv", header: str = "value") -> Path:+ """Write a one-column CSV with a single data cell, CSV-quoting it correctly.++ Building the data row through ``csv.writer`` (rather than an f-string) keeps a cell+ that contains the delimiter — e.g. the comma-decimal ``1,5`` — a single field instead+ of silently splitting it into two columns. An empty ``cell`` is written as a quoted+ ``""`` (a genuine empty-cell row that reads back as ``['']``), distinct from a blank+ physical line (``[]``) which the codec skips.+ """+ path = directory / name+ with path.open("w", encoding="utf-8", newline="") as csv_file:+ writer = csv.writer(csv_file)+ writer.writerow([header])+ writer.writerow([cell])+ return path+++class TestCsvCodec:+ # ----------------------------------------------------------------------------------+ # read_rows / write_rows primitives+ # ----------------------------------------------------------------------------------++ def test_read_rows_happy(self, tmp_path: Path) -> None:+ path = write_csv_file(tmp_path, "name,age\nAda,36\nGrace,85\n")+ rows = read_rows(path)+ assert rows == [{"name": "Ada", "age": "36"}, {"name": "Grace", "age": "85"}]++ def test_read_rows_custom_delimiter(self, tmp_path: Path) -> None:+ path = write_csv_file(tmp_path, "name;age\nAda;36\n", name="semi.csv")+ rows = read_rows(path, delimiter=";")+ assert rows == [{"name": "Ada", "age": "36"}]++ def test_read_strips_utf8_bom(self, tmp_path: Path) -> None:+ # Excel's "CSV UTF-8" export prepends a BOM. With the default read encoding it must be stripped,+ # so the first header is 'name' (not 'name') and the file parses identically to its plain twin.+ path = tmp_path / "bom.csv"+ path.write_bytes(b"\xef\xbb\xbfname,age\nAda,36\n")+ assert read_rows(path) == [{"name": "Ada", "age": "36"}]++ def test_read_bom_file_into_list_content(self, tmp_path: Path) -> None:+ # The headline use case: a BOM-prefixed file (from Excel) binds to a concept without a column error.+ path = tmp_path / "bom_required.csv"+ path.write_bytes(b"\xef\xbb\xbftext\nhello\n")+ item = list_content_from_csv(path, RequiredTextRow).items[0]+ assert item.text == "hello"++ def test_read_empty_file_raises(self, tmp_path: Path) -> None:+ # A 0-byte file is a distinct path from a missing file: it must raise CsvReadError naming the header.+ path = tmp_path / "empty.csv"+ path.write_text("", encoding="utf-8")+ with pytest.raises(CsvReadError) as exc_info:+ read_rows(path)+ assert "header" in str(exc_info.value).lower()++ def test_read_header_only_into_empty_list_content(self, tmp_path: Path) -> None:+ # A valid header with zero data rows reads as an empty ListContent (the read-side empty path).+ path = write_csv_file(tmp_path, "name,age,height,active,born\n")+ result = list_content_from_csv(path, FlatRow)+ assert result.items == []++ def test_short_row_pads_missing_trailing_cell_to_none(self, tmp_path: Path) -> None:+ # A data row with FEWER cells than the header pads the missing trailing column → None on an+ # optional field (exercises _row_to_dict's padding branch, not just an explicit empty cell).+ path = write_csv_file(tmp_path, "name,age,height,active,born,nickname\nAda,36,1.7,true,1815-12-10\n")+ item = list_content_from_csv(path, FlatRow).items[0]+ assert item.nickname is None+ assert item.name == "Ada"++ def test_read_rows_missing_file_raises(self, tmp_path: Path) -> None:+ with pytest.raises(CsvReadError):+ read_rows(tmp_path / "does_not_exist.csv")++ def test_read_rows_bad_encoding_raises(self, tmp_path: Path) -> None:+ # 0xFF is not valid UTF-8; reading with utf-8 must surface a typed error.+ path = tmp_path / "latin.csv"+ path.write_bytes(b"name\nCaf\xe9\n")+ with pytest.raises(CsvReadError):+ read_rows(path, encoding="utf-8")++ def test_malformed_quote_raises(self, tmp_path: Path) -> None:+ # An unterminated quoted field: strict parsing must surface CsvReadError instead of silently+ # merging the following lines into one cell and running the pipeline on corrupted data.+ path = write_csv_file(tmp_path, 'name\n"unterminated\nAda\n')+ with pytest.raises(CsvReadError):+ read_rows(path)++ def test_duplicate_header_raises(self, tmp_path: Path) -> None:+ path = write_csv_file(tmp_path, "name,name\nAda,Lovelace\n")+ with pytest.raises(CsvColumnError):+ read_rows(path)++ def test_empty_header_row_raises(self, tmp_path: Path) -> None:+ # A leading blank line (the first row names no columns) is a malformed header, not data.+ path = write_csv_file(tmp_path, "\nname\nAda\n")+ with pytest.raises(CsvColumnError) as exc_info:+ read_rows(path)+ assert "empty header" in str(exc_info.value).lower()++ def test_blank_header_raises(self, tmp_path: Path) -> None:+ path = write_csv_file(tmp_path, "name,,country\nAda,x,UK\n")+ with pytest.raises(CsvColumnError):+ read_rows(path)++ def test_write_rows_then_read_rows_round_trips(self, tmp_path: Path) -> None:+ path = tmp_path / "out.csv"+ rows = [{"name": "Ada", "age": "36"}, {"name": "Grace", "age": "85"}]+ write_rows(path, headers=["name", "age"], rows=rows)+ assert read_rows(path) == rows++ def test_write_rows_empty_writes_header_only(self, tmp_path: Path) -> None:+ path = tmp_path / "empty.csv"+ write_rows(path, headers=["name", "age"], rows=[])+ text = path.read_text(encoding="utf-8")+ assert text.splitlines() == ["name,age"]++ # ----------------------------------------------------------------------------------+ # Flatness classifier (shared by read AND write)+ # ----------------------------------------------------------------------------------++ def test_flat_field_names_returns_declared_order(self) -> None:+ assert flat_field_names(FlatRow) == ["name", "age", "height", "active", "born", "nickname"]++ def test_flat_field_names_accepts_literal_scalar(self) -> None:+ assert flat_field_names(LiteralRow) == ["name", "rating"]++ def test_flat_field_names_accepts_str_enum(self) -> None:+ # A str-valued Enum (StrEnum) round-trips through a CSV cell, so it is flat.+ assert flat_field_names(StrEnumRow) == ["name", "grade"]++ @pytest.mark.parametrize(+ ("row_model", "offending_field"),+ [+ (WithList, "tags"),+ (WithDict, "meta"),+ (WithNested, "inner"),+ (WithUnion, "value"),+ (WithAny, "value"),+ # Non-string choices serialize but do not coerce back from a CSV string → not flat.+ (IntLiteralRow, "value"),+ (IntEnumRow, "priority"),+ ],+ )+ def test_flat_field_names_rejects_non_flat(self, row_model: type[StructuredContent], offending_field: str) -> None:+ with pytest.raises(CsvFlatnessError) as exc_info:+ flat_field_names(row_model)+ assert offending_field in str(exc_info.value)++ # ----------------------------------------------------------------------------------+ # list_content_from_csv — coercion + empty-cell semantics+ # ----------------------------------------------------------------------------------++ def test_list_content_from_csv_coerces_scalar_types(self, tmp_path: Path) -> None:+ path = write_csv_file(+ tmp_path,+ "name,age,height,active,born,nickname\nAda,36,1.7,true,1815-12-10,Countess\n",+ )+ list_content = list_content_from_csv(path, FlatRow)+ assert isinstance(list_content, ListContent)+ assert len(list_content.items) == 1+ item = list_content.items[0]+ assert item.name == "Ada"+ assert item.age == 36+ assert item.height == 1.7+ assert item.active is True+ assert item.born == date(1815, 12, 10)+ assert isinstance(item.born, date)+ assert item.nickname == "Countess"++ def test_empty_cell_on_optional_field_becomes_none(self, tmp_path: Path) -> None:+ path = write_csv_file(+ tmp_path,+ "name,age,height,active,born,nickname\nVint,82,1.8,true,1943-06-23,\n",+ )+ item = list_content_from_csv(path, FlatRow).items[0]+ assert item.nickname is None++ def test_empty_cell_on_required_field_raises(self, tmp_path: Path) -> None:+ # A genuine quoted-empty cell (not a blank line) -> None -> required int fails coercion.+ path = write_value_csv(tmp_path, "", name="req.csv")+ with pytest.raises(CsvCoercionError):+ list_content_from_csv(path, IntRow)++ @pytest.mark.parametrize(+ ("row_model", "cell", "expected"),+ [+ (BoolRow, "true", True),+ (BoolRow, "false", False),+ (BoolRow, "1", True),+ (BoolRow, "0", False),+ (BoolRow, "yes", True),+ (BoolRow, "no", False),+ (BoolRow, "on", True),+ (BoolRow, "off", False),+ (IntRow, "42", 42),+ (FloatRow, "3.14", 3.14),+ (DateRow, "1815-12-10", date(1815, 12, 10)),+ ],+ )+ def test_coercion_accept_table(self, tmp_path: Path, row_model: type[StructuredContent], cell: str, expected: object) -> None:+ path = write_value_csv(tmp_path, cell)+ item = list_content_from_csv(path, row_model).items[0]+ assert item.model_dump()["value"] == expected++ @pytest.mark.parametrize(+ ("row_model", "cell"),+ [+ (IntRow, "abc"),+ (FloatRow, "1,5"), # comma-decimal must be rejected, not silently parsed+ (DateRow, "31/12/2020"), # ambiguous / non-ISO date must be rejected+ ],+ )+ def test_coercion_reject_table(self, tmp_path: Path, row_model: type[StructuredContent], cell: str) -> None:+ path = write_value_csv(tmp_path, cell)+ with pytest.raises(CsvCoercionError):+ list_content_from_csv(path, row_model)++ def test_coercion_failure_names_row_and_field(self, tmp_path: Path) -> None:+ path = write_csv_file(tmp_path, "value\n42\noops\n")+ with pytest.raises(CsvCoercionError) as exc_info:+ list_content_from_csv(path, IntRow)+ message = str(exc_info.value)+ assert "value" in message+ assert "row 3" in message # 'oops' is on physical CSV line 3 (header=1, '42'=2, 'oops'=3)++ def test_coercion_error_does_not_leak_cell_value(self, tmp_path: Path) -> None:+ # CsvError is caller-facing (survives STRICT disclosure), so the coercion message must NOT echo+ # the raw offending cell content — only row/field identifiers and the expected-type reason.+ path = write_csv_file(tmp_path, "value\nSUPERSECRET_TOKEN_xyz\n")+ with pytest.raises(CsvCoercionError) as exc_info:+ list_content_from_csv(path, IntRow)+ message = str(exc_info.value)+ assert "SUPERSECRET_TOKEN_xyz" not in message+ assert "value" in message # the field is still named so the author can locate it++ def test_string_literal_and_str_enum_round_trip_on_read(self, tmp_path: Path) -> None:+ # The read-side coercion of accepted string Literal / StrEnum cells (flat_field_names accepts+ # them; this confirms the cell actually coerces to the member through list_content_from_csv).+ literal_item = list_content_from_csv(write_csv_file(tmp_path, "name,rating\nAda,high\n", name="lit.csv"), LiteralRow).items[0]+ assert literal_item.rating == "high"+ enum_item = list_content_from_csv(write_csv_file(tmp_path, "name,grade\nAda,pass\n", name="enum.csv"), StrEnumRow).items[0]+ assert enum_item.grade == Grade.PASS++ @pytest.mark.parametrize(("row_model", "bad_cell"), [(LiteralRow, "name,rating\nAda,medium\n"), (StrEnumRow, "name,grade\nAda,unknown\n")])+ def test_out_of_choice_value_raises_coercion(self, tmp_path: Path, row_model: type[StructuredContent], bad_cell: str) -> None:+ # An out-of-choice value for a Literal/StrEnum column fails coercion with a typed error.+ with pytest.raises(CsvCoercionError):+ list_content_from_csv(write_csv_file(tmp_path, bad_cell), row_model)++ # ----------------------------------------------------------------------------------+ # Strict columns+ # ----------------------------------------------------------------------------------++ def test_extra_column_raises(self, tmp_path: Path) -> None:+ path = write_csv_file(tmp_path, "value,surprise\n42,boom\n")+ with pytest.raises(CsvColumnError):+ list_content_from_csv(path, IntRow)++ def test_missing_required_column_raises(self, tmp_path: Path) -> None:+ # 'nickname' is optional so its absence is fine; drop a REQUIRED column ('name') instead.+ path = write_csv_file(tmp_path, "age,height,active,born,nickname\n36,1.7,true,1815-12-10,x\n", name="missing_req.csv")+ with pytest.raises(CsvColumnError):+ list_content_from_csv(path, FlatRow)++ def test_missing_optional_column_sets_field_none(self, tmp_path: Path) -> None:+ # 'nickname' (optional) column omitted entirely → None for every row (CT2 lenient).+ path = write_csv_file(tmp_path, "name,age,height,active,born\nAda,36,1.7,true,1815-12-10\n")+ item = list_content_from_csv(path, FlatRow).items[0]+ assert item.nickname is None+ assert item.name == "Ada"++ def test_missing_optional_column_is_none_not_model_default(self, tmp_path: Path) -> None:+ # CT2: an omitted NULLABLE column is None for every row even when the field has a NON-None+ # default ('anon') — the CSV is the source of truth, not the row model's construction default.+ path = write_csv_file(tmp_path, "name\nAda\n")+ item = list_content_from_csv(path, OptDefaultRow).items[0]+ assert item.nickname is None++ def test_missing_non_nullable_defaulted_column_keeps_default(self, tmp_path: Path) -> None:+ # A non-required but NON-nullable field (count: int = 0) keeps its own default when its column+ # is omitted — forcing None there would fail validation rather than honor the absence.+ path = write_csv_file(tmp_path, "name\nAda\n")+ item = list_content_from_csv(path, IntDefaultRow).items[0]+ assert item.count == 0++ # ----------------------------------------------------------------------------------+ # csv_from_list_content — write side+ # ----------------------------------------------------------------------------------++ def test_csv_from_list_content_writes_declared_header_and_rows(self, tmp_path: Path) -> None:+ path = tmp_path / "people.csv"+ list_content: ListContent[FlatRow] = ListContent(+ items=[+ FlatRow(name="Ada", age=36, height=1.7, active=True, born=date(1815, 12, 10), nickname="Countess"),+ FlatRow(name="Grace", age=85, height=1.6, active=False, born=date(1906, 12, 9), nickname=None),+ ]+ )+ csv_from_list_content(list_content, row_model=FlatRow, path=path)+ lines = path.read_text(encoding="utf-8").splitlines()+ assert lines[0] == "name,age,height,active,born,nickname"+ # date serialized via model_dump(mode="json") → ISO string.+ assert lines[1].startswith("Ada,36,1.7,")+ assert "1815-12-10" in lines[1]+ # None must serialize to an empty cell, never the string "None".+ assert lines[2].endswith(",")+ assert "None" not in lines[2]++ def test_csv_from_empty_list_writes_header_only_from_model(self, tmp_path: Path) -> None:+ # CRITICAL: header comes from the declared model, never items[0].+ path = tmp_path / "empty.csv"+ empty: ListContent[FlatRow] = ListContent(items=[])+ csv_from_list_content(empty, row_model=FlatRow, path=path)+ assert path.read_text(encoding="utf-8").splitlines() == ["name,age,height,active,born,nickname"]++ def test_round_trip_stability(self, tmp_path: Path) -> None:+ path = tmp_path / "rt.csv"+ original: ListContent[FlatRow] = ListContent(+ items=[+ FlatRow(name="Ada", age=36, height=1.7, active=True, born=date(1815, 12, 10), nickname="Countess"),+ FlatRow(name="Grace", age=85, height=1.6, active=False, born=date(1906, 12, 9), nickname=None),+ ]+ )+ csv_from_list_content(original, row_model=FlatRow, path=path)+ reloaded = list_content_from_csv(path, FlatRow)+ assert reloaded.items == original.items++ # ----------------------------------------------------------------------------------+ # None <-> empty-cell, both sides+ # ----------------------------------------------------------------------------------++ def test_none_and_empty_string_both_write_blank_and_read_back_none(self, tmp_path: Path) -> None:+ path = tmp_path / "opt.csv"+ list_content: ListContent[OptionalTextRow] = ListContent(items=[OptionalTextRow(text=None), OptionalTextRow(text="")])+ csv_from_list_content(list_content, row_model=OptionalTextRow, path=path)+ reloaded = list_content_from_csv(path, OptionalTextRow)+ # Empty-string text is indistinguishable from None → both read back as None (documented).+ assert [item.text for item in reloaded.items] == [None, None]++ def test_required_text_empty_cell_rejected(self, tmp_path: Path) -> None:+ path = write_value_csv(tmp_path, "", name="reqtext.csv", header="text")+ with pytest.raises(CsvCoercionError):+ list_content_from_csv(path, RequiredTextRow)++ # ----------------------------------------------------------------------------------+ # Phase-2 review findings (#2 blank lines, #3 over-wide rows, #4 delimiter/encoding,+ # #5 write item-type guard, #7 model-level coercion label)+ # ----------------------------------------------------------------------------------++ def test_blank_line_in_body_is_skipped(self, tmp_path: Path) -> None:+ # A blank physical line (csv yields []) is skipped, not turned into a phantom all-None row.+ path = write_csv_file(tmp_path, "a,b\nx,y\n\nz,w\n")+ items = list_content_from_csv(path, OptPairRow).items+ assert [(item.a, item.b) for item in items] == [("x", "y"), ("z", "w")]++ def test_row_wider_than_header_raises(self, tmp_path: Path) -> None:+ # A data row with more cells than the header has no column for the surplus → error, not silent drop.+ path = write_csv_file(tmp_path, "name,age\nAda,36,EXTRA\n")+ with pytest.raises(CsvColumnError):+ read_rows(path)++ # "\n"/"\r"/'"' are one character (so they pass the length check) but csv rejects them with a raw+ # ValueError; the codec must turn that into a typed CsvError, not let it escape the boundary.+ @pytest.mark.parametrize("bad_delimiter", ["", "||", "\n", "\r", '"'])+ def test_bad_delimiter_raises_on_read_and_write(self, tmp_path: Path, bad_delimiter: str) -> None:+ read_path = write_csv_file(tmp_path, "a,b\n1,2\n")+ with pytest.raises(CsvError):+ read_rows(read_path, delimiter=bad_delimiter)+ with pytest.raises(CsvError):+ write_rows(tmp_path / "out.csv", headers=["a"], rows=[], delimiter=bad_delimiter)++ def test_unknown_encoding_raises(self, tmp_path: Path) -> None:+ path = write_csv_file(tmp_path, "a\n1\n")+ with pytest.raises(CsvReadError):+ read_rows(path, encoding="not-a-real-codec")++ def test_write_non_encodable_cell_raises(self, tmp_path: Path) -> None:+ # A cell that the target encoding can't represent must surface a typed CsvError on write,+ # not let a raw UnicodeEncodeError escape the codec boundary.+ with pytest.raises(CsvError):+ write_rows(tmp_path / "out.csv", headers=["name"], rows=[{"name": "café"}], encoding="ascii")++ def test_write_rejects_item_not_matching_row_model(self, tmp_path: Path) -> None:+ # An item whose class isn't the declared row_model would write silent empty cells → reject loudly.+ list_content: ListContent[StructuredContent] = ListContent(items=[IntRow(value=1)])+ with pytest.raises(CsvError):+ csv_from_list_content(list_content, row_model=RequiredTextRow, path=tmp_path / "x.csv")++ def test_coercion_error_names_model_level_failure(self, tmp_path: Path) -> None:+ # A model_validator failure has an empty pydantic loc; the message must still name the+ # row and concept (not the unhelpful "field(s) ?").+ path = write_csv_file(tmp_path, "a,b\n5,1\n")+ with pytest.raises(CsvCoercionError) as exc_info:+ list_content_from_csv(path, OrderedPairRow)+ message = str(exc_info.value)+ assert "field(s) ?" not in message+ # The offending row is the first data row, which is physical CSV line 2 (the header is line 1).+ assert "row 2" in message+ assert "OrderedPairRow" in message++ def test_coercion_error_row_number_after_multiline_cell(self, tmp_path: Path) -> None:+ # A valid quoted cell spanning two physical lines must not skew the line number of a later+ # bad row: the failing 'bad,oops' record starts on physical line 4 (header=1, multiline cell+ # =2-3), so the error must say "row 4", not "row 3".+ path = write_csv_file(tmp_path, 'label,number\n"multi\nline",1\nbad,oops\n')+ with pytest.raises(CsvCoercionError) as exc_info:+ list_content_from_csv(path, LabeledIntRow)+ assert "row 4" in str(exc_info.value)++ def test_coercion_error_row_number_counts_skipped_blank_lines(self, tmp_path: Path) -> None:+ # The reported row number is the physical CSV line, even when a blank line was skipped.+ path = write_csv_file(tmp_path, "value\n42\n\noops\n")+ with pytest.raises(CsvCoercionError) as exc_info:+ list_content_from_csv(path, IntRow)+ # 'oops' is on physical CSV line 4 (line 1 header, line 2 '42', line 3 blank, line 4 'oops').+ assert "row 4" in str(exc_info.value)++ # ----------------------------------------------------------------------------------+ # Format seam+ # ----------------------------------------------------------------------------------++ def test_supported_suffix_accepts_csv(self, tmp_path: Path) -> None:+ # No raise for .csv.+ assert_supported_table_suffix(tmp_path / "data.csv")++ def test_xlsx_suffix_points_at_tabular_extra(self, tmp_path: Path) -> None:+ with pytest.raises(CsvError) as exc_info:+ assert_supported_table_suffix(tmp_path / "data.xlsx")+ assert "pipelex[tabular]" in str(exc_info.value)++ def test_unsupported_suffix_raises(self, tmp_path: Path) -> None:+ with pytest.raises(CsvError):+ assert_supported_table_suffix(tmp_path / "data.parquet")diff --git a/wip/README.md b/wip/README.mdindex ed9e8862e..89b87c2dc 100644--- a/wip/README.md+++ b/wip/README.md@@ -13,6 +13,7 @@ This folder holds the **active** planning, design, and current-state docs for th- **[distributed-execution/](distributed-execution/)** — running MTHDS methods as Temporal workflows across separate worker processes. Its [README](distributed-execution/README.md) is the priority-ordered plan; the folder also holds the tracing/cost-reporting as-built reference, the crate-packaging phases (6a/6b) and crate-first vision, the implemented Temporal IDs/naming design, and the deferred Temporal-exception and schema-reconstruction hardening proposals. **This is the active headline track.**- **[error-handling/](error-handling/)** — the current-state reference for error handling across Pipelex, track-organized (metadata model, worker classification, extract/classify/render, retry, CLI delivery, Temporal integration, testing). Mostly landed; its [README](error-handling/README.md) lists what's still open.- **[concurrency/](concurrency/)** — direct-mode batching / backpressure design. The retry-jitter quick win shipped; the remaining design work is split into fan-out scheduling, rate limiting, and batch partial-failure docs. Status: design scoping.+- **[csv-support/](csv-support/)** — reading `.csv` files into typed `ListContent` and writing flat list outputs back out via `--save-csv`. Its [README](csv-support/README.md) is the feature + reviewer reference (codec, core input hook, CLI output hook, the I/O contract, locked decisions); the folder also holds the design rationale and the deferred review/security follow-ups. Landed in #955.## Standalone docsdiff --git a/wip/csv-support/README.md b/wip/csv-support/README.mdnew file mode 100644index 000000000..95d3083b2--- /dev/null+++ b/wip/csv-support/README.md@@ -0,0 +1,97 @@+# CSV Support — Feature Guide & PR-Review Reference++Branch `feature/Support-csv` (worktree `_csv`), targeting `dev`. This document describes the shipped CSV input/output feature so a PR-review agent can review the diff against intent without re-deriving the design. Design rationale lives in [`design.md`](design.md); deferred follow-ups in the other `*.md` files in this folder.++**Status: v1 complete — PR [#955](https://github.com/Pipelex/pipelex/pull/955) finalized, ready to merge.** All phases done; `make agent-check` green (pyright 0 / mypy 0 / ruff + plxt clean); full `make agent-test` green; docs + changelog landed. The PR went through 9 automated review rounds (greptile / cubic / codex) to convergence, then a pre-landing gstack `/review` (5 specialists + Codex adversarial) — which caught 3 issues the bot rounds missed (BOM/Excel input, a coercion-error data leak under STRICT, and `--save-csv` error framing) — and the bot re-review of those fixes also converged. All review threads resolved; CI green; mergeable. Deferred items are documented in this folder (security-followups, phase2/phase3-4 followups).++---++## What this feature does++A CSV (or, behind a not-yet-built seam, an Excel `.xlsx`) file maps to a `ListContent[YourConcept]`: each row is one instance of a flat concept, each column is one of its fields. The MTHDS language surface is unchanged — there is **no new native content type or concept**. You describe the row concept exactly as any structured concept.++- **Input:** an `inputs.json` reference whose `content.url` ends in a tabular suffix, under a flat structured concept, is read into a typed `ListContent` (one file → one list; the concept names the *row* type). Reachable from the CLI, the runner API, and programmatic callers, because the hook lives in core, not the CLI.+- **Output:** `--save-csv <path>` on `pipelex run pipe|bundle|method` writes the main stuff to a literal, cwd-relative CSV path when that output is a flat list.++The headline acceptance test is a full round trip: `people.csv` → `PipeBatch` → `PipeSequence`(`PipeLLM` + `PipeCompose`) → `summaries.csv`, green in dry mode via both the in-process runner and a real CLI subprocess.++User-facing docs: [`docs/building-methods/pipes/csv-input-and-output.md`](../../docs/building-methods/pipes/csv-input-and-output.md) + `--save-csv` in [`docs/tools/cli/run.md`](../../docs/tools/cli/run.md); CHANGELOG `[Unreleased]`.++---++## Code map (what to review, where)++**Codec — `pipelex/tools/tabular/csv_codec.py`** (stdlib `csv`, one module per CQ2; no codec/binding split until `.xlsx` earns it):++- `read_rows` / `write_rows` — low-level primitives over an internal `_read_table` (explicit header + raw rows) / `_row_to_dict` (pads short/blank rows). `read_rows` has no suffix gate.+- `flat_field_names(row_model)` — the **single shared flatness classifier** used by BOTH read and write. `_is_flat_annotation` unwraps `Optional`/PEP-604 unions, ACCEPTS `str/int/float/bool/date/datetime` + **string-valued** `Literal`/`Enum` (only those round-trip through a CSV cell), REJECTS genuine multi-arm unions, non-string `Literal`/`Enum`, list/dict/nested-model/`Any`. Raises `CsvFlatnessError` naming field + concept; returns declared field order.+- `list_content_from_csv(path, row_model, *, delimiter, encoding)` — read path. Maps `""`→`None` before validate; extra/missing-required column → `CsvColumnError`; missing-optional column → field absent → pydantic default (`None`); per-row `ValidationError` → `CsvCoercionError` (1-based row + field).+- `csv_from_list_content(list_content, row_model, path, *, delimiter, encoding)` — write path. Headers from the **declared `row_model`** (so an empty list still writes a header-only file); serializes via `model_dump(mode="json")`; `_to_cell` maps `None`→`""` and bool→`true`/`false`.+- `is_tabular_path(path)` — suffix membership in `_TABULAR_SUFFIXES` (`.csv`, `.xlsx`).+- `assert_supported_table_suffix(path)` — the format seam: `.csv` ok; `.xlsx` → `CsvError` naming `pipelex[tabular]`; else generic `CsvError`.+- `delimiter`/`encoding` are function params (`,` / `utf-8`); no main-config touched (D1).++**Error taxonomy — `pipelex/tools/tabular/exceptions.py`:** `CsvError(ToolError)` base (`ErrorDomain.INPUT`) + `CsvReadError` / `CsvFlatnessError` / `CsvColumnError` / `CsvCoercionError`. All are `PipelexError`s, so the CLI prints them cleanly (no traceback) via `execute_run`'s `except PipelexError`. The codec catches every raw `OSError` / `UnicodeDecodeError` / `csv.Error` / pydantic `ValidationError` at its boundary and re-raises typed, carrying file path + 1-based row + concept + field — no raw third-party exception escapes into core/runner.++**Input hook — `pipelex/core/stuffs/stuff_factory.py`:** `_try_make_csv_list_stuff`, called from **Case 2.5** of `make_stuff_from_stuff_content_or_data` BEFORE the record `model_validate`. Detection: content has a `url` str key whose **path** (via `urlsplit(url).path`, so a query-string/fragment can't hide the suffix) is tabular AND the concept is **non-native** (`Concept.is_native_concept` False — native file concepts like Image/PDF keep their own url handling). Remote guard: `resolve_uri(url)` must be a `ResolvedLocalPath` AND `"://" not in resolved.path` (the `://` clause catches `s3://`/`gs://`, which `resolve_uri` currently mislabels as local) — else `CsvError` "local file paths only in v1". Row model = `concept.get_structure_class()` (wrapped so an unregistered structure class raises a typed `CsvError`, not a raw `ConceptValueError`). Returns `None` for ordinary record dicts (no url / non-tabular suffix / native) → falls through to normal Case 2.5.++**Output hook — `pipelex/cli/commands/run/_run_core.py`:** `_execute_run` / `execute_run` take keyword-only `save_csv: str | None = None`. The three run commands (`pipe_cmd.py` / `bundle_cmd.py` / `method_cmd.py`) declare `--save-csv` and forward it. The save block (after the working-memory save): rejects empty path, no-main-stuff, and non-`ListContent` main stuff with `typer.secho` + `Exit(1)`; creates the parent dir; writes via `csv_from_list_content(...)` at the literal cwd path (CQ1 — NOT under `--output-dir`); the codec's `CsvFlatnessError` covers non-flat rows.++---++## Test map++- **Unit — `tests/unit/pipelex/tools/tabular/test_csv_codec.py`:** read→dicts, custom delimiter, missing-file/bad-encoding → `CsvReadError`, duplicate/blank header → `CsvColumnError`, write+read round-trip, empty→header-only; flatness classifier (declared order + `Literal` accept; list/dict/nested/`Union`/`Any` reject); coercion incl. `date`→`datetime.date`; empty-cell→`None` (optional) / required→`CsvCoercionError`; **T3 coercion accept/reject tables** (bool `true/false/1/0/yes/no/on/off`, int, float, ISO date / reject `abc`, `1,5`, `31/12/2020`); strict columns (extra/missing-required error, missing-optional all-`None`); `None`↔`""` both sides.+- **Integration — `tests/integration/pipelex/csv/test_csv_roundtrip.py`** (split per T1): (a) input-codec WorkingMemory build → `ListContent[Person]` + `death_year is None` for Vint Cerf; (b) output-codec hand-built `ListContent[PersonSummary]` → header + rows; (c) full pipeline via `PipelexRunner(...).execute_pipeline("summarize_people", ...)` → 3 items, names/countries from real rows (dry-mode-safe).+- **Integration — `tests/integration/pipelex/csv/test_csv_input_detection.py`:** A3/CT1 pins (`.csv`-url column reads as a table; non-`.csv` url stays a single record), remote-url rejection (`http(s)`/`s3://`/`gs://`, incl. query-string/fragment), eager-IO refresh-after-mutation.+- **E2E — `tests/e2e/pipelex/cli/test_csv_run.py`** (`@pytest.mark.gha_disabled`, subprocess): `run pipe --save-csv --dry-run` → exit 0 + file at literal cwd path + header + one row per person; **T2 cheap wiring checks** that all three run commands declare `save_csv` and forward it; clean missing-`.csv` error (non-zero exit, no traceback, names the file).+- **Fixtures — `tests/integration/pipelex/csv/`:** the runnable demo lives in `csv_demo/` (`csv_demo.mthds` with flat `Person`/`PersonSummary`, `people.csv`, `inputs.json`); the url-field detection fixture is split out into `url_field_concept/` (`url_field_concept.mthds` with the pipe-less `Link` concept that declares its own `url` field, plus `links.csv`). Inline `[concept.X.structure]` blocks auto-generate registered `StructuredContent` subclasses (no `test_structures.py`).++---++## The contract (input/output semantics — do not silently change)++- **Flat concepts only.** Scalar fields: `text`→`str`, `integer`→`int`, `number`→`float`, `boolean`→`bool`, `date`→`datetime.datetime` (the MTHDS `date` type generates `datetime.datetime`; the codec also accepts a hand-built `datetime.date`). Optionals allowed; string-valued `Literal`/`StrEnum` choices allowed. Nested/list/dict/`Union`/concept-typed/`Any`/non-string `Literal`/`Enum` → `CsvFlatnessError` naming the field. No silent flatten, no JSON-in-cell, no dropped cells.+- **Input headers:** required; must match field names exactly (no implicit remap). **Extra column → error; missing required column → error; missing *nullable* column → that field is `None` for all rows; missing non-nullable-but-defaulted column → that field keeps its own default** (CT2 lenient). Empty cell → `None` before pydantic (field must be optional).+- **Coercion:** through pydantic lax validation; the accept/reject set is pinned by T3 (see test map). Don't hand-roll coercion.+- **Output:** `--save-csv` triggers explicitly (no auto-emit). Header from the declared row model (empty list → header-only). `None`→`""` (never `"None"`); bool→lowercase. Literal cwd-relative path, parent dir created.+- **CSV is an I/O codec, not a prompt-render format** — it must NOT join the `rendered_plain/markdown/html/json` `TextFormat` router.++---++## Locked design decisions (do not re-litigate — see design doc §6 + plan-eng-review)++- **Typed lists only**; full round-trip in v1; **no new native `Table` concept**.+- **stdlib `csv` in core; `.xlsx` deferred** behind `pipelex[tabular]`, reached through the thin suffix seam (`assert_supported_table_suffix`). Don't add openpyxl or build xlsx now.+- **Input hook in CORE** (`stuff_factory` Case 2.5), not the CLI, so it's reachable from the runner API. Eager local-disk read accepted there; codec stays a pure module.+- **v1 = local filesystem paths only.** Remote tabular `url` (`http(s)`/`s3`/`gs`/signed) actively rejected, never opened.+- **No main-config in v1 (D1).** Delimiter/encoding are codec params, never guessed; no TOML/CLI surface yet. `make tb` not needed.+- **`--save-csv` is literal/cwd-relative** (CQ1), keyword-only in `_execute_run`. No `--csv-path`.+- **A3/CT1:** keep `url` + tabular-suffix detection; accept the narrow residual (a flat concept with a `url` field given a `.csv` value reads as a table) — pinned by tests, documented as a limitation.++---++## Documented limitations (written up in the guide page)++- **Local paths only** — remote tabular URLs rejected.+- **`.xlsx` recognized but not implemented** — routed to the `pipelex[tabular]` message; backend not built.+- **Empty string indistinguishable from `None`** — empty cell ⇄ `None` both directions.+- **Single-key `{"url": "x.csv"}` residual (A3/CT1)** — a bare `{"url": "x.csv"}` reference under a flat concept reads as a table, not a single record. Detection is gated to the exact single-key wrapper, so a record with sibling keys (`{"label": "Home", "url": "x.csv"}`) stays a record — its fields are never dropped (was a silent-drop; fixed in PR #955 review round 1).+- **No formula-injection escaping (CT3)** — cells written verbatim for data fidelity; opt-in escape deferred.++---++## Deferred follow-ups (NOT in v1 — captured, do not expand scope into them)++Out of scope for v1 (design §10): `.xlsx` via openpyxl; streaming/out-of-core; Google Sheets/Parquet; delimiter/dialect auto-detection; a native `Table` concept; remote `.csv` URLs.++Captured review follow-ups (decisions deferred, not bugs to fix in this PR):++- [`phase3-4-review-followups.md`](phase3-4-review-followups.md): (A) `resolve_uri` doesn't classify `s3://`/`gs://` — a cross-caller `tools/uri` gap the `"://"` heuristic papers over; fix is its own change. The same root cause is the `?`/`#`-in-local-filename mis-strip (niche, graceful). (B) factory-altitude eager I/O + cwd-relative paths for non-CLI callers — design decision. **(C) RESOLVED in PR #955 review round 1** — detection tightened to the single-key `{"url": ...}` wrapper, so a multi-field record no longer silently drops siblings. Plus minor cleanups (output-side error framing, suffix-vocabulary drift, header-only-vs-empty-list parity, redundant casts).+- [`phase2-review-followups.md`](phase2-review-followups.md): non-string `Literal`/`Enum` flatness — **RESOLVED in PR #955 review round 3** (Option A: the gate now accepts only string-valued `Literal`/`Enum`, rejecting `Literal[int]`/`IntEnum` that serialize but don't coerce back).++Also addressed in PR #955 review rounds (not deferred): error row numbers now point at the physical CSV line (incl. across multiline quoted cells, via `csv.reader.line_num`); CSV detection gated to the single-key `{"url": ...}` wrapper; `--save-csv` validates before `mkdir` and frames its own failures as "Failed to --save-csv" (not a pipeline failure); newline + quote delimiters rejected with a typed `CsvError`; strict CSV parsing (malformed quotes → `CsvReadError`); `write_rows` wraps `UnicodeEncodeError`; an omitted *nullable* column is `None` for all rows regardless of the row-model default while a non-nullable defaulted column keeps its own default (CT2); `CsvError` opts into `_authors_caller_facing_message` so STRICT disclosure keeps its caller-facing message, and that message sanitizes the offending url (strips query/fragment/userinfo, guards malformed port + bad-IPv6-bracket) so a signed/token-bearing url never leaks; the CLI `s3://` remote-rejection bypass (`is_relative_local_path`) is fixed; **reads default to `utf-8-sig` so Excel-exported BOM CSVs parse**; and the coercion error no longer echoes the raw cell value (no data leak under STRICT).++Deferred (documented): local paths containing URL-reserved `?`/`#` (cubic agreed; entangled with `resolve_uri` scheme-classification, item A); nullable-but-required columns stay mandatory (phase2-followups #2). Security review follow-ups (gstack `/review`) in [`security-followups.md`](security-followups.md): platform-wide untrusted-`url` local-file-read confinement (pre-existing input model, not CSV-specific), CSV formula-injection (locked CT3 document-only decision), embedded-NUL path error-hygiene (niche).++Deferred feature TODOs: user-facing delimiter/encoding config (D1); opt-in CSV/Excel formula-escape flag (CT3); `SaveOptions` struct for `_execute_run`'s param clump; multiplicity-gated CSV detection (CT1 option C, only if the residual proves real).diff --git a/wip/csv-support/design.md b/wip/csv-support/design.mdnew file mode 100644index 000000000..188a3ba58--- /dev/null+++ b/wip/csv-support/design.md@@ -0,0 +1,145 @@+# CSV Support — Design++Status: **design / pre-plan**. This document frames the problem and the design space so we can choose an approach before writing an implementation plan. It deliberately leaves the key decisions open (see "Decisions to make").++## 1. Goal++Let Pipelex methods take **CSV files as input** and produce **CSV files as output**, so tabular data flows in and out of pipelines as a first-class citizen rather than being shoehorned into a generic document blob or a hand-rolled `PipeFunc`.++Concrete user stories that should "just work":++- "I have a `contacts.csv`; run my enrichment pipe over each row and give me back an enriched CSV."+- "My pipeline produces a list of extracted records; save them as a `.csv` I can open in Excel."+- "Hand this spreadsheet to an LLM step as context."++## 2. Where this lands in the current architecture++Findings from the codebase (file references for the eventual plan):++- **Content/stuff type system** — every value is a `StuffContent` subclass wrapped in a `Stuff` (`pipelex/core/stuffs/`). Concrete types: `TextContent`, `ImageContent`, `DocumentContent`, `JSONContent`, `HtmlContent`, `NumberContent`, `ListContent[T]`, `StructuredContent` (base for custom concepts), `TextAndImagesContent`, `SearchResultContent`, `PageContent`, `MermaidContent`, `DynamicContent`.+- **Common interface** — `rendered_plain / rendered_html / rendered_markdown / rendered_json` (+ async variants), `smart_dump()`, `rendered_pretty()`. Defined on `StuffContent` (`pipelex/core/stuffs/stuff_content.py`).+- **Native concepts** — enumerated in `pipelex/core/concepts/native/concept_native.py` (`NativeConceptCode` → `structure_class`). Native concepts are part of the **MTHDS language surface**, not just runtime sugar.+- **Lists & batch** — `ListContent[T]` implements the Python list protocols; `PipeBatch` (`pipelex/pipe_controllers/batch/pipe_batch.py`) runs a branch pipe over each item of a `ListContent` and recombines outputs into a `ListContent`. `[]` multiplicity (`Person[]`, `Text[3]`) is declared in `.mthds` inputs/outputs.+- **Input path** — `pipelex run --inputs <json|path>` → `inputs.json` may reference local files by `url`; `_inputs_path_resolver.py` resolves relative paths; `working_memory_factory.make_from_pipeline_inputs` builds the `WorkingMemory`. File-backed types (`Document`, `Image`) store a **URL/reference**, not parsed bytes.+- **Output path** — `--save-main-stuff` writes `main_stuff.{json,md,html}`; `--save-working-memory` dumps the whole memory as JSON (`pipelex/cli/commands/run/_run_core.py`).+- **No tabular code exists** — no `csv`/`pandas`/`polars`/`openpyxl` dependency; `.xlsx` is only registered as a MIME type and treated as an opaque document.++**Key reframing:** a CSV table is structurally a `ListContent` of rows. So "CSV support" is less "add a blob type" and more **"a serialization bridge between CSV files and lists of (optionally typed) rows"** — the same role JSON already plays for structured content.++## 3. The central modeling fork++There are two coherent ways to model a CSV in the type system. They are **not** mutually exclusive — they can be layered — but v1 should pick a primary.++### Mode A — Typed rows: CSV ⇄ `ListContent[Concept]`++A CSV binds to a declared concept. `contacts.csv` under concept `Person` becomes `ListContent[Person]`; header cells map to concept fields; cell strings are coerced to field types.++- **Input:** `{ "contacts": { "concept": "myDomain.Person", "content": { "url": "contacts.csv" } } }` → parsed into typed rows.+- **Output:** any `ListContent[StructuredContent]` → CSV by flattening fields to columns.+- **Pros:** deeply native; rows immediately flow through `PipeBatch` and typed pipes; type-safe; round-trips with the rest of the system.+- **Cons:** needs a column↔field mapping rule, type coercion, and a policy for nested/complex fields (flatten with dotted keys / JSON-encode the cell / reject). Header semantics must be pinned down.++### Mode B — Table primitive: CSV ⇄ `TableContent`++A new content type holding the table verbatim: `headers: list[str]` + `rows` (list of string rows, or list of dicts). Renders to markdown/HTML/JSON table. No schema, lossless strings.++- **Pros:** simple; schema-free; ideal for "give the LLM a table" and "carry a spreadsheet through unchanged"; minimal coercion logic.+- **Cons:** untyped (everything is strings); to do typed per-row processing you still need a separate "table → typed list" step. Adds a **new native concept** to the MTHDS language surface (enum + registry + `Stuff` helpers + tests + arguably the `mthds/` spec).++### How they relate++Mode B is the **lossless, schema-free primitive**; Mode A is the **typed bridge**. A clean end state has both: read a CSV as a `Table` when you just want the grid, or bind it to a concept when you want typed rows; write any list/table back out. The decision is which is the *canonical* v1 surface and whether we add a native concept at all.++## 4. I/O surface design space++Independent of the modeling fork, each direction has surface choices (non-exclusive):++**Output**++- **(O1) Render format** — add a CSV rendering (`rendered_csv()` / a `csv` `TextFormat`). Cheapest; makes CSV "just another rendering of a list/table". But CSV doesn't fit the prompt-format router cleanly and is awkward for arbitrarily nested structures.+- **(O2) CLI export** — extend `--save-main-stuff` to also emit `main_stuff.csv` when the main stuff is a list/table, or add `--save-csv <path>`. Best for "run pipeline → get a CSV".+- **(O3) Dedicated pipe operator** — an explicit "write list/table to CSV artifact" operator. Most explicit, most plumbing; probably overkill for v1.++**Input**++- **(I1) `inputs.json` references a `.csv`** — extend the input loader/factory to parse `.csv` URLs into the chosen content (Table or typed list). Most consistent with how `Document`/`Image` inputs work today.+- **(I2) CLI sugar** — e.g. `pipelex run my_pipe --csv data.csv`. Nice-to-have, later.+- **(I3) Programmatic factory** — `TableContent.make_from_csv_path(...)` / `ListContentFactory.from_csv(path, concept=...)` for library users and tests.++## 5. Dependency question++Core has no DataFrame dependency and is intentionally lean. Options:++- **Stdlib `csv` only** — covers comma/delimiter parsing, quoting, headers. No `.xlsx`. Keeps core lean. **Default recommendation.**+- **pandas/polars** — richer (dtypes, Excel, big files) but a heavy core dependency we'd rather not impose. If Excel is wanted, prefer an **optional extra** (`pipelex[tabular]`) over a hard dep.++This choice bounds the Excel/`.xlsx` ambition, so it's coupled to the scope decision below.++## 6. Chosen approach (locked)++Decisions taken (see §7 for the record):++- **Model — typed lists only (Mode A).** A CSV is a **serialization format for `ListContent[Concept]`**, never a blob and never a new content type. No native `Table` concept; the MTHDS language surface is unchanged. Rows are instances of a declared concept; columns are that concept's fields. This reuses `ListContent`, typed concepts, and `PipeBatch` directly.+- **Scope — full round-trip.** CSV in → process (typically batch over rows) → CSV out, both directions in v1.+- **Dependency — stdlib `csv` in core, `.xlsx` behind an optional extra.** v1 reads/writes CSV with the stdlib `csv` module (no DataFrame dep). The reader/writer sits behind a small **format seam** so an `openpyxl`-backed `.xlsx` codec can be added later under `pipelex[tabular]` without touching callers.+- **Nested cells — reject non-flat.** A row concept used for CSV must have **scalar fields only** (`str`/`int`/`float`/`bool`/optional thereof). A nested structure or list field raises a clear error telling the author to project to a flat concept first. No silent flattening, no JSON-in-cell, no dropped data.++**Design principle that falls out of this:** CSV is an **I/O codec**, not a prompt-render format. It does **not** join the `rendered_plain/markdown/html/json` `TextFormat` router (that family is about LLM/template/display rendering). Reading and writing CSV files is a separate codec surface that the CLI and the input loader call into.++## 7. Worked example — the round-trip++The headline "enrich a CSV" flow, end to end:++1. **Input** — `inputs.json` points a row concept at a `.csv`:++ ```json+ { "contacts": { "concept": "acme.Contact", "content": { "url": "contacts.csv" } } }+ ```++ The input loader sees a `.csv` URL, parses each data row into an `acme.Contact`, and yields `ListContent[Contact]`. (A CSV is inherently a list, so one `.csv` → one list; the named concept is the **row** concept.)++2. **Process** — the pipe declares `contacts = "Contact[]"` and uses `PipeBatch` to run a branch pipe `enrich_contact` over each `Contact`, recombining into `ListContent[EnrichedContact]` as the main stuff.++3. **Output** — the CLI writes the flat `ListContent[EnrichedContact]` back to `enriched.csv` (columns = `EnrichedContact`'s scalar fields).++No new operators, no new concept type — just the CSV codec bolted onto the existing list/batch machinery at the two edges.++## 8. Surfaces & semantics++**Input (I1 + I3).** `inputs.json` `.csv` references are parsed in the input loader; underneath sits a programmatic codec (`from_csv(path, row_concept) -> ListContent`) that tests and library users can call directly. Semantics:++- The `.csv` must have a **header row**; column headers must **match the row concept's field names** (strict — no implicit remapping).+- Cell strings are coerced to field types via pydantic's lax validation (e.g. `"42"` → `int`). An **empty cell → `None`** (and must therefore map to an optional field).+- **Extra columns** (no matching field) and **missing columns** for required fields are errors — predictable, no silent reshaping. (Consistent with the reject-non-flat stance.)++**Output (O2 + codec).** A flat `ListContent[StructuredContent]` is written to CSV; underneath the same codec (`to_csv(list_content, path)`). Trigger surface (finer point, see below): either auto-emit `main_stuff.csv` from `--save-main-stuff` when the main stuff is a flat list, or an explicit `--save-csv <path>`. Semantics:++- Columns are the row concept's scalar fields, in declared order; one CSV row per item.+- A non-flat row concept (any nested/list field) is **rejected with a clear error** at write time.++**Codec & format seam.** A single small tabular-codec module owns read/write, dispatched by file extension/format. CSV is the only built-in in v1; the seam's shape (a reader fn + writer fn keyed by format) is what lets `.xlsx` arrive later as an optional extra with zero changes to the loader or CLI.++## 9. Finer points (proposed defaults — speak up to change)++These don't change the shape of the feature; listed so the plan can lock them.++- **`inputs.json` convention:** the `concept` names the **row** concept; a `.csv` content URL implies list multiplicity (one CSV → `ListContent[row-concept]`). Pipe inputs are therefore declared with `[]`.+- **Output trigger:** prefer an explicit `--save-csv <path>` over auto-emitting from `--save-main-stuff` (explicit, no surprise files). Open to auto-emit if you'd rather.+- **Empty cell → `None`**; requires the field to be optional.+- **Strict columns:** exact header↔field correspondence; extra/missing columns error.+- **Dialect:** UTF-8, standard comma dialect; **delimiter and encoding configurable**, never auto-guessed.++## 10. Non-goals (v1)++- Streaming / out-of-core CSVs too large for memory.+- `.xlsx` in core — it's explicitly designed-for but ships later behind `pipelex[tabular]`; Google Sheets / Parquet are out.+- Delimiter/dialect **auto-detection** (configurable, not guessed).+- Schema/type **inference** beyond what the bound concept declares.+- A native `Table` concept / schema-free table primitive (deliberately not in this design; would be a separate, additive language-surface decision).++## 11. Risks / watch-outs++- **Lossy coercion** — string→typed coercion can change data (e.g. leading zeros, locale numbers). Document the rules; lean on pydantic's well-defined coercion rather than ad-hoc parsing.+- **Header collisions / unnamed columns** — define behavior for duplicate or blank headers up front (recommend: error).+- **Flat-only friction** — real concepts are often nested; the reject-non-flat rule means authors must add a "project to flat row" step. That's intended (predictable), but the error message must name the offending field and suggest the fix.+- **Format-seam over-engineering** — keep the seam minimal (two functions keyed by format); don't build a plugin framework before the second format exists.diff --git a/wip/csv-support/phase2-review-followups.md b/wip/csv-support/phase2-review-followups.mdnew file mode 100644index 000000000..db7b4fed6--- /dev/null+++ b/wip/csv-support/phase2-review-followups.md@@ -0,0 +1,49 @@+# CSV Support — Phase 2 code-review follow-ups (topics to revisit)++Status: **deferred — captured for a deliberate decision later, NOT a blocker for Phase 2.** Produced by an extra-high-recall `/code-review` of the Phase-2 codec implementation (`pipelex/tools/tabular/csv_codec.py`) on branch `feature/Support-csv` (worktree `_csv`).++This file holds review findings that are real but involve a **design tradeoff** rather than a rote fix — they shouldn't be patched reflexively. The mechanical/silent-correctness findings from the same review (blank-line phantom rows, over-wide-row truncation, delimiter/encoding boundary leak, write item-type guard, model-level coercion label) were the "fix-now" set and have since been resolved (see this folder's [`README.md`](README.md)).++Line numbers are approximate anchors — grep the quoted token if they have drifted.++---++## #2 — Nullable-but-required column: mandatory vs. backfill-None (DEFERRED, PR #955 review round 6)++cubic flagged that a *nullable required* field (`x: str | None` with **no** default — pydantic reports `is_required() == True`) has a mandatory CSV column: omitting it raises `CsvColumnError` rather than backfilling `None`. cubic argues nullable columns should always be backfillable to `None`.++**Decision: keep current behavior (deferred, not adopted).** The rule "a column is mandatory ⟺ the field is pydantic-required" is intentional and *respects the author's signal*: writing `x: str | None` **without** `= None` is a deliberate "must be provided" choice (pydantic gives `X | None` no implicit default). Backfilling `None` would silently override that. The current code already maps an omitted *non-required* nullable column to `None` (round-4 `_annotation_allows_none`); only the *required* nullable case errors, which is correct. This is also unreachable from `.mthds`: a `required = false` field always compiles to `Optional[T] = None` (nullable WITH a None default → non-required → already `None` on omission), and a `required = true` field is non-nullable. So the distinction only arises for hand-built Python models. Revisit only if a real use-case wants "nullable ⟹ optional column" semantics.++---++## #1 — Non-string `Literal` / `Enum` row fields — RESOLVED (Option A, PR #955 review round 3)++**Resolution:** went with Option A (reject). `_is_flat_annotation` now accepts a `Literal` only when every arg is a `str`, and an `Enum` only when every member value is a `str`; a non-string `Literal`/`IntEnum` falls through to `CsvFlatnessError`. This keeps "flat ⇒ round-trippable" honest. Pinned by `IntLiteralRow`/`IntEnumRow` rejection cases + a `StrEnumRow` acceptance case. greptile + cubic both flagged it across rounds. The original tradeoff analysis is kept below for context.++---++### Original analysis (for context)++### Non-string `Literal` / `Enum` row fields: reject vs. coerce-on-read++**The finding.** The flatness gate `_is_flat_annotation` (`pipelex/tools/tabular/csv_codec.py:67`, `if origin is Literal: return True`; `:70`, `issubclass(annotation, Enum) → True`) accepts *any* `Literal` and *any* `Enum` as CSV-flat. But the codec can only round-trip **string-valued** ones:++- `_to_cell` writes the value as a bare string (`1`) or lowercased bool (`true`).+- On read, pydantic's `Literal`/`Enum` validator does **not** coerce that string back to the typed member.++Verified: `Literal[1,2,3]` writes `1` → read raises `CsvCoercionError`; `Literal[True,False]` writes `true` → fails; an `IntEnum` likewise. `Literal["low","high"]` (and `StrEnum` / str-valued `Enum`) round-trip fine. So a concept the gate declares "flat" can silently fail its own round-trip.++**Why it's deferred (the tradeoff).** Non-string `Literal`/`Enum` fields are **not reachable from `.mthds` today**, so this is not a live data-corruption risk:++- The structure generator *always* emits **string** `Literal`s — even integer/number "choices" compile to `Literal['1','2',…]`: `pipelex/core/concepts/structure_generation/generator.py:318-320` (`python_type = f"Literal[{', '.join(repr(c) for c in field_blueprint.choices)}]"`, fed by string choices).+- `choices` is validated as `list[str]`: `pipelex/core/concepts/concept_structure_blueprint.py:60` (`choices: list[str] | None`), and is only allowed on text/integer/number types: `pipelex/builder/concept/concept_spec.py:150-162`.+- The generator never produces `Enum`/`StrEnum` subclasses at all.++So non-string `Literal`/`Enum` only arise from hand-built `StructuredContent` subclasses (e.g. tests). Two ways to resolve, with different futures:++- **Option A — reject (simple, safe now).** Tighten the gate to accept only string-valued `Literal` (`all(isinstance(arg, str) for arg in get_args(...))`) and str-valued `Enum` (`all(isinstance(m.value, str) for m in enum)`); non-string members fall through to the existing `CsvFlatnessError`. Breaks no reachable concept, keeps "flat ⇒ round-trippable" honest. **Downside:** forecloses (or makes the codec inconsistent with) a future where integer/number choices materialize as real `Literal[int]` / `IntEnum` and someone wants them in CSV.+- **Option B — coerce-on-read (preserves future support).** For a `Literal`/`Enum` field, coerce the cell string against the member's underlying scalar type (or match by `str(member)`/`member.value`) before/within validation, so `1` → `Literal[1,2,3]` works. More code; partly duplicates pydantic; needs care for mixed-type literals.++**Recommended default if forced to pick now:** Option A (reject) — it's the smallest correct change and matches the current `.mthds` reality. Revisit toward Option B **only if** the structure generator ever emits typed (non-string) `Literal`/`Enum` for choices, at which point CSV support for them should be a deliberate feature with its own coercion + round-trip tests.++**Decision owner:** revisit when (a) touching the choices/structure-generation typing, or (b) a user asks for integer/enum-typed CSV columns. Until then this is a latent gap behind an unreachable code path, intentionally left as-is.diff --git a/wip/csv-support/phase3-4-review-followups.md b/wip/csv-support/phase3-4-review-followups.mdnew file mode 100644index 000000000..50d5d76fa--- /dev/null+++ b/wip/csv-support/phase3-4-review-followups.md@@ -0,0 +1,46 @@+# CSV support — Phase 3+4 code-review follow-ups++Deferred items from the xhigh code review of the Phase 3 (input hook) + Phase 4 (`--save-csv`) work (2026-06-02). The two confirmed bugs (#1 query-string remote-rejection, #2 `ConceptValueError` wrap) and the cheap UX guards (#7 empty `--save-csv`, #8 parent-dir creation) were **fixed in that session** — see `stuff_factory._try_make_csv_list_stuff` and `_run_core._execute_run`. This file captures the items deliberately **not** patched: design-tradeoff / right-depth calls that deserve a decision rather than a reflexive fix.++## Deferred — design / altitude++### A. `resolve_uri` does not classify non-file remote schemes (`s3://`, `gs://`, …)++`resolve_uri` only special-cases `data:`, `pipelex-storage://`, `file://`, and `http(s)://`; **every other scheme falls through to `ResolvedLocalPath(path=<uri>)`** (`pipelex/tools/uri/uri_resolver.py` final `return`). So `s3://bucket/x.csv` is reported as a *local path* whose `.path` still contains `s3://...`.++The CSV input hook works around this with a string heuristic — `not isinstance(resolved, ResolvedLocalPath) or "://" in resolved.path` — to reject those as remote. That works, but:++- It encodes a fragile assumption about `resolve_uri`'s fallthrough; the `"://"` clause silently becomes dead/contradictory the day a real `ResolvedS3Url` variant is added.+- **The same blind spot lives in other callers** that trust `resolve_uri`'s `ResolvedLocalPath` to mean "local file" — e.g. the input normalizer and the image/document renderers (`load_binary_async(Path("s3://…"))`), and `_inputs_path_resolver.is_relative_local_path` (which would treat `s3://…` as a relative local path and prepend `base_dir`).++**Right-depth fix:** teach `resolve_uri` / `UriKind` to classify an unknown `<scheme>://` as a distinct non-local kind, and expose a single `is_local_uri(resolved) -> bool` predicate. Then the CSV hook and `_inputs_path_resolver` share one authoritative classifier (matching the project's "match/case over a typed enum, no string-sniffing" rule) and the `"://"` heuristic disappears. This is a `tools/uri` change with cross-caller impact → scope and test it on its own, not inside the CSV feature.++**Update (PR #955 review round 9):** the blind spot became a concrete bug — `_inputs_path_resolver.is_relative_local_path` was rewriting `s3://bucket/x.csv` into `<base_dir>/s3:/bucket/x.csv` on the CLI JSON-file path, defeating the CSV remote-url guard (codex/cubic). Fixed by adding the same `"://"`-in-resolved-path stopgap to `is_relative_local_path` (so scheme-bearing refs are left untouched and the remote guard fires). The `"://"` heuristic now lives in **two** callers (the CSV hook + `is_relative_local_path`); item A's typed `is_local_uri` predicate should replace both at once.++**Same root cause — local paths containing URL-reserved `?`/`#` (cubic, PR #955 round 1).** Suffix detection runs on `urlsplit(url).path` so a remote presigned `…csv?X-Amz-…` can't hide its suffix and slip past the local-only guard. The cost: a *local* file whose name literally contains `?` or `#` (POSIX-legal, Windows-illegal) has its suffix mis-stripped, so `data#1.csv` is not detected as tabular and falls through to record handling — a clear validation error, no data loss. The principled fix is the same `resolve_uri` scheme-classification above: resolve first, then take the suffix from the *resolved local path* (`?`/`#` are ordinary filename bytes there) for local refs and from `urlsplit` only for true URLs. Deferred with item A rather than bolted on as a second string heuristic. Niche (pathological local filename) and graceful, so not a v1 blocker.++### B. CSV detection + eager file I/O live in `StuffFactory`, the universal stuff constructor++A1 deliberately put the hook in `StuffFactory.make_stuff_from_stuff_content_or_data` (Case 2.5) so CSV input is reachable from the runner API and programmatic callers, not just the CLI. The consequence to weigh:++- `StuffFactory` is reached by **every** path that builds a stuff — CLI runs, runner-API input building, `WorkingMemory` re-hydration from a serialized payload, mock-input construction. All of them now perform eager local-disk I/O whenever a content dict carries a tabular-suffixed `url`.+- The CLI rewrites relative `url`s to absolute via `_inputs_path_resolver` **before** the factory sees them; non-CLI callers do **not**. A runner/programmatic caller passing `{"url": "data.csv"}` therefore opens a path resolved against the **process cwd**, and a re-hydrated `WorkingMemory` re-opens a path that may not exist on that host. (`test_eager_read_reflects_file_mutation` pins the disk-read-on-every-build behavior as intended.)++**Decision needed:** is eager file I/O at the factory altitude acceptable for all non-CLI callers, or should CSV reference resolution move up into a dedicated input-resolution layer (the design doc §2/§8 originally placed CSV parsing "in the input loader")? If the factory stays, document the relative-path-vs-cwd contract for non-CLI callers and decide whether re-hydration should re-read or carry the parsed list. (Note: Temporal is not shipped — don't over-weight the worker re-hydration angle, but the runner-API relative-path case is real today.)++## Deferred — documentation (fold into the Phase 5 docs page)++### C. `url`-field residual extends to multi-field records (A3/CT1) — RESOLVED (option b, PR #955 review round 1)++The documented A3/CT1 residual was "a single-record concept with a `url`-named field whose value ends `.csv` is read as a table". The hook originally keyed off `content.get("url")` **only**, so a content dict with *other* keys too — `{"label": "Home", "url": "report.csv"}` — was also hijacked into a table read, **silently dropping the sibling keys**.++**Fixed** by tightening detection to the single-key wrapper: `_try_make_csv_list_stuff` now returns `None` unless `set(content) == {"url"}`. A record with any sibling key stays an ordinary record (pinned by `test_record_with_csv_url_field_and_siblings_stays_record`). The residual is now strictly "a one-field `url` concept given `{"url": <tabular>}`", matching the documented limitation exactly. Codex + cubic both flagged the silent-drop; this removes it.++## Deferred — minor / cleanup (low priority, no decision needed)++- **Output-side error UX:** a non-flat / column-mismatched output raises `CsvFlatnessError`/`CsvColumnError` from `csv_from_list_content` **after** the pipeline succeeded; it surfaces via `execute_run`'s generic `except PipelexError` as "Failed to execute pipeline" rather than a tailored "Failed to --save-csv: …". Consider catching `CsvError` around the write block and re-emitting with the save-csv framing.+- **Suffix vocabulary drift:** `_TABULAR_SUFFIXES` (drives `is_tabular_path`) and `assert_supported_table_suffix`'s inline `.csv`/`.xlsx` literals are two independent authorities; express the latter in terms of the former before the `.xlsx`/`.tsv` codec ships.+- **Header-only CSV** yields an empty `ListContent` accepted by the hook, whereas the literal-empty-list path (Case 1.5) rejects empty lists — decide whether an empty CSV is a valid empty batch or should fail fast for parity.+- **Refined native concepts:** a concept that *refines* `Image`/`Document` has a non-native `concept_ref`, so `Concept.is_native_concept` is False and a `.csv`-suffixed `url` triggers the hook → `CsvFlatnessError` on the native content class. Niche; only matters if someone gives such a concept a `.csv` url.+- **Native-row list output:** `--save-csv` on a `Text[]` output writes a degenerate single-column `text` CSV instead of signalling the "flat structured list" contract. Acceptable or worth a clearer message — decide.+- **Small cleanups:** redundant `cast("dict[str, Any]", content)` in Case 2.5 and `cast("ListContent[StuffContent]", …)` in the save block; `get_optional_main_stuff()` vs the `pipe_output.optional_main_stuff` property (two spellings of one lookup); the two `if save_csv is not None:` blocks could be consolidated.diff --git a/wip/csv-support/security-followups.md b/wip/csv-support/security-followups.mdnew file mode 100644index 000000000..9a4461742--- /dev/null+++ b/wip/csv-support/security-followups.md@@ -0,0 +1,25 @@+# CSV support — security review follow-ups (gstack /review, 2026-06-02)++The pre-landing `/review` (5 Claude specialists + Codex adversarial) confirmed the codec is well-hardened after 9 bot rounds. Three security-flavored items are captured here as deliberate decisions / deferred follow-ups rather than blockers for PR #955. The fixes the review *did* apply (BOM read, coercion-message redaction, `--save-csv` error framing) are in the PR.++## 1. Local-file read via a `url` input is the existing input model, not a CSV-specific hole — DEFERRED (platform-wide)++**Finding (security + Codex):** the CSV input hook opens a local file from an attacker-controllable `url` with no path-traversal/absolute-path confinement. `{"url": "../../some.csv"}` or `{"url": "/abs/some.csv"}` is read. Reachable from `WorkingMemoryFactory.make_from_pipeline_inputs`, the path the hosted runner API uses to build working memory from request inputs.++**Why it's not a CSV-specific blocker:** reading a local file from a `url` is the **pre-existing input model** — native `Image`/`Document` concepts already auto-populate from a local-file `url` path and load it (`image_content.py`/`document_content.py` "Auto-populate filename from url when it is a local file path", and the renderers/extractors call `load_binary` on it). The CSV hook is consistent with that model; it only extends it to flat structured concepts for `.csv`/`.xlsx`. It explicitly defers native concepts to their own handling. So "an untrusted `url` can name a local file" is a platform-wide property that pre-exists this PR, not something CSV introduces.++**The real concern (platform-wide):** *if* the hosted runner API accepts untrusted `url` inputs that reach `make_from_pipeline_inputs` without confinement, then any `url`-bearing input type (Image/Document today, CSV now) is a local-file-read primitive. CSV narrows it to `*.csv`/`*.xlsx` files specifically (the suffix is checked on the real path), but that still includes other tenants' CSVs / config CSVs on the host.++**Recommendation (own change, not this PR):** decide the runner's input-trust model and confine file-path inputs for untrusted callers at the input-resolution layer — reject absolute paths and paths escaping an allowlisted base dir — applied uniformly across **all** `url` inputs (Image/Document/CSV), not just CSV. Doing it CSV-only would be inconsistent and would also break the legitimate CLI use case (the CLI deliberately reads local files by relative/absolute path; `_inputs_path_resolver` resolves relative `url`s against the inputs-file dir). This belongs with the platform/runner input-handling, paired with the `tools/uri` scheme-classification follow-up (item A in `phase3-4-review-followups.md`).++## 2. CSV/Excel formula injection — DEFERRED by design (CT3, documented limitation)++**Finding (Codex, "block merge"):** `csv_from_list_content` writes cell values verbatim, so a value starting with `=`/`+`/`-`/`@` executes when opened in Excel/Sheets.++**Decision: this is the locked CT3 product decision, not a new bug.** plan-eng-review chose document-only on injection: do NOT auto-escape formula-leading cells (escaping mutates data and hurts fidelity), document it as a known v1 limitation, defer an opt-in `escape_formulas` flag. It is already written up in the user docs ("No formula-injection escaping") and the design. Codex flags it because it can't see the product decision. v1's primary surface is the local CLI (the user writes and opens their own CSV). The opt-in escape flag is the planned follow-up for when generated CSVs ship to third parties via the hosted runner. Not a blocker; no change in this PR.++## 3. Embedded-NUL path → raw `ValueError` escapes the codec boundary — DEFERRED (niche)++**Finding (security, sub-threshold):** a `url` whose resolved path contains an embedded NUL byte (`x\x00.csv`) makes `path.open()` raise `ValueError("embedded null byte")`, which `_read_table` doesn't catch (it wraps `OSError`/`UnicodeDecodeError`/`LookupError`/`csv.Error`). No file is opened, no traversal, no sensitive data in the message — it's an error-hygiene gap, not a security issue.++**Decision: defer.** Very niche (embedded NUL in a CSV path), and a broad `except ValueError` in `_read_table` risks masking real bugs. If we tighten it later, catch it narrowly around `path.open()` only. Low priority.