Build AI methods with your coding agent, then run them anywhere: as an MCP for chatbots, as a webapp for people, or via API for your software.
3/3 Pipelex kernel — Phase 3: the memory boundary and run-scoped state parity (#1083)
Louis Choquelcca466ac2ad0
diff --git a/.drift/acks/pipelex-kernel-docs.toml b/.drift/acks/pipelex-kernel-docs.tomlnew file mode 100644index 000000000..925e36e42--- /dev/null+++ b/.drift/acks/pipelex-kernel-docs.toml@@ -0,0 +1,27 @@+contract = "pipelex-kernel-docs"+digest = "sha256:e52331097d39ad087ae7b63711abf7409c99f9ccc7e68d5da2ef7175eaed12ca"+reviewed_by = "Claude (Opus 5)"+reviewed_at = "2026-08-09T15:59:15Z"+rationale = "Two failures, one cause. The contract had no ack and a stray ack file sat under its pre-rename id: the class rename swept drift.toml and the ack file's own 'contract' field but not the FILE's name, and the tool keys acks by filename. Renamed the file with git mv rather than hand-editing it, which restored the recorded 2026-08-03 review and reduced the failure to the honest one — triggers changed since that ack. Those changes are rename-only: kernel/__init__.py, pipelex_kernel.py and the boot-contract test each carry the class and module rename and nothing else, and the review target was already swept in the same commit. Reviewed docs/under-the-hood/pipelex-kernel.md against the page's own standing rule (every kernel entry point owes the boot-contract subprocess an arm) by enumerating the package rather than reading the table: all 29 public module-level functions across 18 modules are accounted for, and the boot-contract subprocess imports every operator entry point (run_*, build_*, assemble_*) plus the facade, with the remaining names being internal helpers those ops call transitively. One real gap found and fixed: the table under 'What a programmatic caller imports' had a row for every module except prompt_references, while assemble_llm_prompt and assemble_img_gen_prompt both require the caller to construct ImageReference / DocumentReference values — so the page omitted a module its own entry points oblige a caller to import. Added the row. Pre-existing rather than trigger-induced; fixed because the table states a completeness claim."++[trigger_files]+"pipelex/kernel/__init__.py" = "blob:e513f4ddae722980c268c0d908f55b7752332cec"+"pipelex/kernel/compose_ops.py" = "blob:28e80623d5e454a1b8f0612a27a28a6ddc73c5de"+"pipelex/kernel/compose_results.py" = "blob:a837ae9680c7d4d0eb4dee37d4657c410e5b5cea"+"pipelex/kernel/exceptions.py" = "blob:45e9ad60c2699f7b57cad0a46c13f0b07b4cfbc9"+"pipelex/kernel/extract_ops.py" = "blob:cd596c03e45ab57eccca6fe1bf332e5977d1f2b2"+"pipelex/kernel/extract_results.py" = "blob:79405a56760f3d4b483660d76441fff95f58c64d"+"pipelex/kernel/func_ops.py" = "blob:8718fb7d05bf57ee7686427dd75c2930ace7aecc"+"pipelex/kernel/func_results.py" = "blob:a7425a0ddc21571f23c53b92f5d7559ffdd08b71"+"pipelex/kernel/img_gen_ops.py" = "blob:b22869bee7dfbb624e3c406a0631969fc8695b7c"+"pipelex/kernel/img_gen_prompt.py" = "blob:90c33eae95298aa76d036f256ffc7d6523249166"+"pipelex/kernel/img_gen_results.py" = "blob:46b4ca11c70d906ae6d58ce66461f9b3414f1bfa"+"pipelex/kernel/llm_ops.py" = "blob:36a2e2f3a002e0c50ef4f645937b66573e46aa51"+"pipelex/kernel/llm_prompt_content.py" = "blob:d1d44863aec6e3c4a1451bb25b9586e6d24ef06c"+"pipelex/kernel/llm_results.py" = "blob:6e3b8a2878b5bb284ece5fac10262fb59ac2029d"+"pipelex/kernel/memory_ops.py" = "blob:c4d7fee0a5ac2c1292ad39d2c188b2a02b47d15b"+"pipelex/kernel/pipelex_kernel.py" = "blob:abcdf8ca64f29666c3c51fba61287ef80865e5d1"+"pipelex/kernel/prompt_references.py" = "blob:9af4077adcb49abacef98d0100ab9cab14d24880"+"pipelex/kernel/search_ops.py" = "blob:01be4bef28f98342678cba16ac8c4b21fadcaae1"+"pipelex/kernel/search_results.py" = "blob:101b70dc96cc5ee1721c9f6b1cb725a5ca78ff70"+"tests/unit/pipelex/kernel/test_kernel_boot_contract.py" = "blob:bd9f512eca935f9fae1288fb7a6aa41342f9289e"diff --git a/docs/under-the-hood/architecture-overview.md b/docs/under-the-hood/architecture-overview.mdindex 4ef5528cc..07547f48e 100644--- a/docs/under-the-hood/architecture-overview.md+++ b/docs/under-the-hood/architecture-overview.md@@ -46,7 +46,7 @@ Operators perform concrete actions:- **PipeCompose** - Compose content from templates or construct structured objects- **PipeSearch** - Search the web for information-What an operator *does* — model-deck resolution, prompt assembly, generation, memory write-back — lives one layer down in [`pipelex/kernel/`](https://github.com/Pipelex/pipelex/tree/main/pipelex/kernel), as plain functions that take everything method-specific as an explicit argument and read only the runtime services a boot stands up, such as the model deck and the content generator. The operator classes above hold what is specific to being a step in a method: blueprint resolution, input validation, error context, and execution-graph tracing. The split exists so the same semantics can be called without a loaded method, and so there is one implementation rather than two that drift.+What an operator *does* — model-deck resolution, prompt assembly, generation, memory write-back — lives one layer down in [`pipelex/kernel/`](https://github.com/Pipelex/pipelex/tree/main/pipelex/kernel), as plain functions that take everything method-specific as an explicit argument and read only the runtime services a boot stands up, such as the model deck and the content generator. The operator classes above hold what is specific to being a step in a method: blueprint resolution, input validation, error context, and execution-graph tracing. The split exists so the same semantics can be called without a loaded method, and so there is one implementation rather than two that drift. See [The Pipelex Kernel](./pipelex-kernel.md) for the layering contract, what a programmatic caller imports, and how it boots.### Core Domaindiff --git a/docs/under-the-hood/index.md b/docs/under-the-hood/index.mdindex 72499e1b3..74c130e9c 100644--- a/docs/under-the-hood/index.md+++ b/docs/under-the-hood/index.md@@ -10,6 +10,7 @@ Welcome to the technical deep-dives of Pipelex. This section is for contributors## What You'll Find Here- **Architecture Overview** - The two-layer design and how components fit together+- **The Pipelex Kernel** - Operator semantics as importable functions, and how to call them without a loaded method- **Build-time Elaboration** - How shorthand directives like `structuring_method = "preliminary_text"` are rewritten into concrete pipe trees before any pipe runs- **Execution Graph Tracing** - How pipeline executions are captured as graphs for visualization- **Image Handling in LLM Prompts** - How images flow from inputs to LLM calls@@ -35,6 +36,7 @@ Welcome to the technical deep-dives of Pipelex. This section is for contributors## Start Exploring- [:material-sitemap: Architecture Overview](./architecture-overview.md){ .md-button .md-button--primary }+- [:material-engine-outline: The Pipelex Kernel](./pipelex-kernel.md){ .md-button }- [:material-source-branch: Build-time Elaboration](./build-time-elaboration.md){ .md-button }- [:material-graph: Execution Graph Tracing](./execution-graph-tracing.md){ .md-button }- [:material-image-multiple: Image Handling in LLM Prompts](./image-handling-in-llm-prompts.md){ .md-button }diff --git a/docs/under-the-hood/pipelex-kernel.md b/docs/under-the-hood/pipelex-kernel.mdnew file mode 100644index 000000000..01e5190f4--- /dev/null+++ b/docs/under-the-hood/pipelex-kernel.md@@ -0,0 +1,162 @@+---+title: "The Pipelex Kernel"+description: "Operator semantics as importable functions — what pipelex/kernel/ is, the layering contract that keeps it callable without a loaded method, and how a programmatic caller boots it."+---++# The Pipelex Kernel++This page is for contributors working on Pipelex internals, and for anyone embedding the runtime directly rather than running `.mthds` methods. For how the operator classes above it fit into the whole, see [Architecture Overview](./architecture-overview.md).++What a `PipeLLM` step actually *does* — resolve a model off the deck, derive a prompting style, assemble the prompt, generate, write the result into memory — used to be reachable only through a fully booted interpreter with a method loaded. [`pipelex/kernel/`](https://github.com/Pipelex/pipelex/tree/main/pipelex/kernel) holds that semantics as plain functions, so it has **one implementation with two kinds of caller**:++- the interpreter's operator classes (`PipeLLM`, `PipeExtract`, `PipeImgGen`, `PipeSearch`, `PipeCompose`, `PipeFunc`), which resolve blueprints, validate inputs, wrap errors and trace, then call the kernel;+- any **programmatic caller** embedding the runtime, which calls the same functions on a process with zero `.mthds` loaded.++Single-sourcing is the whole point. Two callers with two copies of "what an LLM step means" drift, and nothing tells you when they have.++---++## The boot contract++Every kernel call must be servable on `RuntimeBoot.make()` ([`pipelex/runtime_boot.py`](https://github.com/Pipelex/pipelex/blob/main/pipelex/runtime_boot.py)) — the **runtime-only** composition root, with no interpreter constructed and no library loaded.++```python+from pipelex.runtime_boot import RuntimeBoot+from pipelex.kernel.pipelex_kernel import PipelexKernel++RuntimeBoot.make()+kernel = PipelexKernel.make(user_id="my-service")+```++That boot stands up the model deck, the content generator, the class registry, the reporting delegate and the plugin registries — the machinery inference needs. It does **not** stand up the library manager, the pipe router or the pipeline manager, because a kernel caller has no method to load.++`needs_inference=False` boots keyless (no credentials, no model-deck validation) and sets the forced-DRY flag: every run the process initiates is coerced to `run_mode=DRY`, so the leaves mock instead of calling a provider. `PipelexKernel.make` applies that rule through the same `runtime_hub.resolve_run_mode_for_boot` the pipe tier uses — a second copy of the rule at a second factory is how the two would drift apart.++---++## Layering: what the kernel may and may not touch++| May | May not |+|---|---|+| `pipelex.runtime_hub` — the model deck, the content generator, the reporting delegate | `pipelex.interpreter_hub`, directly or transitively |+| `pipelex.core`, `pipelex.cogt`, `pipelex.tools`, `pipelex.tracing` | `pipelex.libraries`, `pipelex.pipe_operators`, `pipelex.pipe_controllers`, `pipelex.pipe_run`, `pipelex.pipeline`, `pipelex.mthds_parsing`, … |+| Definition-site imports | `pipelex.exceptions` or any other cross-layer re-export aggregate |+| Module-top-level imports | Function-local imports (invisible to the static graph *and* to the import-closure test at once) |++The **caller-facing** API is stricter still: hub-free. Everything method-specific arrives as an explicit argument — the concept, the concrete output class, the resolved setting, the working memory — and never through an ambient lookup. Concept compatibility, when a kernel path needs it at all, goes through the pure tiers (`Concept.are_compatible_by_declaration`, `are_structure_classes_compatible`), never through `ConceptLibrary.is_compatible`.++Four gates hold this, and each covers something the others miss — see [Hub Layering](../contribute/hub-layering.md) for the full picture:++| Gate | What it proves |+|---|---|+| `pipelex-dev check-hub-layering` | No kernel *module* imports the interpreter hub |+| `tests/unit/pipelex/test_runtime_layer_import_closure.py` | A kernel entry point *imports* clean |+| `tests/unit/pipelex/test_runtime_layer_exceptions_aggregate_gate.py` | No kernel module reaches the exceptions aggregate — imports and bare strings alike |+| `tests/unit/pipelex/kernel/test_kernel_boot_contract.py` | Every kernel entry point **runs** on a keyless boot, swept afterwards — except the three `resolve_*_setting` helpers, which read the model deck (a separate question from this one) |++Only the last one can see a function-local interpreter import, and it is **per-function**: it catches one inside `run_search` only by calling `run_search`. Every new kernel entry point owes it an arm.++---++## What a programmatic caller imports++Module-level functions carry the semantics. `PipelexKernel` is a thin façade over the LLM pair, holding the per-run state a caller would otherwise thread through every call; every other operator is called directly.++Both façade calls take the concept and the output class the caller wants, defaulting to `Text` and `TextContent` when it wants neither. `llm_text` accepts them because a text step is not always a *native*-`Text` step: a method may declare its output as a concept refining `Text`, and a façade that hardcoded the native one would write a different concept into memory than the interpreter writes from the same authored declaration.++| Module | Entry points |+|---|---|+| `pipelex.kernel.pipelex_kernel` | `PipelexKernel.make`, `.llm_text`, `.llm_object`, `.make_step_metadata` |+| `pipelex.kernel.llm_ops` | `resolve_llm_setting_for_text` / `_for_object`, `derive_templating_style`, `derive_structure_prompt`, `generate_object_content`, `run_llm_text`, `run_llm_object` |+| `pipelex.kernel.extract_ops` | `resolve_extract_setting`, `build_extract_job_params`, `run_extract` |+| `pipelex.kernel.img_gen_ops` | `resolve_img_gen_setting`, `resolve_default_size`, `build_img_gen_job_params`, `run_img_gen` |+| `pipelex.kernel.search_ops` | `resolve_search_setting`, `run_search` |+| `pipelex.kernel.compose_ops` | `build_compose_context`, `build_composed_content`, `run_compose_template` |+| `pipelex.kernel.func_ops` | `call_registered_function`, `run_func` |+| `pipelex.kernel.memory_ops` | `shape_inputs`, `store_result`, `extract_main_content` / `extract_named_content`, `extract_main_content_as_list` / `extract_named_content_as_list` |+| `pipelex.kernel.llm_prompt_content` | `LlmPromptContent`, `assemble_llm_prompt` |+| `pipelex.kernel.img_gen_prompt` | `assemble_img_gen_prompt` |+| `pipelex.kernel.prompt_references` | `ImageReference` / `ImageReferenceKind`, `DocumentReference` / `DocumentReferenceKind` |+| `pipelex.kernel.*_results` | The typed result envelopes |++The two `assemble_*` functions are there because `run_llm_text` and `run_img_gen` both take a *ready* prompt. A caller that could not build one would be holding an operator it cannot reach, which is what image generation was until `assemble_img_gen_prompt` existed: its only builder was an interpreter-layer blueprint. What they own is the part a caller must not re-derive — resolving `ImageReference` and `DocumentReference` out of working memory, and, on the image side, keeping the `[Image N]` tokens numbered from the same registry that orders `input_images`, since a mismatch mislabels which image the prompt is describing and nothing downstream can detect it.++There are **no re-exports**: `pipelex/kernel/__init__.py` holds doctrine and nothing else, and every symbol is imported from the module that defines it. For this package that is a layering property rather than a style one — a module that re-exports across layers is a layer boundary with the sign filed off.++Every kernel function is **fully keyword-only**, with zero entries in `subject_grants.toml`. Call sites name every argument.++---++## The memory boundary++`WorkingMemory` is threaded explicitly: a call takes it and returns it. The contract, which both kinds of caller must read the same way:++!!! warning "Treat the returned memory as the result"+ A kernel call may mutate the memory it was passed **and** returns it. Callers must use the returned one and must not rely on the two being the same object — inline execution aliases them today, and a serialization boundary will not.++`pipelex.kernel.memory_ops` holds the three ends of that boundary — shape in, write back, read out:++- **`shape_inputs`** — interpret raw values against the specs declared for them (Smart Inputs: a bare string becomes the declared concept, a dict validates against a structured one, a list shapes element-wise). It takes a `ConceptProviderAbstract` explicitly, because resolving concepts is what a loaded method's library is for and the kernel must stay callable without one. The interpreter hands over its concept library; a library-free caller supplies its own provider (the boot-contract test shows the smallest one that works — native concepts from `ConceptFactory`, compatibility from the declaration tier, structure classes from the class registry a boot fills).+- **`store_result`** — the write-back every operator's ops end with, and the one place the memory contract is implemented.+- **`extract_main_content` / `extract_named_content`** — the typed read. Needed even though every result envelope already carries the produced content, because those fields are annotated with the base `StuffContent`: pass the class you asked for and get it back narrowed.+- **`extract_main_content_as_list` / `extract_named_content_as_list`** — the same typed read for a call that produced several objects. A multiple-output call stores one `ListContent`, which the single-content reads cannot narrow: the bare item class raises, and `ListContent[item_type]` is rejected by design. These verify every item against `item_type`, so the list comes back typed all the way down.++```python+memory = shape_inputs(inputs={"topic": "kernels"}, concept_provider=provider, input_specs=specs)+result = await kernel.llm_object(memory=memory, output_class=Summary, concept=summary_concept, model=model, user="Summarize $topic", result="summary")+summary = extract_main_content(memory=result.memory, content_type=Summary)+```++Reach for the list pair whenever the call asked for several — `is_multiple_output=True` or `fixed_nb_output=n`:++```python+result = await kernel.llm_object(memory=memory, output_class=Summary, concept=summary_concept, model=model, user="Summarize $topic", result="summaries", is_multiple_output=True)+summaries = extract_main_content_as_list(memory=result.memory, item_type=Summary).items+```++---++## Run-scoped state, and who owns the usage lifecycle++`PipelexKernel` holds exactly two things, both run-scoped identity:++- **`job_metadata`** — the run-level metadata. It is not what a step runs under: every call mints a per-step copy through `make_step_metadata()`, carrying a fresh `pipe_run_id` and inheriting the trace context, so trace and usage attribution stay per-step. This mirrors the interpreter's pass-down-a-modified-copy pattern.+- **`cogt_run_params`** — the execution-mode contract (`run_mode`, and the DRY-only `is_mock_usage` sub-flag) that every cogt leaf reads off the assignment it is handed.++Nothing derived from config or the model deck is cached on the instance — resolved settings and prompting styles are computed per call, because cached derived state would shadow a later config or deck change and break per-call variation.++**Cost and usage reporting is the caller's lifecycle, not the kernel's.** The interpreter's run machinery opens a graph tracer, builds an event log, registers it on the report delegate and closes all three in a `finally`, because it has a run boundary to hang that on. A kernel call has no such boundary — it is one step, and a caller may make one or a thousand. So the kernel takes a `TraceContext` and does exactly one thing with it: stamp it onto every step's `JobMetadata`, which is what the cogt leaf reads to decide whether to emit a usage event.++Everything else is yours:++```python+from pipelex.runtime_hub import get_report_delegate+from pipelex.system.trace_context import TraceContext+from pipelex.tracing.in_memory_event_log import InMemoryEventLog+from pipelex.tracing.usage_aggregator import UsageAggregator++event_log = InMemoryEventLog()+trace_context = TraceContext(graph_id=run_id, data_inclusion=data_inclusion, emit_graph_events=False, emit_usage_events=True)+get_report_delegate().set_event_log(context_key=trace_context.lookup_key, event_log=event_log, workflow_id="direct", pipeline_run_id=run_id)+try:+ kernel = PipelexKernel.make(user_id="my-service", trace_context=trace_context)+ ...+ tokens_usages = UsageAggregator.aggregate(event_log.read_events(run_id))+finally:+ get_report_delegate().clear_event_log(context_key=trace_context.lookup_key)+```++Passing a `trace_context` adopts its `graph_id` as the run's `pipeline_run_id`. The two are one identity: letting them diverge would scatter a single run's usage events across two ids, because the registered-context emit path stamps the event log's id while the runner fallback stamps the metadata's — and a read-back keyed on either would silently miss the other's.++`pipelex.tracing` holds both halves a caller needs (`make_event_log` for a configured backend, `UsageAggregator` for the read-back) and is runtime-layer, so none of this costs the boot contract. The records that come out are the same [`TokensUsage` wire records](./tokens-usage-wire-records.md) an `/execute` response carries — pinned by an integration test that runs the same step through both callers and compares them.++---++## What the kernel deliberately does not cover++Two arms of the interpreter stayed interpreter-side, both because moving them would cost more than the caller gains:++- **`PipeCompose`'s construct mode.** Its semantics are `StructuredContentComposer` over a `ConstructBlueprint`, and a blueprint is an MTHDS language artifact with a language-side consumer. A programmatic caller holds real Python and builds its structured object directly rather than describing the construction declaratively. The template path *is* fully extracted, and the one thing both paths share — the three-layer context ordering — is single-sourced in `build_compose_context`.+- **`PipeFunc`'s pluggable executor seam.** The protocol and its DTOs are typed on interpreter models (`PipeRunParams`, `LibraryCrate`), so the kernel cannot name them. What running a function *means* — registry lookup, async-vs-sync dispatch, content coercion — is single-sourced in `call_registered_function`, which both the in-process executor and the kernel's `run_func` ride. What stays outside is *where* the function runs, which is configured deployment machinery rather than operator semantics. So a kernel `run_func` always runs in this process.++Beyond those: controllers are out of scope entirely (`pipelex/pipe_controllers/` is the interpreter's), the kernel is not separately installable from PyPI, and "activity-shaped" is a design constraint on the call signatures rather than a distributed-execution deliverable.diff --git a/drift.toml b/drift.tomlindex 7adbbc20d..91213fefa 100644--- a/drift.toml+++ b/drift.toml@@ -71,3 +71,20 @@ review = ["docs/contribute/hub-layering.md",]verify_commands = []++[contracts.pipelex-kernel-docs]+description = "The pipelex-kernel doc is the kernel's specification for programmatic callers and must track the kernel surface."+# The page tabulates the surface module by module (every entry point a caller imports) and states+# what each of the four layering gates covers. A kernel function added, renamed or moved silently+# falsifies the first table; a change to the boot-contract script's coverage silently falsifies the+# second. Neither is mechanizable — "does this prose still describe the surface" is a judgement — so+# it is a review obligation rather than a derived check. No verify command: the gates the page+# describes already run in `make check` and CI.+triggers = [+ "pipelex/kernel/**/*.py",+ "tests/unit/pipelex/kernel/test_kernel_boot_contract.py",+]+review = [+ "docs/under-the-hood/pipelex-kernel.md",+]+verify_commands = []diff --git a/mkdocs.yml b/mkdocs.ymlindex 40d964088..889a2b8b9 100644--- a/mkdocs.yml+++ b/mkdocs.yml@@ -310,6 +310,7 @@ plugins:Under the Hood:- under-the-hood/index.md: "Under the Hood Overview"- under-the-hood/architecture-overview.md: "Architecture Overview"+ - under-the-hood/pipelex-kernel.md: "The Pipelex Kernel"- under-the-hood/build-time-elaboration.md: "Build-time Elaboration"- under-the-hood/codegen-projections.md: "Codegen Projections"- under-the-hood/dry-run-mock-generation.md: "Dry Run Mock Generation"@@ -550,6 +551,7 @@ nav:- Under the Hood:- Overview: under-the-hood/index.md- Architecture Overview: under-the-hood/architecture-overview.md+ - The Pipelex Kernel: under-the-hood/pipelex-kernel.md- Build-time Elaboration: under-the-hood/build-time-elaboration.md- Codegen Projections: under-the-hood/codegen-projections.md- Dry Run Mock Generation: under-the-hood/dry-run-mock-generation.mddiff --git a/pipelex/kernel/llm_results.py b/pipelex/kernel/llm_results.pyindex 2df6462b6..6e3b8a287 100644--- a/pipelex/kernel/llm_results.py+++ b/pipelex/kernel/llm_results.py@@ -29,7 +29,15 @@ from pipelex.core.stuffs.stuff_content import StuffContentclass StructuringPath(StrEnum):- """How the output was produced — the tracer's vocabulary, owned here rather than at each caller."""+ """Which kernel LLM path produced the output, owned here rather than restated at each caller.++ It names the *LLM* paths, not the whole `structuring_path` key the tracer consumes: `PipeStructure`+ writes a bare `"structure"` into that same key, and deliberately has no member here. It calls the+ kernel's pieces (`generate_object_content` and friends) rather than an entry point that returns one+ of these, so no kernel op can produce `"structure"` — adding the member would put a value into a+ "what the kernel op did" type that nothing here can return. The key is `dict[str, Any]` end to end+ and this is a `StrEnum`, so both forms are the same JSON string downstream.+ """TEXT = "text"OBJECT_DIRECT = "object_direct"diff --git a/pipelex/kernel/memory_ops.py b/pipelex/kernel/memory_ops.pyindex 18af07a64..c4d7fee0a 100644--- a/pipelex/kernel/memory_ops.py+++ b/pipelex/kernel/memory_ops.py@@ -1,17 +1,70 @@-"""Memory write-back, shared by every operator's kernel ops.+"""The memory boundary of a kernel call: shape inputs in, write results back, read results out.-Each operator ends the same way — wrap the produced content in a `Stuff` and make it the memory's new-main stuff — so that ending lives here once rather than at each domain's ops module. It is also where-the memory contract is actually implemented, which is why a domain module never inlines the two calls:-the day the contract stops aliasing argument and return, exactly one function changes.+`store_result` is the write-back every operator's kernel ops end with — wrap the produced content in+a `Stuff` and make it the memory's new main stuff — so that ending lives here once rather than at+each domain's ops module. It is also where the memory contract is actually implemented, which is why+a domain module never inlines the two calls: the day the contract stops aliasing argument and return,+exactly one function changes.++`shape_inputs` and the extraction helpers are the boundary's other two ends. They are thin over+machinery that already exists (`InputShaper`, `WorkingMemory`'s typed accessors) and are deliberately+kept that way — the value is not new behavior, it is that a programmatic caller reads the whole+memory boundary off one module whose every function is keyword-only, hub-free and boot-contract+tested, instead of assembling it from a `core/` factory, a `core/` classmethod and two `WorkingMemory`+methods with positional parameters. Both callers of the shaping half go through the same function,+which is the property that stops them drifting."""+from pathlib import Path++from mthds.protocol.pipeline_inputs import PipelineInputs+from pipelex.core.concepts.concept import Concept+from pipelex.core.concepts.concept_provider_abstract import ConceptProviderAbstract+from pipelex.core.memory.input_shaper import InputShaperfrom pipelex.core.memory.working_memory import WorkingMemory-from pipelex.core.stuffs.stuff_content import StuffContent+from pipelex.core.pipes.inputs.input_stuff_specs import InputStuffSpecs+from pipelex.core.stuffs.list_content import ListContent+from pipelex.core.stuffs.stuff_content import StuffContent, StuffContentTypefrom pipelex.core.stuffs.stuff_factory import StuffFactory+def shape_inputs(+ *,+ inputs: PipelineInputs,+ concept_provider: ConceptProviderAbstract,+ input_specs: InputStuffSpecs,+ search_domain_codes: list[str] | None = None,+ inputs_base_dir: Path | None = None,+) -> WorkingMemory:+ """Interpret a caller's raw inputs against their declared specs and return the memory to run on.++ Signature-driven shaping only (Smart Inputs): each value is read top-down against the concept its+ spec declares. `input_specs` is required rather than optional, which is the one place this+ function is narrower than `WorkingMemoryFactory.make_from_pipeline_inputs` — the bottom-up,+ no-signature arm infers a concept from each value's own shape, and inference is exactly the kind+ of ambient guess a kernel entry point should not make on a caller's behalf. A caller with no+ declared specs builds its stuffs directly and hands over a `WorkingMemory`.++ `concept_provider` is taken explicitly, never looked up: resolving concepts is what a loaded+ method's library is for, and the kernel must stay callable without one.++ A malformed input is rejected here with a typed, locator-bearing error rather than coerced — this+ is the boundary where a caller's raw values stop being raw, so it is the right place to refuse.++ Raises:+ UnknownInputNameError: a provided name is not declared in `input_specs` (D8).+ InputShapingError subclasses: a provided value cannot be shaped to its declared concept (D4).+ """+ return InputShaper.shape(+ inputs,+ concept_provider=concept_provider,+ input_specs=input_specs,+ search_domain_codes=search_domain_codes,+ inputs_base_dir=inputs_base_dir,+ )++def store_result(*,memory: WorkingMemory,@@ -29,3 +82,63 @@ def store_result(stuff = StuffFactory.make_stuff(concept=concept, content=content, name=result_name, code=result_code)memory.set_new_main_stuff(stuff=stuff, name=result_name)return memory+++def extract_main_content(*, memory: WorkingMemory, content_type: type[StuffContentType]) -> StuffContentType:+ """Read the memory's main stuff as `content_type`.++ This is the typed read of what a kernel call just produced. It is needed even though every result+ envelope already carries the content, because the envelopes annotate that field with the base+ `StuffContent` (`LlmObjectResult.content` and its siblings) — so a caller holding a result has the+ object but not the type. Pass the class you asked the op to produce and get it back narrowed.++ Read from the memory the call **returned**, never from the one you passed in: they alias today and+ the contract says they need not.++ Raises:+ WorkingMemoryStuffNotFoundError: nothing has been stored as main stuff yet.+ StuffContentTypeError: the stored content is not a `content_type`.+ """+ return memory.main_stuff_as(content_type)+++def extract_named_content(*, memory: WorkingMemory, name: str, content_type: type[StuffContentType]) -> StuffContentType:+ """Read a named slot as `content_type` — the same typed read as `extract_main_content`, by name.++ Named slots are how a caller reads back anything other than the last result: an input it shaped,+ or an earlier step's output stored under its own `result_name`.++ Raises:+ WorkingMemoryStuffNotFoundError: no stuff (and no alias) under that name.+ StuffContentTypeError: the stored content is not a `content_type`.+ """+ return memory.get_stuff_as(name=name, content_type=content_type)+++def extract_main_content_as_list(*, memory: WorkingMemory, item_type: type[StuffContentType]) -> ListContent[StuffContentType]:+ """Read the memory's main stuff as a list of `item_type` — the typed read for a multi-output call.++ A kernel op that produces several objects (`llm_object(is_multiple_output=True)`) stores them as one+ `ListContent`, and the single-content read above cannot narrow that: passing the bare item class+ raises, and passing `ListContent[item_type]` is rejected by design. Without this, the one shape the+ kernel produces that its own read half could not narrow would send the caller back out to+ `WorkingMemory`'s positional accessors — and the bare-`ListContent` workaround loses the item type,+ which is the whole point of asking.++ Every item is verified against `item_type`, so the returned list is typed all the way down.++ Raises:+ WorkingMemoryStuffNotFoundError: nothing has been stored as main stuff yet.+ StuffContentTypeError: the stored content is not a list, or an item is not an `item_type`.+ """+ return memory.main_stuff_as_list(item_type)+++def extract_named_content_as_list(*, memory: WorkingMemory, name: str, item_type: type[StuffContentType]) -> ListContent[StuffContentType]:+ """Read a named slot as a list of `item_type` — the same typed read as `extract_main_content_as_list`, by name.++ Raises:+ WorkingMemoryStuffNotFoundError: no stuff (and no alias) under that name.+ StuffContentTypeError: the stored content is not a list, or an item is not an `item_type`.+ """+ return memory.get_stuff_as_list(name, item_type=item_type)diff --git a/pipelex/kernel/pipelex_kernel.py b/pipelex/kernel/pipelex_kernel.pyindex 97bac6c48..abcdf8ca6 100644--- a/pipelex/kernel/pipelex_kernel.py+++ b/pipelex/kernel/pipelex_kernel.py@@ -15,12 +15,24 @@ What it deliberately does **not** hold is anything derived from config or the moLLM settings, prompting style. Those are computed per call and never cached here, exactly as``pipe_llm.py`` derives them per run today: cached derived state is hidden shared state, it makes alater config or deck change invisible to a live kernel, and it breaks per-call variation.++**Who owns the usage/trace lifecycle for a kernel-driven run: the caller, not the kernel.** The+interpreter's run machinery opens a graph tracer, builds the event log, registers it on the report+delegate and closes all three in a ``finally`` — because it has a run boundary to hang that on. A+kernel call has no such boundary: it is one step, and a caller may make one or a thousand. So the+kernel takes a ``TraceContext`` as an argument and does exactly one thing with it — stamp it onto+every step's :class:`JobMetadata`, which is what the cogt leaf reads to decide whether and where to+emit a usage event. Opening the event log, registering it (``get_report_delegate().set_event_log``),+reading the events back and clearing the registration stay the caller's, and they are the whole of+what stands between a kernel run and the interpreter's cost reporting. ``pipelex/tracing/`` holds+both halves a caller needs (``make_event_log``, ``UsageAggregator``) and is runtime-layer, so none of+this costs the boot contract. See ``docs/under-the-hood/pipelex-kernel.md``."""from typing import Selffrom uuid import uuid4-from pipelex.cogt.content_generation.cogt_run_params import CogtRunParams+from pipelex.cogt.content_generation.cogt_run_params import CogtRunParams, check_mock_usage_requires_dryfrom pipelex.cogt.llm.llm_setting import LLMModelChoicefrom pipelex.core.concepts.concept import Conceptfrom pipelex.core.concepts.concept_factory import ConceptFactory@@ -40,6 +52,7 @@ from pipelex.kernel.llm_results import LlmObjectResult, LlmTextResultfrom pipelex.runtime_hub import resolve_run_mode_for_bootfrom pipelex.system.job_metadata import JobMetadatafrom pipelex.system.pipe_run_mode import PipeRunMode+from pipelex.system.trace_context import TraceContextclass PipelexKernel:@@ -50,18 +63,40 @@ class PipelexKernel:self.cogt_run_params = cogt_run_params@classmethod- def make(cls, *, run_mode: PipeRunMode = PipeRunMode.LIVE, user_id: str) -> Self:- """Mint a kernel for one run: a fresh run id, and the execution-mode contract for it.+ def make(+ cls,+ *,+ run_mode: PipeRunMode = PipeRunMode.LIVE,+ user_id: str,+ is_mock_usage: bool = False,+ trace_context: TraceContext | None = None,+ ) -> Self:+ """Mint a kernel for one run: a run id, the execution-mode contract, and optional tracing.This is a run-params factory, so it owes the same keyless-boot contract as the pipe tier's:a process booted with ``needs_inference=False`` forces every run it initiates to DRY, and akernel that skipped that would spend real money on the exact boot this package documents asits target. Applied through the shared ``resolve_run_mode_for_boot`` — a second copy of the- rule at a second factory is how the two would drift.+ rule at a second factory is how the two would drift. And, like the pipe tier's factory, the+ REQUESTED mode is validated before that coercion: ``is_mock_usage`` is a DRY-only sub-flag,+ so an illegal LIVE request must fail loud on every boot rather than be silently legalised by+ a keyless process forcing it to DRY.++ ``trace_context`` is what makes cost/usage reporting reach parity with an interpreter run:+ the cogt leaf emits a usage event only when the metadata it is handed carries one. Passing it+ adopts its ``graph_id`` as this run's ``pipeline_run_id`` rather than minting a fresh id —+ the two are one identity, and letting them diverge would scatter a single run's usage events+ across two ids (the registered-context emit path stamps the event log's id, the runner+ fallback stamps the metadata's), so the read-back would silently miss half of them."""+ check_mock_usage_requires_dry(run_mode=run_mode, is_mock_usage=is_mock_usage)return cls(- job_metadata=JobMetadata(user_id=user_id, pipeline_run_id=str(uuid4())),- cogt_run_params=CogtRunParams(run_mode=resolve_run_mode_for_boot(requested=run_mode)),+ job_metadata=JobMetadata(+ user_id=user_id,+ pipeline_run_id=trace_context.graph_id if trace_context is not None else str(uuid4()),+ trace_context=trace_context,+ ),+ cogt_run_params=CogtRunParams(run_mode=resolve_run_mode_for_boot(requested=run_mode), is_mock_usage=is_mock_usage),)def make_step_metadata(self) -> JobMetadata:@@ -69,8 +104,9 @@ class PipelexKernel:``otel_context`` is passed explicitly as ``None`` rather than left to inherit, which is thecontract ``copy_with_update`` states: the field is computed fresh per step by whoever opens- the span, and a kernel call opens none. Span and trace-context wiring for a kernel-driven- run is a separate concern from minting the identity, and is not wired here.+ the span, and a kernel call opens none. ``trace_context`` is left to inherit, which is the+ same method's other contract and the reason usage events from every step of a kernel run+ attribute to the one context the caller supplied."""return self.job_metadata.copy_with_update(otel_context=None, pipe_run_id=str(uuid4()))diff --git a/pipelex/pipeline/execution_seams.py b/pipelex/pipeline/execution_seams.pyindex 54728b180..fd042fc0f 100644--- a/pipelex/pipeline/execution_seams.py+++ b/pipelex/pipeline/execution_seams.py@@ -36,6 +36,7 @@ from pipelex.interpreter_hub import (resolve_library_dirs,set_current_library,)+from pipelex.kernel.memory_ops import shape_inputsfrom pipelex.mthds_parsing.parser import MthdsParserfrom pipelex.pipe_machinery.pipe_abstract import PipeAbstractfrom pipelex.pipe_run.pipe_job import PipeJob@@ -199,9 +200,10 @@ async def prepare_pipe_job(else:# Thread the pipe's declared inputs so each value is shaped top-down against the# signature (Smart Inputs). `pipe.inputs` is the method-boundary contract — the same- # source the Optionals pass reads below — not the aggregated `needed_inputs()`.- working_memory = WorkingMemoryFactory.make_from_pipeline_inputs(- pipeline_inputs=inputs,+ # source the Optionals pass reads below — not the aggregated `needed_inputs()`. Through+ # the kernel op so the interpreter and a programmatic caller shape inputs identically.+ working_memory = shape_inputs(+ inputs=inputs,concept_provider=get_concept_library(),input_specs=pipe.inputs,search_domain_codes=search_domain_codes,diff --git a/tests/integration/pipelex/kernel/test_usage_parity.py b/tests/integration/pipelex/kernel/test_usage_parity.pynew file mode 100644index 000000000..d28e86b59--- /dev/null+++ b/tests/integration/pipelex/kernel/test_usage_parity.py@@ -0,0 +1,145 @@+"""Usage/cost reporting reaches parity between an interpreter run and direct kernel calls.++The kernel's promise is that operator semantics have one implementation with two callers. Usage+reporting is where that promise is easiest to break silently: the records are assembled from trace+events, and a kernel call that forgot to carry a ``TraceContext`` onto its per-step ``JobMetadata``+would simply emit nothing — no error, an empty cost report, and a suite that stays green because no+test compares the two callers' output.++So this measures rather than asserts. Both sides run the same step (one LLM text generation) under+``run_mode=DRY`` with ``is_mock_usage=True``, which keeps the leaf offline while making it report+*non-zero* synthetic usage under a sentinel model. The two ``TokensUsageRecord`` lists — the wire+shape ``/execute`` returns on ``pipe_output.tokens_usages`` and durable runs persist as+``tokens_usages.json`` — are then compared field by field.++``pipe_code`` is the one field that would legitimately differ — a kernel call has no pipe to name —+and this measurement cannot see it: the interpreter stamps it in ``live_run_pipe``, which a DRY run+never reaches, so both sides report ``None``. That is asserted rather than excluded quietly, because+it is the honest scope of what the comparison proves, and it is a pre-existing interpreter gap in its+own right (a ``--dry-run --mock-usage`` cost report has no pipe attribution — see KF-13 in+``wip/kernel/deferred-follow-ups.md``). Timestamps differ because the two runs happen at different+times.++Transport ownership is visible in the setup itself. The interpreter half needs NDJSON tracing enabled+because its run machinery builds the event log off the config; the kernel half hands its own+``InMemoryEventLog`` to the report delegate, because for a kernel-driven run the caller owns that+lifecycle — the kernel only stamps the context it is given onto every step.+"""++from typing import Any++import pytest+from pytest_mock import MockerFixture++from pipelex.cogt.content_generation.dry_mock import MOCK_USAGE_MODEL_NAME+from pipelex.cogt.llm.llm_setting import LLMSetting+from pipelex.config import get_config+from pipelex.core.memory.working_memory_factory import WorkingMemoryFactory+from pipelex.kernel.pipelex_kernel import PipelexKernel+from pipelex.pipeline.runner import PipelexMTHDSProtocol+from pipelex.reporting.reporting_types import AnyTokensUsage+from pipelex.reporting.usage_records import TokensUsageRecord, make_tokens_usage_record+from pipelex.runtime_hub import get_report_delegate+from pipelex.system.configuration.configs import NdjsonTracingConfig, TracingBackend+from pipelex.system.pipe_run_mode import PipeRunMode+from pipelex.system.trace_context import TraceContext+from pipelex.tracing.in_memory_event_log import InMemoryEventLog+from pipelex.tracing.usage_aggregator import UsageAggregator++_DOMAIN = "kernel_usage_parity"+_PIPE_CODE = "write_text"+_USER_PROMPT = "Write about the kernel."+_MTHDS = f"""+domain = "{_DOMAIN}"+description = "One LLM text step, the smallest thing both callers can run"++[pipe.{_PIPE_CODE}]+type = "PipeLLM"+description = "Pipe that outputs plain text"+output = "Text"+prompt = "{_USER_PROMPT}"+"""++#: The record fields that carry the run's identity or its wall-clock, which two separate runs cannot+#: share. Everything else is the shape parity is measured on.+_RUN_SPECIFIC_FIELDS = ("pipe_code", "started_at", "completed_at")++[email protected](loop_scope="class")+class TestKernelUsageParity:+ def _comparable(self, record: TokensUsageRecord) -> dict[str, Any]:+ return {key: value for key, value in record.model_dump().items() if key not in _RUN_SPECIFIC_FIELDS}++ def _records(self, tokens_usages: list[AnyTokensUsage]) -> list[TokensUsageRecord]:+ return [make_tokens_usage_record(tokens_usage) for tokens_usage in tokens_usages]++ async def _run_through_the_interpreter(self, mocker: MockerFixture, traces_dir: str) -> list[TokensUsageRecord]:+ tracing_config = get_config().pipelex.tracing_config+ mocker.patch.object(tracing_config, "is_enabled", True)+ mocker.patch.object(tracing_config, "backend", TracingBackend.NDJSON)+ mocker.patch.object(tracing_config, "ndjson", NdjsonTracingConfig(traces_dir=traces_dir))++ execution_config = get_config().pipelex.pipeline_execution_config.with_execution_overrides(+ generate_graph=False,+ generate_usage=True,+ )+ runner = PipelexMTHDSProtocol(pipe_run_mode=PipeRunMode.DRY, is_mock_usage=True, execution_config=execution_config)+ response = await runner.execute(pipe_code=_PIPE_CODE, mthds_contents=[_MTHDS])++ assert response.pipe_output.tokens_usages is not None, "the interpreter run assembled no usage at all"+ return self._records(response.pipe_output.tokens_usages)++ async def _run_through_the_kernel(self) -> list[TokensUsageRecord]:+ # The caller's transport, and the caller's lifecycle: build the event log, register it for the+ # run, read the events back, clear the registration. The kernel is handed only the context.+ event_log = InMemoryEventLog()+ trace_context = TraceContext(+ graph_id="kernel-usage-parity",+ data_inclusion=get_config().pipelex.pipeline_execution_config.graph_config.data_inclusion,+ emit_graph_events=False,+ emit_usage_events=True,+ )+ get_report_delegate().set_event_log(+ context_key=trace_context.lookup_key,+ event_log=event_log,+ workflow_id="direct",+ pipeline_run_id=trace_context.graph_id,+ )+ try:+ kernel = PipelexKernel.make(+ run_mode=PipeRunMode.DRY,+ user_id="kernel-usage-parity",+ is_mock_usage=True,+ trace_context=trace_context,+ )+ await kernel.llm_text(+ memory=WorkingMemoryFactory.make_empty(),+ model=LLMSetting(model="kernel-usage-parity-model", temperature=0.5),+ user=_USER_PROMPT,+ result="written",+ )+ return self._records(UsageAggregator.aggregate(event_log.read_events(trace_context.graph_id)))+ finally:+ get_report_delegate().clear_event_log(context_key=trace_context.lookup_key)++ async def test_both_callers_produce_the_same_usage_records(self, tmp_path_factory: pytest.TempPathFactory, mocker: MockerFixture) -> None:+ """One LLM step, two callers, one usage shape."""+ interpreter_records = await self._run_through_the_interpreter(mocker, str(tmp_path_factory.mktemp("traces_parity")))+ kernel_records = await self._run_through_the_kernel()++ assert len(kernel_records) == len(interpreter_records) == 1, (+ f"one LLM step must report exactly one usage record on each side, got interpreter={len(interpreter_records)} kernel={len(kernel_records)}"+ )+ assert self._comparable(kernel_records[0]) == self._comparable(interpreter_records[0])++ # Non-zero under the sentinel model on both sides: a parity of two empty records would prove+ # nothing, and zero tokens are what a plain dry run reports.+ for record in (*interpreter_records, *kernel_records):+ assert record.inference_model_name == MOCK_USAGE_MODEL_NAME+ assert sum(record.nb_tokens_by_category.values()) > 0++ # The scope of what was measured, pinned rather than left implicit: the field a kernel run+ # could never fill is also unfilled on the interpreter's DRY path, so parity here does not+ # cover it (KF-13).+ assert interpreter_records[0].pipe_code is None+ assert kernel_records[0].pipe_code is Nonediff --git a/tests/integration/pipelex/system/test_keyless_boot_forced_dry.py b/tests/integration/pipelex/system/test_keyless_boot_forced_dry.pyindex 50351c028..ec246f102 100644--- a/tests/integration/pipelex/system/test_keyless_boot_forced_dry.py+++ b/tests/integration/pipelex/system/test_keyless_boot_forced_dry.py@@ -102,6 +102,22 @@ class TestKeylessBootForcedDry:finally:Pipelex.teardown_if_needed()+ def test_keyless_boot_does_not_legalise_mock_usage_on_a_live_request(self) -> None:+ """The DRY-only sub-flag is validated against the REQUESTED mode, before the coercion.++ Same rule and same ordering as ``PipeRunParamsFactory.make_run_params``. Without it, the one+ illegal combination would raise on a keyed boot and pass silently on a keyless one — a+ contract violation whose visibility depended on whether the process happened to hold keys.+ """+ try:+ self._boot_keyless()+ assert is_dry_run_forced()++ with pytest.raises(ValueError, match="is_mock_usage"):+ PipelexKernel.make(run_mode=PipeRunMode.LIVE, user_id="test-user", is_mock_usage=True)+ finally:+ Pipelex.teardown_if_needed()+def test_keyed_boot_does_not_force_dry(self) -> None:"""Control arm: a normal boot leaves the flag unset."""try:diff --git a/tests/unit/pipelex/kernel/test_kernel_boot_contract.py b/tests/unit/pipelex/kernel/test_kernel_boot_contract.pyindex b98bd7d49..bd9f512ec 100644--- a/tests/unit/pipelex/kernel/test_kernel_boot_contract.py+++ b/tests/unit/pipelex/kernel/test_kernel_boot_contract.py@@ -15,12 +15,19 @@ interpreter through a function-local import, which is invisible to the static grimport-closure test *at once*. The sweep below therefore runs **after** the kernel call, which iswhat makes this test strictly stronger than the import-time one it complements.-**Every operator's entry point is called, not a representative one.** The blind spot this test exists-for is per-function, so covering one op says nothing about the next: a function-local import inside-`run_search` is caught only by calling `run_search`. That also makes this the only gate the ops-modules other than `llm_ops` have — `PipelexKernel` is an LLM-era façade that does not import them, so-`test_runtime_layer_import_closure.py`'s `pipelex.kernel.pipelex_kernel` entry point does not reach-them even at import time.+**Every kernel entry point is called, not a representative one** — the seven operator arms and the+memory-boundary ops (`shape_inputs` and the four extraction helpers), with the three+`resolve_*_setting` helpers the sole exception, for the deck reason given below. The blind spot this test+exists for is per-function, so covering one op says nothing about the next: a function-local import+inside `run_search` is caught only by calling `run_search`. That also makes this the only gate the+ops modules other than `llm_ops` have — `PipelexKernel` is an LLM-era façade that does not import+them, so `test_runtime_layer_import_closure.py`'s `pipelex.kernel.pipelex_kernel` entry point does not+reach them even at import time.++The memory-boundary arm carries a second proof the operator arms cannot: `shape_inputs` needs a+`ConceptProviderAbstract`, and the only implementation in the tree is the interpreter's concept+library. The script therefore supplies its own, which is what a library-free caller has to do — so+the arm demonstrates the shape of that caller as well as pinning the closure.Run in a **subprocess** for the reason its two siblings state: a suite-level boot already owns theprocess singletons, and both hubs are sticky class attributes that `teardown` deliberately does not@@ -65,20 +72,27 @@ _KERNEL_CALL_SCRIPT = textwrap.dedent(import asyncioimport sys+ from typing_extensions import override+from pipelex.cogt.extract.extract_input import ExtractInputfrom pipelex.cogt.extract.extract_setting import ExtractSettingfrom pipelex.cogt.img_gen.img_gen_setting import ImgGenSettingfrom pipelex.cogt.llm.llm_setting import LLMSettingfrom pipelex.cogt.search.search_setting import SearchSettingfrom pipelex.cogt.templating.template_blueprint import TemplateBlueprint+ from pipelex.core.concepts.concept import Conceptfrom pipelex.core.concepts.concept_factory import ConceptFactory+ from pipelex.core.concepts.concept_provider_abstract import ConceptProviderAbstractfrom pipelex.core.concepts.native.concept_native import NativeConceptCodefrom pipelex.core.memory.working_memory import WorkingMemoryfrom pipelex.core.memory.working_memory_factory import WorkingMemoryFactory+ from pipelex.core.pipes.inputs.input_stuff_specs import InputStuffSpecs+ from pipelex.core.pipes.stuff_spec.stuff_spec import StuffSpecfrom pipelex.core.stuffs.image_content import ImageContentfrom pipelex.core.stuffs.list_content import ListContentfrom pipelex.core.stuffs.number_content import NumberContentfrom pipelex.core.stuffs.search_result_content import SearchResultContent+ from pipelex.core.stuffs.stuff_content import StuffContentfrom pipelex.core.stuffs.stuff_factory import StuffFactoryfrom pipelex.core.stuffs.text_content import TextContentfrom pipelex.kernel.compose_ops import run_compose_template@@ -87,11 +101,19 @@ _KERNEL_CALL_SCRIPT = textwrap.dedent(from pipelex.kernel.img_gen_ops import build_img_gen_job_params, run_img_genfrom pipelex.kernel.img_gen_prompt import assemble_img_gen_promptfrom pipelex.kernel.llm_results import LlmObjectResult, LlmTextResult+ from pipelex.kernel.memory_ops import (+ extract_main_content,+ extract_main_content_as_list,+ extract_named_content,+ extract_named_content_as_list,+ shape_inputs,+ )from pipelex.kernel.pipelex_kernel import PipelexKernelfrom pipelex.kernel.prompt_references import ImageReference, ImageReferenceKindfrom pipelex.kernel.search_ops import run_searchfrom pipelex.runtime_boot import RuntimeBootfrom pipelex.system.pipe_run_mode import PipeRunMode+ from pipelex.system.registries.class_registry_access import get_class_registryfrom pipelex.system.registries.func_registry import func_registryfrom pipelex.system.runtime import IntegrationModefrom pipelex.tools.jinja2.template_category import TemplateCategory@@ -117,15 +139,54 @@ _KERNEL_CALL_SCRIPT = textwrap.dedent(if not isinstance(stored, expected_type):fail(f"the memory returned by {label} carries {type(stored).__name__}, not the {expected_type.__name__} it produced")+ class NativeOnlyConceptProvider(ConceptProviderAbstract):+ # A caller without a loaded library still has to answer concept questions, and this is what+ # that costs on a runtime-only boot: native concepts from the pure factory, compatibility+ # from the declaration tier the kernel doctrine routes to, and structure classes from the+ # class registry a RuntimeBoot fills. `shape_inputs` takes the provider explicitly for+ # exactly this reason — the interpreter hands over its library, a programmatic caller hands+ # over something like this.+ @override+ def get_required_concept(self, concept_ref):+ return ConceptFactory.make_native_concept(native_concept_code=NativeConceptCode(concept_ref.split(".")[-1]))++ @override+ def get_native_concept(self, native_concept):+ return ConceptFactory.make_native_concept(native_concept_code=native_concept)++ @override+ def get_required_concept_from_concept_ref_or_code(self, concept_ref_or_code, *, search_domain_codes=None):+ return self.get_required_concept(concept_ref_or_code)++ @override+ def is_compatible(self, *, tested_concept, wanted_concept, strict=False):+ return Concept.are_compatible_by_declaration(concept_1=tested_concept, concept_2=wanted_concept)++ @override+ def get_structure_class(self, *, concept):+ return get_class_registry().get_required_subclass(name=concept.structure_class_name, base_class=StuffContent)+RuntimeBoot.make(integration_mode=IntegrationMode.PYTEST, needs_inference=False)kernel = PipelexKernel.make(run_mode=PipeRunMode.DRY, user_id="kernel-boot-contract")model = LLMSetting(model="kernel-boot-contract-model", temperature=0.5)text_concept = ConceptFactory.make_native_concept(native_concept_code=NativeConceptCode.TEXT)+ # The input end of the memory boundary, driven the way a library-free caller drives it: raw values+ # plus the specs it declares for them. The shaped memory is then what the first op runs on, so the+ # two ends of the boundary are proven against each other rather than in isolation.+ SHAPED_TOPIC = "what a runtime-only boot can run"+ shaped_memory = shape_inputs(+ inputs={"topic": SHAPED_TOPIC},+ concept_provider=NativeOnlyConceptProvider(),+ input_specs=InputStuffSpecs(root={"topic": StuffSpec(concept=text_concept)}),+ )+ if extract_named_content(memory=shaped_memory, name="topic", content_type=TextContent).text != SHAPED_TOPIC:+ fail("shape_inputs did not land the provided value under its declared name")+object_result = asyncio.run(kernel.llm_object(- memory=WorkingMemoryFactory.make_empty(),+ memory=shaped_memory,output_class=NumberContent,concept=ConceptFactory.make_native_concept(native_concept_code=NativeConceptCode.NUMBER),model=model,@@ -141,6 +202,40 @@ _KERNEL_CALL_SCRIPT = textwrap.dedent(if object_result.memory.get_main_stuff().content is not object_result.content:fail("the memory returned by llm_object does not carry the produced content as its main stuff")+ # The read end of the boundary, on the memory the call returned: the produced content narrowed to+ # the class it was asked for (the result envelope annotates that field as the base StuffContent),+ # and the shaped input still readable beside it.+ if extract_main_content(memory=object_result.memory, content_type=NumberContent) is not object_result.content:+ fail("extract_main_content did not return the content llm_object produced")+ if extract_named_content(memory=object_result.memory, name="topic", content_type=TextContent).text != SHAPED_TOPIC:+ fail("the shaped input did not survive the kernel call it was passed into")++ # The multi-output shape, which the single-content reads above cannot narrow: it is stored as one+ # ListContent, so it needs the list reads to come back typed down to the item.+ list_result = asyncio.run(+ kernel.llm_object(+ memory=WorkingMemoryFactory.make_empty(),+ output_class=NumberContent,+ concept=ConceptFactory.make_native_concept(native_concept_code=NativeConceptCode.NUMBER),+ model=model,+ user="Pick a few numbers.",+ result="answers",+ is_multiple_output=True,+ )+ )++ if not isinstance(list_result.content, ListContent):+ fail(f"llm_object(is_multiple_output=True) produced {type(list_result.content).__name__}, not a ListContent")+ for extracted_list in (+ extract_main_content_as_list(memory=list_result.memory, item_type=NumberContent),+ extract_named_content_as_list(memory=list_result.memory, name="answers", item_type=NumberContent),+ ):+ if not extracted_list.items:+ fail("a list extraction helper returned no items for a multiple-output result")+ for extracted_item in extracted_list.items:+ if not isinstance(extracted_item, NumberContent):+ fail(f"a list extraction helper yielded a {type(extracted_item).__name__}, not the NumberContent it was asked for")+text_result = asyncio.run(kernel.llm_text(memory=WorkingMemoryFactory.make_empty(),diff --git a/tests/unit/pipelex/kernel/test_pipelex_kernel_run_state.py b/tests/unit/pipelex/kernel/test_pipelex_kernel_run_state.pynew file mode 100644index 000000000..e1cc71e76--- /dev/null+++ b/tests/unit/pipelex/kernel/test_pipelex_kernel_run_state.py@@ -0,0 +1,124 @@+"""The run-scoped state `PipelexKernel` holds, and what each step inherits from it.++`test_usage_parity.py` measures the end of this chain — that a kernel run's usage records match an+interpreter run's. These pin the links, because a parity failure there says "no events" without+saying which link dropped: the run-level metadata, the per-step copy, or the run mode.+"""++import pytest++from pipelex.core.concepts.concept_factory import ConceptFactory+from pipelex.core.concepts.native.concept_native import NativeConceptCode+from pipelex.core.memory.working_memory_factory import WorkingMemoryFactory+from pipelex.core.stuffs.exceptions import StuffContentTypeError+from pipelex.core.stuffs.list_content import ListContent+from pipelex.core.stuffs.text_content import TextContent+from pipelex.kernel.memory_ops import (+ extract_main_content,+ extract_main_content_as_list,+ extract_named_content,+ extract_named_content_as_list,+ store_result,+)+from pipelex.kernel.pipelex_kernel import PipelexKernel+from pipelex.system.data_inclusion_config import DataInclusionConfig+from pipelex.system.pipe_run_mode import PipeRunMode+from pipelex.system.trace_context import TraceContext++_GRAPH_ID = "kernel-run-state"+++def _trace_context() -> TraceContext:+ return TraceContext(+ graph_id=_GRAPH_ID,+ data_inclusion=DataInclusionConfig(+ stuff_json_content=False,+ stuff_text_content=False,+ stuff_html_content=False,+ error_stack_traces=False,+ pipe_and_concept_registry=False,+ ),+ emit_graph_events=False,+ emit_usage_events=True,+ )+++class TestPipelexKernelRunState:+ def test_a_supplied_trace_context_becomes_the_runs_identity(self) -> None:+ """Passing a context adopts its graph_id as the run id — the two are one identity.++ Letting them diverge would scatter a single run's usage events across two ids: the+ registered-context emit path stamps the event log's, the runner fallback stamps the+ metadata's, so a read-back keyed on either would silently miss the other's.+ """+ trace_context = _trace_context()++ kernel = PipelexKernel.make(run_mode=PipeRunMode.DRY, user_id="test-user", trace_context=trace_context)++ assert kernel.job_metadata.pipeline_run_id == _GRAPH_ID+ assert kernel.job_metadata.trace_context == trace_context++ def test_without_a_trace_context_the_run_mints_its_own_id_and_traces_nothing(self) -> None:+ """The default stays what it was: a fresh run id, and no context for the leaf to emit against."""+ first = PipelexKernel.make(run_mode=PipeRunMode.DRY, user_id="test-user")+ second = PipelexKernel.make(run_mode=PipeRunMode.DRY, user_id="test-user")++ assert first.job_metadata.trace_context is None+ assert first.job_metadata.pipeline_run_id != second.job_metadata.pipeline_run_id++ def test_each_step_inherits_the_trace_context_and_carries_its_own_run_id(self) -> None:+ """The per-step copy: same trace context (so every step attributes to it), distinct pipe_run_id."""+ kernel = PipelexKernel.make(run_mode=PipeRunMode.DRY, user_id="test-user", trace_context=_trace_context())++ first_step = kernel.make_step_metadata()+ second_step = kernel.make_step_metadata()++ assert first_step.trace_context == kernel.job_metadata.trace_context+ assert second_step.trace_context == kernel.job_metadata.trace_context+ assert first_step.pipeline_run_id == second_step.pipeline_run_id == _GRAPH_ID+ assert first_step.pipe_run_id != second_step.pipe_run_id+ # Computed fresh per step by whoever opens the span, and a kernel call opens none.+ assert first_step.otel_context is None++ def test_mock_usage_rides_the_execution_mode_contract(self) -> None:+ """The DRY sub-flag lands on the carrier every cogt leaf reads off its assignment."""+ kernel = PipelexKernel.make(run_mode=PipeRunMode.DRY, user_id="test-user", is_mock_usage=True)++ assert kernel.cogt_run_params.run_mode.is_dry+ assert kernel.cogt_run_params.is_mock_usage++ def test_the_memory_boundary_reads_back_what_was_written(self) -> None:+ """`store_result` writes the main stuff; both extraction helpers read it typed."""+ concept = ConceptFactory.make_native_concept(native_concept_code=NativeConceptCode.TEXT)+ memory = store_result(+ memory=WorkingMemoryFactory.make_empty(),+ concept=concept,+ content=TextContent(text="written by the kernel"),+ result_name="written",+ )++ assert extract_main_content(memory=memory, content_type=TextContent).text == "written by the kernel"+ assert extract_named_content(memory=memory, name="written", content_type=TextContent).text == "written by the kernel"++ def test_the_memory_boundary_reads_back_a_multiple_output_result_typed(self) -> None:+ """The list reads narrow a `ListContent` down to its items, which the single-content reads cannot.++ That is the whole reason they exist: a multiple-output kernel call stores one `ListContent`, and+ asking for the bare item class raises rather than unwrapping it.+ """+ concept = ConceptFactory.make_native_concept(native_concept_code=NativeConceptCode.TEXT)+ memory = store_result(+ memory=WorkingMemoryFactory.make_empty(),+ concept=concept,+ content=ListContent(items=[TextContent(text="first"), TextContent(text="second")]),+ result_name="written",+ )++ for extracted in (+ extract_main_content_as_list(memory=memory, item_type=TextContent),+ extract_named_content_as_list(memory=memory, name="written", item_type=TextContent),+ ):+ assert [item.text for item in extracted.items] == ["first", "second"]++ with pytest.raises(StuffContentTypeError):+ extract_main_content(memory=memory, content_type=TextContent)diff --git a/wip/drift-contracts/dogfood-log.md b/wip/drift-contracts/dogfood-log.mdindex 3bc3b6f43..5771dedf8 100644--- a/wip/drift-contracts/dogfood-log.md+++ b/wip/drift-contracts/dogfood-log.md@@ -66,3 +66,13 @@ That distinction matters for the recommendation running through this log. A contBoth prescribed mechanical checks were run rather than reasoned about, and both passed. The Enforcement enumeration is clean for the **second consecutive round** after seven rounds of rot — it now names all 15 declared packages, `pipelex.kernel` among them, which is the payoff of that package having been declared in the commit that created it (praised as "the rule applied ahead of a breach" when it landed, and now visible as an enumeration that did not need catching up). Accessor coverage: 34 + 32 public hub symbols, all present. The standing automation candidate — a test asserting the doc names every declared package — remains the right call and remains unbuilt; two clean rounds are not evidence it is unnecessary, only that the last two changes were not the kind that rots it.One process note, recorded because the pilot's evidence is only as good as its completeness: **the preceding ack has no entry in this log.** The 2026-08-01 forced-DRY ack (Louis, `resolve_run_mode_for_boot` added to `runtime_hub.py`) is present in `.drift/acks/` with a substantial rationale describing two real doc edits, but the mandatory dogfood entry was never written. That is the second recorded instance of the log being skipped while the ack was recorded (cf. the 2026-07-12 backfill note), and both times the ack itself was sound — so the failure mode is specifically the *second* step, the one with no tooling behind it. `drift ack` gates on its verify commands and stages its own file; nothing gates the log entry. If the pilot wants this evidence to be trustworthy, that asymmetry is the thing to fix.++## 2026-08-09 · pipelex-kernel-docs — a rename that renamed everything except the filename++**Verdict: real-catch (one row), on a contract that first had to be un-broken.** The contract failed twice over: `pipelex-kernel-docs` had no ack at all, and a stray ack file under the contract's pre-rename id matched no contract. One cause — the class rename swept `drift.toml` and the ack file's own `contract` field, but not the ack **file's name**, and the tool keys acks by filename. So a review that had genuinely happened (Louis, 2026-08-03) was invisible, and the contract read as never-reviewed. `git mv` restored it; nothing was hand-edited.++Worth recording as a manifest-rot shape the reference doesn't list. `drift check`'s documented rot modes are dead trigger globs, zero-match review targets, orphan acks and id/filename mismatch — and this presented as *two* of them at once (orphan + missing ack) while being a single half-applied rename. The tell is that the orphan file's interior already named the new contract: an orphan ack whose `contract` field points at a live contract is not an orphan, it is a misfiled ack, and the fix is a rename rather than a deletion plus a fresh review. Deleting it — the obvious reading of "orphan" — would have silently discarded a real review and forced a re-review of triggers nobody had touched.++The review proper then ran on rename-only trigger diffs, and the page had already been swept in the same commit. But the page carries a standing rule — *every kernel entry point owes the boot-contract subprocess an arm* — so it was checked by enumerating the package rather than by reading the table: 29 public module-level functions across 18 modules. **Three of the four mechanical checks I wrote produced false positives, and that is the entry's most transferable content.** `__all__` was empty, so the first check compared an empty list and reported a vacuous "none missing" — a green that proved nothing. The second flagged 15 functions as lacking a boot-contract arm; the test imports the operator entry points (`run_*`, `build_*`, `assemble_*`) and the façade, and the rest are internal helpers reached transitively. The third flagged seven modules as absent from the table; the table covers them with a `*_results` glob row and a shorthand (`resolve_llm_setting_for_text` / `_for_object`). Every one of those would have been a fabricated finding had it been reported without opening the file it accused. A mechanical check over prose is a *lead generator*, never a verdict — the opposite failure from the one `hub-layering-convention` keeps logging, where the mechanical checks are blind rather than noisy.++The surviving finding is real and small: the table under "What a programmatic caller imports" had a row for every kernel module except `prompt_references`, while `assemble_llm_prompt` and `assemble_img_gen_prompt` both oblige the caller to construct `ImageReference` / `DocumentReference` values. A table that states what a caller imports, omitting a module its own listed entry points force the caller to import, is wrong by its own claim. Pre-existing rather than trigger-induced — it survived the 2026-08-03 review — which is the same lesson `hub-layering-convention` logged on 2026-07-27: a review target's value is bounded by what the reviewer reads, not by what opened the contract.diff --git a/wip/kernel/deferred-follow-ups.md b/wip/kernel/deferred-follow-ups.mdindex df42cc9fa..39fff150e 100644--- a/wip/kernel/deferred-follow-ups.md+++ b/wip/kernel/deferred-follow-ups.md@@ -16,6 +16,8 @@ Items surfaced during the plan's engineering review that are deliberately **not***Why it is deferred rather than fixed here.** The fix is a caller-facing memory-boundary concern, and Phase 3 is where that boundary gets shaped for real (3.1 `shape_inputs`, 3.2 result extraction). Adding a concept-minting factory in Phase 1 would be designing that surface from one test's convenience rather than from the caller experience Phase 3 defines. Revisit at 3.2: if result extraction stays typed on the class, this may need nothing at all; if it keys on concepts, a library-free minting path becomes load-bearing and should be designed then.+**Phase-3 verdict: nothing is owed, and the deferral was right.** Result extraction stayed typed on the class — `extract_main_content` / `extract_named_content` take a `content_type`, never a concept — so no kernel *read* path asks a concept question, and the branch KF-2 was watching for ("if it keys on concepts") did not happen. The input side turned out to have a shape the item did not anticipate: `shape_inputs` takes a `ConceptProviderAbstract` explicitly, so a library-free caller supplies its own, and `test_kernel_boot_contract.py` now carries the smallest one that works (native concepts from `ConceptFactory`, compatibility from `Concept.are_compatible_by_declaration`, structure classes from the class registry a boot fills). That is a *provider*, not a concept-minting factory — a caller wanting a genuinely non-native concept for its own class still has to declare one, which is what an author writing `.mthds` does. Nothing in the kernel needs it, so nothing is built. Revisit only if a kernel path ever answers a compatibility question about a caller's own concept, which the 0.1e routing rule would then govern.+## KF-3 — `LLMPromptBlueprint.make_llm_prompt` is now production-dead, held up entirely by its tests**Surfaced by** the Checkpoint-A cold review, which reported the narrower symptom: the `log.verbose(f"User text with {output_concept_ref=}:\n {user_text}")` diagnostic inside it no longer fires on a live run. That is true, but it is a consequence, not the finding. Verified the real shape: `make_llm_prompt` is defined at `pipelex/pipe_operators/llm/llm_prompt_blueprint.py:63` and called from **tests only** — 38 call sites across `tests/integration/pipelex/pipes/llm_prompt_inputs/` and `tests/unit/pipelex/pipe_operators/pipe_llm/`, and zero in `pipelex/`. The re-point is what killed it: `PipeLLM._live_run_operator_pipe` now calls `to_prompt_content()` and hands that to the kernel, so the blueprint's own assembly method is bypassed.@@ -26,6 +28,8 @@ Items surfaced during the plan's engineering review that are deliberately **not***Scope note, because this item is about a method and not a class.** What is production-dead is `make_llm_prompt`, the assembly method — *not* `LLMPromptBlueprint` itself, which `PipeLLM` holds as a field and which parse-and-validate still needs. An earlier revision of this paragraph named `PipeStructure` as the blueprint's remaining consumer. That was never true on any branch, including `dev` before this work: `pipe_structure.py` has never referenced `LLMPromptBlueprint`. Whoever picks this up should look at `PipeLLM`, and should be deleting one method, not a model.+**Scheduling note after Phase 3.** Its trigger — "the end of Phase 2, when the blueprint's surviving role is settled" — passed unexercised, and Phase 3 touches neither the blueprint nor its tests, so nothing schedules this now. The blueprint's role *is* settled (Phase 2 moved no prompt assembly onto it, and `PipeLLM` still holds it as `llm_prompt_spec`), so the question is answerable: `make_llm_prompt` is genuinely dead in production and its tests belong on `LlmPromptContent`, while the blueprint model itself stays. Do it in the same standalone tidy as KF-4, after this series lands — both are mechanical, both are cheaper once no PR is claiming a verbatim move.+**If the diagnostic itself is wanted back sooner** — independent of the above — it is a one-line addition in `run_llm_text`/`run_llm_object` in `pipelex/kernel/llm_ops.py`. It was dropped deliberately (recorded under the plan's Deviations: moving it meant either duplicating it or carrying a log-only parameter across the layer boundary), and no one has asked for it; the kernel would log without the `output_concept_ref` the old line carried, since the text arm has no concept ref to name.## KF-4 — Two `TestClass`es per module in the moved reference-type tests@@ -52,7 +56,9 @@ So the kernel result is behaving exactly like the type it mirrors, and the proje**When it becomes real.** Phase 3 shapes the caller-facing boundary (3.1 `shape_inputs`, 3.2 result extraction). If kernel results are ever handed across a process, API or cache boundary there, this is the moment to decide deliberately — and the answer should almost certainly be "serialize with `kajson`, like everything else that crosses", not a bespoke annotation on one field. Revisit at 3.2 alongside KF-2, which is the same boundary seen from the input side.-## KF-6 — `StructuringPath` owns three quarters of the tracer's `structuring_path` vocabulary+**Phase-3 verdict: unchanged, because 3.2 read the results in-process.** `extract_main_content` / `extract_named_content` narrow a result's content by type on the caller's own stack; nothing in Phase 3 dumps or transports a kernel result, so the path this describes still has no caller. The docstring note stays as the caller-facing warning it was. The trigger is now genuinely external — the first time a host hands a kernel result across a process, API or cache boundary — rather than a phase of this plan.++## ✅ KF-6 — `StructuringPath` owns three quarters of the tracer's `structuring_path` vocabulary — SETTLED IN PHASE 3**Surfaced by** the PR #1081 arbitration pass, reviewing the re-pointed operators for behavior parity. `StructuringPath`'s docstring calls itself "the tracer's vocabulary, owned here rather than at each caller", but `PipeStructure` writes a bare `"structure"` into the same `execution_data["structuring_path"]` key (`pipelex/pipe_operators/structure/pipe_structure.py:172`), and that value has no enum member. So the key the tracer consumes is fed from an enum on one path and a string literal on another.@@ -60,7 +66,9 @@ So the kernel result is behaving exactly like the type it mirrors, and the proje**Scheduling note — this has no trigger unless one is added.** An earlier draft said "when Phase 2 re-points `PipeStructure` fully, the answer falls out". Phase 2's task list (2.1–2.5) has no `PipeStructure` task, and the plan's own Deviations argue *against* routing it through `run_llm_object`. So nothing currently schedules this. Fold it into whichever Phase-2 task touches `PipeStructure`'s execution data, or settle it at Phase 3 when the kernel docs page has to describe the vocabulary. Extra evidence for leaving the value alone: our graph UI removed its Structuring row on the grounds that `structuring_path` never carried information, so no renderer branches on it in any repo.-**Why it is deferred rather than fixed here.** The right fix depends on a question Phase 2 answers, not this PR: is `"structure"` a *fourth kernel structuring path*, or a pipe-level label that merely shares a dict key? Today it is the latter — `PipeStructure` calls `generate_object_content` directly and builds its own execution data, so the kernel never produces `"structure"` and adding the member now would put a value into a "what the kernel op did" type that no kernel op can return. When Phase 2 re-points `PipeStructure` fully, the answer falls out: either it starts flowing through a kernel entry point that legitimately returns a fourth path (add the member, drop the literal), or it stays a caller-side label (narrow the enum's docstring to say it names the *LLM* paths, not the whole key). Cheap either way; wrong to guess now.+**Why it was deferred rather than fixed then.** The right fix depended on a question Phase 2 answers: is `"structure"` a *fourth kernel structuring path*, or a pipe-level label that merely shares a dict key? Either it starts flowing through a kernel entry point that legitimately returns a fourth path (add the member, drop the literal), or it stays a caller-side label (narrow the enum's docstring to say it names the *LLM* paths, not the whole key). Cheap either way; wrong to guess before the answer was in.++**What landed (Phase 3).** The second arm, because Phase 2 settled the premise: its Deviations record `PipeStructure` riding the kernel's *pieces* rather than `run_llm_object`, so it calls `generate_object_content` and builds its own execution data — no kernel op can return `"structure"`, and adding the member would put a value into a "what the kernel op did" type that nothing here produces. `StructuringPath`'s docstring now says it names the LLM paths and states why the sibling literal has no member. The literal in `pipe_structure.py` is untouched: the key is `dict[str, Any]` end to end, `StructuringPath` is a `StrEnum`, and nothing downstream can tell the two forms apart — our graph UI removed its Structuring row precisely because the value never carried information.## KF-7 — `PipeCompose`'s construct mode is not kernel-covered, and covering it means relocating an MTHDS blueprint@@ -93,6 +101,8 @@ Phase 1's `LLMPromptBlueprint` precedent looks like it points the other way, and**When to revisit.** Whenever `pipe_run_params` on the executor seam is settled on its own merits — the question is whether a serialized PipeFunc request needs the run params at all, and it wants an answer coordinated across the three repos, not a drive-by narrowing here. Phase 3's `JobMetadata` task (3.3) is the natural moment: it is already the task that asks what run-scoped state a kernel call carries, and this is the same question asked at the one seam that serializes it.+**Phase-3 verdict: still deferred, and 3.3 sharpened the argument rather than answering it.** 3.3 settled what run-scoped state a kernel call carries — a per-step `JobMetadata` (identity plus an optional `TraceContext`) and a `CogtRunParams`, and *not* a `PipeRunParams`. That is a point in favour of narrowing the executor protocol, since the pipe-tier params on it are the one thing the kernel could never supply — but it does not answer the question the narrowing rests on ("does anything downstream read them?"), which is only answerable from the two out-of-tree implementers. Unchanged: this is a cross-repo wire-format decision, not a kernel one. The trigger is a coordinated pass over pipelex + our Daytona and Temporal plugins.+## KF-9 — Two constructors build `ImgGenJobParams` from the same config defaults with different derivation rules**Surfaced by** cubic on PR #1082, against `build_img_gen_job_params` in `pipelex/kernel/img_gen_ops.py`: a transparent background with no step-level `output_format` builds params that fail `ImgGenJobParams.validate_background_vs_output_format`.@@ -115,6 +125,8 @@ Phase 1's `LLMPromptBlueprint` precedent looks like it points the other way, and**Why it is deferred rather than fixed here, and why the obvious fix is the wrong one.** Revalidating the copy (the bot's suggestion) would raise a pydantic `ValidationError` deep inside the search run path, with no `.mthds` locator to tell the author which step is wrong — trading an unvalidated value for a bad diagnostic. The constraint belongs on `PipeSearchBlueprint.max_results` as `Field(default=None, ge=1)`, where it is caught at validation time with a proper error. That changes the MTHDS JSON Schema and drags the committed downstream copies (`mthds/`, `vscode-pipelex/`, `mthds-ui/`) with it, which is unambiguously outside a zero-behavior-change refactor. Do it as a standalone language-surface change with the schema regeneration and the cross-repo sweep in the same PR.+**Phase-3 verdict (shared with KF-12): the conclusion holds, and 3.1 did not turn out to be its home.** Both items were scheduled onto task 3.1 as "the task that decides what a kernel entry point does with a malformed argument". 3.1 answered that for what it actually governs — the *memory* boundary — and the answer is: reject at the boundary with a typed error that names the variable and renders the expected shape, which is what `InputShaper` already does and what `shape_inputs` now hands both callers. Neither `max_results` nor `nb_images` passes through that boundary: they are operator arguments, not memory inputs, so 3.1 has no reach over them. The conclusion both items had already reached is unchanged, and nothing in this plan blocks it any longer — the constraint belongs upstream on the blueprint, caught at validation time with a locator, in one standalone language-surface PR that carries the schema regeneration and the cross-repo sweep.+## KF-11 — `PipeImgGen`'s `seed` is documented language surface that no image-generation provider reads**Surfaced by** verifying cubic's P2 against `build_img_gen_job_params`, which reported that `seed or img_gen_param_defaults.seed` discarded an explicit `0`. That part was true and is **fixed in PR #1082** — the line now reads `seed if seed is not None else …`, matching the `is_raw` line below it. The larger fact came out of checking the claimed impact.@@ -134,6 +146,39 @@ Phase 1's `LLMPromptBlueprint` precedent looks like it points the other way, and**Why it is deferred rather than fixed here, and this one is genuine doubt rather than scope.** There is no zero-behavior-change fix available. Unlike the `run_search` signature narrowing in this PR — where the ignorable fields could simply be removed from the signature — `nb_images` is an `int` and plain Python cannot express `int >= 1` at the boundary, so any fix *adds an error path* to a PR whose contract is that it adds none. And the fix's shape is not obvious: `nb_images=0` could defensibly mean "reject this request" or "produce an empty `ListContent`", and picking between them from a bot comment rather than from the caller experience is exactly the mistake KF-2 records. Phase 3's task 3.1 (`shape_inputs`) is the task that decides what a kernel entry point does with a malformed argument; settle it there, for all the ops at once, rather than one `int` at a time.**Worth noting on the way.** The interpreter path can reach it too — nothing between a `.mthds` `nb_output` and `nb_images` constrains the value — so if the answer turns out to be "reject", the constraint most likely belongs upstream on the blueprint, the same conclusion KF-10 reaches about `max_results`. The two should be settled together.++**Phase-3 verdict: see KF-10.** 3.1 settled the malformed-argument question for the memory boundary only, which `nb_images` does not cross. Both items stay together, and their shared home is the standalone blueprint-constraint PR described there.++## KF-13 — a DRY run's usage records carry no `pipe_code`, so a `--mock-usage` cost report has no pipe attribution++**Surfaced by** writing task 3.3's parity test, which expected `pipe_code` to be the one field an interpreter run fills and a kernel run cannot. It is not filled on either side, and the reason is on the interpreter's side.++**The gap.** `pipe_code` reaches a usage record through `JobMetadata`, and the only place it is stamped is `PipeAbstract.live_run_pipe`, which mints the child metadata with `pipe_code=self.code`. A `run_mode=DRY` run never reaches it — `_run_pipe_traced` matches on the run mode and routes to `dry_run_pipe`, which passes the metadata down untouched. So every usage record a dry run produces has `pipe_code=None`, and a `--dry-run --mock-usage` cost report attributes nothing to a pipe.++**Why it is not urgent.** It bites only the mock-usage cost report, which exists to validate cross-worker cost-report *rendering* cheaply — the numbers are synthetic either way. A LIVE run, where attribution matters and money is real, stamps it correctly. Nothing silently misattributes; the field is simply absent.++**Why it is deferred rather than fixed here.** It is pre-existing, entirely interpreter-side, and untouched by the kernel extraction — verified on the base branch, where `dry_run_pipe` has the same shape. Fixing it means moving (or duplicating) the child-metadata mint above the LIVE/DRY fork in `_run_pipe_traced`, which changes what every dry run's metadata carries: a behavior change inside a zero-behavior-change PR, on a path the graph tracer also reads. It belongs with whoever next works on the dry-run observability surface. The parity test pins the current reality on both sides so the fix will show up there as a diff rather than as a silent divergence.++## KF-14 — `UsageReportEvent.node_id` has no readers, which is why a kernel run's flat node state is not a bug yet++**Surfaced by** cubic on PR #1083 (P2), which argued that `PipelexKernel.make_step_metadata` inherits the run-level `TraceContext` unchanged, so every step of a multi-call kernel run carries `node_sequence=0` and `parent_node_id=None` — and concluded that a multi-step run's usage records would collide onto one node. The cold `/code-review` raised the same point independently.++**Rejected as a code change, on measurement.** Two kernel LLM calls under one `TraceContext` produce two events and two records with the correct token total, on both emit paths (the registered-context fast path and the runner fallback). Nothing collides, because nothing on that path keys on the node: the event log's dedup key is `(workflow_id, writer_id, event type, sequence)` with `sequence` coming from a lock-guarded per-log counter, `UsageAggregator.aggregate` is a flat filter-and-map with no grouping, and `TokensUsageRecord` has no `node_id` field at all — so it never reaches `/execute`'s `tokens_usages` or `tokens_usages.json`.++**The fact worth recording, because it is what makes the whole class of finding moot.** `UsageReportEvent.node_id` has no *production* reader. The only other consumer of `UsageReportEvent` in the tree is `GraphSpecAssembler`, which matches it and does nothing (`pass # Handled by UsageAggregator`); across the sibling repos, the sole non-test hits declare the type or list it in an ignore tuple. Two tests do read it — `tests/unit/pipelex/reporting/test_reporting_event_emission.py` asserts both the `"unknown"` fallback and a supplied `graph:node_42` — which strengthens rather than weakens the point: the first of those is the existing pin on exactly the behavior this item defends. A kernel run also emits no graph events — the graph tracer is interpreter-side and a leaf call does not pass through it — so the field's value is unobservable from a kernel call by any route available today.++**Why advancing the sequence would be worse than leaving it.** A kernel run has no graph. `"unknown"` states that honestly; minting `graph_id:node_0`, `graph_id:node_1` would manufacture references to nodes that exist in no `GraphSpec`, and put them into a field nothing reads. Two of cubic's supporting mechanisms were also wrong on inspection: `copy_with_update` applies its update values by reference *after* the deep copy rather than deep-copying them, and `make_node_id()` is never called on the usage path at all.++**What would reopen it.** Node identity becomes load-bearing the moment something reads it — graph-shaped tracing extended to kernel runs, or a consumer that groups usage events by node. At that point the answer is not a per-step sequence invented by the kernel but a decision about what a node *means* for a caller that has no graph, and it should be settled alongside whoever introduces the reader.++## KF-15 — `Stuff.as_list_of_fixed_content_type` documents an exception it does not raise++**Surfaced by** wiring the kernel's list-extraction helpers onto it, which meant reading what it actually raises in order to write their own `Raises:` sections honestly.++**The gap.** Its docstring documents `Raises: TypeError` (`pipelex/core/stuffs/stuff.py`). It raises `StuffContentTypeError`, via `content_as` and `verify_content_type`, like every other typed accessor on `Stuff`. A caller who writes `except TypeError` around it catches nothing — `StuffContentTypeError` descends from `StuffError`, not from `TypeError`.++**Why it is deferred rather than fixed here.** Pre-existing, one line, and in `pipelex/core/stuffs/` rather than on the kernel surface this PR is scoped to. Worth folding into whatever next touches those accessors; the same pass should check the sibling docstrings in that module for the same error, since this one was found by reading a single function rather than by sweeping.+## KF-16 — The model-derived prompting style is obsolete and should become an explicit choice**Numbered 16, not 7, deliberately.** This item lands on the *first* PR of a three-PR stack, but the stacked Phase 2 and Phase 3 PRs already add KF-7 through KF-15. Taking the next free number instead of the next sequential one keeps every one of those entries — and the cross-references to them in the plan — exactly as written, so folding this in churns neither of the PRs above it. The gap below is theirs to fill.diff --git a/wip/kernel/kernel-extraction-plan.md b/wip/kernel/kernel-extraction-plan.mdindex e46fa2c8c..8bbdf4eac 100644--- a/wip/kernel/kernel-extraction-plan.md+++ b/wip/kernel/kernel-extraction-plan.md@@ -2,15 +2,18 @@## Status block (update at every checkpoint)-- **Current phase:** **Phase 2 COMPLETE — CHECKPOINT B reached. Two PRs are open against `dev` and NEITHER may be merged** (the series is stacked: 1/3 = PR #1081 on `refactor/Kernel`, 2/3 = this phase on `refactor/Kernel-phase2` based on it, Phase 3 to follow as 3/3). **Commits are named by subject here, not by SHA.** This branch is rebased every time `refactor/Kernel` moves, which rewrites every hash on it — so a SHA written into this block goes stale the next time that happens, and this block has already been caught carrying stale ones. Phase 1's record, on `refactor/Kernel`: the Phase 0 scaffolding, tasks 1.1–1.6, task 1.7, then three review-driven commits (the forced-DRY fix below among them). Phase 2, in order: the extract / img-gen / search / compose operator extraction (tasks 2.1–2.4); the function-call semantics plus the per-operator boot contract (task 2.5, with the extension the cold review forced); the bot-triage fixes; then the parity-gaps fold — `pipelex/kernel/img_gen_prompt.py` for that track's gap 2.3, and the boot-contract arm it turned out to owe — and the doc corrections from the bot round after it. Phase 3 is untouched.-- **Gates at Phase 2 head:** `make agent-check` **green**, full `make agent-test` **green**, `make drift-check` **green** — last full-suite run at the boot-contract-arm commit, everything after it being documentation only (not reopened — Phase 2 touched no hub module and no guard file, because the executor seam stayed interpreter-side; see the 2.5 deviation). CI green on every check on PR #1082.+- **Current phase:** **Phase 3 COMPLETE — CHECKPOINT C reached, and the plan is done. Three PRs are open against `dev` and NONE may be merged** (the series is stacked: 1/3 = PR #1081 on `refactor/Kernel`, 2/3 = PR #1082 on `refactor/Kernel-phase2`, 3/3 = PR #1083 on `refactor/Kernel-phase3` based on it). **Commits are named by subject here, not by SHA.** Each branch in this series is rebased every time the one below it moves, which rewrites every hash on it — so a SHA written into this block goes stale the next time that happens, and this block has already been caught carrying stale ones. Phase 1's record, on `refactor/Kernel`: the Phase 0 scaffolding, tasks 1.1–1.6, task 1.7, then three review-driven commits (the forced-DRY fix below among them). Phase 2, in order: the extract / img-gen / search / compose operator extraction (tasks 2.1–2.4); the function-call semantics plus the per-operator boot contract (task 2.5, with the extension the cold review forced); the bot-triage fixes; the doc levelling; then the parity-gaps fold — `pipelex/kernel/img_gen_prompt.py` for that track's gap 2.3, and the boot-contract arm it turned out to owe — and the doc corrections from the bot round after it.+- **Gates at Phase 3 head:** `make agent-check` **green**, full `make agent-test` **green**, `make drift-check` **green**.+- **Gates at Phase 2 head, kept for the record:** `make agent-check` **green**, full `make agent-test` **green**, `make drift-check` **green** (not reopened — Phase 2 touched no hub module and no guard file, because the executor seam stayed interpreter-side; see the 2.5 deviation). CI green on every check on PR #1082.- **Gates at Phase 1 head (`c0815cf7f`), kept for the record:** all three green, `hub-layering-convention` acked twice — the forced-DRY fix reopened it. CI green on every check. Two gate failures surfaced at Checkpoint A and both are fixed — see "What Phase 1 taught Phase 2" below. Note for the record: this doc once claimed `make agent-check` was green at `3a11d113a`. It was not; the keyword-only gate was red there.- **One real bug was found by review and fixed — the PR's only behavior change.** `PipelexKernel.make` minted a LIVE `CogtRunParams` on a keyless boot (`needs_inference=False`), bypassing the forced-DRY coercion `PipeRunParamsFactory` applies. Reachable, not hypothetical: `run_mode` alone gates mocking, so the call reached a real provider, and `needs_inference=False` means backends load *leniently* rather than credential-free. The hazard was the **default** — `PipelexKernel.make(user_id=…)` names no mode. Root cause is this branch's own doing: the factory's docstring called itself "the single writer" of `run_mode`, which stopped being true when the kernel became a second run-params factory. The rule now lives once in `runtime_hub.resolve_run_mode_for_boot`, called by both factories, pinned by a third arm on the D4 contract module. **Phase 2 lesson: any new kernel-side factory of run-scoped state owes the same audit — ask which process-wide invariants the pipe tier applies that the kernel tier now skips.**- **Review record — Phase 1 (PR #1081):** a cold `/code-review` (no inherited context) found no correctness bug, no layering violation and no stale reference, and confirmed the three past bug fixes in this code (templating-style mutation, the multi-image/document dict-key collision, LLM-error cause unwrapping) preserved verbatim. Bots: Greptile 5/5 on head; cubic found the forced-DRY bug above; Codex raised one item (polymorphic serialization of `LlmObjectResult.content`) four times, independently verified and rejected twice — see KF-5.- **Review record — Phase 2 (PR #1082):** the cold `/code-review` found the boot-contract gap (see "What the cold review changed" below), which was the one substantive finding. Bots across two rounds: Greptile pass 5/5 both times with no actionable defects; cubic found four items and came back "no issues" after the fixes; Codex found three. **Three were fixed**, in *honour an explicit seed of 0 and stop run_search taking fields it ignores* — named by subject rather than hash, because this branch is rebased whenever its parent moves and the hash this line used to carry now resolves from no branch at all: the img-gen seed read with `or`, swallowing an explicit `0` that `ImgGenJobParams` accepts under its own `ge=0`; `run_search` taking a `TemplateBlueprint` and using two of its four fields; and an architecture-overview sentence *this branch introduced* claiming kernel ops do "no hub lookups" (they read `runtime_hub` freely — what they never touch is the interpreter layer). **Four were deferred** as KF-9..KF-12, each verified pre-existing against `git show refactor/Kernel:<path>` first. **One was rejected**: a P1 asking for a single-consumer test fixture to move to a `conftest.py` that does not exist, against a rule whose own text says "for easy sharing". Two of the deferred four are worth carrying into Phase 3 as one item, not two — KF-10 (`max_results`) and KF-12 (`nb_images`) are the same defect, a value flowing from blueprint to provider with no bound anywhere on the path, and in both cases the constraint belongs upstream on the blueprint rather than where the bot pointed.-- **Next action:** Phase 3, starting at task 3.1. Two things Phase 2 leaves on its doorstep: `pipelex/kernel/memory_ops.py` already exists and is where 3.1 (`shape_inputs`) and 3.2 (result extraction) belong; and 3.3's run-scoped-state question is the natural moment to settle KF-8, since the PipeFunc executor seam is the one place run params actually serialize. "What Phase 1 taught Phase 2" below still applies verbatim to Phase 3 — the four move-consequence classes are properties of moving code, not of Phase 2.+- **Review record — Phase 3 (PR #1083):** CI green on every check. Greptile 5/5 with no comments, vouching specifically for trace identity and per-step metadata propagation; cubic found two; Codex found one, the same as cubic's second. **One was fixed**: the memory boundary could not read back a multiple-output result — `llm_object(is_multiple_output=True)` stores a `ListContent`, and the single-content reads cannot narrow it, so the boundary sent the caller back to `WorkingMemory`'s positional accessors via an error message it raised itself. Two bots and the cold `/code-review` converged on it independently, and the producing side ships in this very diff, which is what made it a hole rather than a want. Fixing it also caught a doc bug this branch introduced: all four `extract_*` docstrings named `WorkingMemoryTypeError` where the accessors raise `StuffContentTypeError`. **One was rejected on measurement** and recorded as KF-14 — see there for the two calls / two records / correct total probe, and for the fact that makes the class of finding moot (`UsageReportEvent.node_id` has no readers anywhere, and a kernel run emits no graph events).+- **Next action: none in this plan — it is finished.** What remains is landing the stack (merge 1/3, then 2/3, then 3/3, in that order and only when each is approved) and the deferred items in [`deferred-follow-ups.md`](deferred-follow-ups.md) — none of which blocks landing this plan. (Named by file rather than by numeric range: the range went stale twice as later rounds appended entries, which is the same self-staling shape as naming commits by hash.) Some are *scheduled after* the series lands, which is a different thing from blocking it: KF-3 and KF-4 are worth doing together as a standalone tidy once it does, both mechanical and both cheaper once no PR is claiming a verbatim move; KF-10 and KF-12 share one home, a blueprint-constraint PR carrying the schema regeneration and the cross-repo sweep.+- **What Phase 3 turned out to be, in one line:** 3.1 and 3.2 are small on purpose — the memory boundary was already single-sourced machinery, so what they add is a named, keyword-only, boot-contract-tested surface for it rather than new behavior; 3.3 is where the real work was, because "usage reporting reaches parity" turned out to be a question about *who owns a lifecycle the kernel structurally cannot own*.- **What Phase 2 turned out to be, in one line:** four operators were genuine extractions (2.1–2.4 moved deck resolution, job-params assembly, generation and write-back into `pipelex/kernel/`); the fifth (2.5) was not a move and did not become one — its seam is typed on interpreter models, so what moved was the *call*, not the pluggable indirection.-- **Open decisions:** none blocking. The kernel construction shape (0.2), structure-prompt provenance and the deferred runtime-boot orchestrator question are settled under Decisions, alongside every deviation from the plan's letter taken in Phases 1 and 2. Deferred items are KF-1..KF-12 in [`deferred-follow-ups.md`](deferred-follow-ups.md).+- **Open decisions:** none blocking. The kernel construction shape (0.2), structure-prompt provenance and the deferred runtime-boot orchestrator question are settled under Decisions, alongside every deviation from the plan's letter taken in Phases 1, 2 and 3. Deferred items are in [`deferred-follow-ups.md`](deferred-follow-ups.md), which is the count of record.### What Phase 1 taught Phase 2@@ -43,7 +46,7 @@ Read this before starting 2.1. **Moving code has consequences no compiler report| `llm_prompt_content.py` | `LlmPromptContent` (+ `make_from_text`) and `assemble_llm_prompt` — the whole former `LLMPromptBlueprint.make_llm_prompt` body, images/documents/registry/placeholders included. || `llm_results.py` | `StructuringPath` StrEnum, `LlmTextResult`, `LlmObjectResult`. || `llm_ops.py` | `resolve_llm_setting_for_text`, `resolve_llm_setting_for_object`, `derive_templating_style`, `derive_structure_prompt`, `generate_object_content`, `run_llm_text`, `run_llm_object`. |-| `memory_ops.py` | `store_result` — the write-back every operator ends with. **Phase 2 moved it out of `llm_ops.py`** (see Deviations); this is also where Phase 3's 3.1 and 3.2 belong. |+| `memory_ops.py` | The whole memory boundary: `shape_inputs` (in), `store_result` (write-back, which **Phase 2 moved out of `llm_ops.py`**), `extract_main_content` / `extract_named_content` (typed read) and their `_as_list` pair (the multiple-output shape). || `extract_ops.py` / `extract_results.py` | `resolve_extract_setting`, `build_extract_job_params`, `run_extract`; `ExtractResult`. || `img_gen_ops.py` / `img_gen_results.py` | `resolve_img_gen_setting`, `resolve_default_size`, `build_img_gen_job_params`, `run_img_gen`; `ImgGenResult`. || `search_ops.py` / `search_results.py` | `resolve_search_setting`, `run_search`; `SearchResult`. |@@ -207,14 +210,17 @@ Two things surfaced while writing it, both recorded where they belong rather tha## Phase 3 — memory boundary and run-scoped state parity-**Standing rule, learned the hard way in Phase 2:** every new kernel entry point gets called in `test_kernel_boot_contract.py`'s subprocess. It is the only gate that sees a function-local interpreter import, it is per-function, and neither static guard will tell you when it has fallen behind — Phase 2 added five ops and both gates stayed green while covering none of them. Adding the call is a few lines; noticing it was never added took a cold review.+**Standing rule, learned the hard way in Phase 2:** every new kernel entry point gets called in `test_kernel_boot_contract.py`'s subprocess — the standing exception being the `resolve_*_setting` helpers, which read the model deck, a question the boot contract deliberately does not ask. It is the only gate that sees a function-local interpreter import, it is per-function, and neither static guard will tell you when it has fallen behind — Phase 2 added five ops and both gates stayed green while covering none of them. Adding the call is a few lines; noticing it was never added took a cold review.-- [ ] 3.1 Boundary shaping: kernel `shape_inputs` over the existing `InputShaper` (per-signature specialization is out of scope).-- [ ] 3.2 Result extraction helpers (main-stuff and named-slot, typed).-- [ ] 3.3 Per-step `JobMetadata` semantics (per-step `pipe_run_id` minted from the instance's run-level metadata via `copy_with_update`, per the 0.2 constraint) as a kernel duty; optional trace-context wiring so cost/usage reporting reaches parity when a caller provides one. This task must also answer the lifecycle-ownership question it implies: the interpreter's run machinery is what opens and closes usage reporting today, so state who owns that lifecycle for a kernel-driven run. **Parity bar, measured not asserted:** a test runs an equivalent step through the re-pointed interpreter and through direct kernel calls (mock usage) and asserts the two produced `TokensUsageRecord` lists ([`pipelex/reporting/usage_records.py`](../../pipelex/reporting/usage_records.py)) match in shape and count — the list `/execute` returns on `pipe_output.tokens_usages` and durable runs persist as `tokens_usages.json`.-- [ ] 3.4 Docs: a `docs/` page for the kernel — public-API doctrine, the layering contract, what a programmatic caller may import and how it boots.+- [x] 3.1 Boundary shaping: kernel `shape_inputs` over the existing `InputShaper` (per-signature specialization is out of scope). Landed in `pipelex/kernel/memory_ops.py`, beside `store_result` — the two ends of the same boundary. `pipeline/execution_seams.py` calls it instead of `WorkingMemoryFactory.make_from_pipeline_inputs`, so the interpreter is one caller. **Narrower than the factory it re-points, deliberately:** `input_specs` is required, so the kernel has no bottom-up no-signature arm — see Deviations.+- [x] 3.2 Result extraction helpers (main-stuff and named-slot, typed). `extract_main_content` / `extract_named_content` in `memory_ops.py`. They earn their place on one fact: every result envelope annotates its content field with the base `StuffContent`, so a caller holding an `LlmObjectResult` has the object but not its type — the helper takes the class the op was asked for and hands it back narrowed. Review added `extract_main_content_as_list` / `extract_named_content_as_list`: a multiple-output call stores one `ListContent`, the single-content read cannot narrow it (the bare item class raises, `ListContent[item_type]` is rejected by design), and the bare-`ListContent` workaround loses the item type — so without them the kernel produced one shape its own read half could not read.+- [x] 3.3 Per-step `JobMetadata` semantics (per-step `pipe_run_id` minted from the instance's run-level metadata via `copy_with_update`, per the 0.2 constraint) as a kernel duty; optional trace-context wiring so cost/usage reporting reaches parity when a caller provides one. This task must also answer the lifecycle-ownership question it implies: the interpreter's run machinery is what opens and closes usage reporting today, so state who owns that lifecycle for a kernel-driven run. **Parity bar, measured not asserted:** a test runs an equivalent step through the re-pointed interpreter and through direct kernel calls (mock usage) and asserts the two produced `TokensUsageRecord` lists ([`pipelex/reporting/usage_records.py`](../../pipelex/reporting/usage_records.py)) match in shape and count — the list `/execute` returns on `pipe_output.tokens_usages` and durable runs persist as `tokens_usages.json`.-**CHECKPOINT C = done** — interpreter operator semantics fully single-sourced in the kernel; smoke run live on it; full gates; this doc updated and the `docs/` page landed.+ **Answered: the caller owns the lifecycle; the kernel owns the stamp.** `PipelexKernel.make` grew `trace_context` and `is_mock_usage`; the context rides the run-level `JobMetadata` and every `make_step_metadata()` copy inherits it, which is the whole of what the cogt leaf needs to emit a usage event. Opening the event log, registering it on the report delegate, reading the events back and clearing the registration stay outside — a kernel call has no run boundary to hang a `finally` on, because it is one step and a caller may make one or a thousand. `pipelex/tracing/` gives a caller both halves it needs (`make_event_log`, `UsageAggregator`) and is runtime-layer, so this costs the boot contract nothing. The parity bar is met by `tests/integration/pipelex/kernel/test_usage_parity.py`, and it is **non-vacuous by mutation**: dropping the trace-context stamp in `PipelexKernel.make` turns 1 record into 0 and the test red.++- [x] 3.4 Docs: a `docs/` page for the kernel — public-API doctrine, the layering contract, what a programmatic caller may import and how it boots. `docs/under-the-hood/pipelex-kernel.md`, in both mkdocs navs and the section index, written *against* `architecture-overview.md` (which now links to it) and `tokens-usage-wire-records.md` rather than beside them, as trap #3 asked.++**CHECKPOINT C — PASSED.** Interpreter operator semantics single-sourced in the kernel and its memory boundary named at both ends; `make agent-check`, full `make agent-test` and `make drift-check` green; the `docs/` page landed; this doc updated. Scope stated honestly: "fully single-sourced" carries the two Phase-2 exceptions unchanged (`PipeCompose`'s construct mode, `PipeFunc`'s executor seam — KF-7, KF-8), and the boot contract now covers twelve entry points rather than seven (ten at the first push, plus the two list-extraction helpers review added).## Decisions@@ -222,6 +228,15 @@ Two things surfaced while writing it, both recorded where they belong rather tha- **The deferred runtime-boot orchestrator question — SETTLED: recorded, not guarded.** `runtime_boot.py` deferred the external-interpreter-orchestrator half-application hole to "the first caller of a runtime-only boot", which is now the kernel's boot-contract test. That caller does not trigger it, and not by luck: the gate is only reached when `plugins.boot_orchestrator` is non-`None`, and the test names none — as no kernel caller will, since the kernel's doctrine forbids it from reading `HubSlot` at all and Phase 2's `PipeFunc` op takes its executor as an explicit argument for that reason. The remedy stays unbuilt: both candidates are real machinery (a `HubSlot.is_interpreter_slot` property plus a `honours_interpreter_slots` class attribute, or a widened `build_registrar` signature) bought for a path with still no production caller. Reassess when a runtime-only boot is offered to real callers — that is when naming an orchestrator becomes reachable. Written up in [`wip/boot-split/runtime-boot-external-interpreter-orchestrator.md`](../boot-split/runtime-boot-external-interpreter-orchestrator.md), and `runtime_boot.py`'s comment now points at the settlement rather than at an open question.- **Kernel construction shape** (task 0.2) — **SETTLED as implemented.** `PipelexKernel` holds exactly `job_metadata` (run-level, fresh `uuid4` `pipeline_run_id`) and `cogt_run_params`. `make_step_metadata()` mints the per-step copy with a fresh `pipe_run_id` and an explicit `otel_context=None`. Nothing config- or deck-derived is cached.+### Deviations from the plan's letter, taken during Phase 3++- **`shape_inputs` requires `input_specs`; the bottom-up no-signature arm is not kernel business.** 3.1 said "over the existing `InputShaper`", and that is literally what landed — the kernel does not wrap `WorkingMemoryFactory.make_from_pipeline_inputs`, whose signature-driven arm *is* `InputShaper.shape` and whose other arm infers each value's concept from its own shape. Inference is exactly the ambient guess a kernel entry point must not make for a caller, so `input_specs` is required rather than optional. A caller with no declared specs builds its stuffs and hands over a `WorkingMemory` directly.+- **`WorkingMemoryFactory.make_from_pipeline_inputs` was re-pointed at, not moved.** The instinct — move the body into the kernel and delete the factory method, as 2.2 did with `resolve_default_size` — is wrong here, and checking is what showed it: the method is on the sanctioned transport surface, pinned by `docs/specs/pipelex-transport-boundary.md` — which lives at the **workspace root**, not in this repo, alongside the `conformance/` suite that enforces it — and is called from `pipelex-transport/pipelex_transport/bridge.py`, `pipelex-mistralai-workflows` and both demo repos. Deleting it is a cross-repo breaking change, which is not what a zero-behavior-change refactor gets to do. So the interpreter's one call site moved to the kernel op and the factory kept its surface.+- **`PipelexKernel.make` grew two parameters, and the ordering of one of them is load-bearing.** `trace_context` was the task's ask; `is_mock_usage` came with it, because the DRY-only sub-flag is what makes a measurable parity comparison possible at all (a plain dry run reports zero tokens on both sides, which would compare equal while proving nothing). It is validated against the **requested** mode, before `resolve_run_mode_for_boot`, mirroring `PipeRunParamsFactory` — otherwise the one illegal combination would raise on a keyed boot and pass silently on a keyless one, i.e. a contract violation whose visibility depended on whether the process held API keys. Pinned in `test_keyless_boot_forced_dry.py`.+- **A supplied `TraceContext` sets the run id rather than sitting beside one.** `PipelexKernel.make(trace_context=…)` adopts its `graph_id` as `pipeline_run_id`. The alternative (a separate optional `pipeline_run_id` parameter) lets the two diverge, and divergence is not cosmetic: the registered-context emit path stamps the event log's id while the runner fallback stamps the metadata's, so a run whose two ids differ scatters its usage events across both and a read-back keyed on either silently misses half. One identity, one parameter.+- **The parity test measures `pipe_code` as *absent on both sides*, not as the one honest difference.** The plan's expectation was that the interpreter fills it and a kernel call cannot. It does not: `pipe_code` is stamped in `live_run_pipe`, which a DRY run never reaches, so the measurement cannot observe the one field that would legitimately differ. Asserted as-is rather than excluded quietly, because that is the honest scope of what the comparison proves — and recorded as KF-13, since a `--dry-run --mock-usage` cost report having no pipe attribution is a real (pre-existing, interpreter-side) gap.+- **KF-6 was settled, KF-2 and KF-5 were closed out as needing nothing, KF-10/KF-12 were re-homed.** Four deferred items named a Phase-3 task as their trigger. Three resolved without code (see `deferred-follow-ups.md`); KF-6 resolved with a docstring, narrowing `StructuringPath` to say it names the *LLM* paths and why `PipeStructure`'s sibling literal has no member. KF-10 and KF-12 had been scheduled onto 3.1 as "the task that decides what a kernel entry point does with a malformed argument" — 3.1 answered that for the memory boundary (reject with a typed, locator-bearing error, which is what `InputShaper` already does), but `max_results` and `nb_images` are operator arguments that never cross it, so their conclusion is unchanged and their home is a standalone blueprint-constraint PR.+### Deviations from the plan's letter, taken during Phase 2- **`PipeFunc`'s pluggable executor seam stays interpreter-side (task 2.5).** The task asked the kernel op to take the executor as an explicit protocol-typed argument, carrying both arms. Neither arm's type survives the layer boundary: `PipeFuncExecutorProtocol.run_pipe_func` declares `pipe_run_params: PipeRunParams` and `run_pipe_func_transported` is typed on `LibraryCrate`, both interpreter models the task itself rules out moving. That leaves two ways to satisfy the letter, and both were rejected on cost. *Re-typing the protocol* breaks two out-of-tree implementers (our Daytona and Temporal plugins) which thread `pipe_run_params` onto the transported request — for a parameter nothing in either repo actually reads, which makes it a wire-format question, not a cleanup. *A kernel-side protocol plus an interpreter adapter* buys a second protocol for one seam, a wrapper class, and a model conversion at the boundary — to compose two calls. So the composition (`run_func`) is the kernel's in-process path, and what is genuinely shared — the call itself and the write-back — is single-sourced in `call_registered_function` and `store_result`, which is what the interpreter's default executor and operator now ride. Consequence to hold onto: a programmatic caller cannot dispatch a kernel `run_func` to a sandbox, and `run_func` has no in-tree caller, which is why it is unit-tested directly. Recorded in [`deferred-follow-ups.md`](deferred-follow-ups.md) as KF-8.