Skip to content

Build AI methods with your coding agent, then run them anywhere: as an MCP for chatbots, as a webapp for people, or via API for your software.

feat(job-metadata): split RunMetadata out of JobMetadata, and carry the job on a result (#1141)

Thomas Hebrard5c8cb813a469
diff --git a/.drift/acks/pipelex-kernel-docs.toml b/.drift/acks/pipelex-kernel-docs.toml
index b565fb654..3ab41abb9 100644
--- a/.drift/acks/pipelex-kernel-docs.toml
+++ b/.drift/acks/pipelex-kernel-docs.toml
@@ -1,8 +1,8 @@
contract = "pipelex-kernel-docs"
-digest = "sha256:c87b2054eef43e624731aed0ac71cf54ca4f7dfbc011e4e72f14b54493ae3a23"
+digest = "sha256:ba88545c68191606d49339b145a8ed109fda948f3b16e4a4ed31d6a783e420a2"
reviewed_by = "thomashebrard"
-reviewed_at = "2026-08-18T11:19:04Z"
-rationale = "PipelexKernel.make gained a required storage_scope beside a user_id that is now also required (both had a fallback path before: user_id coalesced to the tracing placeholder 'anonymous' deep in pipeline_run_setup, and the storage prefix was derived from it). Both code examples on the page called make(user_id=...) and would now be a TypeError, so both are corrected. Added a paragraph on the two fields to the boot section: what storage_scope is (an opaque prefix the caller composes from its own tenancy model, which the kernel composes leaves onto and never parses), that it is validated at construction because it becomes a storage key prefix, and why neither field defaults. Checked the module table at line 69 still lists the kernel's public surface - make/.llm_text/.llm_object/.make_step_metadata are unchanged in name and the only signature that moved is make's. The step_id_source paragraph is untouched by this change."
+reviewed_at = "2026-08-20T16:42:26Z"
+rationale = "PipelexKernel.make's signature is unchanged: user_id, storage_scope and trace_context are still flat keyword arguments, so both code examples on the page (lines 28 and 148) remain correct as written, and the paragraph at line 33 describing the two fields and why neither defaults is untouched by this change. What did change is internal to make(): it now builds JobMetadata(run_metadata=RunMetadata(...)) because JobMetadata split its run-constant half (user_id, pipeline_run_id, storage_scope, request_id) into a nested RunMetadata. That is invisible at the call site but NOT to a programmatic caller reading a field off kernel.job_metadata, and this page is the kernel's specification for exactly those callers - so the job_metadata bullet now states which half a field lives in, that the path is job_metadata.run_metadata.pipeline_run_id rather than job_metadata.pipeline_run_id, and that there is no read-through accessor. The split also formalises the run-level-vs-per-step distinction that bullet already drew informally, which is noted. Checked the rest: make_step_metadata still mints a per-step copy with a fresh pipe_run_id (pipe_run_id stays on JobMetadata, not RunMetadata), so that half of the bullet is unchanged; the trace_context paragraph at line 155 still holds because graph_id is still adopted as the run's pipeline_run_id, only nested; and the module table's public surface (make / llm_text / llm_object / make_step_metadata) is unchanged in both name and signature."
[trigger_files]
"pipelex/kernel/__init__.py" = "blob:c4f1b0fe2ccc17db9110733f71b612f4ac1ba4ce"
@@ -20,7 +20,7 @@ rationale = "PipelexKernel.make gained a required storage_scope beside a user_id
"pipelex/kernel/llm_prompt_content.py" = "blob:82c34283dacd2bfd9a2c263915e744736d399875"
"pipelex/kernel/llm_results.py" = "blob:6e3b8a2878b5bb284ece5fac10262fb59ac2029d"
"pipelex/kernel/memory_ops.py" = "blob:f6a5745e804ac08f19c70f1beb1f78baf50d51cd"
-"pipelex/kernel/pipelex_kernel.py" = "blob:47009beeb3cef974eb2b738cca0aaa853cccddde"
+"pipelex/kernel/pipelex_kernel.py" = "blob:93b02410886a1f81d4a32c135bcbc35896417af3"
"pipelex/kernel/prompt_references.py" = "blob:03ec9ef67ae84f141d072ca2d2c3ef888549ccc0"
"pipelex/kernel/search_ops.py" = "blob:ed0a8c7953aca1e38de852d0ddef11731b819539"
"pipelex/kernel/search_results.py" = "blob:101b70dc96cc5ee1721c9f6b1cb725a5ca78ff70"
diff --git a/.github/workflows/guard-branches.yml b/.github/workflows/guard-branches.yml
index 93f7fba9f..25f72ddd2 100644
--- a/.github/workflows/guard-branches.yml
+++ b/.github/workflows/guard-branches.yml
@@ -36,8 +36,8 @@ jobs:
if [[ "$HEAD" == "dev" ]]; then
exit 0
fi
- if [[ ! "$HEAD" =~ ^(fix|feature|refactor|chore|docs|ci-cd|changelog|codex)\/[A-Za-z0-9._\/\<\>\=\-]+$ ]]; then
- echo "::error::Branch must start with fix/, feature/, refactor/, chore/, docs/, or ci-cd/."
+ if [[ ! "$HEAD" =~ ^(fix|feature|feat|refactor|chore|docs|ci-cd|changelog|codex)\/[A-Za-z0-9._\/\<\>\=\-]+$ ]]; then
+ echo "::error::Branch must start with fix/, feature/, feat/, refactor/, chore/, docs/, or ci-cd/."
exit 1
fi
diff --git a/CHANGELOG.md b/CHANGELOG.md
index 6c0fc0206..200a05b2d 100644
--- a/CHANGELOG.md
+++ b/CHANGELOG.md
@@ -1,9 +1,34 @@
# Changelog
+## Unreleased
+
+- **Breaking: `JobMetadata` splits its run-constant half into a new `RunMetadata`.** `user_id`, `pipeline_run_id`, `storage_scope` and `request_id` are constant for a whole run; `pipe_code`, `pipe_run_id`, `otel_context` and `content_generation_job_id` change at every step. They sat side by side with nothing saying which was which, so a copy made for a nested pipe had to know, field by field, what to carry through. They are now `job_metadata.run_metadata.*`, and the copy carries one object.
+
+ There is deliberately no read-through accessor on `JobMetadata`: one spelling for one fact. Every call site reads the nested path.
+
+- **A pipe's result now carries the job it was produced under.** `PipeOutput.job_metadata` and `TracingAssembly.run_metadata` are set once at the run boundary, so a transport that offloads an oversized result to storage can key it inside the run's own namespace. Both are **private attributes behind properties, not model fields**: `PipeOutput` is on the wire — `pipelex-api` returns it and publishes its schema — so as a field this would have put `user_id`, `request_id` and `otel_context` into a public API's response body and its documented OpenAPI artifact. A transport reads the value by attribute on the live object before serializing, so it never needed to be on the wire at all.
+
+ The gap was one-directional and easy to miss: every payload a run sends *in* carries a `job_metadata` to resolve a scope through, and nothing it sent *back* did. Under a hosted orchestrator that meant the two largest payloads an ordinary run produces — the pipe output and the tracing assembly, both around 350 KB — were stored outside every prefix the host's erasure cascade deletes, and survived deletion of the run that produced them. The same bytes were also stored a second time, correctly scoped, inside the delivery argument that carries them.
+
+ Set at the run boundary and **not** at the ~17 `PipeOutput(...)` construction sites: only the top-level output crosses a transport boundary, and stamping every constructor to serve one of them is the same "remember to add it everywhere" fragility that caused the original bug. Optional throughout — dry runs, signature stubs and tests build outputs with no job in hand — and invisible on the wire, since `serialize_completed_output` names the fields it returns explicitly.
+
+- **The `error_type` values a validation verdict can carry are now an enumerable, closed registry**: a `.mthds` bundle that fails validation reports each fault under an `error_type`, and until now there was no way to ask the runtime which faults exist. The values were reachable only by reading raise sites, and the only curated list of them lived outside this repo, in a hand-maintained conformance suite that could silently fall behind. `pipelex/validation_error_types.py` now holds the vocabulary in full and enumerates it as `VALIDATION_ERROR_TYPES`, so a consumer building coverage over the language surface — a test corpus, a client mapping faults onto its own UI — reads the registry instead of collecting strings from whichever diagnostics it happened to have seen. It is a **union of the enums the runtime already raises**, never a second list beside them: the two stage vocabularies plus a new `ValidationResidualErrorType` naming the one residual channel that had no enum of its own and rode an inline string. A member added to any of them is in the registry the moment it is declared. `ValidationErrorItem.error_type` is typed against their union, which is what makes the registry closed rather than merely documented — an unregistered string can no longer be constructed or parsed onto an item — and which publishes the vocabulary into the OpenAPI schema served for `/validate`, where the field was previously an open `string`. **No wire value changed**, including the dry-run residual's `DryRunError`: it keeps the exception-class spelling it has always had, because normalizing it would break every consumer that pins the string and would buy nothing an enumeration does not already give. Membership means a value is reachable on the wire, not that it is worth exercising — the advisory-only warning type and the two `unknown_*` fallbacks are in the registry and a coverage consumer excludes them on its own side, with a reason, rather than pruning the runtime's truth.
+
## [v0.49.0] - 2026-08-20
### Added
+- **Breaking: `JobMetadata` splits its run-constant half into a new `RunMetadata`.** `user_id`, `pipeline_run_id`, `storage_scope` and `request_id` are constant for a whole run; `pipe_code`, `pipe_run_id`, `otel_context` and `content_generation_job_id` change at every step. They sat side by side with nothing saying which was which, so a copy made for a nested pipe had to know, field by field, what to carry through. They are now `job_metadata.run_metadata.*`, and the copy carries one object.
+
+ There is deliberately no read-through accessor on `JobMetadata`: one spelling for one fact. Every call site reads the nested path.
+
+- **A pipe's result now carries the job it was produced under.** `PipeOutput.job_metadata` and `TracingAssembly.run_metadata` are set once at the run boundary, so a transport that offloads an oversized result to storage can key it inside the run's own namespace. Both are **private attributes behind properties, not model fields**: `PipeOutput` is on the wire — `pipelex-api` returns it and publishes its schema — so as a field this would have put `user_id`, `request_id` and `otel_context` into a public API's response body and its documented OpenAPI artifact. A transport reads the value by attribute on the live object before serializing, so it never needed to be on the wire at all.
+
+ The gap was one-directional and easy to miss: every payload a run sends _in_ carries a `job_metadata` to resolve a scope through, and nothing it sent _back_ did. Under a hosted orchestrator that meant the two largest payloads an ordinary run produces — the pipe output and the tracing assembly, both around 350 KB — were stored outside every prefix the host's erasure cascade deletes, and survived deletion of the run that produced them. The same bytes were also stored a second time, correctly scoped, inside the delivery argument that carries them.
+
+ Set at the run boundary and **not** at the ~17 `PipeOutput(...)` construction sites: only the top-level output crosses a transport boundary, and stamping every constructor to serve one of them is the same "remember to add it everywhere" fragility that caused the original bug. Optional throughout — dry runs, signature stubs and tests build outputs with no job in hand — and invisible on the wire, since `serialize_completed_output` names the fields it returns explicitly.
+
+- **The `error_type` values a validation verdict can carry are now an enumerable, closed registry**: a `.mthds` bundle that fails validation reports each fault under an `error_type`, and until now there was no way to ask the runtime which faults exist. The values were reachable only by reading raise sites, and the only curated list of them lived outside this repo, in a hand-maintained conformance suite that could silently fall behind. `pipelex/validation_error_types.py` now holds the vocabulary in full and enumerates it as `VALIDATION_ERROR_TYPES`, so a consumer building coverage over the language surface — a test corpus, a client mapping faults onto its own UI — reads the registry instead of collecting strings from whichever diagnostics it happened to have seen. It is a **union of the enums the runtime already raises**, never a second list beside them: the two stage vocabularies plus a new `ValidationResidualErrorType` naming the one residual channel that had no enum of its own and rode an inline string. A member added to any of them is in the registry the moment it is declared. `ValidationErrorItem.error_type` is typed against their union, which is what makes the registry closed rather than merely documented — an unregistered string can no longer be constructed or parsed onto an item — and which publishes the vocabulary into the OpenAPI schema served for `/validate`, where the field was previously an open `string`. **No wire value changed**, including the dry-run residual's `DryRunError`: it keeps the exception-class spelling it has always had, because normalizing it would break every consumer that pins the string and would buy nothing an enumeration does not already give. Membership means a value is reachable on the wire, not that it is worth exercising — the advisory-only warning type and the two `unknown_*` fallbacks are in the registry and a coverage consumer excludes them on its own side, with a reason, rather than pruning the runtime's truth.
- **Closed registry for validation error types**: `VALIDATION_ERROR_TYPES`, in the new `pipelex/validation_error_types.py`, is an enumerable, closed registry of every `error_type` a validation verdict can carry. It is a union of the enums the runtime already raises — `PipeValidationErrorType`, `PipeFactoryErrorType`, and a new `ValidationResidualErrorType` naming the one residual channel that had no enum of its own — so a member added to any of them is in the registry the moment it is declared. A consumer building coverage over the language surface now reads the registry instead of collecting strings from whichever diagnostics it happened to have seen. No wire value changed.
- **The corpus's invalid axis**: the MTHDS Test Corpus now carries an `error.*` tag namespace generated from `VALIDATION_ERROR_TYPES`, plus a focused invalid entry for every tag in it that a `.mthds` bundle can actually produce — each authored to trigger exactly one error, and gated on doing so. Five codes are excluded, each with the measurement behind it: `circular_dependency_error` and `unknown_concept` are unreachable through bundle validation, `optional_force_redundant` is advisory-only and rides the `warnings` array, and the two `unknown_*` fallbacks name the absence of a diagnosis rather than a language fault.
- **The exhaustivity gate now requires an invalid entry to agree with itself**: such an entry states its fault twice — as `expected_error`, the wire string, and as an `error.*` tag in `covers`, the normalized form — and nothing checked that the two matched. The check joins through the vocabulary's own `code` field rather than re-normalizing, so it carries no second copy of the normalization rule. A valid entry may no longer carry an `error.*` tag, since its bundle produces no diagnostic for the exhaustivity arm to count.
@@ -42,7 +67,7 @@
### Added
-- **`codegen.lock` now declares its format version, so the format can grow without breaking every reader:** the lock is a cross-language interchange format — this CLI writes it, other implementations of the offline check read it — and it validated strictly with no version field at all, so the day it gained any key every consumer pinned to an older reader would have failed CI with an opaque "unknown key" error instead of a drift verdict. Each lock now opens with `lock_version = 1`; a reader refuses a version it does not know and names which side to upgrade, and it reads the version *before* the key set, so a future lock is diagnosed by its version rather than by whichever new key it happens to carry first. A lock written before the field existed is version 1 by definition and needs no migration, but every regeneration rewrites the lock once to add the key — that one-line diff is the change consumers will see. The stamp header stays deliberately unversioned, since it already ignores commented fields it does not recognise.
+- **`codegen.lock` now declares its format version, so the format can grow without breaking every reader:** the lock is a cross-language interchange format — this CLI writes it, other implementations of the offline check read it — and it validated strictly with no version field at all, so the day it gained any key every consumer pinned to an older reader would have failed CI with an opaque "unknown key" error instead of a drift verdict. Each lock now opens with `lock_version = 1`; a reader refuses a version it does not know and names which side to upgrade, and it reads the version _before_ the key set, so a future lock is diagnosed by its version rather than by whichever new key it happens to carry first. A lock written before the field existed is version 1 by definition and needs no migration, but every regeneration rewrites the lock once to add the key — that one-line diff is the change consumers will see. The stamp header stays deliberately unversioned, since it already ignores commented fields it does not recognise.
- **Line endings are pinned to LF in the git index and on checkout, on every platform:** this repo carries a lot of generated-but-committed text — `CLAUDE.md` and `AGENTS.md`, `subject_grants.toml`, the drift acks, `.test_durations`, the error-identity snapshot, the generated error pages, the migration goldens, the gateway model references — and each is produced by a writer that passes no explicit `newline`, so on Windows Python translated every `\n` to CRLF and the whole set churned in version control with no change of content. No gate would have caught it: they all compare with universal-newline reads, which fold CRLF back to LF before the comparison. A `.gitattributes` carrying `* text=auto eol=lf` closes that class for tracked files whichever writer produced them. `text=auto` leaves git to tell text from binary, so images and other binaries are untouched, and no tracked file held CRLF when this landed — it introduces no renormalisation diff, it pins the invariant. The complementary writer-by-writer fix is the structural half and is tracked separately.
- **Documentation:** `docs/under-the-hood/codegen-projections.md` now covers the strict stamp-header rules, the JSON conformance the `options` payload must meet, the policy for evolving the lock format under `lock_version`, and the cross-platform line-ending guarantee.
@@ -94,11 +119,11 @@
### Fixed
-- **The codegen stamp header is now read strictly:** `pipelex codegen check` verified a generated file as pristine even when a line had been injected between the stamp's fence markers, because the content hash covers only the body *below* the fence — so an executable statement could sit inside a header that says `DO NOT EDIT` and the tree would still report as current. Any line inside the header that does not carry the file's comment prefix now makes the stamp unparseable, which the check already reports as `hand-edited`. The `options` value must also be conformant JSON: Python's non-standard `NaN` / `Infinity` / `-Infinity` literals are refused, so a stamp cannot hold something only a Python reader accepts — the header is a cross-language interchange format, and the second implementation of the offline check (in `@pipelex/sdk`) already rejected them. Commented `key: value` lines a reader does not recognise are still ignored, so the header can gain fields without breaking older readers. One consequence to know: a file whose header was tampered with is no longer one `pipelex codegen types` will overwrite — it now refuses a destination it cannot prove it owns rather than clobbering work, so such a file has to be removed before regenerating, which is what the check's advice already tells you to do.
+- **The codegen stamp header is now read strictly:** `pipelex codegen check` verified a generated file as pristine even when a line had been injected between the stamp's fence markers, because the content hash covers only the body _below_ the fence — so an executable statement could sit inside a header that says `DO NOT EDIT` and the tree would still report as current. Any line inside the header that does not carry the file's comment prefix now makes the stamp unparseable, which the check already reports as `hand-edited`. The `options` value must also be conformant JSON: Python's non-standard `NaN` / `Infinity` / `-Infinity` literals are refused, so a stamp cannot hold something only a Python reader accepts — the header is a cross-language interchange format, and the second implementation of the offline check (in `@pipelex/sdk`) already rejected them. Commented `key: value` lines a reader does not recognise are still ignored, so the header can gain fields without breaking older readers. One consequence to know: a file whose header was tampered with is no longer one `pipelex codegen types` will overwrite — it now refuses a destination it cannot prove it owns rather than clobbering work, so such a file has to be removed before regenerating, which is what the check's advice already tells you to do.
- **`pipelex codegen check` now reports drift in one order, whichever half of the report it comes from:** locked artifacts were listed by their full path compared as a string, while stale orphans came out in the order a directory walk found them — a component-by-component order. For a tree holding a `models/` directory beside a `models.py`, the two halves of a single report therefore sorted paths by different rules, and a second implementation could not mirror both with one comparator. Both halves now use the plain full-string sort, and the ordering is written down as a contract: every locked-artifact drift first, then every orphan, each group by path.
- **Generated files are written with LF line endings on every platform:** `save_text_to_path` let Python translate each newline to `os.linesep`, so a file emitted on Windows landed as CRLF on disk while its recorded content hash had been taken over the LF text it was built from. The verdict survived, because reads translate back, but the promise that regenerating produces byte-identical output held only within one platform — a team mixing Windows and Linux saw generated trees churn in version control with no change of content. Every artifact this writes (generated code, the codegen lock, JSON outputs, input templates) is now the string verbatim.
-- **The scope was validated *after* the first thing that wrote with it.** `prepare_pipe_job` ran the data-url normalizer — which stores bytes at `{storage_scope}/assets/…` — and only then constructed the `JobMetadata` whose validator is supposed to make that key safe. A scope carrying `..` therefore escaped the tenant's namespace, and the guard meant to stop it ran afterwards, on damage already done. The defect was ordering, not absence: this file's own "safe by construction" claim did not hold on the one path that mattered. The scope is now validated at the top of the seam, before anything composes a key from it. Pinned by a test that asserts the normalizer was never called, with real inputs — the empty-inputs harness skips the normalizer on its own, so the same test written that way passes with the guard deleted.
+- **The scope was validated _after_ the first thing that wrote with it.** `prepare_pipe_job` ran the data-url normalizer — which stores bytes at `{storage_scope}/assets/…` — and only then constructed the `JobMetadata` whose validator is supposed to make that key safe. A scope carrying `..` therefore escaped the tenant's namespace, and the guard meant to stop it ran afterwards, on damage already done. The defect was ordering, not absence: this file's own "safe by construction" claim did not hold on the one path that mattered. The scope is now validated at the top of the seam, before anything composes a key from it. Pinned by a test that asserts the normalizer was never called, with real inputs — the empty-inputs harness skips the normalizer on its own, so the same test written that way passes with the guard deleted.
- **Generated content had no leaf: it landed loose at the root of the scope.** Three things write during a run and each built its key separately — the data-url normalizer under `assets/`, storage delivery under `results/`, and generated bytes (an image a pipe produced) directly at `{storage_scope}/<hash>.<ext>`, because `uri_format` rendered the whole key and shipped without one. A file under no leaf is invisible to anything listing a run's content by category, cannot be retained or dropped separately from the rest, and shares the root with every leaf added later. Generated bytes now land under `generated/`.
diff --git a/docs/under-the-hood/pipelex-kernel.md b/docs/under-the-hood/pipelex-kernel.md
index 8dd66eaf4..0c567e48a 100644
--- a/docs/under-the-hood/pipelex-kernel.md
+++ b/docs/under-the-hood/pipelex-kernel.md
@@ -125,7 +125,7 @@ summaries = extract_main_content_as_list(memory=result.memory, item_type=Summary
`PipelexKernel` holds run-scoped identity, plus the one seam that identity needs:
-- **`job_metadata`** — the run-level metadata. It is not what a step runs under: every call mints a per-step copy through `make_step_metadata()`, carrying a fresh `pipe_run_id` and inheriting the trace context, so trace and usage attribution stay per-step. This mirrors the interpreter's pass-down-a-modified-copy pattern. `make_step_metadata(pipe_code=…)` names the pipe the step is running — mirroring what the interpreter stamps on its live and dry paths — so log correlation, usage accounting, and the per-step labelling a distributed backend derives see a named step. When no `pipe_code` is supplied the key is omitted from the update rather than passed as `None`, so a run-level `pipe_code` is never silently erased; the direct-call façade (`llm_text`/`llm_object`) stays deliberately anonymous, because the caller ran no pipe.
+- **`job_metadata`** — the run-level metadata. **The two halves are now separate types, and a caller reading a field off this object must know which half it is in.** `JobMetadata.run_metadata` holds what is constant for the whole run — `user_id`, `pipeline_run_id`, `storage_scope`, `request_id` — and the rest of `JobMetadata` holds what changes per step (`pipe_code`, `pipe_run_id`, `otel_context`). So it is `kernel.job_metadata.run_metadata.pipeline_run_id`, not `kernel.job_metadata.pipeline_run_id`; there is deliberately no read-through accessor, so one fact has one spelling. The distinction this bullet already drew informally — run-level metadata, per-step copy — is what the split makes explicit, and it is why the per-step copy can carry the run half through untouched. It is not what a step runs under: every call mints a per-step copy through `make_step_metadata()`, carrying a fresh `pipe_run_id` and inheriting the trace context, so trace and usage attribution stay per-step. This mirrors the interpreter's pass-down-a-modified-copy pattern. `make_step_metadata(pipe_code=…)` names the pipe the step is running — mirroring what the interpreter stamps on its live and dry paths — so log correlation, usage accounting, and the per-step labelling a distributed backend derives see a named step. When no `pipe_code` is supplied the key is omitted from the update rather than passed as `None`, so a run-level `pipe_code` is never silently erased; the direct-call façade (`llm_text`/`llm_object`) stays deliberately anonymous, because the caller ran no pipe.
- **`cogt_run_params`** — the execution-mode contract (`run_mode`, and the DRY-only `is_mock_usage` sub-flag) that every cogt leaf reads off the assignment it is handed.
- **`step_id_source`** — where each step's `pipe_run_id` comes from, defaulting to a fresh `uuid4`, which is what every in-process run wants. It exists for a kernel hosted inside a replay-based executor, where ids minted from a non-replay-safe source take different values each re-execution; such a host injects its own replay-safe source via `PipelexKernel.make(step_id_source=…)`. It is a seam, not state — the kernel never inspects what it returns.
diff --git a/pipelex/cogt/content_generation/dry_mock.py b/pipelex/cogt/content_generation/dry_mock.py
index 452a13714..5a30b80cb 100644
--- a/pipelex/cogt/content_generation/dry_mock.py
+++ b/pipelex/cogt/content_generation/dry_mock.py
@@ -293,7 +293,7 @@ def _dry_text_gen_truncate_length() -> int:
def dry_llm_gen_text(llm_assignment: LLMAssignment) -> str:
"""Dry leaf for ``llm_gen_text``: synthetic job report + a ``DRY RUN:`` marker string."""
job_metadata = llm_assignment.job_metadata
- log.verbose(f"🤡 DRY RUN: llm_gen_text for '{job_metadata.pipeline_run_id}'")
+ log.verbose(f"🤡 DRY RUN: llm_gen_text for '{job_metadata.run_metadata.pipeline_run_id}'")
report_func = _dry_report_func(llm_assignment.cogt_run_params)
report_func(job_metadata, llm_setting=llm_assignment.llm_setting, llm_prompt=llm_assignment.llm_prompt)
prompt_truncated = llm_assignment.llm_prompt.desc(truncate_text_length=_dry_text_gen_truncate_length())
@@ -402,7 +402,7 @@ def dry_render_page_views(render_assignment: RenderPageViewsAssignment) -> list[
Fake URLs come from ``inference.dry_run.image_urls`` (validated non-empty) — the single configured
source of truth for dry fake images, same as the img-gen mock.
"""
- log.verbose(f"🤡 DRY RUN: render_page_views for '{render_assignment.job_metadata.pipeline_run_id}'")
+ log.verbose(f"🤡 DRY RUN: render_page_views for '{render_assignment.job_metadata.run_metadata.pipeline_run_id}'")
nb_pages = get_config().inference.dry_run.nb_extract_pages
image_urls = get_config().inference.dry_run.image_urls
return [_dry_image_content(image_url=image_urls[page_index % len(image_urls)]) for page_index in range(nb_pages)]
diff --git a/pipelex/cogt/content_generation/extract_generate.py b/pipelex/cogt/content_generation/extract_generate.py
index 53dd0d100..034c3971c 100644
--- a/pipelex/cogt/content_generation/extract_generate.py
+++ b/pipelex/cogt/content_generation/extract_generate.py
@@ -33,6 +33,6 @@ async def extract_gen_pages_and_store(
return dry_extract_page_contents(extract_assignment)
extract_output = await extract_gen_pages(extract_assignment)
return await generated_content_factory.make_page_contents(
- storage_scope=extract_assignment.job_metadata.storage_scope,
+ storage_scope=extract_assignment.job_metadata.run_metadata.storage_scope,
extract_output=extract_output,
)
diff --git a/pipelex/cogt/content_generation/img_gen_generate.py b/pipelex/cogt/content_generation/img_gen_generate.py
index 1b625311e..710937579 100644
--- a/pipelex/cogt/content_generation/img_gen_generate.py
+++ b/pipelex/cogt/content_generation/img_gen_generate.py
@@ -54,7 +54,7 @@ async def img_gen_single_image_and_store(
return dry_img_gen_image_contents(img_gen_assignment.model_copy(update={"nb_images": 1}))[0]
generated_image = await img_gen_single_image(img_gen_assignment)
image_content = await generated_content_factory.make_image_content(
- storage_scope=img_gen_assignment.job_metadata.storage_scope,
+ storage_scope=img_gen_assignment.job_metadata.run_metadata.storage_scope,
raw_details=generated_image,
)
image_content.source_prompt = img_gen_assignment.img_gen_prompt.positive_text
@@ -77,7 +77,7 @@ async def img_gen_image_list_and_store(
image_contents: list[ImageContent] = []
for raw_details in generated_image_list:
image_content = await generated_content_factory.make_image_content(
- storage_scope=img_gen_assignment.job_metadata.storage_scope,
+ storage_scope=img_gen_assignment.job_metadata.run_metadata.storage_scope,
raw_details=raw_details,
)
image_content.source_prompt = img_gen_assignment.img_gen_prompt.positive_text
diff --git a/pipelex/cogt/content_generation/render_generate.py b/pipelex/cogt/content_generation/render_generate.py
index 4c3f48b6d..850f88882 100644
--- a/pipelex/cogt/content_generation/render_generate.py
+++ b/pipelex/cogt/content_generation/render_generate.py
@@ -43,7 +43,7 @@ async def render_page_views_and_store(
# otherwise holds every page bitmap in memory until the whole store loop finishes.
page_view_image.close()
image_content = await generated_content_factory.make_image_content(
- storage_scope=render_assignment.job_metadata.storage_scope,
+ storage_scope=render_assignment.job_metadata.run_metadata.storage_scope,
raw_details=raw_details,
)
image_contents.append(image_content)
diff --git a/pipelex/cogt/llm/llm_worker_abstract.py b/pipelex/cogt/llm/llm_worker_abstract.py
index f9e79425f..b387fc96f 100644
--- a/pipelex/cogt/llm/llm_worker_abstract.py
+++ b/pipelex/cogt/llm/llm_worker_abstract.py
@@ -118,7 +118,7 @@ class LLMWorkerAbstract(InferenceWorkerAbstract, ABC):
if not pipe_code:
msg = "pipe_code is required for LLM span"
raise JobMetadataError(msg)
- pipeline_run_id = job_metadata.pipeline_run_id
+ pipeline_run_id = job_metadata.run_metadata.pipeline_run_id
# Build output description for span name (full values - exporter handles redaction)
output_desc: str
@@ -243,7 +243,7 @@ class LLMWorkerAbstract(InferenceWorkerAbstract, ABC):
log.verbose(
f"[OTel] LLM SPAN ENDING:\n"
f" pipe_code='{job_metadata.pipe_code}'\n"
- f" pipeline_run_id='{job_metadata.pipeline_run_id}'\n"
+ f" pipeline_run_id='{job_metadata.run_metadata.pipeline_run_id}'\n"
f" trace_id={span_ctx.trace_id:032x}\n"
f" span_id={span_ctx.span_id:016x}"
)
@@ -274,7 +274,7 @@ class LLMWorkerAbstract(InferenceWorkerAbstract, ABC):
log.verbose(
f"[OTel] LLM SPAN ENDING:\n"
f" pipe_code='{job_metadata.pipe_code}'\n"
- f" pipeline_run_id='{job_metadata.pipeline_run_id}'\n"
+ f" pipeline_run_id='{job_metadata.run_metadata.pipeline_run_id}'\n"
f" trace_id={span_ctx.trace_id:032x}\n"
f" span_id={span_ctx.span_id:016x}"
)
@@ -306,7 +306,7 @@ class LLMWorkerAbstract(InferenceWorkerAbstract, ABC):
log.verbose(
f"[OTel] LLM SPAN ENDING WITH ERROR:\n"
f" pipe_code='{job_metadata.pipe_code}'\n"
- f" pipeline_run_id='{job_metadata.pipeline_run_id}'\n"
+ f" pipeline_run_id='{job_metadata.run_metadata.pipeline_run_id}'\n"
f" trace_id={span_ctx.trace_id:032x}\n"
f" span_id={span_ctx.span_id:016x}"
)
diff --git a/pipelex/core/pipes/pipe_output.py b/pipelex/core/pipes/pipe_output.py
index e04dddaf2..b7a9dc1ed 100644
--- a/pipelex/core/pipes/pipe_output.py
+++ b/pipelex/core/pipes/pipe_output.py
@@ -1,7 +1,7 @@
from typing import Any, TypeVar
from mthds.protocol.pipe_output import PipeOutputAbstract
-from pydantic import Field
+from pydantic import Field, PrivateAttr
from pipelex.core.memory.working_memory import WorkingMemory
from pipelex.core.stuffs.date_content import DateContent
@@ -17,7 +17,7 @@ from pipelex.core.stuffs.text_content import TextContent
from pipelex.core.stuffs.yes_no_content import YesNoContent
from pipelex.graph.graphspec import GraphSpec
from pipelex.reporting.reporting_types import AnyTokensUsage
-from pipelex.system.job_metadata import SpecialPipelineId
+from pipelex.system.job_metadata import JobMetadata, SpecialPipelineId
class PipeOutput(PipeOutputAbstract[WorkingMemory]):
@@ -33,6 +33,43 @@ class PipeOutput(PipeOutputAbstract[WorkingMemory]):
tokens_usages: list[AnyTokensUsage] | None = None
usage_assembly_error: str | None = None
+ # The job this output was produced under, when there was one.
+ #
+ # Carried so a transport that offloads an oversized result to storage can key
+ # it inside the run's own namespace instead of at the root of a bucket. Every
+ # payload a run sends IN carries a `job_metadata` for a transport to resolve a
+ # scope through; the results did not, so the two largest payloads an ordinary
+ # run produces landed outside every prefix a host's erasure cascade deletes
+ # and survived deletion of their own run.
+ #
+ # **A private attribute behind a property, NOT a model field, and that is the
+ # whole point.** `PipeOutput` is on the wire: `pipelex-api` returns it inside
+ # `PipelexApiExecuteResponse` and publishes its schema in a committed OpenAPI
+ # artifact. As a field, this would put `user_id`, `request_id`, `otel_context`
+ # and `trace_context` into a public API's response body and its documented
+ # schema — which `test_openapi_contract.py` exists to prevent, and which it
+ # caught. As a private attribute it is readable by a transport resolving a
+ # scope (attribute access on the live object, before serialization) and absent
+ # from `model_dump`, from `model_json_schema`, and therefore from the wire.
+ #
+ # It does not survive serialization, and does not need to: a transport reads it
+ # on the object it is about to serialize, in the process that produced it.
+ _job_metadata: JobMetadata | None = PrivateAttr(default=None)
+
+ @property
+ def job_metadata(self) -> JobMetadata | None:
+ return self._job_metadata
+
+ def set_job_metadata(self, *, job_metadata: JobMetadata | None) -> None:
+ """Set the job_metadata a transport reads to place this result's bytes.
+
+ A METHOD, not a `@job_metadata.setter`: this repo's keyword-only convention
+ rewrites a positional setter parameter into a keyword-only one, which is not
+ a valid property-setter signature. A named method takes the keyword happily
+ and says what it does at the call site.
+ """
+ self._job_metadata = job_metadata
+
@property
def main_stuff(self) -> Stuff:
return self.working_memory.get_main_stuff()
diff --git a/pipelex/kernel/pipelex_kernel.py b/pipelex/kernel/pipelex_kernel.py
index 47009beeb..93b024108 100644
--- a/pipelex/kernel/pipelex_kernel.py
+++ b/pipelex/kernel/pipelex_kernel.py
@@ -54,7 +54,7 @@ from pipelex.kernel.llm_prompt_content import LlmPromptContent
from pipelex.kernel.llm_results import LlmObjectResult, LlmTextResult
from pipelex.kernel.templating_style_ops import resolve_templating_style
from pipelex.runtime_hub import resolve_run_mode_for_boot
-from pipelex.system.job_metadata import JobMetadata
+from pipelex.system.job_metadata import JobMetadata, RunMetadata
from pipelex.system.pipe_run_mode import PipeRunMode
from pipelex.system.trace_context import TraceContext
from pipelex.tools.templating.templating_style import TemplatingStyle
@@ -118,9 +118,11 @@ class PipelexKernel:
check_mock_usage_requires_dry(run_mode=run_mode, is_mock_usage=is_mock_usage)
return cls(
job_metadata=JobMetadata(
- user_id=user_id,
- storage_scope=storage_scope,
- pipeline_run_id=trace_context.graph_id if trace_context is not None else str(uuid4()),
+ run_metadata=RunMetadata(
+ user_id=user_id,
+ storage_scope=storage_scope,
+ pipeline_run_id=trace_context.graph_id if trace_context is not None else str(uuid4()),
+ ),
trace_context=trace_context,
),
cogt_run_params=CogtRunParams(run_mode=resolve_run_mode_for_boot(requested=run_mode), is_mock_usage=is_mock_usage),
diff --git a/pipelex/pipe_controllers/batch/pipe_batch.py b/pipelex/pipe_controllers/batch/pipe_batch.py
index df9ace0e3..784343760 100644
--- a/pipelex/pipe_controllers/batch/pipe_batch.py
+++ b/pipelex/pipe_controllers/batch/pipe_batch.py
@@ -254,7 +254,7 @@ class PipeBatch(PipeController):
return PipeOutput(
working_memory=working_memory,
- pipeline_run_id=job_metadata.pipeline_run_id,
+ pipeline_run_id=job_metadata.run_metadata.pipeline_run_id,
)
@override
diff --git a/pipelex/pipe_controllers/condition/pipe_condition.py b/pipelex/pipe_controllers/condition/pipe_condition.py
index 19dbfe519..02c7f3d10 100644
--- a/pipelex/pipe_controllers/condition/pipe_condition.py
+++ b/pipelex/pipe_controllers/condition/pipe_condition.py
@@ -309,7 +309,7 @@ class PipeCondition(PipeController):
output_name=output_name,
reason=f"PipeCondition '{self.code}' resolved to 'continue' for evaluated expression '{evaluated_expression}'",
)
- return PipeOutput(working_memory=working_memory, pipeline_run_id=job_metadata.pipeline_run_id)
+ return PipeOutput(working_memory=working_memory, pipeline_run_id=job_metadata.run_metadata.pipeline_run_id)
if SpecialOutcome.is_fail(outcome):
self._register_execution_data(job_metadata=job_metadata, execution_data=execution_data_dict)
@@ -433,7 +433,7 @@ class PipeCondition(PipeController):
output_name=output_name,
reason=f"dry run of PipeCondition '{self.code}': all outcomes are special, the declared output resolves absent",
)
- return PipeOutput(working_memory=working_memory, pipeline_run_id=job_metadata.pipeline_run_id)
+ return PipeOutput(working_memory=working_memory, pipeline_run_id=job_metadata.run_metadata.pipeline_run_id)
def _record_declared_absent_output(self, *, working_memory: WorkingMemory, output_name: str | None, reason: str) -> None:
"""Resolve this pipe's declared output as a declared-absent record (the `continue` arm)."""
diff --git a/pipelex/pipe_controllers/parallel/pipe_parallel.py b/pipelex/pipe_controllers/parallel/pipe_parallel.py
index 1292c0722..38a4ed78c 100644
--- a/pipelex/pipe_controllers/parallel/pipe_parallel.py
+++ b/pipelex/pipe_controllers/parallel/pipe_parallel.py
@@ -549,7 +549,7 @@ class PipeParallel(PipeController):
return PipeOutput(
working_memory=working_memory,
- pipeline_run_id=job_metadata.pipeline_run_id,
+ pipeline_run_id=job_metadata.run_metadata.pipeline_run_id,
)
@override
diff --git a/pipelex/pipe_controllers/sequence/pipe_sequence.py b/pipelex/pipe_controllers/sequence/pipe_sequence.py
index 0cf76482e..bcd721f43 100644
--- a/pipelex/pipe_controllers/sequence/pipe_sequence.py
+++ b/pipelex/pipe_controllers/sequence/pipe_sequence.py
@@ -390,7 +390,7 @@ class PipeSequence(PipeController):
return PipeOutput(
working_memory=evolving_memory,
- pipeline_run_id=job_metadata.pipeline_run_id,
+ pipeline_run_id=job_metadata.run_metadata.pipeline_run_id,
)
@override
diff --git a/pipelex/pipe_machinery/pipe_abstract.py b/pipelex/pipe_machinery/pipe_abstract.py
index 770ee9161..ee4a15219 100644
--- a/pipelex/pipe_machinery/pipe_abstract.py
+++ b/pipelex/pipe_machinery/pipe_abstract.py
@@ -914,7 +914,7 @@ class PipeAbstract(ABC, BaseModel):
else:
working_memory.record_resolved_absence(companion_record)
- return PipeOutput(working_memory=working_memory, pipeline_run_id=job_metadata.pipeline_run_id)
+ return PipeOutput(working_memory=working_memory, pipeline_run_id=job_metadata.run_metadata.pipeline_run_id)
@final
async def live_run_pipe(
@@ -944,7 +944,7 @@ class PipeAbstract(ABC, BaseModel):
# Start OTel span first
span, is_root_span = self._start_pipe_span(
parent_otel_context=parent_otel_context,
- pipeline_run_id=job_metadata.pipeline_run_id,
+ pipeline_run_id=job_metadata.run_metadata.pipeline_run_id,
working_memory=working_memory,
)
# Get the actual span_id from OTel (OTel generates its own span_id)
diff --git a/pipelex/pipe_operators/compose/pipe_compose.py b/pipelex/pipe_operators/compose/pipe_compose.py
index bd614af41..ff3548d3a 100644
--- a/pipelex/pipe_operators/compose/pipe_compose.py
+++ b/pipelex/pipe_operators/compose/pipe_compose.py
@@ -222,7 +222,7 @@ class PipeCompose(PipeOperator[PipeComposeOutput]):
return PipeComposeOutput(
working_memory=working_memory,
- pipeline_run_id=job_metadata.pipeline_run_id,
+ pipeline_run_id=job_metadata.run_metadata.pipeline_run_id,
)
async def _run_construct_mode(
@@ -275,7 +275,7 @@ class PipeCompose(PipeOperator[PipeComposeOutput]):
return PipeComposeOutput(
working_memory=working_memory,
- pipeline_run_id=job_metadata.pipeline_run_id,
+ pipeline_run_id=job_metadata.run_metadata.pipeline_run_id,
)
@override
@@ -337,7 +337,7 @@ class PipeCompose(PipeOperator[PipeComposeOutput]):
working_memory.set_new_main_stuff(stuff=output_stuff, name=output_name)
return PipeComposeOutput(
working_memory=working_memory,
- pipeline_run_id=job_metadata.pipeline_run_id,
+ pipeline_run_id=job_metadata.run_metadata.pipeline_run_id,
)
@override
diff --git a/pipelex/pipe_operators/extract/pipe_extract.py b/pipelex/pipe_operators/extract/pipe_extract.py
index 8cd6ea825..771c415b0 100644
--- a/pipelex/pipe_operators/extract/pipe_extract.py
+++ b/pipelex/pipe_operators/extract/pipe_extract.py
@@ -159,7 +159,7 @@ class PipeExtract(PipeOperator[PipeExtractOutput]):
self._register_execution_data(job_metadata=job_metadata, execution_data=execution_data_dict)
return PipeExtractOutput(
working_memory=working_memory,
- pipeline_run_id=job_metadata.pipeline_run_id,
+ pipeline_run_id=job_metadata.run_metadata.pipeline_run_id,
)
@override
diff --git a/pipelex/pipe_operators/func/pipe_func.py b/pipelex/pipe_operators/func/pipe_func.py
index 6fa15b4cb..2ee9917ee 100644
--- a/pipelex/pipe_operators/func/pipe_func.py
+++ b/pipelex/pipe_operators/func/pipe_func.py
@@ -250,7 +250,7 @@ class PipeFunc(PipeOperator[PipeFuncOutput]):
return PipeFuncOutput(
working_memory=working_memory,
- pipeline_run_id=job_metadata.pipeline_run_id,
+ pipeline_run_id=job_metadata.run_metadata.pipeline_run_id,
)
@override
@@ -327,7 +327,7 @@ class PipeFunc(PipeOperator[PipeFuncOutput]):
return PipeFuncOutput(
working_memory=working_memory,
- pipeline_run_id=job_metadata.pipeline_run_id,
+ pipeline_run_id=job_metadata.run_metadata.pipeline_run_id,
)
@override
diff --git a/pipelex/pipe_operators/img_gen/pipe_img_gen.py b/pipelex/pipe_operators/img_gen/pipe_img_gen.py
index ad0fbff16..ea0d5af2a 100644
--- a/pipelex/pipe_operators/img_gen/pipe_img_gen.py
+++ b/pipelex/pipe_operators/img_gen/pipe_img_gen.py
@@ -255,7 +255,7 @@ class PipeImgGen(PipeOperator[PipeImgGenOutput]):
self._register_execution_data(job_metadata=job_metadata, execution_data=execution_data_dict)
return PipeImgGenOutput(
working_memory=working_memory,
- pipeline_run_id=job_metadata.pipeline_run_id,
+ pipeline_run_id=job_metadata.run_metadata.pipeline_run_id,
)
@override
diff --git a/pipelex/pipe_operators/llm/pipe_llm.py b/pipelex/pipe_operators/llm/pipe_llm.py
index b539fc6a6..dd7d72929 100644
--- a/pipelex/pipe_operators/llm/pipe_llm.py
+++ b/pipelex/pipe_operators/llm/pipe_llm.py
@@ -325,7 +325,7 @@ class PipeLLM(PipeOperator[PipeLLMOutput]):
self._register_execution_data(job_metadata=job_metadata, execution_data=execution_data_dict)
return PipeLLMOutput(
working_memory=working_memory,
- pipeline_run_id=job_metadata.pipeline_run_id,
+ pipeline_run_id=job_metadata.run_metadata.pipeline_run_id,
)
def _format_error_location(self, pipe_run_params: PipeRunParams) -> str:
diff --git a/pipelex/pipe_operators/search/pipe_search.py b/pipelex/pipe_operators/search/pipe_search.py
index 0a4d10219..050f439f1 100644
--- a/pipelex/pipe_operators/search/pipe_search.py
+++ b/pipelex/pipe_operators/search/pipe_search.py
@@ -137,7 +137,7 @@ class PipeSearch(PipeOperator[PipeSearchOutput]):
self._register_execution_data(job_metadata=job_metadata, execution_data=execution_data_dict)
return PipeSearchOutput(
working_memory=working_memory,
- pipeline_run_id=job_metadata.pipeline_run_id,
+ pipeline_run_id=job_metadata.run_metadata.pipeline_run_id,
)
@override
diff --git a/pipelex/pipe_operators/structure/pipe_structure.py b/pipelex/pipe_operators/structure/pipe_structure.py
index ae56b6813..599b60fef 100644
--- a/pipelex/pipe_operators/structure/pipe_structure.py
+++ b/pipelex/pipe_operators/structure/pipe_structure.py
@@ -182,7 +182,7 @@ class PipeStructure(PipeOperator[PipeStructureOutput]):
return PipeStructureOutput(
working_memory=working_memory,
- pipeline_run_id=job_metadata.pipeline_run_id,
+ pipeline_run_id=job_metadata.run_metadata.pipeline_run_id,
)
def _format_error_location(self, pipe_run_params: PipeRunParams) -> str:
diff --git a/pipelex/pipe_run/pipe_router_protocol.py b/pipelex/pipe_run/pipe_router_protocol.py
index 33bbe890d..bfccf8aa1 100644
--- a/pipelex/pipe_run/pipe_router_protocol.py
+++ b/pipelex/pipe_run/pipe_router_protocol.py
@@ -17,7 +17,7 @@ class PipeRouterProtocol(Protocol):
pipe_job: PipeJob,
) -> None:
payload: PayloadType = {
- PayloadKey.PIPELINE_RUN_ID: pipe_job.job_metadata.pipeline_run_id,
+ PayloadKey.PIPELINE_RUN_ID: pipe_job.job_metadata.run_metadata.pipeline_run_id,
PayloadKey.PIPE_JOB: pipe_job,
}
await self.observer.observe_before_run(payload)
@@ -29,7 +29,7 @@ class PipeRouterProtocol(Protocol):
pipe_output: PipeOutput,
) -> None:
payload: PayloadType = {
- PayloadKey.PIPELINE_RUN_ID: pipe_job.job_metadata.pipeline_run_id,
+ PayloadKey.PIPELINE_RUN_ID: pipe_job.job_metadata.run_metadata.pipeline_run_id,
PayloadKey.PIPE_JOB: pipe_job,
PayloadKey.PIPE_OUTPUT: pipe_output,
}
@@ -42,7 +42,7 @@ class PipeRouterProtocol(Protocol):
error: Exception,
) -> None:
payload: PayloadType = {
- PayloadKey.PIPELINE_RUN_ID: pipe_job.job_metadata.pipeline_run_id,
+ PayloadKey.PIPELINE_RUN_ID: pipe_job.job_metadata.run_metadata.pipeline_run_id,
PayloadKey.PIPE_JOB: pipe_job,
PayloadKey.ERROR: error,
}
diff --git a/pipelex/pipe_run/pipe_run.py b/pipelex/pipe_run/pipe_run.py
index 505321c75..0bbc220eb 100644
--- a/pipelex/pipe_run/pipe_run.py
+++ b/pipelex/pipe_run/pipe_run.py
@@ -33,7 +33,7 @@ class PipeRun(PipeRunProtocol):
*,
delivery_assignment: DeliveryAssignment | None = None,
) -> PipeOutput:
- pipeline_run_id: str = pipe_job.job_metadata.pipeline_run_id
+ pipeline_run_id: str = pipe_job.job_metadata.run_metadata.pipeline_run_id
status: DeliveryStatus = DeliveryStatus.COMPLETED
pipe_output: PipeOutput | None = None
execution_error: Exception | None = None
@@ -41,6 +41,11 @@ class PipeRun(PipeRunProtocol):
try:
pipe_output = await self._pipe_router.run(pipe_job)
+ # Stamp the job onto the run's OWN output, here and nowhere else.
+ # Only the top-level output crosses a transport boundary; a sub-pipe's
+ # output stays in-process, so stamping at every `PipeOutput(...)` site
+ # would be noise on ~17 constructors to serve one of them.
+ pipe_output.set_job_metadata(job_metadata=pipe_job.job_metadata)
except Exception as exc: # noqa: BLE001
# Observe-and-reraise: records FAILED status so the finally delivery sees it, then re-raises the original error below.
status = DeliveryStatus.FAILED
@@ -89,12 +94,12 @@ class PipeRun(PipeRunProtocol):
try:
await self._delivery_executor.execute(
pipe_output=pipe_output,
- storage_scope=pipe_job.job_metadata.storage_scope,
+ storage_scope=pipe_job.job_metadata.run_metadata.storage_scope,
pipeline_run_id=pipeline_run_id,
delivery_assignment=delivery_assignment,
status=status,
error_report=error_report,
- request_id=pipe_job.job_metadata.request_id,
+ request_id=pipe_job.job_metadata.run_metadata.request_id,
)
except DeliveryError as delivery_error:
if execution_error is None:
diff --git a/pipelex/pipe_run/tracing_assembly.py b/pipelex/pipe_run/tracing_assembly.py
index ce46186ad..98c3c5df9 100644
--- a/pipelex/pipe_run/tracing_assembly.py
+++ b/pipelex/pipe_run/tracing_assembly.py
@@ -12,7 +12,7 @@ that crosses the Temporal activity boundary (returned by ``act_assemble_tracing`
import json
-from pydantic import BaseModel, ConfigDict, ValidationError
+from pydantic import BaseModel, ConfigDict, PrivateAttr, ValidationError
from pipelex import log
from pipelex.base_exceptions import PipelexConfigError
@@ -22,6 +22,7 @@ from pipelex.graph.graphspec import GraphSpec, PipelineRef
from pipelex.reporting.reporting_types import AnyTokensUsage
from pipelex.runtime_hub import get_event_log_override
from pipelex.system.exceptions import MissingDependencyError
+from pipelex.system.job_metadata import RunMetadata
from pipelex.system.pipe_run_mode import PipeRunMode
from pipelex.tracing.event_log_factory import make_event_log
from pipelex.tracing.exceptions import EventLogError
@@ -48,6 +49,29 @@ class TracingAssembly(BaseModel):
tokens_usages: list[AnyTokensUsage] | None = None
usage_assembly_error: str | None = None
+ # The run this assembly belongs to. Same purpose as `PipeOutput.job_metadata`
+ # and, for the same reason, the same shape: a private attribute behind a
+ # property rather than a model field, so it is readable by a transport
+ # resolving a storage scope and absent from every serialization and schema.
+ #
+ # `RunMetadata` rather than the whole `JobMetadata`: an assembly belongs to the
+ # RUN, not to any one job within it, and the per-job half would be a lie here.
+ _run_metadata: RunMetadata | None = PrivateAttr(default=None)
+
+ @property
+ def run_metadata(self) -> RunMetadata | None:
+ return self._run_metadata
+
+ def set_run_metadata(self, *, run_metadata: RunMetadata | None) -> None:
+ """Set the run_metadata a transport reads to place this result's bytes.
+
+ A METHOD, not a `@run_metadata.setter`: this repo's keyword-only convention
+ rewrites a positional setter parameter into a keyword-only one, which is not
+ a valid property-setter signature. A named method takes the keyword happily
+ and says what it does at the call site.
+ """
+ self._run_metadata = run_metadata
+
def assemble_tracing(
pipeline_run_id: str,
@@ -57,6 +81,7 @@ def assemble_tracing(
domain_code: str | None = None,
main_pipe_code: str | None = None,
run_mode: PipeRunMode = PipeRunMode.LIVE,
+ run_metadata: RunMetadata | None = None,
) -> TracingAssembly:
"""Read the trace events for ``pipeline_run_id`` once and assemble the requested artifacts.
@@ -79,11 +104,15 @@ def assemble_tracing(
domain_code: Domain code for the graph's pipeline ref.
main_pipe_code: Main pipe code for the graph's pipeline ref.
run_mode: Final pipe run mode used to stamp GraphSpec provenance.
+ run_metadata: The run this assembly belongs to, stamped onto the result so a
+ transport that offloads it keys it inside the run's namespace. None for a
+ run with no host-supplied scope.
Returns:
A TracingAssembly carrying whichever artifacts were requested and succeeded.
"""
result = TracingAssembly()
+ result.set_run_metadata(run_metadata=run_metadata)
tracing_config = get_config().runtime.tracing
# A scoped override (see hub.scoped_event_log) is the run's transport and implies
# tracing-enabled (D1) — it must not be skipped by the is_enabled early-return.
@@ -175,6 +204,8 @@ def assemble_tracing_on_output(
domain_code=domain_code,
main_pipe_code=main_pipe_code,
run_mode=run_mode,
+ # Inherited from the output this assembly decorates, so both key together.
+ run_metadata=pipe_output.job_metadata.run_metadata if pipe_output.job_metadata else None,
)
if result.graph_spec is not None:
pipe_output.graph_spec = result.graph_spec
diff --git a/pipelex/pipe_signature/pipe_signature.py b/pipelex/pipe_signature/pipe_signature.py
index 509c6b213..4235454c1 100644
--- a/pipelex/pipe_signature/pipe_signature.py
+++ b/pipelex/pipe_signature/pipe_signature.py
@@ -107,4 +107,4 @@ class PipeSignature(PipeAbstract):
typed_named = WorkingMemoryFactory.convert_stuff_spec_to_typed_named(stuff_spec=self.output, name=mock_name)
mock_stuff = WorkingMemoryFactory.make_mock_stuff(typed_named_stuff_spec=typed_named)
working_memory.set_new_main_stuff(stuff=mock_stuff, name=mock_name)
- return PipeOutput(working_memory=working_memory, pipeline_run_id=job_metadata.pipeline_run_id)
+ return PipeOutput(working_memory=working_memory, pipeline_run_id=job_metadata.run_metadata.pipeline_run_id)
diff --git a/pipelex/pipeline/execution_seams.py b/pipelex/pipeline/execution_seams.py
index b98915fc1..3fa3ba0e3 100644
--- a/pipelex/pipeline/execution_seams.py
+++ b/pipelex/pipeline/execution_seams.py
@@ -46,7 +46,7 @@ from pipelex.pipe_run.pipe_run_params_factory import PipeRunParamsFactory
from pipelex.pipeline.blueprint_selection import select_primary_blueprint
from pipelex.pipeline.input_normalizer import normalize_data_urls_to_storage
from pipelex.system.configuration.configs import PipelineExecutionConfig
-from pipelex.system.job_metadata import JobMetadata, OtelContext
+from pipelex.system.job_metadata import JobMetadata, OtelContext, RunMetadata
from pipelex.system.pipe_run_mode import PipeRunMode
from pipelex.system.storage_scope import validate_storage_scope
from pipelex.tools.misc.file_utils import reject_bare_str_or_path
@@ -278,12 +278,9 @@ async def prepare_pipe_job(
working_memory = await normalize_data_urls_to_storage(working_memory, storage_scope=storage_scope)
job_metadata = JobMetadata(
- user_id=user_id,
- storage_scope=storage_scope,
- pipeline_run_id=pipeline_run_id,
+ run_metadata=RunMetadata(user_id=user_id, storage_scope=storage_scope, pipeline_run_id=pipeline_run_id, request_id=request_id),
otel_context=otel_context,
trace_context=trace_context,
- request_id=request_id,
)
pipe_run_params = PipeRunParamsFactory.make_run_params(
diff --git a/pipelex/reporting/reporting_manager.py b/pipelex/reporting/reporting_manager.py
index 2d2824b82..fbd77230d 100644
--- a/pipelex/reporting/reporting_manager.py
+++ b/pipelex/reporting/reporting_manager.py
@@ -296,7 +296,7 @@ class ReportingManager(ReportingProtocol):
seq = process_event_log.next_sequence()
event = UsageReportEvent(
- pipeline_run_id=inference_job.job_metadata.pipeline_run_id,
+ pipeline_run_id=inference_job.job_metadata.run_metadata.pipeline_run_id,
workflow_id=workflow_id,
writer_id=process_event_log.writer_id,
timestamp=datetime.now(UTC),
diff --git a/pipelex/system/job_metadata.py b/pipelex/system/job_metadata.py
index bbe174fa5..910d41df0 100644
--- a/pipelex/system/job_metadata.py
+++ b/pipelex/system/job_metadata.py
@@ -50,7 +50,23 @@ class UnitJobId(StrEnum):
return "Search"
-class JobMetadata(BaseModel):
+class RunMetadata(BaseModel):
+ """Who is running, which run it is, and where its bytes go.
+
+ **The four facts that are constant for a whole run**, split out from
+ :class:`JobMetadata` — which mixes them with per-job facts that change at
+ every step (``pipe_code``, ``pipe_run_id``, ``otel_context``,
+ ``content_generation_job_id``). Grouping them says which is which: a copy
+ made for a nested pipe replaces the job half and carries this half through
+ unchanged.
+
+ It also gives the run identity somewhere to live on a RESULT. A
+ :class:`~pipelex.core.pipes.pipe_output.PipeOutput` carries the
+ ``JobMetadata`` it was produced under, so a transport that offloads an
+ oversized result to storage can key it inside the run's own namespace
+ instead of at the root of a bucket.
+ """
+
user_id: str
pipeline_run_id: str
@@ -67,6 +83,40 @@ class JobMetadata(BaseModel):
# `DRY_RUN_STORAGE_SCOPE`.
storage_scope: str
+ # The API-inbound ``X-Request-ID`` (set by the dispatcher when an external
+ # HTTP request enters Pipelex). Rides here so it crosses the Temporal
+ # serialization boundary — every activity / workflow can correlate logs and
+ # the resulting ``ErrorReport`` back to the originating request. Distinct
+ # from :class:`pipelex.cogt.inference.error_classification.ProviderErrorMetadata.request_id`,
+ # which is the *provider*-side request id (OpenAI ``x-request-id`` etc.) —
+ # both can appear together when the API surfaces a provider failure.
+ # Constrained at the wire-format boundary (printable ASCII only, max 128
+ # chars) so an unsanitized upstream value cannot inject newlines or control
+ # characters into the log lines or ``ErrorReport`` envelopes that quote it.
+ request_id: str | None = Field(default=None, max_length=128, pattern=r"^[\x20-\x7E]+$")
+
+ @field_validator("storage_scope")
+ @classmethod
+ def _validate_storage_scope(cls, value: str) -> str:
+ """Refuse a scope that would escape its namespace, at construction.
+
+ On the TYPE rather than at the call sites: the value becomes a storage
+ key prefix, so a `..` or a leading slash in it is a traversal into
+ another tenant's data. Validating here means every key the runtime
+ derives is safe by construction. `model_copy` does NOT re-run
+ validators, which is fine — `copy_with_update` only ever carries an
+ already-validated scope forward, and nothing constructs a scope from
+ request data after this point.
+ """
+ return validate_storage_scope(value=value)
+
+
+class JobMetadata(BaseModel):
+ # The run-constant half: who, which run, where its bytes go, which request
+ # started it. Nested rather than flattened so a copy for a nested pipe can
+ # carry it through untouched while replacing the per-job fields below.
+ run_metadata: RunMetadata
+
pipe_code: str | None = None
# Per-process Pipelex session id (``Config.session_id``) captured at the
@@ -76,18 +126,6 @@ class JobMetadata(BaseModel):
# so cannot cause a non-determinism mismatch on child-workflow starts.
session_id: str | None = None
- # The API-inbound ``X-Request-ID`` (set by the dispatcher when an external
- # HTTP request enters Pipelex). Rides on ``JobMetadata`` so it crosses the
- # Temporal serialization boundary — every activity / workflow can correlate
- # logs and the resulting ``ErrorReport`` back to the originating request.
- # Distinct from :class:`pipelex.cogt.inference.error_classification.ProviderErrorMetadata.request_id`,
- # which is the *provider*-side request id (OpenAI ``x-request-id`` etc.) —
- # both can appear together when the API surfaces a provider failure.
- # Constrained at the wire-format boundary (printable ASCII only, max 128
- # chars) so an unsanitized upstream value cannot inject newlines or control
- # characters into the log lines or ``ErrorReport`` envelopes that quote it.
- request_id: str | None = Field(default=None, max_length=128, pattern=r"^[\x20-\x7E]+$")
-
# Business ID for the current pipe execution (16-char hex string).
# Always set during pipe runs for tracking purposes.
pipe_run_id: str | None = None
@@ -106,21 +144,6 @@ class JobMetadata(BaseModel):
started_at: datetime | None = Field(default_factory=datetime.now)
completed_at: datetime | None = None
- @field_validator("storage_scope")
- @classmethod
- def _validate_storage_scope(cls, value: str) -> str:
- """Refuse a scope that would escape its namespace, at construction.
-
- On the TYPE rather than at the call sites: the value becomes a storage
- key prefix, so a `..` or a leading slash in it is a traversal into
- another tenant's data. Validating here means every key the runtime
- derives is safe by construction. `model_copy` does NOT re-run
- validators, which is fine — `copy_with_update` only ever carries an
- already-validated scope forward, and nothing constructs a scope from
- request data after this point.
- """
- return validate_storage_scope(value=value)
-
@property
def duration(self) -> float | None:
if self.started_at is not None and self.completed_at is not None:
diff --git a/tests/conftest.py b/tests/conftest.py
index a65c77cb3..050a28d0d 100644
--- a/tests/conftest.py
+++ b/tests/conftest.py
@@ -9,7 +9,7 @@ from pytest_mock import MockerFixture
from pipelex import log
from pipelex.interpreter_hub import clear_current_library, get_current_library_id_or_none, get_library_manager, set_current_library
from pipelex.pipelex import Pipelex
-from pipelex.system.job_metadata import JobMetadata
+from pipelex.system.job_metadata import JobMetadata, RunMetadata
from pipelex.system.pipelex_service.pipelex_service_config import (
PipelexServiceConfig,
)
@@ -198,8 +198,4 @@ def job_metadata(request: pytest.FixtureRequest) -> JobMetadata:
random_code: str = shortuuid.uuid()[:5]
pipeline_run_id: str = f"{test_id}-{random_code}"
- return JobMetadata(
- storage_scope="test/scope",
- user_id="pytest",
- pipeline_run_id=pipeline_run_id,
- )
+ return JobMetadata(run_metadata=RunMetadata(storage_scope="test/scope", user_id="pytest", pipeline_run_id=pipeline_run_id))
diff --git a/tests/integration/pipelex/cogt/content_generation/test_leaf_dry_object_mocks.py b/tests/integration/pipelex/cogt/content_generation/test_leaf_dry_object_mocks.py
index 00cb01e79..59eabbffd 100644
--- a/tests/integration/pipelex/cogt/content_generation/test_leaf_dry_object_mocks.py
+++ b/tests/integration/pipelex/cogt/content_generation/test_leaf_dry_object_mocks.py
@@ -34,7 +34,7 @@ from pipelex.cogt.llm.llm_setting import LLMSetting
from pipelex.cogt.search.search_setting import SearchSetting
from pipelex.config import get_config
from pipelex.core.stuffs.structured_content import StructuredContent
-from pipelex.system.job_metadata import JobMetadata
+from pipelex.system.job_metadata import JobMetadata, RunMetadata
from pipelex.system.pipe_run_mode import PipeRunMode
from tests.integration.pipelex.cogt.content_generation.test_data import ConstrainedName
@@ -89,7 +89,7 @@ class SpecWithPipeCode(StructuredContent):
def _dry_object_assignment(object_class: type[StructuredContent], nb_items: int | None = None) -> ObjectAssignment:
llm_assignment = LLMAssignment(
- job_metadata=JobMetadata(storage_scope="test/scope", user_id="u", pipeline_run_id="run_dry_objects"),
+ job_metadata=JobMetadata(run_metadata=RunMetadata(storage_scope="test/scope", user_id="u", pipeline_run_id="run_dry_objects")),
cogt_run_params=CogtRunParams(run_mode=PipeRunMode.DRY),
llm_setting=LLMSetting(model="gpt-4o", temperature=0.5),
llm_prompt=LLMPrompt(user_text="make objects"),
@@ -130,7 +130,7 @@ class TestLeafDryObjectMocks:
async def test_dry_object_list_stamps_regardless_of_mock_usage(self) -> None:
"""The stamp is unconditional on the is_mock_usage sub-flag — it only changes reporting (D3)."""
llm_assignment = LLMAssignment(
- job_metadata=JobMetadata(storage_scope="test/scope", user_id="u", pipeline_run_id="run_mock_usage_stamp"),
+ job_metadata=JobMetadata(run_metadata=RunMetadata(storage_scope="test/scope", user_id="u", pipeline_run_id="run_mock_usage_stamp")),
cogt_run_params=CogtRunParams(run_mode=PipeRunMode.DRY, is_mock_usage=True),
llm_setting=LLMSetting(model="gpt-4o", temperature=0.5),
llm_prompt=LLMPrompt(user_text="make specs"),
@@ -232,7 +232,7 @@ class TestLeafDryObjectMocks:
async def test_boundary_dry_search_structured_dict_validates_against_original_class(self) -> None:
"""The structured-search dry boundary leaf returns a dict the original output class accepts."""
search_assignment = SearchAssignment(
- job_metadata=JobMetadata(storage_scope="test/scope", user_id="u", pipeline_run_id="run_dry_search_structured"),
+ job_metadata=JobMetadata(run_metadata=RunMetadata(storage_scope="test/scope", user_id="u", pipeline_run_id="run_dry_search_structured")),
cogt_run_params=CogtRunParams(run_mode=PipeRunMode.DRY),
query="what is pipelex?",
search_setting=SearchSetting(model="mock-search-handle"),
diff --git a/tests/integration/pipelex/cogt/content_generation/test_object_class_passthrough.py b/tests/integration/pipelex/cogt/content_generation/test_object_class_passthrough.py
index 4e0585f61..1ce453ec2 100644
--- a/tests/integration/pipelex/cogt/content_generation/test_object_class_passthrough.py
+++ b/tests/integration/pipelex/cogt/content_generation/test_object_class_passthrough.py
@@ -33,7 +33,7 @@ from pipelex.cogt.llm.llm_prompt import LLMPrompt
from pipelex.cogt.llm.llm_setting import LLMSetting
from pipelex.cogt.search.search_setting import SearchSetting
from pipelex.core.stuffs.structured_content import StructuredContent
-from pipelex.system.job_metadata import JobMetadata
+from pipelex.system.job_metadata import JobMetadata, RunMetadata
from pipelex.system.pipe_run_mode import PipeRunMode
@@ -80,7 +80,7 @@ class _StubListResponse(BaseModel):
def _live_object_assignment() -> ObjectAssignment:
llm_assignment = LLMAssignment(
- job_metadata=JobMetadata(storage_scope="test/scope", user_id="u", pipeline_run_id="run_object_class_passthrough"),
+ job_metadata=JobMetadata(run_metadata=RunMetadata(storage_scope="test/scope", user_id="u", pipeline_run_id="run_object_class_passthrough")),
cogt_run_params=CogtRunParams(run_mode=PipeRunMode.LIVE),
llm_setting=LLMSetting(model="test-model", temperature=0.5),
llm_prompt=LLMPrompt(user_text="make a name"),
@@ -90,7 +90,9 @@ def _live_object_assignment() -> ObjectAssignment:
def _dry_object_assignment() -> ObjectAssignment:
llm_assignment = LLMAssignment(
- job_metadata=JobMetadata(storage_scope="test/scope", user_id="u", pipeline_run_id="run_object_class_passthrough_dry"),
+ job_metadata=JobMetadata(
+ run_metadata=RunMetadata(storage_scope="test/scope", user_id="u", pipeline_run_id="run_object_class_passthrough_dry")
+ ),
cogt_run_params=CogtRunParams(run_mode=PipeRunMode.DRY),
llm_setting=LLMSetting(model="test-model", temperature=0.5),
llm_prompt=LLMPrompt(user_text="make a name"),
@@ -100,7 +102,7 @@ def _dry_object_assignment() -> ObjectAssignment:
def _live_search_assignment() -> SearchAssignment:
return SearchAssignment(
- job_metadata=JobMetadata(storage_scope="test/scope", user_id="u", pipeline_run_id="run_search_class_passthrough"),
+ job_metadata=JobMetadata(run_metadata=RunMetadata(storage_scope="test/scope", user_id="u", pipeline_run_id="run_search_class_passthrough")),
cogt_run_params=CogtRunParams(run_mode=PipeRunMode.LIVE),
query="make a name",
search_setting=SearchSetting(model="mock-search-handle"),
@@ -236,7 +238,7 @@ class TestObjectClassPassthrough:
ObjectAssignment.make_for_class(
object_class=NormalizedReference,
llm_assignment=LLMAssignment(
- job_metadata=JobMetadata(storage_scope="test/scope", user_id="u", pipeline_run_id="run_wrapped"),
+ job_metadata=JobMetadata(run_metadata=RunMetadata(storage_scope="test/scope", user_id="u", pipeline_run_id="run_wrapped")),
cogt_run_params=CogtRunParams(run_mode=PipeRunMode.LIVE),
llm_setting=LLMSetting(model="test-model", temperature=0.5),
llm_prompt=LLMPrompt(user_text="make a reference"),
diff --git a/tests/integration/pipelex/cogt/test_extract.py b/tests/integration/pipelex/cogt/test_extract.py
index 0d29d2d8e..0a84de183 100644
--- a/tests/integration/pipelex/cogt/test_extract.py
+++ b/tests/integration/pipelex/cogt/test_extract.py
@@ -41,7 +41,7 @@ class TestExtract:
extract_output = await extract_worker.extract_pages(extract_job=extract_job)
assert extract_output.pages
page_contents = await generated_content_factory.make_page_contents(
- storage_scope=job_metadata.storage_scope,
+ storage_scope=job_metadata.run_metadata.storage_scope,
extract_output=extract_output,
)
assert page_contents
@@ -156,7 +156,7 @@ class TestExtract:
if page.raw_html:
pretty_print(page.raw_html[:500], title="Raw HTML (first 500 chars)")
page_contents = await generated_content_factory.make_page_contents(
- storage_scope=job_metadata.storage_scope,
+ storage_scope=job_metadata.run_metadata.storage_scope,
extract_output=extract_output,
)
assert page_contents
diff --git a/tests/integration/pipelex/cogt/test_llm_generate_security.py b/tests/integration/pipelex/cogt/test_llm_generate_security.py
index e9dfffc52..e47854c37 100644
--- a/tests/integration/pipelex/cogt/test_llm_generate_security.py
+++ b/tests/integration/pipelex/cogt/test_llm_generate_security.py
@@ -15,13 +15,13 @@ from pipelex.cogt.content_generation.exceptions import UnsafeSchemaError
from pipelex.cogt.content_generation.llm_generate import llm_gen_object, llm_gen_object_list
from pipelex.cogt.llm.llm_prompt import LLMPrompt
from pipelex.cogt.llm.llm_setting import LLMSetting
-from pipelex.system.job_metadata import JobMetadata
+from pipelex.system.job_metadata import JobMetadata, RunMetadata
from pipelex.system.pipe_run_mode import PipeRunMode
def _make_stub_llm_assignment() -> LLMAssignment:
return LLMAssignment(
- job_metadata=JobMetadata(storage_scope="test/scope", user_id="test-user", pipeline_run_id="test-run"),
+ job_metadata=JobMetadata(run_metadata=RunMetadata(storage_scope="test/scope", user_id="test-user", pipeline_run_id="test-run")),
cogt_run_params=CogtRunParams(run_mode=PipeRunMode.LIVE),
llm_setting=LLMSetting(model="test-model", temperature=0.7),
llm_prompt=LLMPrompt(user_text="test prompt"),
diff --git a/tests/integration/pipelex/fixtures/pipe_job_helpers.py b/tests/integration/pipelex/fixtures/pipe_job_helpers.py
index ee5a805c0..a51e866ce 100644
--- a/tests/integration/pipelex/fixtures/pipe_job_helpers.py
+++ b/tests/integration/pipelex/fixtures/pipe_job_helpers.py
@@ -12,7 +12,7 @@ from pipelex.pipe_run.pipe_job import PipeJob
from pipelex.pipe_run.pipe_job_factory import PipeJobFactory
from pipelex.pipe_run.pipe_run_params import PipeRunParams
from pipelex.pipe_run.pipe_run_params_factory import PipeRunParamsFactory
-from pipelex.system.job_metadata import JobMetadata, SpecialPipelineId
+from pipelex.system.job_metadata import JobMetadata, RunMetadata, SpecialPipelineId
from pipelex.system.pipe_run_mode import PipeRunMode
from pipelex.system.storage_scope import DRY_RUN_USER_ID
@@ -49,7 +49,9 @@ def build_pipe_job(
return PipeJobFactory.make_pipe_job(
pipe=pipe,
pipe_run_params=PipeRunParamsFactory.make_run_params(pipe_run_mode=pipe_run_mode),
- job_metadata=JobMetadata(storage_scope="test/scope", user_id=DRY_RUN_USER_ID, pipeline_run_id=resolved_pipeline_run_id),
+ job_metadata=JobMetadata(
+ run_metadata=RunMetadata(storage_scope="test/scope", user_id=DRY_RUN_USER_ID, pipeline_run_id=resolved_pipeline_run_id)
+ ),
working_memory=WorkingMemoryFactory.make_empty(),
library_crate=library_crate,
)
diff --git a/tests/integration/pipelex/pipe_signature/test_pipe_signature.py b/tests/integration/pipelex/pipe_signature/test_pipe_signature.py
index 196ab5585..4defc9471 100644
--- a/tests/integration/pipelex/pipe_signature/test_pipe_signature.py
+++ b/tests/integration/pipelex/pipe_signature/test_pipe_signature.py
@@ -11,7 +11,7 @@ from pipelex.pipe_signature.exceptions import PipeSignatureNotExecutableError
from pipelex.pipe_signature.pipe_signature import PipeSignature
from pipelex.pipe_signature.pipe_signature_blueprint import PipeSignatureBlueprint
from pipelex.pipeline.bundle_validator import BundleValidator, DryRunStatus
-from pipelex.system.job_metadata import JobMetadata, SpecialPipelineId
+from pipelex.system.job_metadata import JobMetadata, RunMetadata, SpecialPipelineId
from pipelex.system.pipe_run_mode import PipeRunMode
from pipelex.system.storage_scope import DRY_RUN_USER_ID
from tests.integration.pipelex.pipe_signature.conftest import SIGNATURES_DOMAIN_CODE
@@ -101,7 +101,9 @@ class TestPipeSignature:
needed = WorkingMemoryFactory.convert_to_working_memory_format(needed_inputs_spec=runtime.needed_inputs())
working_memory = WorkingMemoryFactory.make_mock_inputs(needed_inputs=needed)
await runtime.run_pipe(
- job_metadata=JobMetadata(storage_scope="test/scope", user_id=DRY_RUN_USER_ID, pipeline_run_id=SpecialPipelineId.DRY_RUN_UNTITLED),
+ job_metadata=JobMetadata(
+ run_metadata=RunMetadata(storage_scope="test/scope", user_id=DRY_RUN_USER_ID, pipeline_run_id=SpecialPipelineId.DRY_RUN_UNTITLED)
+ ),
working_memory=working_memory,
pipe_run_params=PipeRunParamsFactory.make_run_params(pipe_run_mode=PipeRunMode.DRY),
)
@@ -120,7 +122,9 @@ class TestPipeSignature:
needed = WorkingMemoryFactory.convert_to_working_memory_format(needed_inputs_spec=runtime.needed_inputs())
working_memory = WorkingMemoryFactory.make_mock_inputs(needed_inputs=needed)
await runtime.run_pipe(
- job_metadata=JobMetadata(storage_scope="test/scope", user_id=DRY_RUN_USER_ID, pipeline_run_id=SpecialPipelineId.DRY_RUN_UNTITLED),
+ job_metadata=JobMetadata(
+ run_metadata=RunMetadata(storage_scope="test/scope", user_id=DRY_RUN_USER_ID, pipeline_run_id=SpecialPipelineId.DRY_RUN_UNTITLED)
+ ),
working_memory=working_memory,
pipe_run_params=PipeRunParamsFactory.make_run_params(pipe_run_mode=PipeRunMode.DRY),
)
@@ -141,7 +145,9 @@ class TestPipeSignature:
working_memory = WorkingMemoryFactory.make_empty()
with pytest.raises(PipeSignatureNotExecutableError) as exc_info:
await runtime._live_run_pipe( # noqa: SLF001 # pyright: ignore[reportPrivateUsage]
- job_metadata=JobMetadata(storage_scope="test/scope", user_id=DRY_RUN_USER_ID, pipeline_run_id=SpecialPipelineId.UNTITLED),
+ job_metadata=JobMetadata(
+ run_metadata=RunMetadata(storage_scope="test/scope", user_id=DRY_RUN_USER_ID, pipeline_run_id=SpecialPipelineId.UNTITLED)
+ ),
working_memory=working_memory,
pipe_run_params=PipeRunParamsFactory.make_run_params(pipe_run_mode=PipeRunMode.LIVE),
)
@@ -162,7 +168,9 @@ class TestPipeSignature:
working_memory = WorkingMemoryFactory.make_empty()
with pytest.raises(PipeSignatureNotExecutableError) as exc_info:
await runtime.run_pipe(
- job_metadata=JobMetadata(storage_scope="test/scope", user_id=DRY_RUN_USER_ID, pipeline_run_id=SpecialPipelineId.UNTITLED),
+ job_metadata=JobMetadata(
+ run_metadata=RunMetadata(storage_scope="test/scope", user_id=DRY_RUN_USER_ID, pipeline_run_id=SpecialPipelineId.UNTITLED)
+ ),
working_memory=working_memory,
pipe_run_params=PipeRunParamsFactory.make_run_params(pipe_run_mode=PipeRunMode.LIVE),
)
diff --git a/tests/integration/pipelex/pipeline/test_execution_seams.py b/tests/integration/pipelex/pipeline/test_execution_seams.py
index bc373e800..76ed78840 100644
--- a/tests/integration/pipelex/pipeline/test_execution_seams.py
+++ b/tests/integration/pipelex/pipeline/test_execution_seams.py
@@ -153,7 +153,7 @@ class TestExecutionSeams:
)
assert pipe_job.pipe.code == "echo_topic"
assert pipe_job.pipe_run_params.run_mode.is_dry
- assert pipe_job.job_metadata.pipeline_run_id == "seams-prepare-run-id"
+ assert pipe_job.job_metadata.run_metadata.pipeline_run_id == "seams-prepare-run-id"
assert pipe_job.library_crate is not None
# Mock input materialized for the pipe's needed input.
assert "subject" in pipe_job.get_working_memory().root
diff --git a/tests/integration/pipelex/pipeline/test_pipeline_run_setup_emit_gates.py b/tests/integration/pipelex/pipeline/test_pipeline_run_setup_emit_gates.py
index c0c2dd201..9fe2f8830 100644
--- a/tests/integration/pipelex/pipeline/test_pipeline_run_setup_emit_gates.py
+++ b/tests/integration/pipelex/pipeline/test_pipeline_run_setup_emit_gates.py
@@ -148,7 +148,7 @@ class TestTheLocalScopeIsPerRun:
pipe_run_mode=PipeRunMode.DRY,
)
try:
- scope = pipe_job.job_metadata.storage_scope
+ scope = pipe_job.job_metadata.run_metadata.storage_scope
# The sentinel must NOT survive onto the job — it is "no host
# supplied a scope", not a prefix, and a scope shared by every
# run is the shared-key state this repairs.
@@ -176,6 +176,6 @@ class TestTheLocalScopeIsPerRun:
pipe_run_mode=PipeRunMode.DRY,
)
try:
- assert pipe_job.job_metadata.storage_scope == "org_a/mt_b/run_c"
+ assert pipe_job.job_metadata.run_metadata.storage_scope == "org_a/mt_b/run_c"
finally:
_cleanup(pipeline_run_id, library_id)
diff --git a/tests/integration/pipelex/pipeline/test_pipeline_run_setup_request_id.py b/tests/integration/pipelex/pipeline/test_pipeline_run_setup_request_id.py
index bfa6797ec..3815b6106 100644
--- a/tests/integration/pipelex/pipeline/test_pipeline_run_setup_request_id.py
+++ b/tests/integration/pipelex/pipeline/test_pipeline_run_setup_request_id.py
@@ -2,7 +2,7 @@
onto :class:`JobMetadata`.
The ``X-Request-ID`` reaches the worker by riding the workflow input
-(``arg.pipe_job.job_metadata.request_id``); there is no ContextVar layer. This
+(``arg.pipe_job.job_metadata.run_metadata.request_id``); there is no ContextVar layer. This
test pins the dispatcher-side contract: passing ``request_id`` to
``pipeline_run_setup`` produces a :class:`PipeJob` whose ``job_metadata``
carries it.
@@ -35,7 +35,7 @@ prompt = "Echo the $subject as a topic"
@pytest.mark.asyncio(loop_scope="class")
class TestPipelineRunSetupRequestId:
async def test_request_id_threads_onto_job_metadata(self) -> None:
- """``pipeline_run_setup(..., request_id=...)`` puts the value on ``job_metadata.request_id``."""
+ """``pipeline_run_setup(..., request_id=...)`` puts the value on ``job_metadata.run_metadata.request_id``."""
execution_config = get_config().interpreter.pipeline_execution.with_execution_overrides(
generate_graph=False,
)
@@ -47,10 +47,10 @@ class TestPipelineRunSetupRequestId:
pipe_code="echo_topic",
request_id="r-pipeline-setup-abc-123",
)
- assert pipe_job.job_metadata.request_id == "r-pipeline-setup-abc-123"
+ assert pipe_job.job_metadata.run_metadata.request_id == "r-pipeline-setup-abc-123"
async def test_request_id_absent_defaults_to_none(self) -> None:
- """Omitting ``request_id`` leaves ``job_metadata.request_id`` as ``None``."""
+ """Omitting ``request_id`` leaves ``job_metadata.run_metadata.request_id`` as ``None``."""
execution_config = get_config().interpreter.pipeline_execution.with_execution_overrides(
generate_graph=False,
)
@@ -61,4 +61,4 @@ class TestPipelineRunSetupRequestId:
mthds_contents=[_MINIMAL_MTHDS],
pipe_code="echo_topic",
)
- assert pipe_job.job_metadata.request_id is None
+ assert pipe_job.job_metadata.run_metadata.request_id is None
diff --git a/tests/integration/pipelex/pipes/controller/pipe_condition/test_pipe_condition_continue_delivery.py b/tests/integration/pipelex/pipes/controller/pipe_condition/test_pipe_condition_continue_delivery.py
index 34ff6642b..ca3640266 100644
--- a/tests/integration/pipelex/pipes/controller/pipe_condition/test_pipe_condition_continue_delivery.py
+++ b/tests/integration/pipelex/pipes/controller/pipe_condition/test_pipe_condition_continue_delivery.py
@@ -88,7 +88,7 @@ class TestPipeConditionContinueDelivery:
# Run identity must survive the absence path: the continue arm builds its own PipeOutput,
# so it must stamp pipeline_run_id like every other controller — otherwise the blocking
# bridge serializes SpecialPipelineId.UNTITLED and callers track the wrong run.
- assert pipe_output.pipeline_run_id == job_metadata.pipeline_run_id
+ assert pipe_output.pipeline_run_id == job_metadata.run_metadata.pipeline_run_id
async def test_continue_supersedes_previous_main_stuff(self, job_metadata: JobMetadata, load_test_library: Callable[[list[Path]], None]):
"""A previous main stuff no longer passes through: it stays under its own name while the
@@ -114,7 +114,7 @@ class TestPipeConditionContinueDelivery:
previous = result_memory.get_optional_stuff("input_text")
assert previous is not None
assert previous.stuff_code == input_text_stuff.stuff_code
- assert pipe_output.pipeline_run_id == job_metadata.pipeline_run_id
+ assert pipe_output.pipeline_run_id == job_metadata.run_metadata.pipeline_run_id
async def test_dry_run_all_special_outcomes_records_absence(self, job_metadata: JobMetadata, load_test_library: Callable[[list[Path]], None]):
"""Dry-run parity: an all-special-outcomes condition records the declared output absent
@@ -135,7 +135,7 @@ class TestPipeConditionContinueDelivery:
assert isinstance(resolved_main, AbsenceRecord)
assert resolved_main.kind == AbsenceKind.DECLARED_ABSENT
assert resolved_main.producing_pipe == "continue_gate"
- assert pipe_output.pipeline_run_id == job_metadata.pipeline_run_id
+ assert pipe_output.pipeline_run_id == job_metadata.run_metadata.pipeline_run_id
async def test_dry_run_fail_only_condition_raises(self, job_metadata: JobMetadata, load_test_library: Callable[[list[Path]], None]):
"""A fail-only condition has no absent arm — every live path raises — so dry-run must
@@ -188,4 +188,4 @@ class TestPipeConditionContinueDelivery:
result_memory = pipe_output.working_memory
assert isinstance(result_memory.resolve_main_stuff(), AbsenceRecord)
assert result_memory.get_optional_stuff("input_text") is not None
- assert pipe_output.pipeline_run_id == job_metadata.pipeline_run_id
+ assert pipe_output.pipeline_run_id == job_metadata.run_metadata.pipeline_run_id
diff --git a/tests/integration/pipelex/pipes/controller/pipe_parallel/test_pipe_parallel_absence.py b/tests/integration/pipelex/pipes/controller/pipe_parallel/test_pipe_parallel_absence.py
index 15f17f41b..6c2473a3b 100644
--- a/tests/integration/pipelex/pipes/controller/pipe_parallel/test_pipe_parallel_absence.py
+++ b/tests/integration/pipelex/pipes/controller/pipe_parallel/test_pipe_parallel_absence.py
@@ -186,7 +186,7 @@ class TestPipeParallelAbsence:
assert note.kind == AbsenceKind.SKIPPED
assert note.producing_pipe == "opt_par_find"
assert note.origin().producing_pipe == "extract_penalty_clause"
- assert pipe_output.pipeline_run_id == job_metadata.pipeline_run_id
+ assert pipe_output.pipeline_run_id == job_metadata.run_metadata.pipeline_run_id
async def test_structured_output_optional_field_absorbs_as_none(self, job_metadata: JobMetadata, load_empty_library: Callable[[], str]):
"""Structured output: a non-required field absorbs the absent branch as field-level None."""
diff --git a/tests/integration/pipelex/pipes/optionals/test_lift_skip_trichotomy.py b/tests/integration/pipelex/pipes/optionals/test_lift_skip_trichotomy.py
index bedcdf7ef..f5c8ce886 100644
--- a/tests/integration/pipelex/pipes/optionals/test_lift_skip_trichotomy.py
+++ b/tests/integration/pipelex/pipes/optionals/test_lift_skip_trichotomy.py
@@ -24,7 +24,7 @@ from pipelex.pipe_machinery.pipe_factory import PipeFactory
from pipelex.pipe_operators.func.pipe_func import PipeFunc
from pipelex.pipe_operators.func.pipe_func_blueprint import PipeFuncBlueprint
from pipelex.pipe_run.pipe_run_params import PipeRunParams
-from pipelex.system.job_metadata import JobMetadata
+from pipelex.system.job_metadata import JobMetadata, RunMetadata
from pipelex.system.pipe_run_mode import PipeRunMode
from pipelex.system.registries.func_registry import func_registry
from tests.integration.pipelex.fixtures.pipe_job_helpers import make_mode_guarded_run_params
@@ -127,7 +127,7 @@ class TestLiftSkipTrichotomy:
assert isinstance(resolved_main, AbsenceRecord)
assert result_memory.get_optional_main_stuff() is None
# Run identity is stamped like on every other path.
- assert pipe_output.pipeline_run_id == job_metadata.pipeline_run_id
+ assert pipe_output.pipeline_run_id == job_metadata.run_metadata.pipeline_run_id
async def test_lift_chain_provenance(self, job_metadata: JobMetadata, load_empty_library: Callable[[], str]):
"""Two lifted pipes in a row chain their skipped-absence records back to the origin."""
@@ -295,10 +295,8 @@ class TestLiftSkipTrichotomy:
return_value=mock_manager,
)
traced_metadata = JobMetadata(
- storage_scope="test/scope",
- user_id="pytest",
- pipeline_run_id=job_metadata.pipeline_run_id,
- trace_context=make_trace_context(graph_id=job_metadata.pipeline_run_id),
+ run_metadata=RunMetadata(storage_scope="test/scope", user_id="pytest", pipeline_run_id=job_metadata.run_metadata.pipeline_run_id),
+ trace_context=make_trace_context(graph_id=job_metadata.run_metadata.pipeline_run_id),
)
pipe_output = await pipe.run_pipe(
diff --git a/tests/unit/pipelex/cogt/content_generation/test_assignment_models_schema.py b/tests/unit/pipelex/cogt/content_generation/test_assignment_models_schema.py
index 2eae63c96..7bfae4da3 100644
--- a/tests/unit/pipelex/cogt/content_generation/test_assignment_models_schema.py
+++ b/tests/unit/pipelex/cogt/content_generation/test_assignment_models_schema.py
@@ -6,7 +6,7 @@ from pipelex.cogt.content_generation.assignment_models import LLMAssignment, Obj
from pipelex.cogt.content_generation.cogt_run_params import CogtRunParams
from pipelex.cogt.llm.llm_prompt import LLMPrompt
from pipelex.cogt.llm.llm_setting import LLMSetting
-from pipelex.system.job_metadata import JobMetadata
+from pipelex.system.job_metadata import JobMetadata, RunMetadata
from pipelex.system.pipe_run_mode import PipeRunMode
@@ -16,11 +16,7 @@ class SampleOutputModel(BaseModel):
def _make_stub_job_metadata() -> JobMetadata:
- return JobMetadata(
- storage_scope="test/scope",
- user_id="test-user",
- pipeline_run_id="test-run",
- )
+ return JobMetadata(run_metadata=RunMetadata(storage_scope="test/scope", user_id="test-user", pipeline_run_id="test-run"))
def _make_stub_llm_assignment() -> LLMAssignment:
diff --git a/tests/unit/pipelex/cogt/content_generation/test_assignment_models_security.py b/tests/unit/pipelex/cogt/content_generation/test_assignment_models_security.py
index 9cd81cf5b..5559d8721 100644
--- a/tests/unit/pipelex/cogt/content_generation/test_assignment_models_security.py
+++ b/tests/unit/pipelex/cogt/content_generation/test_assignment_models_security.py
@@ -13,16 +13,12 @@ from pipelex.cogt.content_generation.exceptions import UnsafeSchemaError
from pipelex.cogt.content_generation.schema_to_model_factory import SchemaToModelFactory
from pipelex.cogt.llm.llm_prompt import LLMPrompt
from pipelex.cogt.llm.llm_setting import LLMSetting
-from pipelex.system.job_metadata import JobMetadata
+from pipelex.system.job_metadata import JobMetadata, RunMetadata
from pipelex.system.pipe_run_mode import PipeRunMode
def _make_stub_job_metadata() -> JobMetadata:
- return JobMetadata(
- storage_scope="test/scope",
- user_id="test-user",
- pipeline_run_id="test-run",
- )
+ return JobMetadata(run_metadata=RunMetadata(storage_scope="test/scope", user_id="test-user", pipeline_run_id="test-run"))
def _make_stub_llm_assignment() -> LLMAssignment:
diff --git a/tests/unit/pipelex/cogt/content_generation/test_dry_mock.py b/tests/unit/pipelex/cogt/content_generation/test_dry_mock.py
index 24294610a..9f4c4a8fd 100644
--- a/tests/unit/pipelex/cogt/content_generation/test_dry_mock.py
+++ b/tests/unit/pipelex/cogt/content_generation/test_dry_mock.py
@@ -25,13 +25,13 @@ from pipelex.cogt.llm.llm_prompt import LLMPrompt
from pipelex.cogt.llm.llm_setting import LLMSetting
from pipelex.cogt.usage.cost_registry import CostRegistry
from pipelex.core.stuffs.text_content import TextContent
-from pipelex.system.job_metadata import JobMetadata
+from pipelex.system.job_metadata import JobMetadata, RunMetadata
from pipelex.system.pipe_run_mode import PipeRunMode
def _dry_object_assignment() -> ObjectAssignment:
llm_assignment = LLMAssignment(
- job_metadata=JobMetadata(storage_scope="test/scope", user_id="u", pipeline_run_id="run_dry_mock"),
+ job_metadata=JobMetadata(run_metadata=RunMetadata(storage_scope="test/scope", user_id="u", pipeline_run_id="run_dry_mock")),
cogt_run_params=CogtRunParams(run_mode=PipeRunMode.DRY),
llm_setting=LLMSetting(model="gpt-4o", temperature=0.5),
llm_prompt=LLMPrompt(),
@@ -56,7 +56,7 @@ class TestDryMock:
"""report_mock_usage_llm_job emits non-zero usage under the mock_usage sentinel -> reportable."""
captured = self._capture_reported_job(mocker)
report_mock_usage_llm_job(
- job_metadata=JobMetadata(storage_scope="test/scope", user_id="u", pipeline_run_id="run_mock"),
+ job_metadata=JobMetadata(run_metadata=RunMetadata(storage_scope="test/scope", user_id="u", pipeline_run_id="run_mock")),
llm_setting=LLMSetting(model="gpt-4o", temperature=0.5),
llm_prompt=LLMPrompt(),
)
@@ -73,7 +73,7 @@ class TestDryMock:
"""report_dry_llm_job stays zero-token under the dry_run sentinel -> NOT reportable (suppressed)."""
captured = self._capture_reported_job(mocker)
report_dry_llm_job(
- job_metadata=JobMetadata(storage_scope="test/scope", user_id="u", pipeline_run_id="run_dry"),
+ job_metadata=JobMetadata(run_metadata=RunMetadata(storage_scope="test/scope", user_id="u", pipeline_run_id="run_dry")),
llm_setting=LLMSetting(model="gpt-4o", temperature=0.5),
llm_prompt=LLMPrompt(),
)
diff --git a/tests/unit/pipelex/cogt/content_generation/test_dry_report_tz.py b/tests/unit/pipelex/cogt/content_generation/test_dry_report_tz.py
index 7b17455b1..a7f0262a9 100644
--- a/tests/unit/pipelex/cogt/content_generation/test_dry_report_tz.py
+++ b/tests/unit/pipelex/cogt/content_generation/test_dry_report_tz.py
@@ -16,7 +16,7 @@ from pipelex.cogt.content_generation.dry_mock import report_dry_llm_job
from pipelex.cogt.llm.llm_job import LLMJob
from pipelex.cogt.llm.llm_prompt import LLMPrompt
from pipelex.cogt.llm.llm_setting import LLMSetting
-from pipelex.system.job_metadata import JobMetadata
+from pipelex.system.job_metadata import JobMetadata, RunMetadata
class TestDryReportTimezone:
@@ -47,10 +47,7 @@ class TestDryReportTimezone:
)
job_metadata = JobMetadata(
- storage_scope="test/scope",
- user_id="test_user",
- pipeline_run_id="run_dry_tz",
- started_at=started_at,
+ run_metadata=RunMetadata(storage_scope="test/scope", user_id="test_user", pipeline_run_id="run_dry_tz"), started_at=started_at
)
report_dry_llm_job(
diff --git a/tests/unit/pipelex/cogt/content_generation/test_extract_generate_dry_branch.py b/tests/unit/pipelex/cogt/content_generation/test_extract_generate_dry_branch.py
index 29f0de896..188e8e3b4 100644
--- a/tests/unit/pipelex/cogt/content_generation/test_extract_generate_dry_branch.py
+++ b/tests/unit/pipelex/cogt/content_generation/test_extract_generate_dry_branch.py
@@ -15,14 +15,14 @@ from pipelex.cogt.content_generation.extract_generate import extract_gen_pages_a
from pipelex.cogt.extract.extract_input import ExtractInput
from pipelex.cogt.extract.extract_job_components import ExtractJobConfig, ExtractJobParams
from pipelex.config import get_config
-from pipelex.system.job_metadata import JobMetadata
+from pipelex.system.job_metadata import JobMetadata, RunMetadata
from pipelex.system.pipe_run_mode import PipeRunMode
class TestExtractGenerateDryBranch:
def _assignment(self, *, run_mode: PipeRunMode, extract_input: ExtractInput) -> ExtractAssignment:
return ExtractAssignment(
- job_metadata=JobMetadata(storage_scope="test/scope", user_id="u", pipeline_run_id="run_extract_dry"),
+ job_metadata=JobMetadata(run_metadata=RunMetadata(storage_scope="test/scope", user_id="u", pipeline_run_id="run_extract_dry")),
cogt_run_params=CogtRunParams(run_mode=run_mode),
extract_handle="mock-extract-handle",
extract_input=extract_input,
diff --git a/tests/unit/pipelex/cogt/content_generation/test_img_gen_generate_dry_branch.py b/tests/unit/pipelex/cogt/content_generation/test_img_gen_generate_dry_branch.py
index 53c698b77..fd6b9503c 100644
--- a/tests/unit/pipelex/cogt/content_generation/test_img_gen_generate_dry_branch.py
+++ b/tests/unit/pipelex/cogt/content_generation/test_img_gen_generate_dry_branch.py
@@ -14,14 +14,14 @@ from pipelex.cogt.content_generation.cogt_run_params import CogtRunParams
from pipelex.cogt.content_generation.img_gen_generate import img_gen_image_list_and_store, img_gen_single_image_and_store
from pipelex.cogt.img_gen.img_gen_job_components import AspectRatio, Background, ImgGenJobConfig, ImgGenJobParams
from pipelex.cogt.img_gen.img_gen_prompt import ImgGenPrompt
-from pipelex.system.job_metadata import JobMetadata
+from pipelex.system.job_metadata import JobMetadata, RunMetadata
from pipelex.system.pipe_run_mode import PipeRunMode
class TestImgGenGenerateDryBranch:
def _assignment(self, *, run_mode: PipeRunMode, nb_images: int = 1) -> ImgGenAssignment:
return ImgGenAssignment(
- job_metadata=JobMetadata(storage_scope="test/scope", user_id="u", pipeline_run_id="run_img_dry"),
+ job_metadata=JobMetadata(run_metadata=RunMetadata(storage_scope="test/scope", user_id="u", pipeline_run_id="run_img_dry")),
cogt_run_params=CogtRunParams(run_mode=run_mode),
img_gen_handle="mock-img-handle",
img_gen_prompt=ImgGenPrompt(positive_text="a red apple", negative_text="no worms"),
diff --git a/tests/unit/pipelex/cogt/content_generation/test_llm_generate_dry_branch.py b/tests/unit/pipelex/cogt/content_generation/test_llm_generate_dry_branch.py
index 2ac244d98..a26623818 100644
--- a/tests/unit/pipelex/cogt/content_generation/test_llm_generate_dry_branch.py
+++ b/tests/unit/pipelex/cogt/content_generation/test_llm_generate_dry_branch.py
@@ -22,14 +22,14 @@ from pipelex.cogt.content_generation.llm_generate import llm_gen_text
from pipelex.cogt.llm.llm_prompt import LLMPrompt
from pipelex.cogt.llm.llm_setting import LLMSetting
from pipelex.cogt.usage.cost_registry import CostRegistry
-from pipelex.system.job_metadata import JobMetadata
+from pipelex.system.job_metadata import JobMetadata, RunMetadata
from pipelex.system.pipe_run_mode import PipeRunMode
class TestLlmGenerateDryBranch:
def _assignment(self, *, run_mode: PipeRunMode, is_mock_usage: bool = False) -> LLMAssignment:
return LLMAssignment(
- job_metadata=JobMetadata(storage_scope="test/scope", user_id="u", pipeline_run_id="run_dry_branch"),
+ job_metadata=JobMetadata(run_metadata=RunMetadata(storage_scope="test/scope", user_id="u", pipeline_run_id="run_dry_branch")),
cogt_run_params=CogtRunParams(run_mode=run_mode, is_mock_usage=is_mock_usage),
llm_setting=LLMSetting(model="gpt-4o", temperature=0.5),
llm_prompt=LLMPrompt(),
diff --git a/tests/unit/pipelex/cogt/content_generation/test_render_generate_dry_branch.py b/tests/unit/pipelex/cogt/content_generation/test_render_generate_dry_branch.py
index e2f9976aa..d572de423 100644
--- a/tests/unit/pipelex/cogt/content_generation/test_render_generate_dry_branch.py
+++ b/tests/unit/pipelex/cogt/content_generation/test_render_generate_dry_branch.py
@@ -11,14 +11,14 @@ from pipelex.cogt.content_generation.assignment_models import RenderPageViewsAss
from pipelex.cogt.content_generation.cogt_run_params import CogtRunParams
from pipelex.cogt.content_generation.render_generate import render_page_views_and_store
from pipelex.config import get_config
-from pipelex.system.job_metadata import JobMetadata
+from pipelex.system.job_metadata import JobMetadata, RunMetadata
from pipelex.system.pipe_run_mode import PipeRunMode
class TestRenderGenerateDryBranch:
def _assignment(self, *, run_mode: PipeRunMode) -> RenderPageViewsAssignment:
return RenderPageViewsAssignment(
- job_metadata=JobMetadata(storage_scope="test/scope", user_id="u", pipeline_run_id="run_render_dry"),
+ job_metadata=JobMetadata(run_metadata=RunMetadata(storage_scope="test/scope", user_id="u", pipeline_run_id="run_render_dry")),
cogt_run_params=CogtRunParams(run_mode=run_mode),
document_uri="file:///tmp/doc.pdf",
page_views_dpi=72,
diff --git a/tests/unit/pipelex/cogt/content_generation/test_search_generate_dry_branch.py b/tests/unit/pipelex/cogt/content_generation/test_search_generate_dry_branch.py
index 877e87c8e..6249e3b5a 100644
--- a/tests/unit/pipelex/cogt/content_generation/test_search_generate_dry_branch.py
+++ b/tests/unit/pipelex/cogt/content_generation/test_search_generate_dry_branch.py
@@ -15,14 +15,14 @@ from pipelex.cogt.content_generation.search_generate import search_gen_sourced_a
from pipelex.cogt.search.search_setting import SearchSetting
from pipelex.config import get_config
from pipelex.core.stuffs.search_result_content import SearchResultContent
-from pipelex.system.job_metadata import JobMetadata
+from pipelex.system.job_metadata import JobMetadata, RunMetadata
from pipelex.system.pipe_run_mode import PipeRunMode
class TestSearchGenerateDryBranch:
def _assignment(self, *, run_mode: PipeRunMode) -> SearchAssignment:
return SearchAssignment(
- job_metadata=JobMetadata(storage_scope="test/scope", user_id="u", pipeline_run_id="run_search_dry"),
+ job_metadata=JobMetadata(run_metadata=RunMetadata(storage_scope="test/scope", user_id="u", pipeline_run_id="run_search_dry")),
cogt_run_params=CogtRunParams(run_mode=run_mode),
query="what is pipelex?",
search_setting=SearchSetting(model="mock-search-handle"),
diff --git a/tests/unit/pipelex/cogt/content_generation/test_templating_generate_dry_branch.py b/tests/unit/pipelex/cogt/content_generation/test_templating_generate_dry_branch.py
index d178e0cb2..027b785e7 100644
--- a/tests/unit/pipelex/cogt/content_generation/test_templating_generate_dry_branch.py
+++ b/tests/unit/pipelex/cogt/content_generation/test_templating_generate_dry_branch.py
@@ -11,7 +11,7 @@ from pytest_mock import MockerFixture
from pipelex.cogt.content_generation.assignment_models import TemplatingAssignment
from pipelex.cogt.content_generation.cogt_run_params import CogtRunParams
from pipelex.cogt.content_generation.templating_generate import templating_gen_text
-from pipelex.system.job_metadata import JobMetadata
+from pipelex.system.job_metadata import JobMetadata, RunMetadata
from pipelex.system.pipe_run_mode import PipeRunMode
from pipelex.tools.jinja2.exceptions import Jinja2TemplateSyntaxError
from pipelex.tools.jinja2.template_category import TemplateCategory
@@ -20,7 +20,7 @@ from pipelex.tools.jinja2.template_category import TemplateCategory
class TestTemplatingGenerateDryBranch:
def _assignment(self, *, run_mode: PipeRunMode, template: str) -> TemplatingAssignment:
return TemplatingAssignment(
- job_metadata=JobMetadata(storage_scope="test/scope", user_id="u", pipeline_run_id="run_templating_dry"),
+ job_metadata=JobMetadata(run_metadata=RunMetadata(storage_scope="test/scope", user_id="u", pipeline_run_id="run_templating_dry")),
cogt_run_params=CogtRunParams(run_mode=run_mode),
context={"the_answer": "42"},
template=template,
diff --git a/tests/unit/pipelex/cogt/extract/test_extract_worker_usage_capture.py b/tests/unit/pipelex/cogt/extract/test_extract_worker_usage_capture.py
index af0e87029..1cec28067 100644
--- a/tests/unit/pipelex/cogt/extract/test_extract_worker_usage_capture.py
+++ b/tests/unit/pipelex/cogt/extract/test_extract_worker_usage_capture.py
@@ -23,7 +23,7 @@ from pipelex.cogt.model_backends.model_spec import InferenceModelSpec
from pipelex.cogt.model_backends.model_type import ModelType
from pipelex.cogt.usage.cost_category import CostCategory
from pipelex.cogt.usage.token_category import NbTokensByCategoryDict, TokenCategory
-from pipelex.system.job_metadata import JobMetadata
+from pipelex.system.job_metadata import JobMetadata, RunMetadata
_PER_PAGE_TOKENS = 1_000_000
@@ -65,7 +65,7 @@ def _make_extract_job() -> ExtractJob:
job_params=ExtractJobParams.make_default_extract_job_params(),
job_config=ExtractJobConfig(),
job_report=ExtractJobReport(),
- job_metadata=JobMetadata(storage_scope="test/scope", user_id="test-user", pipeline_run_id="test-run"),
+ job_metadata=JobMetadata(run_metadata=RunMetadata(storage_scope="test/scope", user_id="test-user", pipeline_run_id="test-run")),
)
diff --git a/tests/unit/pipelex/cogt/img_gen/conftest.py b/tests/unit/pipelex/cogt/img_gen/conftest.py
index 23f77f258..9e077719c 100644
--- a/tests/unit/pipelex/cogt/img_gen/conftest.py
+++ b/tests/unit/pipelex/cogt/img_gen/conftest.py
@@ -15,7 +15,7 @@ from pipelex.cogt.img_gen.img_gen_job_components import (
Quality,
)
from pipelex.cogt.img_gen.img_gen_prompt import ImgGenPrompt
-from pipelex.system.job_metadata import JobMetadata
+from pipelex.system.job_metadata import JobMetadata, RunMetadata
from pipelex.tools.misc.image_utils import ImageFormat
if TYPE_CHECKING:
@@ -62,5 +62,5 @@ def make_img_gen_job(
),
job_config=ImgGenJobConfig(is_sync_mode=False),
job_report=ImgGenJobReport(),
- job_metadata=JobMetadata(storage_scope="test/scope", user_id="test-user", pipeline_run_id="test-run"),
+ job_metadata=JobMetadata(run_metadata=RunMetadata(storage_scope="test/scope", user_id="test-user", pipeline_run_id="test-run")),
)
diff --git a/tests/unit/pipelex/cogt/img_gen/test_img_gen_args_factory.py b/tests/unit/pipelex/cogt/img_gen/test_img_gen_args_factory.py
index 226f9bd21..444de1d5b 100644
--- a/tests/unit/pipelex/cogt/img_gen/test_img_gen_args_factory.py
+++ b/tests/unit/pipelex/cogt/img_gen/test_img_gen_args_factory.py
@@ -46,7 +46,7 @@ from pipelex.cogt.model_backends.model_spec import InferenceModelSpec
from pipelex.cogt.model_backends.model_type import ModelType
from pipelex.cogt.usage.cost_category import CostCategory
from pipelex.providers.openai.openai_img_gen_worker import OpenAIImgGenWorker
-from pipelex.system.job_metadata import JobMetadata
+from pipelex.system.job_metadata import JobMetadata, RunMetadata
from pipelex.tools.misc.filetype_utils import FileType
from pipelex.tools.misc.image_utils import ImageFormat
from pipelex.tools.uri.prepared_file import PreparedFileBase64
@@ -81,7 +81,7 @@ class TestImgGenArgsFactory:
),
job_config=ImgGenJobConfig(is_sync_mode=False),
job_report=ImgGenJobReport(),
- job_metadata=JobMetadata(storage_scope="test/scope", user_id="test-user", pipeline_run_id="test-run"),
+ job_metadata=JobMetadata(run_metadata=RunMetadata(storage_scope="test/scope", user_id="test-user", pipeline_run_id="test-run")),
)
@staticmethod
diff --git a/tests/unit/pipelex/cogt/usage/test_data.py b/tests/unit/pipelex/cogt/usage/test_data.py
index 3b7233380..48165e2f3 100644
--- a/tests/unit/pipelex/cogt/usage/test_data.py
+++ b/tests/unit/pipelex/cogt/usage/test_data.py
@@ -7,7 +7,7 @@ from pipelex.cogt.search.search_report import SearchTokensUsage
from pipelex.cogt.usage.cost_category import CostCategory, CostsByCategoryDict
from pipelex.cogt.usage.token_category import NbTokensByCategoryDict, TokenCategory
from pipelex.reporting.reporting_types import AnyTokensUsage
-from pipelex.system.job_metadata import JobCategory, JobMetadata, UnitJobId
+from pipelex.system.job_metadata import JobCategory, JobMetadata, RunMetadata, UnitJobId
RATED_NB_TOKENS: NbTokensByCategoryDict = {
TokenCategory.INPUT: 1000,
@@ -36,12 +36,9 @@ class UsageFixtures:
def full_job_metadata(*, unit_job_id: UnitJobId, job_category: JobCategory) -> JobMetadata:
"""A JobMetadata with EVERY optional field populated, so leak tests prove the trim."""
return JobMetadata(
- storage_scope="test/scope",
- user_id="user-42",
- pipeline_run_id="plr-fixture",
+ run_metadata=RunMetadata(storage_scope="test/scope", user_id="user-42", pipeline_run_id="plr-fixture", request_id="req-123"),
pipe_code="analyze_contract",
session_id="session-abc",
- request_id="req-123",
pipe_run_id="a1b2c3d4e5f60718",
content_generation_job_id="cgj-9",
unit_job_id=unit_job_id,
diff --git a/tests/unit/pipelex/kernel/test_pipelex_kernel_run_state.py b/tests/unit/pipelex/kernel/test_pipelex_kernel_run_state.py
index e2a40a71c..1f64f8242 100644
--- a/tests/unit/pipelex/kernel/test_pipelex_kernel_run_state.py
+++ b/tests/unit/pipelex/kernel/test_pipelex_kernel_run_state.py
@@ -57,7 +57,7 @@ class TestPipelexKernelRunState:
kernel = PipelexKernel.make(storage_scope="test/scope", run_mode=PipeRunMode.DRY, user_id="test-user", trace_context=trace_context)
- assert kernel.job_metadata.pipeline_run_id == _GRAPH_ID
+ assert kernel.job_metadata.run_metadata.pipeline_run_id == _GRAPH_ID
assert kernel.job_metadata.trace_context == trace_context
def test_without_a_trace_context_the_run_mints_its_own_id_and_traces_nothing(self) -> None:
@@ -66,7 +66,7 @@ class TestPipelexKernelRunState:
second = PipelexKernel.make(storage_scope="test/scope", run_mode=PipeRunMode.DRY, user_id="test-user")
assert first.job_metadata.trace_context is None
- assert first.job_metadata.pipeline_run_id != second.job_metadata.pipeline_run_id
+ assert first.job_metadata.run_metadata.pipeline_run_id != second.job_metadata.run_metadata.pipeline_run_id
def test_each_step_inherits_the_trace_context_and_carries_its_own_run_id(self) -> None:
"""The per-step copy: same trace context (so every step attributes to it), distinct pipe_run_id."""
@@ -77,7 +77,7 @@ class TestPipelexKernelRunState:
assert first_step.trace_context == kernel.job_metadata.trace_context
assert second_step.trace_context == kernel.job_metadata.trace_context
- assert first_step.pipeline_run_id == second_step.pipeline_run_id == _GRAPH_ID
+ assert first_step.run_metadata.pipeline_run_id == second_step.run_metadata.pipeline_run_id == _GRAPH_ID
assert first_step.pipe_run_id != second_step.pipe_run_id
# Computed fresh per step by whoever opens the span, and a kernel call opens none.
assert first_step.otel_context is None
diff --git a/tests/unit/pipelex/pipe_machinery/test_run_pipe_tracer_metadata.py b/tests/unit/pipelex/pipe_machinery/test_run_pipe_tracer_metadata.py
index c1d12e760..8f0d3a05d 100644
--- a/tests/unit/pipelex/pipe_machinery/test_run_pipe_tracer_metadata.py
+++ b/tests/unit/pipelex/pipe_machinery/test_run_pipe_tracer_metadata.py
@@ -14,7 +14,7 @@ from pipelex.pipe_machinery.pipe_factory import PipeFactory
from pipelex.pipe_operators.llm.pipe_llm import PipeLLM
from pipelex.pipe_operators.llm.pipe_llm_blueprint import PipeLLMBlueprint
from pipelex.pipe_run.pipe_run_params_factory import PipeRunParamsFactory
-from pipelex.system.job_metadata import JobMetadata
+from pipelex.system.job_metadata import JobMetadata, RunMetadata
from tests.unit.pipelex.graph.conftest import make_trace_context
@@ -66,9 +66,7 @@ class TestRunPipeForwardsTracerMetadata:
)
job_metadata = JobMetadata(
- storage_scope="test/scope",
- user_id="pytest",
- pipeline_run_id="test-run-meta",
+ run_metadata=RunMetadata(storage_scope="test/scope", user_id="pytest", pipeline_run_id="test-run-meta"),
trace_context=make_trace_context(graph_id="test-run-meta"),
)
diff --git a/tests/unit/pipelex/pipe_operators/pipe_func/test_direct_executor_workdir.py b/tests/unit/pipelex/pipe_operators/pipe_func/test_direct_executor_workdir.py
index 678fdb64e..28daef588 100644
--- a/tests/unit/pipelex/pipe_operators/pipe_func/test_direct_executor_workdir.py
+++ b/tests/unit/pipelex/pipe_operators/pipe_func/test_direct_executor_workdir.py
@@ -15,7 +15,7 @@ from pipelex.pipe_operators.func import direct_pipe_func_executor
from pipelex.pipe_operators.func.direct_pipe_func_executor import DirectPipeFuncExecutor
from pipelex.pipe_operators.func.pipe_func_execution_dtos import PipeFuncExecutionRequest
from pipelex.pipe_run.pipe_run_params import PipeRunParams
-from pipelex.system.job_metadata import JobMetadata
+from pipelex.system.job_metadata import JobMetadata, RunMetadata
from pipelex.system.pipe_run_mode import PipeRunMode
# A method whose ONLY custom source is the PipeFunc — its output structure (Greeting) is a concept
@@ -63,7 +63,7 @@ class TestDirectExecutorWorkdir:
working_memory_raw={},
pipe_code="missing_pipe",
function_name="missing_func",
- job_metadata=JobMetadata(storage_scope="test/scope", user_id="user", pipeline_run_id="run"),
+ job_metadata=JobMetadata(run_metadata=RunMetadata(storage_scope="test/scope", user_id="user", pipeline_run_id="run")),
pipe_run_params=PipeRunParams(run_mode=PipeRunMode.DRY, pipe_stack_limit=10, batch_max_concurrency=None),
)
@@ -88,7 +88,7 @@ class TestDirectExecutorWorkdir:
working_memory_raw={},
pipe_code="missing_pipe",
function_name="missing_func",
- job_metadata=JobMetadata(storage_scope="test/scope", user_id="user", pipeline_run_id="run"),
+ job_metadata=JobMetadata(run_metadata=RunMetadata(storage_scope="test/scope", user_id="user", pipeline_run_id="run")),
pipe_run_params=PipeRunParams(run_mode=PipeRunMode.DRY, pipe_stack_limit=10, batch_max_concurrency=None),
)
@@ -140,7 +140,7 @@ class TestDirectExecutorWorkdir:
# its lookup is strict, so a bare code would not resolve on the far side.
pipe_code="greet_demo.greet",
function_name="greet_it",
- job_metadata=JobMetadata(storage_scope="test/scope", user_id="user", pipeline_run_id="run"),
+ job_metadata=JobMetadata(run_metadata=RunMetadata(storage_scope="test/scope", user_id="user", pipeline_run_id="run")),
pipe_run_params=PipeRunParams(run_mode=PipeRunMode.DRY, pipe_stack_limit=10, batch_max_concurrency=None),
)
diff --git a/tests/unit/pipelex/pipe_operators/pipe_func/test_pipe_func_execution_transport.py b/tests/unit/pipelex/pipe_operators/pipe_func/test_pipe_func_execution_transport.py
index 201a553a1..740fa340d 100644
--- a/tests/unit/pipelex/pipe_operators/pipe_func/test_pipe_func_execution_transport.py
+++ b/tests/unit/pipelex/pipe_operators/pipe_func/test_pipe_func_execution_transport.py
@@ -16,7 +16,7 @@ from pipelex.pipe_operators.func.pipe_func_execution_transport import (
PipeFuncExecutionResponse,
)
from pipelex.pipe_run.pipe_run_params import PipeRunParams
-from pipelex.system.job_metadata import JobMetadata
+from pipelex.system.job_metadata import JobMetadata, RunMetadata
from pipelex.system.pipe_run_mode import PipeRunMode
@@ -46,7 +46,7 @@ class TestPipeFuncExecutionTransport:
working_memory_raw={},
pipe_code="my_pipe",
function_name="my_func",
- job_metadata=JobMetadata(storage_scope="test/scope", user_id="user", pipeline_run_id="run"),
+ job_metadata=JobMetadata(run_metadata=RunMetadata(storage_scope="test/scope", user_id="user", pipeline_run_id="run")),
pipe_run_params=PipeRunParams(run_mode=PipeRunMode.DRY, pipe_stack_limit=10, batch_max_concurrency=None),
timeout_seconds=float("inf"),
)
diff --git a/tests/unit/pipelex/pipe_operators/pipe_func/test_pipe_func_hosted_mode.py b/tests/unit/pipelex/pipe_operators/pipe_func/test_pipe_func_hosted_mode.py
index ebb9d556b..0f0dd5e1c 100644
--- a/tests/unit/pipelex/pipe_operators/pipe_func/test_pipe_func_hosted_mode.py
+++ b/tests/unit/pipelex/pipe_operators/pipe_func/test_pipe_func_hosted_mode.py
@@ -10,7 +10,7 @@ from pipelex.pipe_machinery.pipe_factory import PipeFactory
from pipelex.pipe_operators.func.pipe_func import PipeFunc
from pipelex.pipe_operators.func.pipe_func_blueprint import PipeFuncBlueprint
from pipelex.pipe_run.pipe_run_params import PipeRunParams
-from pipelex.system.job_metadata import JobMetadata
+from pipelex.system.job_metadata import JobMetadata, RunMetadata
from pipelex.system.pipe_run_mode import PipeRunMode
_UNREGISTERED = PipeFuncBlueprint(
@@ -88,7 +88,7 @@ class TestPipeFuncHostedMode:
)
pipe_output = await pipe_func._dry_run_operator_pipe( # noqa: SLF001 # pyright: ignore[reportPrivateUsage]
- job_metadata=JobMetadata(storage_scope="test/scope", user_id="user", pipeline_run_id="run"),
+ job_metadata=JobMetadata(run_metadata=RunMetadata(storage_scope="test/scope", user_id="user", pipeline_run_id="run")),
working_memory=WorkingMemoryFactory.make_empty(),
pipe_run_params=PipeRunParams(run_mode=PipeRunMode.DRY, pipe_stack_limit=10, batch_max_concurrency=None),
)
diff --git a/tests/unit/pipelex/pipe_operators/pipe_func/test_pipe_func_transported_ref.py b/tests/unit/pipelex/pipe_operators/pipe_func/test_pipe_func_transported_ref.py
index 0797a1b71..2153a85f2 100644
--- a/tests/unit/pipelex/pipe_operators/pipe_func/test_pipe_func_transported_ref.py
+++ b/tests/unit/pipelex/pipe_operators/pipe_func/test_pipe_func_transported_ref.py
@@ -20,7 +20,7 @@ from pipelex.pipe_operators.func.pipe_func import PipeFunc
from pipelex.pipe_operators.func.pipe_func_blueprint import PipeFuncBlueprint
from pipelex.pipe_operators.func.pipe_func_executor_protocol import PipeFuncExecutionResult
from pipelex.pipe_run.pipe_run_params import PipeRunParams
-from pipelex.system.job_metadata import JobMetadata
+from pipelex.system.job_metadata import JobMetadata, RunMetadata
from pipelex.system.pipe_run_mode import PipeRunMode
from pipelex.system.registries.func_registry import func_registry
@@ -56,7 +56,7 @@ class TestPipeFuncTransportedRef:
mocker.patch("pipelex.pipe_operators.func.pipe_func.get_pipe_func_executor", return_value=executor)
await the_pipe_func._live_run_operator_pipe( # noqa: SLF001 # pyright: ignore[reportPrivateUsage]
- job_metadata=JobMetadata(storage_scope="test/scope", user_id="user", pipeline_run_id="run"),
+ job_metadata=JobMetadata(run_metadata=RunMetadata(storage_scope="test/scope", user_id="user", pipeline_run_id="run")),
working_memory=WorkingMemoryFactory.make_empty(),
pipe_run_params=PipeRunParams(run_mode=PipeRunMode.LIVE, pipe_stack_limit=10, batch_max_concurrency=None),
)
diff --git a/tests/unit/pipelex/pipe_operators/pipe_llm/test_prompt_rendering_purity.py b/tests/unit/pipelex/pipe_operators/pipe_llm/test_prompt_rendering_purity.py
index 6026128d1..0793595d8 100644
--- a/tests/unit/pipelex/pipe_operators/pipe_llm/test_prompt_rendering_purity.py
+++ b/tests/unit/pipelex/pipe_operators/pipe_llm/test_prompt_rendering_purity.py
@@ -32,7 +32,7 @@ from pipelex.pipe_operators.llm.llm_prompt_blueprint import LLMPromptBlueprint
from pipelex.pipe_operators.llm.pipe_llm import PipeLLM
from pipelex.pipe_operators.llm.pipe_llm_blueprint import PipeLLMBlueprint
from pipelex.pipe_run.pipe_run_params import PipeRunParams
-from pipelex.system.job_metadata import JobMetadata
+from pipelex.system.job_metadata import JobMetadata, RunMetadata
from pipelex.system.pipe_run_mode import PipeRunMode
from pipelex.tools.jinja2.template_category import TemplateCategory
from pipelex.tools.templating.templating_style import TagStyle, TemplatingStyle
@@ -142,7 +142,9 @@ class TestPromptRenderingPurity:
snapshot = pipe_llm.model_dump()
await pipe_llm._dry_run_operator_pipe( # noqa: SLF001 # pyright: ignore[reportPrivateUsage]
- job_metadata=JobMetadata(storage_scope="test/scope", user_id="pytest", pipeline_run_id="test_prompt_rendering_purity"),
+ job_metadata=JobMetadata(
+ run_metadata=RunMetadata(storage_scope="test/scope", user_id="pytest", pipeline_run_id="test_prompt_rendering_purity")
+ ),
working_memory=_make_working_memory(),
pipe_run_params=PipeRunParams(run_mode=PipeRunMode.DRY, pipe_stack_limit=10, batch_max_concurrency=None),
)
diff --git a/tests/unit/pipelex/pipe_run/test_delivery_executor.py b/tests/unit/pipelex/pipe_run/test_delivery_executor.py
index 089205225..758bca6be 100644
--- a/tests/unit/pipelex/pipe_run/test_delivery_executor.py
+++ b/tests/unit/pipelex/pipe_run/test_delivery_executor.py
@@ -24,7 +24,7 @@ from pipelex.pipe_run.delivery_assignment import (
)
from pipelex.pipe_run.delivery_executor import DeliveryExecutor
from pipelex.pipe_run.exceptions import PipeJobError, StorageDeliveryError, WebhookDeliveryError
-from pipelex.system.job_metadata import JobMetadata
+from pipelex.system.job_metadata import JobMetadata, RunMetadata
from pipelex.tools.network.exceptions import SsrfBlockedError
@@ -325,7 +325,7 @@ class TestDeliveryExecutor:
mock_output.tokens_usages = [
LLMTokensUsage(
model_type="llm",
- job_metadata=JobMetadata(storage_scope="test/scope", user_id="test-user", pipeline_run_id="plr-usage"),
+ job_metadata=JobMetadata(run_metadata=RunMetadata(storage_scope="test/scope", user_id="test-user", pipeline_run_id="plr-usage")),
inference_model_name="test-model",
inference_model_id="test-model-id",
nb_tokens_by_category={TokenCategory.INPUT: 15, TokenCategory.OUTPUT: 4},
diff --git a/tests/unit/pipelex/pipe_run/test_pipe_job_hydration.py b/tests/unit/pipelex/pipe_run/test_pipe_job_hydration.py
index 59541874b..fc919efef 100644
--- a/tests/unit/pipelex/pipe_run/test_pipe_job_hydration.py
+++ b/tests/unit/pipelex/pipe_run/test_pipe_job_hydration.py
@@ -7,7 +7,7 @@ from pipelex.libraries.library_crate import LibraryCrate
from pipelex.pipe_run.exceptions import PipeJobError
from pipelex.pipe_run.pipe_job import PipeJob
from pipelex.pipe_run.pipe_run_params_factory import PipeRunParamsFactory
-from pipelex.system.job_metadata import JobMetadata
+from pipelex.system.job_metadata import JobMetadata, RunMetadata
if TYPE_CHECKING:
from pipelex.pipe_machinery.pipe_abstract import PipeAbstract
@@ -23,7 +23,7 @@ def _make_pipe_job(
working_memory=working_memory,
working_memory_raw=None,
pipe_run_params=PipeRunParamsFactory.make_run_params(),
- job_metadata=JobMetadata(storage_scope="test/scope", user_id="test-user", pipeline_run_id="test-run"),
+ job_metadata=JobMetadata(run_metadata=RunMetadata(storage_scope="test/scope", user_id="test-user", pipeline_run_id="test-run")),
output_name=None,
library_crate=library_crate,
)
diff --git a/tests/unit/pipelex/pipe_run/test_pipe_router_run.py b/tests/unit/pipelex/pipe_run/test_pipe_router_run.py
index 26d8cbd22..25a038419 100644
--- a/tests/unit/pipelex/pipe_run/test_pipe_router_run.py
+++ b/tests/unit/pipelex/pipe_run/test_pipe_router_run.py
@@ -11,7 +11,7 @@ from pipelex.pipe_run.exceptions import PipeRouterError
from pipelex.pipe_run.pipe_job import PipeJob
from pipelex.pipe_run.pipe_router_protocol import PipeRouterProtocol
from pipelex.pipe_run.pipe_run_params_factory import PipeRunParamsFactory
-from pipelex.system.job_metadata import JobMetadata
+from pipelex.system.job_metadata import JobMetadata, RunMetadata
from pipelex.system.pipe_run_mode import PipeRunMode
if TYPE_CHECKING:
@@ -44,7 +44,7 @@ def _make_pipe_job() -> PipeJob:
working_memory=None,
working_memory_raw=None,
pipe_run_params=PipeRunParamsFactory.make_run_params(pipe_run_mode=PipeRunMode.LIVE),
- job_metadata=JobMetadata(storage_scope="test/scope", user_id="test-user", pipeline_run_id="test-run"),
+ job_metadata=JobMetadata(run_metadata=RunMetadata(storage_scope="test/scope", user_id="test-user", pipeline_run_id="test-run")),
output_name=None,
library_crate=None,
)
diff --git a/tests/unit/pipelex/pipe_run/test_pipe_run.py b/tests/unit/pipelex/pipe_run/test_pipe_run.py
index 872aa9a93..8e5acc449 100644
--- a/tests/unit/pipelex/pipe_run/test_pipe_run.py
+++ b/tests/unit/pipelex/pipe_run/test_pipe_run.py
@@ -4,6 +4,7 @@ from pytest_mock import MockerFixture
from pipelex.pipe_run.delivery_assignment import DeliveryAssignment, StorageTarget
from pipelex.pipe_run.exceptions import PipeRouterError, WebhookDeliveryError
from pipelex.pipe_run.pipe_run import PipeRun
+from pipelex.system.job_metadata import JobMetadata, RunMetadata
from pipelex.system.pipe_run_mode import PipeRunMode
@@ -21,7 +22,10 @@ class TestPipeRun:
mock_executor_instance.execute = mocker.AsyncMock()
mock_job = mocker.MagicMock()
- mock_job.job_metadata.pipeline_run_id = "plr-123"
+ # A real JobMetadata, not a MagicMock attribute: `PipeRun.run` copies it
+ # onto the output, and `assemble_tracing_on_output` then feeds its
+ # `run_metadata` into `TracingAssembly`, which is a typed field.
+ mock_job.job_metadata = JobMetadata(run_metadata=RunMetadata(user_id="pytest", pipeline_run_id="plr-123", storage_scope="test/scope"))
pipe_run = PipeRun(pipe_router=mock_router)
assignment = DeliveryAssignment(storage=StorageTarget())
@@ -33,7 +37,7 @@ class TestPipeRun:
mock_executor_instance.execute.assert_called_once()
async def test_run_forwards_request_id_to_delivery_executor(self, mocker: MockerFixture) -> None:
- """Direct-mode PipeRun must forward pipe_job.job_metadata.request_id to DeliveryExecutor.execute.
+ """Direct-mode PipeRun must forward pipe_job.job_metadata.run_metadata.request_id to DeliveryExecutor.execute.
Parallel to the Temporal path (wf_pipe_run.py:108 → DeliveryActivityArg.request_id):
without this forward, the storage/webhook completion logs lose correlation with the
@@ -50,8 +54,11 @@ class TestPipeRun:
mock_executor_instance.execute = mocker.AsyncMock()
mock_job = mocker.MagicMock()
- mock_job.job_metadata.pipeline_run_id = "plr-req"
- mock_job.job_metadata.request_id = "req-direct-mode"
+ # A real JobMetadata, not a MagicMock attribute: `PipeRun.run` copies it
+ # onto the output, and `assemble_tracing_on_output` then feeds its
+ # `run_metadata` into `TracingAssembly`, which is a typed field.
+ mock_job.job_metadata = JobMetadata(run_metadata=RunMetadata(user_id="pytest", pipeline_run_id="plr-req", storage_scope="test/scope"))
+ mock_job.job_metadata.run_metadata.request_id = "req-direct-mode"
pipe_run = PipeRun(pipe_router=mock_router)
assignment = DeliveryAssignment(storage=StorageTarget())
@@ -75,7 +82,10 @@ class TestPipeRun:
mock_executor_instance.execute = mocker.AsyncMock()
mock_job = mocker.MagicMock()
- mock_job.job_metadata.pipeline_run_id = "plr-no-delivery"
+ # A real JobMetadata, not a MagicMock attribute: `PipeRun.run` copies it
+ # onto the output, and `assemble_tracing_on_output` then feeds its
+ # `run_metadata` into `TracingAssembly`, which is a typed field.
+ mock_job.job_metadata = JobMetadata(run_metadata=RunMetadata(user_id="pytest", pipeline_run_id="plr-no-delivery", storage_scope="test/scope"))
pipe_run = PipeRun(pipe_router=mock_router)
@@ -104,7 +114,10 @@ class TestPipeRun:
mock_executor_instance.execute = mocker.AsyncMock()
mock_job = mocker.MagicMock()
- mock_job.job_metadata.pipeline_run_id = "plr-fail"
+ # A real JobMetadata, not a MagicMock attribute: `PipeRun.run` copies it
+ # onto the output, and `assemble_tracing_on_output` then feeds its
+ # `run_metadata` into `TracingAssembly`, which is a typed field.
+ mock_job.job_metadata = JobMetadata(run_metadata=RunMetadata(user_id="pytest", pipeline_run_id="plr-fail", storage_scope="test/scope"))
pipe_run = PipeRun(pipe_router=mock_router)
assignment = DeliveryAssignment(storage=StorageTarget())
@@ -135,7 +148,10 @@ class TestPipeRun:
mock_executor_instance.execute = mocker.AsyncMock()
mock_job = mocker.MagicMock()
- mock_job.job_metadata.pipeline_run_id = "plr-bare"
+ # A real JobMetadata, not a MagicMock attribute: `PipeRun.run` copies it
+ # onto the output, and `assemble_tracing_on_output` then feeds its
+ # `run_metadata` into `TracingAssembly`, which is a typed field.
+ mock_job.job_metadata = JobMetadata(run_metadata=RunMetadata(user_id="pytest", pipeline_run_id="plr-bare", storage_scope="test/scope"))
pipe_run = PipeRun(pipe_router=mock_router)
assignment = DeliveryAssignment(storage=StorageTarget())
@@ -177,7 +193,10 @@ class TestPipeRun:
mock_executor.return_value.execute = mock_execute
mock_job = mocker.MagicMock()
- mock_job.job_metadata.pipeline_run_id = "plr-order"
+ # A real JobMetadata, not a MagicMock attribute: `PipeRun.run` copies it
+ # onto the output, and `assemble_tracing_on_output` then feeds its
+ # `run_metadata` into `TracingAssembly`, which is a typed field.
+ mock_job.job_metadata = JobMetadata(run_metadata=RunMetadata(user_id="pytest", pipeline_run_id="plr-order", storage_scope="test/scope"))
pipe_run = PipeRun(pipe_router=mock_router)
assignment = DeliveryAssignment(storage=StorageTarget())
@@ -210,7 +229,10 @@ class TestPipeRun:
mock_executor.return_value.execute = mocker.AsyncMock()
mock_job = mocker.MagicMock()
- mock_job.job_metadata.pipeline_run_id = "plr-mask-tracer"
+ # A real JobMetadata, not a MagicMock attribute: `PipeRun.run` copies it
+ # onto the output, and `assemble_tracing_on_output` then feeds its
+ # `run_metadata` into `TracingAssembly`, which is a typed field.
+ mock_job.job_metadata = JobMetadata(run_metadata=RunMetadata(user_id="pytest", pipeline_run_id="plr-mask-tracer", storage_scope="test/scope"))
pipe_run = PipeRun(pipe_router=mock_router)
@@ -238,7 +260,10 @@ class TestPipeRun:
mock_executor.return_value.execute = mocker.AsyncMock()
mock_job = mocker.MagicMock()
- mock_job.job_metadata.pipeline_run_id = "plr-tracer-only"
+ # A real JobMetadata, not a MagicMock attribute: `PipeRun.run` copies it
+ # onto the output, and `assemble_tracing_on_output` then feeds its
+ # `run_metadata` into `TracingAssembly`, which is a typed field.
+ mock_job.job_metadata = JobMetadata(run_metadata=RunMetadata(user_id="pytest", pipeline_run_id="plr-tracer-only", storage_scope="test/scope"))
pipe_run = PipeRun(pipe_router=mock_router)
@@ -272,7 +297,10 @@ class TestPipeRun:
)
mock_job = mocker.MagicMock()
- mock_job.job_metadata.pipeline_run_id = "plr-triple-fail"
+ # A real JobMetadata, not a MagicMock attribute: `PipeRun.run` copies it
+ # onto the output, and `assemble_tracing_on_output` then feeds its
+ # `run_metadata` into `TracingAssembly`, which is a typed field.
+ mock_job.job_metadata = JobMetadata(run_metadata=RunMetadata(user_id="pytest", pipeline_run_id="plr-triple-fail", storage_scope="test/scope"))
pipe_run = PipeRun(pipe_router=mock_router)
assignment = DeliveryAssignment(storage=StorageTarget())
diff --git a/tests/unit/pipelex/pipe_run/test_run_batch_branch_hook.py b/tests/unit/pipelex/pipe_run/test_run_batch_branch_hook.py
index 23a60140c..d3f091e42 100644
--- a/tests/unit/pipelex/pipe_run/test_run_batch_branch_hook.py
+++ b/tests/unit/pipelex/pipe_run/test_run_batch_branch_hook.py
@@ -17,7 +17,7 @@ from pipelex.observer.observer_protocol import ObserverNoOp
from pipelex.pipe_run.pipe_job import PipeJob
from pipelex.pipe_run.pipe_router_protocol import PipeRouterProtocol
from pipelex.pipe_run.pipe_run_params_factory import PipeRunParamsFactory
-from pipelex.system.job_metadata import JobMetadata
+from pipelex.system.job_metadata import JobMetadata, RunMetadata
from pipelex.system.pipe_run_mode import PipeRunMode
if TYPE_CHECKING:
@@ -42,7 +42,7 @@ class _RecordingRouter(PipeRouterProtocol):
self.dispatched_jobs.append(pipe_job)
return PipeOutput(
working_memory=WorkingMemoryFactory.make_empty(),
- pipeline_run_id=pipe_job.job_metadata.pipeline_run_id,
+ pipeline_run_id=pipe_job.job_metadata.run_metadata.pipeline_run_id,
)
@@ -52,7 +52,7 @@ def _make_pipe_job() -> PipeJob:
working_memory=None,
working_memory_raw=None,
pipe_run_params=PipeRunParamsFactory.make_run_params(pipe_run_mode=PipeRunMode.LIVE),
- job_metadata=JobMetadata(storage_scope="test/scope", user_id="test-user", pipeline_run_id="test-run"),
+ job_metadata=JobMetadata(run_metadata=RunMetadata(storage_scope="test/scope", user_id="test-user", pipeline_run_id="test-run")),
output_name=None,
library_crate=None,
)
diff --git a/tests/unit/pipelex/pipe_run/test_tracing_assembly.py b/tests/unit/pipelex/pipe_run/test_tracing_assembly.py
index 82d129bfb..dcbf00b3f 100644
--- a/tests/unit/pipelex/pipe_run/test_tracing_assembly.py
+++ b/tests/unit/pipelex/pipe_run/test_tracing_assembly.py
@@ -13,7 +13,7 @@ from pipelex.core.pipes.pipe_output import PipeOutput
from pipelex.graph.graphspec import GraphSpecMode
from pipelex.pipe_run.tracing_assembly import TracingAssembly, assemble_tracing, assemble_tracing_on_output
from pipelex.system.exceptions import MissingDependencyError
-from pipelex.system.job_metadata import JobMetadata
+from pipelex.system.job_metadata import JobMetadata, RunMetadata
from pipelex.system.pipe_run_mode import PipeRunMode
from pipelex.tracing.exceptions import EventLogReadError, EventLogSetupError
from pipelex.tracing.trace_events import UsageReportEvent
@@ -23,7 +23,7 @@ _MODULE = "pipelex.pipe_run.tracing_assembly"
def _make_usage_event(pipeline_run_id: str, node_id: str) -> UsageReportEvent:
"""A UsageReportEvent carrying a real LLMTokensUsage (so the aggregator returns it)."""
- metadata = JobMetadata(storage_scope="test/scope", user_id="tracing-assembly-test", pipeline_run_id=pipeline_run_id)
+ metadata = JobMetadata(run_metadata=RunMetadata(storage_scope="test/scope", user_id="tracing-assembly-test", pipeline_run_id=pipeline_run_id))
tokens_usage = LLMTokensUsage(
job_metadata=metadata,
inference_model_name="fake_model",
diff --git a/tests/unit/pipelex/providers/openai/test_openai_completions_usage_capture.py b/tests/unit/pipelex/providers/openai/test_openai_completions_usage_capture.py
index 0d2038ed1..40679b325 100644
--- a/tests/unit/pipelex/providers/openai/test_openai_completions_usage_capture.py
+++ b/tests/unit/pipelex/providers/openai/test_openai_completions_usage_capture.py
@@ -22,7 +22,7 @@ from pipelex.cogt.usage.cost_category import CostCategory
from pipelex.cogt.usage.token_category import TokenCategory
from pipelex.providers.openai.openai_completions_factory import OpenAICompletionsFactory
from pipelex.providers.openai.openai_completions_llm_worker import OpenAICompletionsLLMWorker
-from pipelex.system.job_metadata import JobMetadata
+from pipelex.system.job_metadata import JobMetadata, RunMetadata
if TYPE_CHECKING:
from typing import Any
@@ -32,7 +32,7 @@ if TYPE_CHECKING:
def _make_llm_job_with_empty_usage() -> LLMJob:
"""Build an LLMJob whose tokens_usage is seeded empty, mirroring ``llm_job_before_start``."""
- job_metadata = JobMetadata(storage_scope="test/scope", user_id="test-user", pipeline_run_id="test-run")
+ job_metadata = JobMetadata(run_metadata=RunMetadata(storage_scope="test/scope", user_id="test-user", pipeline_run_id="test-run"))
tokens_usage = LLMTokensUsage(
job_metadata=job_metadata,
inference_model_name="test-model",
diff --git a/tests/unit/pipelex/providers/openai/test_openai_img_gen_worker_usage_capture.py b/tests/unit/pipelex/providers/openai/test_openai_img_gen_worker_usage_capture.py
index 76debcee7..fa5cb9a5d 100644
--- a/tests/unit/pipelex/providers/openai/test_openai_img_gen_worker_usage_capture.py
+++ b/tests/unit/pipelex/providers/openai/test_openai_img_gen_worker_usage_capture.py
@@ -29,7 +29,7 @@ from pipelex.cogt.model_backends.model_type import ModelType
from pipelex.cogt.usage.cost_category import CostCategory
from pipelex.cogt.usage.token_category import TokenCategory
from pipelex.providers.openai.openai_img_gen_worker import OpenAIImgGenWorker
-from pipelex.system.job_metadata import JobMetadata
+from pipelex.system.job_metadata import JobMetadata, RunMetadata
from pipelex.tools.misc.image_utils import ImageFormat
if TYPE_CHECKING:
@@ -70,7 +70,7 @@ def _make_img_gen_job() -> ImgGenJob:
),
job_config=ImgGenJobConfig(is_sync_mode=False),
job_report=ImgGenJobReport(),
- job_metadata=JobMetadata(storage_scope="test/scope", user_id="test-user", pipeline_run_id="test-run"),
+ job_metadata=JobMetadata(run_metadata=RunMetadata(storage_scope="test/scope", user_id="test-user", pipeline_run_id="test-run")),
)
diff --git a/tests/unit/pipelex/providers/openai/test_openai_responses_usage_capture.py b/tests/unit/pipelex/providers/openai/test_openai_responses_usage_capture.py
index 8b08d2f28..1f4fee753 100644
--- a/tests/unit/pipelex/providers/openai/test_openai_responses_usage_capture.py
+++ b/tests/unit/pipelex/providers/openai/test_openai_responses_usage_capture.py
@@ -21,7 +21,7 @@ from pipelex.cogt.usage.cost_category import CostCategory
from pipelex.cogt.usage.token_category import TokenCategory
from pipelex.providers.openai.openai_responses_factory import OpenAIResponsesFactory
from pipelex.providers.openai.openai_responses_llm_worker import OpenAIResponsesLLMWorker
-from pipelex.system.job_metadata import JobMetadata
+from pipelex.system.job_metadata import JobMetadata, RunMetadata
if TYPE_CHECKING:
from typing import Any
@@ -30,7 +30,7 @@ if TYPE_CHECKING:
def _make_llm_job_with_empty_usage() -> LLMJob:
- job_metadata = JobMetadata(storage_scope="test/scope", user_id="test-user", pipeline_run_id="test-run")
+ job_metadata = JobMetadata(run_metadata=RunMetadata(storage_scope="test/scope", user_id="test-user", pipeline_run_id="test-run"))
tokens_usage = LLMTokensUsage(
job_metadata=job_metadata,
inference_model_name="test-model",
diff --git a/tests/unit/pipelex/reporting/test_emit_runner_fallback.py b/tests/unit/pipelex/reporting/test_emit_runner_fallback.py
index c40ba2131..b8576fd7b 100644
--- a/tests/unit/pipelex/reporting/test_emit_runner_fallback.py
+++ b/tests/unit/pipelex/reporting/test_emit_runner_fallback.py
@@ -29,7 +29,7 @@ from pipelex.reporting.reporting_manager import ReportingManager
from pipelex.system.configuration.configs import NdjsonTracingConfig, TracingBackend
from pipelex.system.data_inclusion_config import DataInclusionConfig
from pipelex.system.exceptions import MissingDependencyError
-from pipelex.system.job_metadata import JobMetadata, UnitJobId
+from pipelex.system.job_metadata import JobMetadata, RunMetadata, UnitJobId
from pipelex.system.trace_context import TraceContext
from pipelex.tracing import activity_event_log
from pipelex.tracing.activity_event_log import ActivityEventLogCache
@@ -55,9 +55,7 @@ def _make_llm_job(
) -> LLMJob:
now = datetime.now(UTC)
job_metadata = JobMetadata(
- storage_scope="test/scope",
- user_id="test_user",
- pipeline_run_id=pipeline_run_id,
+ run_metadata=RunMetadata(storage_scope="test/scope", user_id="test_user", pipeline_run_id=pipeline_run_id),
trace_context=trace_context,
started_at=now,
completed_at=now + timedelta(seconds=1),
diff --git a/tests/unit/pipelex/reporting/test_emit_runner_fallback_non_llm.py b/tests/unit/pipelex/reporting/test_emit_runner_fallback_non_llm.py
index 886d0a10e..42e148b00 100644
--- a/tests/unit/pipelex/reporting/test_emit_runner_fallback_non_llm.py
+++ b/tests/unit/pipelex/reporting/test_emit_runner_fallback_non_llm.py
@@ -36,7 +36,7 @@ from pipelex.config import get_config
from pipelex.reporting.reporting_manager import ReportingManager
from pipelex.system.configuration.configs import NdjsonTracingConfig, TracingBackend
from pipelex.system.data_inclusion_config import DataInclusionConfig
-from pipelex.system.job_metadata import JobMetadata
+from pipelex.system.job_metadata import JobMetadata, RunMetadata
from pipelex.system.trace_context import TraceContext
from pipelex.tools.misc.image_utils import ImageFormat
from pipelex.tracing.activity_event_log import ActivityEventLogCache
@@ -80,9 +80,7 @@ def _trace_context(run_id: str, tracer_key: str) -> TraceContext:
def _job_metadata(run_id: str, tracer_key: str) -> JobMetadata:
now = datetime.now(UTC)
return JobMetadata(
- storage_scope="test/scope",
- user_id="test-user",
- pipeline_run_id=run_id,
+ run_metadata=RunMetadata(storage_scope="test/scope", user_id="test-user", pipeline_run_id=run_id),
trace_context=_trace_context(run_id, tracer_key),
started_at=now,
completed_at=now + timedelta(seconds=1),
diff --git a/tests/unit/pipelex/reporting/test_emit_usage_event_gating.py b/tests/unit/pipelex/reporting/test_emit_usage_event_gating.py
index 22adbce98..06ba7849c 100644
--- a/tests/unit/pipelex/reporting/test_emit_usage_event_gating.py
+++ b/tests/unit/pipelex/reporting/test_emit_usage_event_gating.py
@@ -22,7 +22,7 @@ from pipelex.config import get_config
from pipelex.reporting.reporting_manager import ReportingManager
from pipelex.system.configuration.configs import NdjsonTracingConfig, TracingBackend
from pipelex.system.data_inclusion_config import DataInclusionConfig
-from pipelex.system.job_metadata import JobMetadata, UnitJobId
+from pipelex.system.job_metadata import JobMetadata, RunMetadata, UnitJobId
from pipelex.system.trace_context import TraceContext
from pipelex.tracing.activity_event_log import ActivityEventLogCache
from pipelex.tracing.ndjson_event_log import NdjsonEventLog
@@ -40,9 +40,7 @@ DATA_INCLUSION_OFF = DataInclusionConfig(
def _make_llm_job(pipeline_run_id: str, trace_context: TraceContext | None) -> LLMJob:
now = datetime.now(UTC)
job_metadata = JobMetadata(
- storage_scope="test/scope",
- user_id="test_user",
- pipeline_run_id=pipeline_run_id,
+ run_metadata=RunMetadata(storage_scope="test/scope", user_id="test_user", pipeline_run_id=pipeline_run_id),
trace_context=trace_context,
started_at=now,
completed_at=now + timedelta(seconds=1),
diff --git a/tests/unit/pipelex/reporting/test_reporting_event_emission.py b/tests/unit/pipelex/reporting/test_reporting_event_emission.py
index e56cafb43..60fb0c853 100644
--- a/tests/unit/pipelex/reporting/test_reporting_event_emission.py
+++ b/tests/unit/pipelex/reporting/test_reporting_event_emission.py
@@ -20,7 +20,7 @@ from pipelex.cogt.usage.token_category import TokenCategory
from pipelex.config import get_config
from pipelex.reporting.reporting_manager import ReportingManager
from pipelex.system.data_inclusion_config import DataInclusionConfig
-from pipelex.system.job_metadata import JobMetadata, UnitJobId
+from pipelex.system.job_metadata import JobMetadata, RunMetadata, UnitJobId
from pipelex.system.trace_context import TraceContext
from pipelex.tracing.in_memory_event_log import InMemoryEventLog
from pipelex.tracing.trace_events import UsageReportEvent
@@ -41,9 +41,7 @@ def _make_test_llm_job(
"""Create a minimal LLMJob for testing event emission."""
now = datetime.now(UTC)
job_metadata = JobMetadata(
- storage_scope="test/scope",
- user_id="test_user",
- pipeline_run_id=pipeline_run_id,
+ run_metadata=RunMetadata(storage_scope="test/scope", user_id="test_user", pipeline_run_id=pipeline_run_id),
trace_context=trace_context,
started_at=now,
completed_at=now + timedelta(seconds=1),
diff --git a/tests/unit/pipelex/runtime_bridge/test_direct_orchestrator.py b/tests/unit/pipelex/runtime_bridge/test_direct_orchestrator.py
index ee6f386f5..62f8bc7e0 100644
--- a/tests/unit/pipelex/runtime_bridge/test_direct_orchestrator.py
+++ b/tests/unit/pipelex/runtime_bridge/test_direct_orchestrator.py
@@ -6,7 +6,7 @@ from pipelex.pipe_run.pipe_job import PipeJob
from pipelex.pipe_run.pipe_run_params_factory import PipeRunParamsFactory
from pipelex.runtime_bridge.direct_orchestrator import DirectOrchestrator
from pipelex.runtime_bridge.exceptions import PipelexBridgeDispatchError
-from pipelex.system.job_metadata import JobMetadata
+from pipelex.system.job_metadata import JobMetadata, RunMetadata
if TYPE_CHECKING:
from pipelex.pipe_machinery.pipe_abstract import PipeAbstract
@@ -25,7 +25,7 @@ def _make_pipe_job() -> PipeJob:
working_memory=None,
working_memory_raw=None,
pipe_run_params=PipeRunParamsFactory.make_run_params(),
- job_metadata=JobMetadata(storage_scope="test/scope", user_id="test-user", pipeline_run_id="test-run"),
+ job_metadata=JobMetadata(run_metadata=RunMetadata(storage_scope="test/scope", user_id="test-user", pipeline_run_id="test-run")),
output_name=None,
library_crate=None,
)
diff --git a/tests/unit/pipelex/system/test_job_metadata_request_id.py b/tests/unit/pipelex/system/test_job_metadata_request_id.py
index a3cfd6b60..e78411139 100644
--- a/tests/unit/pipelex/system/test_job_metadata_request_id.py
+++ b/tests/unit/pipelex/system/test_job_metadata_request_id.py
@@ -5,14 +5,14 @@ that rides the workflow input across the Temporal serialization boundary.
import pytest
from pydantic import ValidationError
-from pipelex.system.job_metadata import JobMetadata
+from pipelex.system.job_metadata import JobMetadata, RunMetadata
class TestJobMetadataRequestId:
def test_request_id_defaults_to_none(self) -> None:
"""``JobMetadata`` constructed without ``request_id`` carries ``None``."""
- meta = JobMetadata(storage_scope="test/scope", user_id="u", pipeline_run_id="r")
- assert meta.request_id is None
+ meta = JobMetadata(run_metadata=RunMetadata(storage_scope="test/scope", user_id="u", pipeline_run_id="r"))
+ assert meta.run_metadata.request_id is None
def test_request_id_round_trips_through_json(self) -> None:
"""``request_id`` survives ``model_dump_json`` / ``model_validate_json`` round-trip.
@@ -20,16 +20,16 @@ class TestJobMetadataRequestId:
Temporal's data converter serializes activity args as JSON; this pins
the contract that ``request_id`` reaches the worker intact.
"""
- meta = JobMetadata(storage_scope="test/scope", user_id="u", pipeline_run_id="r", request_id="r-abc-123")
+ meta = JobMetadata(run_metadata=RunMetadata(storage_scope="test/scope", user_id="u", pipeline_run_id="r", request_id="r-abc-123"))
recovered = JobMetadata.model_validate_json(meta.model_dump_json())
- assert recovered.request_id == "r-abc-123"
+ assert recovered.run_metadata.request_id == "r-abc-123"
assert recovered == meta
def test_copy_with_update_preserves_request_id(self) -> None:
"""``copy_with_update`` inherits ``request_id`` from the parent metadata."""
- parent = JobMetadata(storage_scope="test/scope", user_id="u", pipeline_run_id="r", request_id="r-1")
+ parent = JobMetadata(run_metadata=RunMetadata(storage_scope="test/scope", user_id="u", pipeline_run_id="r", request_id="r-1"))
child = parent.copy_with_update(otel_context=None)
- assert child.request_id == "r-1"
+ assert child.run_metadata.request_id == "r-1"
@pytest.mark.parametrize(
"bad_request_id",
@@ -52,7 +52,7 @@ class TestJobMetadataRequestId:
``ErrorReport`` envelopes that quote it.
"""
with pytest.raises(ValidationError):
- JobMetadata(storage_scope="test/scope", user_id="u", pipeline_run_id="r", request_id=bad_request_id)
+ JobMetadata(run_metadata=RunMetadata(storage_scope="test/scope", user_id="u", pipeline_run_id="r", request_id=bad_request_id))
@pytest.mark.parametrize(
"good_request_id",
@@ -65,5 +65,5 @@ class TestJobMetadataRequestId:
)
def test_request_id_accepts_well_formed_input(self, good_request_id: str) -> None:
"""Common ``X-Request-ID`` shapes (UUID, hyphenated, dot-separated) round-trip cleanly."""
- meta = JobMetadata(storage_scope="test/scope", user_id="u", pipeline_run_id="r", request_id=good_request_id)
- assert meta.request_id == good_request_id
+ meta = JobMetadata(run_metadata=RunMetadata(storage_scope="test/scope", user_id="u", pipeline_run_id="r", request_id=good_request_id))
+ assert meta.run_metadata.request_id == good_request_id
diff --git a/tests/unit/pipelex/system/test_storage_scope.py b/tests/unit/pipelex/system/test_storage_scope.py
index cfe5b965b..014249316 100644
--- a/tests/unit/pipelex/system/test_storage_scope.py
+++ b/tests/unit/pipelex/system/test_storage_scope.py
@@ -16,7 +16,7 @@ from pathlib import Path
import pytest
from pydantic import ValidationError
-from pipelex.system.job_metadata import JobMetadata
+from pipelex.system.job_metadata import JobMetadata, RunMetadata
from pipelex.system.storage_scope import (
DRY_RUN_STORAGE_SCOPE,
DRY_RUN_USER_ID,
@@ -28,7 +28,7 @@ from pipelex.system.telemetry.otel_constants import OTelConstants
def _metadata(storage_scope: str) -> JobMetadata:
- return JobMetadata(user_id="u1", pipeline_run_id="run_1", storage_scope=storage_scope)
+ return JobMetadata(run_metadata=RunMetadata(user_id="u1", pipeline_run_id="run_1", storage_scope=storage_scope))
class TestAcceptedScopes:
@@ -44,7 +44,7 @@ class TestAcceptedScopes:
],
)
def test_one_to_three_path_safe_segments_are_accepted(self, scope: str) -> None:
- assert _metadata(scope).storage_scope == scope
+ assert _metadata(scope).run_metadata.storage_scope == scope
def test_the_two_named_constants_are_valid_scopes(self) -> None:
"""They are used as real field values, so they must satisfy the field.
@@ -105,7 +105,7 @@ class TestRefusedScopes:
perfectly valid scope even when the caller is `tenant`; refusing it
requires knowing who is calling, which this layer deliberately does not.
"""
- assert _metadata("tenant2/run_1").storage_scope == "tenant2/run_1"
+ assert _metadata("tenant2/run_1").run_metadata.storage_scope == "tenant2/run_1"
class TestScopeIsRequired:
@@ -122,7 +122,7 @@ class TestScopeIsRequired:
# Both type checkers reject this call, which is the point being
# tested: the omission is caught statically AND at runtime. The
# ignores are what let the runtime half be asserted at all.
- JobMetadata(user_id="u1", pipeline_run_id="run_1") # type: ignore[call-arg] # pyright: ignore[reportCallIssue]
+ JobMetadata(run_metadata=RunMetadata(user_id="u1", pipeline_run_id="run_1")) # type: ignore[call-arg] # pyright: ignore[reportCallIssue]
class TestTheTelemetryPlaceholderStaysOutOfIdentity:
diff --git a/tests/unit/pipelex/tracing/test_assembler_equivalence.py b/tests/unit/pipelex/tracing/test_assembler_equivalence.py
index 15c0444df..2a225b8a2 100644
--- a/tests/unit/pipelex/tracing/test_assembler_equivalence.py
+++ b/tests/unit/pipelex/tracing/test_assembler_equivalence.py
@@ -19,7 +19,7 @@ from pipelex.cogt.usage.cost_category import CostCategory
from pipelex.cogt.usage.token_category import TokenCategory
from pipelex.graph.graph_tracer import GraphTracer
from pipelex.graph.graphspec import EdgeKind, EdgeSpec, GraphSpec, IOSpec, NodeKind, NodeSpec
-from pipelex.system.job_metadata import JobCategory, JobMetadata, UnitJobId
+from pipelex.system.job_metadata import JobCategory, JobMetadata, RunMetadata, UnitJobId
from pipelex.system.trace_context import TraceContext
from pipelex.tracing.graphspec_assembler import GraphSpecAssembler
from pipelex.tracing.in_memory_event_log import InMemoryEventLog
@@ -473,9 +473,7 @@ def _usage_events_for(*, assembled_child_node_id: str, event_log: InMemoryEventL
node_id=assembled_child_node_id,
tokens_usage=LLMTokensUsage(
job_metadata=JobMetadata(
- storage_scope="test/scope",
- user_id="user_test",
- pipeline_run_id=_PIPELINE_RUN_ID,
+ run_metadata=RunMetadata(storage_scope="test/scope", user_id="user_test", pipeline_run_id=_PIPELINE_RUN_ID),
pipe_code="gen_text",
unit_job_id=UnitJobId.LLM_GEN_TEXT,
job_category=JobCategory.LLM_JOB,
diff --git a/tests/unit/pipelex/tracing/test_assembler_usage_attribution.py b/tests/unit/pipelex/tracing/test_assembler_usage_attribution.py
index 939d6f663..0b64144e2 100644
--- a/tests/unit/pipelex/tracing/test_assembler_usage_attribution.py
+++ b/tests/unit/pipelex/tracing/test_assembler_usage_attribution.py
@@ -9,7 +9,7 @@ from pipelex.cogt.usage.cost_category import CostCategory, CostsByCategoryDict
from pipelex.cogt.usage.token_category import TokenCategory
from pipelex.graph.graphspec import GraphSpec, NodeKind, NodeSpec
from pipelex.reporting.reporting_types import AnyTokensUsage
-from pipelex.system.job_metadata import JobCategory, JobMetadata, UnitJobId
+from pipelex.system.job_metadata import JobCategory, JobMetadata, RunMetadata, UnitJobId
from pipelex.tracing.graphspec_assembler import GraphSpecAssembler
from pipelex.tracing.trace_events import (
UNATTRIBUTED_NODE_ID,
@@ -45,9 +45,7 @@ def _base(sequence: int) -> dict[str, Any]:
def _make_usage(*, unit_costs: CostsByCategoryDict, model_name: str = "test-model") -> AnyTokensUsage:
return LLMTokensUsage(
job_metadata=JobMetadata(
- storage_scope="test/scope",
- user_id="user_test",
- pipeline_run_id=_PIPELINE_RUN_ID,
+ run_metadata=RunMetadata(storage_scope="test/scope", user_id="user_test", pipeline_run_id=_PIPELINE_RUN_ID),
pipe_code="test_pipe",
unit_job_id=UnitJobId.LLM_GEN_TEXT,
job_category=JobCategory.LLM_JOB,
diff --git a/tests/unit/pipelex/tracing/test_non_llm_usage_aggregation.py b/tests/unit/pipelex/tracing/test_non_llm_usage_aggregation.py
index 42accdc97..e757c657a 100644
--- a/tests/unit/pipelex/tracing/test_non_llm_usage_aggregation.py
+++ b/tests/unit/pipelex/tracing/test_non_llm_usage_aggregation.py
@@ -20,13 +20,13 @@ from pipelex.cogt.usage.cost_category import CostCategory
from pipelex.cogt.usage.cost_registry import CostRegistry
from pipelex.cogt.usage.token_category import TokenCategory
from pipelex.reporting.reporting_types import AnyTokensUsage
-from pipelex.system.job_metadata import JobMetadata
+from pipelex.system.job_metadata import JobMetadata, RunMetadata
from pipelex.tracing.trace_events import UsageReportEvent
from pipelex.tracing.usage_aggregator import UsageAggregator
def _job_metadata() -> JobMetadata:
- return JobMetadata(storage_scope="test/scope", user_id="test-user", pipeline_run_id="run-non-llm")
+ return JobMetadata(run_metadata=RunMetadata(storage_scope="test/scope", user_id="test-user", pipeline_run_id="run-non-llm"))
def _usage_event(tokens_usage: AnyTokensUsage, sequence: int) -> UsageReportEvent:
diff --git a/tests/unit/pipelex/tracing/test_trace_events.py b/tests/unit/pipelex/tracing/test_trace_events.py
index 45443c0b9..cc06d852e 100644
--- a/tests/unit/pipelex/tracing/test_trace_events.py
+++ b/tests/unit/pipelex/tracing/test_trace_events.py
@@ -10,7 +10,7 @@ from pipelex.cogt.llm.llm_report import LLMTokensUsage
from pipelex.cogt.usage.cost_category import CostCategory
from pipelex.cogt.usage.token_category import TokenCategory
from pipelex.graph.graphspec import EdgeKind, ErrorSpec, IOSpec, NodeKind
-from pipelex.system.job_metadata import JobCategory, JobMetadata, UnitJobId
+from pipelex.system.job_metadata import JobCategory, JobMetadata, RunMetadata, UnitJobId
from pipelex.tracing.trace_events import (
AnyTraceEvent,
BatchAggregateEvent,
@@ -69,9 +69,7 @@ class _Shared:
@staticmethod
def make_job_metadata() -> JobMetadata:
return JobMetadata(
- storage_scope="test/scope",
- user_id="user_test",
- pipeline_run_id=_Shared.PIPELINE_RUN_ID,
+ run_metadata=RunMetadata(storage_scope="test/scope", user_id="user_test", pipeline_run_id=_Shared.PIPELINE_RUN_ID),
pipe_code="test_pipe",
unit_job_id=UnitJobId.LLM_GEN_TEXT,
job_category=JobCategory.LLM_JOB,
diff --git a/tests/unit/pipelex/tracing/test_usage_aggregator.py b/tests/unit/pipelex/tracing/test_usage_aggregator.py
index 290a4715e..7764fbe24 100644
--- a/tests/unit/pipelex/tracing/test_usage_aggregator.py
+++ b/tests/unit/pipelex/tracing/test_usage_aggregator.py
@@ -7,7 +7,7 @@ from pipelex.cogt.llm.llm_report import LLMTokensUsage
from pipelex.cogt.usage.cost_category import CostCategory
from pipelex.cogt.usage.token_category import TokenCategory
from pipelex.graph.graphspec import NodeKind
-from pipelex.system.job_metadata import JobCategory, JobMetadata, UnitJobId
+from pipelex.system.job_metadata import JobCategory, JobMetadata, RunMetadata, UnitJobId
from pipelex.tracing.trace_events import PipeStartEvent, TraceEvent, UsageReportEvent
from pipelex.tracing.usage_aggregator import UsageAggregator
@@ -19,9 +19,7 @@ _WORKFLOW_ID = "wf_abc"
def _make_llm_tokens_usage(model_name: str = "claude-sonnet") -> LLMTokensUsage:
return LLMTokensUsage(
job_metadata=JobMetadata(
- storage_scope="test/scope",
- user_id="user_test",
- pipeline_run_id=_PIPELINE_RUN_ID,
+ run_metadata=RunMetadata(storage_scope="test/scope", user_id="user_test", pipeline_run_id=_PIPELINE_RUN_ID),
pipe_code="test_pipe",
unit_job_id=UnitJobId.LLM_GEN_TEXT,
job_category=JobCategory.LLM_JOB,
diff --git a/tests/unit/pipelex/tracing/test_usage_attribution.py b/tests/unit/pipelex/tracing/test_usage_attribution.py
index 3f6992bcf..1a7a2d952 100644
--- a/tests/unit/pipelex/tracing/test_usage_attribution.py
+++ b/tests/unit/pipelex/tracing/test_usage_attribution.py
@@ -7,7 +7,7 @@ from pipelex.cogt.usage.cost_category import CostCategory, CostsByCategoryDict
from pipelex.cogt.usage.token_category import NbTokensByCategoryDict, TokenCategory
from pipelex.graph.graphspec import NodeUsageSpec
from pipelex.reporting.reporting_types import AnyTokensUsage
-from pipelex.system.job_metadata import JobCategory, JobMetadata, UnitJobId
+from pipelex.system.job_metadata import JobCategory, JobMetadata, RunMetadata, UnitJobId
from pipelex.tracing.usage_attribution import attribute_usage, make_node_usage_spec, roll_up
# Per-million-token USD, so 100 input + 50 output tokens cost 0.0003 + 0.00075.
@@ -29,9 +29,7 @@ def _make_usage(
) -> AnyTokensUsage:
return LLMTokensUsage(
job_metadata=JobMetadata(
- storage_scope="test/scope",
- user_id="user_test",
- pipeline_run_id="run_001",
+ run_metadata=RunMetadata(storage_scope="test/scope", user_id="user_test", pipeline_run_id="run_001"),
pipe_code="test_pipe",
unit_job_id=UnitJobId.LLM_GEN_TEXT,
job_category=JobCategory.LLM_JOB,