Skip to content

Build AI methods with your coding agent, then run them anywhere: as an MCP for chatbots, as a webapp for people, or via API for your software.

feature/Provenance-carries-displayname · L-261026-6ed3f3 (#1491)

Louis Choquel68643ac39be9
diff --git a/.drift/acks/cli-docs.toml b/.drift/acks/cli-docs.toml
index 977a4a470..26941b004 100644
--- a/.drift/acks/cli-docs.toml
+++ b/.drift/acks/cli-docs.toml
@@ -1,8 +1,8 @@
contract = "cli-docs"
-digest = "sha256:197170f0ab55d758506727daadfedcd91777769b1d964a49a81b6b3ca5c91970"
+digest = "sha256:41cf54e443532e0e23af98acf3c6c31dcaf5d51a006d8b2fad0d2c3b90c8c3c1"
reviewed_by = "Louis Choquel"
-reviewed_at = "2026-10-24T04:25:12Z"
-rationale = "real-catch: both CLIs now report a hosted failure as the problem pipelex-sdk carries, the human CLI printing its rows, items and the run's own problem; the former-shape readings are gone, a bare 401/403/404 is read by the key sent, a run past the wait is client.follow_failed and a failed tree upload reads by its reason; docs/tools/cli/run.md, agent-cli.md and pipelex/cli/agent_cli/CLAUDE.md now say so. No flag or output format changed."
+reviewed_at = "2026-10-26T16:40:50Z"
+rationale = "clean-pass: installed_methods.py changed only its comments, which now say the provenance sidecar records the manifest's display_name; the CLI docs describe what a fetch prints (address, tag, commit), which did not change, and the sidecar's contents are documented in docs/building-methods/packages.md, updated in this change."
[trigger_files]
"pipelex/cli/__init__.py" = "blob:8b137891791fe96927ad78e64b0aad7bded08bdc"
@@ -132,7 +132,7 @@ rationale = "real-catch: both CLIs now report a hosted failure as the problem pi
"pipelex/cli/deck_notice.py" = "blob:542877125e89e35b02bc969cfee952155b0d7c2c"
"pipelex/cli/error_handlers.py" = "blob:649461025ca7a84b2e73bfe63cd2649d472f4088"
"pipelex/cli/exceptions.py" = "blob:5c66f20b0a06c8ae8c20b94f92e2c30b78f5b6c5"
-"pipelex/cli/installed_methods.py" = "blob:7e7f289d50ca6bc446b6891cbc846dc4a8cee6e2"
+"pipelex/cli/installed_methods.py" = "blob:208cb640a3685e5850d9de3f889b4b4cbd895a56"
"pipelex/cli/judged_target.py" = "blob:626ccf65bc3b15e06422d9f16c0b1a8083041b59"
"pipelex/cli/method_resolver.py" = "blob:54fe2e6036b637c2e298f18bbfd10cfc616ef4a8"
"pipelex/cli/readiness.py" = "blob:6d185324ed0784663e1dc78d1194a28312c54798"
diff --git a/CHANGELOG.md b/CHANGELOG.md
index 2db872b23..3f9893434 100644
--- a/CHANGELOG.md
+++ b/CHANGELOG.md
@@ -2,6 +2,10 @@
## [Unreleased]
+### Added
+
+- **A fetched method's provenance carries its declared name**: `MethodProvenance` gains `display_name`, the `display_name` the package's `METHODS.toml` declares at the fetched commit, and `null` when it declares none, so the `method_provenance` of a `method_ref` run on `/v1/execute` and `/v1/start` reports it beside the address, tag, commit and tree digest, and a host can name the run's method after it. A fetched copy's `.provenance.json` sidecar records it too, and a sidecar or a record written without it still reads, its `display_name` null.
+
### Fixed
- **The S3 storage provider reuses one client per event loop**: each event loop opens one aiobotocore client on its first S3 operation, and every later operation on that loop shares it and its pool of `S3_MAX_POOL_CONNECTIONS` connections, where every operation, a presigned URL included, built a client and opened a TLS connection of its own; the client is closed as its loop shuts down or at the runtime's teardown, which now calls `StorageProviderAbstract.teardown()`, a no-op that a storage plugin keeping a client open between operations overrides to close it. Measured from a laptop against a bucket in another region, the API server's read of a method tree of 20 blobs went from 4,261 ms of wall time and 425 ms of CPU to 1,582 ms and 32 ms with this change alone.
diff --git a/api/pipelex_api/method_source.py b/api/pipelex_api/method_source.py
index b3049273d..7e89f9e86 100644
--- a/api/pipelex_api/method_source.py
+++ b/api/pipelex_api/method_source.py
@@ -13,7 +13,8 @@ commit, read from the commit's git objects as stored, with no checkout filter, l
`export-ignore`, so its digest is the same on every machine, and filtered by the membership rule read with that
commit's own `.gitignore` files and the package's own `exclude` patterns. A submodule contributes no file, and a
symbolic link committed in the package is refused unless the rule excludes it. The tree's digest is recorded in the
-run's provenance (`method_provenance.tree_digest`), beside the commit.
+run's provenance (`method_provenance.tree_digest`), beside the commit and the `display_name` the package's manifest
+declares, which the fetched package's own provenance already carries.
The packages a package ships under its own `.mthds/methods/<name>/`, the layout of the local method cache, are out of
its tree (a dot directory), and each becomes a dependency tree of its own, keyed by the address its manifest declares,
@@ -291,5 +292,7 @@ def fetched_method(method_ref: str) -> FetchedMethod:
"""
with _fetched_package(method_ref) as package:
request_method = _read_package_method(package=package)
+ # The package's own provenance already holds the address, the tag, the commit and the manifest's display name;
+ # only the digest of the tree this runner read is added here.
provenance = package.provenance.model_copy(update={"tree_digest": request_method.tree.tree.digest})
return FetchedMethod(request_method=request_method, package=package, provenance=provenance)
diff --git a/api/pipelex_api/routes/pipelex/pipeline.py b/api/pipelex_api/routes/pipelex/pipeline.py
index 3eae4b51b..8e971494c 100644
--- a/api/pipelex_api/routes/pipelex/pipeline.py
+++ b/api/pipelex_api/routes/pipelex/pipeline.py
@@ -1043,8 +1043,9 @@ async def execute(request: Request) -> JSONResponse:
# `TokensUsageRecord` wire shape instead (pipelex owns the shape authority).
response_dump = response.model_dump(mode="json", serialize_as_any=True, by_alias=True)
apply_tokens_usage_wire_shape(response_dump, pipe_output=response.pipe_output)
- # Provenance of a `method_ref` run (address, tag, resolved commit SHA) — attached only when
- # the run was selected by reference, so the classic response stays byte-identical.
+ # Provenance of a `method_ref` run (address, tag, resolved commit SHA, tree digest, the manifest's
+ # display name) — attached only when the run was selected by reference, so the classic response
+ # stays byte-identical.
if source.method_provenance is not None:
response_dump["method_provenance"] = source.method_provenance.model_dump(mode="json")
return JSONResponse(content=response_dump)
@@ -1140,9 +1141,10 @@ async def start(
requested_orchestration_mode=extras.orchestration_mode,
callback_problem_view=problem_view_of(request),
)
- # The ack plus this server's provenance extension: `(address, tag, commit_sha)` for a
- # `method_ref` run, null otherwise — the caller (and, on hosted, the platform's Run row)
- # learns exactly what was fetched and at which commit.
+ # The ack plus this server's provenance extension: `(address, tag, commit_sha, tree_digest,
+ # display_name)` for a `method_ref` run, null otherwise — the caller (and, on hosted, the
+ # platform's Run row) learns exactly what was fetched, at which commit, and what the package
+ # calls itself.
return PipelexApiStartResponse(
**dict(start_result),
method_provenance=source.method_provenance,
diff --git a/api/pipelex_api/schemas/models.py b/api/pipelex_api/schemas/models.py
index 5fccf9f54..d2ddaf32a 100644
--- a/api/pipelex_api/schemas/models.py
+++ b/api/pipelex_api/schemas/models.py
@@ -615,9 +615,10 @@ class PipeOutputWire(PipeOutput):
_METHOD_PROVENANCE_DESCRIPTION = (
"PIPELEX-API EXTENSION (not part of the MTHDS Protocol) — provenance of a `method_ref` run: the package's "
- "resolved full address, the requested tag (null for a bare address), and the commit SHA that was actually "
- "fetched. The SHA is what keeps the run explainable when a tag moves. Absent (or null) for runs from inline "
- "source or a bundle."
+ "resolved full address, the requested tag (null for a bare address), the commit SHA that was actually "
+ "fetched, the digest of the method tree the run loaded, and the `display_name` the package's manifest declares "
+ "(null when it declares none). The SHA is what keeps the run explainable when a tag moves. Absent (or null) for "
+ "runs from inline source or a bundle."
)
diff --git a/api/tests/unit/test_method_ref_run.py b/api/tests/unit/test_method_ref_run.py
index 3d81236f8..89af30150 100644
--- a/api/tests/unit/test_method_ref_run.py
+++ b/api/tests/unit/test_method_ref_run.py
@@ -5,7 +5,7 @@ with a local git repository (`install_method_package` in conftest), so these pin
layer's contract: the fetched package is read from its commit's git objects as a method tree,
written as the run's one library directory, the caller's own; the entry pipe defaults to the
manifest's `main_pipe` (a request `pipe_code` overrides); provenance `{address, tag,
-commit_sha, tree_digest}` lands on the response; the exclusivity rules 422; the
+commit_sha, tree_digest, display_name}` lands on the response; the exclusivity rules 422; the
execution-locus gate refuses Python where it must; and each resolution failure maps to its
distinct code + status as RFC 7807 `problem+json`, the status being the code's.
"""
@@ -39,6 +39,13 @@ _TREE_NS = "pipelex_api.method_tree_request"
# The digest of the stub package's tree: its manifest and its one bundle.
_STUB_TREE_DIGEST = digest_of_contents({"METHODS.toml": STUB_METHOD_MANIFEST.encode("utf-8"), "documents.mthds": VALID_MTHDS.encode("utf-8")})
+# The stub manifest declaring the human-readable name a host names the run's method by.
+_DISPLAY_NAME = "Document Smoke Test"
+_MANIFEST_WITH_DISPLAY_NAME = STUB_METHOD_MANIFEST.replace('name = "documents"\n', f'name = "documents"\ndisplay_name = "{_DISPLAY_NAME}"\n', 1)
+_DISPLAY_NAME_TREE_DIGEST = digest_of_contents(
+ {"METHODS.toml": _MANIFEST_WITH_DISPLAY_NAME.encode("utf-8"), "documents.mthds": VALID_MTHDS.encode("utf-8")}
+)
+
_PIPE_FUNC_PY = "def echo(working_memory):\n return 'hi'\n"
_STRUCTURES_PY = """\
@@ -121,12 +128,14 @@ class TestMethodRefRun:
assert snapshot["pipe_code"] == "smoke.echo"
# The tree is removed once the request returns.
assert not Path(snapshot["library_dirs"][0]).exists()
- # Provenance: resolved full address, the requested tag, the fetched commit SHA, the digest of the tree it ran.
+ # Provenance: resolved full address, the requested tag, the fetched commit SHA, the digest of the tree it ran, and
+ # no display name, which the stub manifest does not declare.
assert response.json()["method_provenance"] == {
"address": STUB_METHOD_ADDRESS,
"tag": "v0.1.0",
"commit_sha": STUB_METHOD_COMMIT_SHA,
"tree_digest": _STUB_TREE_DIGEST,
+ "display_name": None,
}
@pytest.mark.parametrize(
@@ -180,6 +189,24 @@ class TestMethodRefRun:
"tag": None,
"commit_sha": STUB_METHOD_COMMIT_SHA,
"tree_digest": _STUB_TREE_DIGEST,
+ "display_name": None,
+ }
+
+ @pytest.mark.parametrize(("route", "expected_status"), [("/v1/execute", 200), ("/v1/start", 202)])
+ def test_provenance_carries_the_display_name_the_manifest_declares(
+ self, mocker: MockerFixture, install_method_package: Callable[..., Path], route: str, expected_status: int
+ ):
+ """Both run routes report the package's declared display name, which a host names the run's method by."""
+ install_method_package(files={"documents.mthds": VALID_MTHDS}, manifest_toml=_MANIFEST_WITH_DISPLAY_NAME)
+ client, _ = _build_client(mocker)
+ response = client.post(route, json={"method_ref": _METHOD_REF, "inputs": {"text": "hi"}})
+ assert response.status_code == expected_status, response.text
+ assert response.json()["method_provenance"] == {
+ "address": STUB_METHOD_ADDRESS,
+ "tag": "v0.1.0",
+ "commit_sha": STUB_METHOD_COMMIT_SHA,
+ "tree_digest": _DISPLAY_NAME_TREE_DIGEST,
+ "display_name": _DISPLAY_NAME,
}
def test_inline_runs_carry_no_provenance(self, mocker: MockerFixture):
diff --git a/docs/api-server/openapi/pipelex-api.openapi.yaml b/docs/api-server/openapi/pipelex-api.openapi.yaml
index 50488caef..b4678b7ba 100644
--- a/docs/api-server/openapi/pipelex-api.openapi.yaml
+++ b/docs/api-server/openapi/pipelex-api.openapi.yaml
@@ -4235,12 +4235,20 @@ components:
title: Tree Digest
description: The digest of the package's method tree, `sha256:` and 64 lowercase hex characters, as read from the
commit's git objects, set by a runner that reads the package as a tree, and null otherwise.
+ display_name:
+ anyOf:
+ - type: string
+ - type: 'null'
+ title: Display Name
+ description: The package's human-readable name, the `display_name` its `METHODS.toml` manifest declares at the fetched
+ commit, for a host to name the run's method by, and null when the manifest declares none.
type: object
required:
- address
- commit_sha
title: MethodProvenance
- description: 'What a fetched-method run must record: address, tag, and the resolved commit SHA.'
+ description: 'What a fetched-method run must record: address, tag, the resolved commit SHA, and the name the package
+ declares.'
MethodTreeRequest:
properties:
files:
@@ -6853,9 +6861,10 @@ components:
- $ref: '#/components/schemas/MethodProvenance'
- type: 'null'
description: 'PIPELEX-API EXTENSION (not part of the MTHDS Protocol) — provenance of a `method_ref` run: the package''s
- resolved full address, the requested tag (null for a bare address), and the commit SHA that was actually fetched.
- The SHA is what keeps the run explainable when a tag moves. Absent (or null) for runs from inline source or a
- bundle.'
+ resolved full address, the requested tag (null for a bare address), the commit SHA that was actually fetched,
+ the digest of the method tree the run loaded, and the `display_name` the package''s manifest declares (null when
+ it declares none). The SHA is what keeps the run explainable when a tag moves. Absent (or null) for runs from
+ inline source or a bundle.'
additionalProperties: true
type: object
required:
@@ -6910,9 +6919,10 @@ components:
- $ref: '#/components/schemas/MethodProvenance'
- type: 'null'
description: 'PIPELEX-API EXTENSION (not part of the MTHDS Protocol) — provenance of a `method_ref` run: the package''s
- resolved full address, the requested tag (null for a bare address), and the commit SHA that was actually fetched.
- The SHA is what keeps the run explainable when a tag moves. Absent (or null) for runs from inline source or a
- bundle.'
+ resolved full address, the requested tag (null for a bare address), the commit SHA that was actually fetched,
+ the digest of the method tree the run loaded, and the `display_name` the package''s manifest declares (null when
+ it declares none). The SHA is what keeps the run explainable when a tag moves. Absent (or null) for runs from
+ inline source or a bundle.'
additionalProperties: true
type: object
required:
diff --git a/docs/api-server/pipe-run.md b/docs/api-server/pipe-run.md
index 5924b1799..93ec4ed8b 100644
--- a/docs/api-server/pipe-run.md
+++ b/docs/api-server/pipe-run.md
@@ -87,7 +87,7 @@ Execute a Pipelex pipeline with flexible inputs and wait for completion.
- `pipe_output.tokens_usages` (array | null): Per-inference-call token usage in the client wire shape (`TokensUsageRecord`): `model_type`, model name/id, `pipe_code`, job-kind fields, `nb_tokens_by_category`, computed USD `cost` (`null` when the model has no rate table), and ISO timestamps. `null` when usage assembly was off for the run, `[]` when no inference happened. `pipe_output.usage_assembly_error` (string | null) is non-null when usage assembly failed. See the pipelex runtime's [TokensUsage Wire Records](https://docs.pipelex.com/under-the-hood/tokens-usage-wire-records/) for the full field reference.
- `pipe_output.graph_spec` (object | null): Best-effort execution graph of the run — the nodes and edges the run actually traversed. `null` when graph generation was off for the run, which is the `interpreter.pipeline_execution.is_generate_graph` setting (`true` in the packaged config); it does not depend on `runtime.tracing`, whose `is_enabled` selects a trace-emission backend rather than gating assembly. `pipe_output.graph_assembly_error` (string | null) is non-null when assembly ran and failed, which is what distinguishes a failed assembly from generation having been off.
- `pipe_output.pipe_io_artifacts` (object | null): The I/O artifacts describing the graph's data, carried beside `graph_spec` so that wherever the graph travels its description travels with it. It groups the same artifacts `/validate` produces — `pipe_io_contracts`, `input_form` and `output_form` — each keyed by namespaced `pipe_ref`, over the **run library's own pipes**: a dependency package's pipes are keyed by alias in the library but by bare `pipe_ref` in the artifacts, where they would collide with the host's, so they are left out and a graph node contributed by a dependency has no entry here. Holding it with the graph is what lets a viewer show a data node's value rather than only its shape. It is built only when the run generated a graph **and** `interpreter.pipeline_execution.graph.graphs_inclusion.graphspec_json` is on (both default to `true`), and only when `graph_spec` itself came out non-null — so a run with `graphspec_json = false` returns a graph with no artifacts beside it and no error, which is the one case where the two fields disagree. `pipe_output.pipe_io_artifacts_error` (string | null) is non-null when the build ran and failed. See [Pipe Validate](pipe-validate.md) for the artifacts' own field reference.
-- `method_provenance` (object, only on `method_ref` runs): `{address, tag, commit_sha}` — the package's resolved full address, the requested tag (`null` for a bare address), and the commit SHA that was actually fetched. Absent for runs from inline source or a bundle. See [Running a method by address](#running-a-method-by-address-method_ref).
+- `method_provenance` (object, only on `method_ref` runs): `{address, tag, commit_sha, tree_digest, display_name}` — the package's resolved full address, the requested tag (`null` for a bare address), the commit SHA that was actually fetched, the digest of the method tree the run loaded, and the `display_name` the package's `METHODS.toml` declares (`null` when it declares none). Absent for runs from inline source or a bundle. See [Running a method by address](#running-a-method-by-address-method_ref).
**Errors** are returned as [RFC 9457 `application/problem+json`](error-responses.md) bodies with HTTP 4xx/5xx status codes. The successful response body has no `status`/`error` field — the HTTP status code is the source of truth. An invalid bundle is refused before any pipe runs with a `422` `method.invalid` problem whose `errors` are the same located items validating it gives ([Structured validation errors](error-responses.md#structured-validation-errors)), and a run that fails reports its root fault at the pipe that failed ([Run failures](error-responses.md#run-failures-the-root-fault-located-at-the-failing-pipe)).
@@ -174,7 +174,7 @@ Start a pipeline execution and get its `pipeline_run_id` back with a `202` ack.
- `main_stuff_name` (null): Always `null`; populated only on the eventual completion callback.
- `pipe_output` (null): Always `null`; the result isn't ready yet.
- `workflow_id` (string | null): The async orchestrator's workflow ID. A `202` ack is only ever returned by an async-capable backend (e.g. a Temporal flavor), so this carries that orchestrator's id; the in-process `direct` base never acks here — it returns a `501` `runtime.unsupported` instead.
-- `method_provenance` (object | null): `{address, tag, commit_sha}` for a `method_ref` run — see [Running a method by address](#running-a-method-by-address-method_ref); `null` for runs from inline source or a bundle.
+- `method_provenance` (object | null): `{address, tag, commit_sha, tree_digest, display_name}` for a `method_ref` run — see [Running a method by address](#running-a-method-by-address-method_ref); `null` for runs from inline source or a bundle.
- `crate_fingerprint` (string | null): **Pipelex extension.** The fingerprint of the library crate the run's job was built from, as on the `/execute` result, known as soon as the run is queued, so a host records it from the acknowledgement.
**Errors** follow the same convention as `/execute`: HTTP 4xx/5xx with an [RFC 9457 `application/problem+json`](error-responses.md) body.
@@ -351,7 +351,7 @@ Both `/execute` and `/start` accept **`method_ref`** — a globally resolvable a
**Entry pipe.** The pipe to run is the tree's entry pipe by the protocol's rule, the package manifest's `main_pipe`; an explicit `pipe_code` in the request overrides it (to run another pipe from the fetched package).
-**Provenance.** The response carries `method_provenance = {address, tag, commit_sha, tree_digest}` — on `/execute` only for `method_ref` runs, on the `/start` ack as a nullable field. Tags can move; the recorded SHA is what keeps a run explainable after the fact, and `tree_digest` names the exact tree the run loaded.
+**Provenance.** The response carries `method_provenance = {address, tag, commit_sha, tree_digest, display_name}` — on `/execute` only for `method_ref` runs, on the `/start` ack as a nullable field. Tags can move; the recorded SHA is what keeps a run explainable after the fact, and `tree_digest` names the exact tree the run loaded. `display_name` is the human-readable name the package's `METHODS.toml` declares at that commit, such as `Slide Designer`, which a host can name the run's method by, and `null` when the manifest declares none.
**Custom Python follows the tree rules, plus one more.** What decides is *where the code would execute*: `.mthds` content is data, always acceptable. On a deployment that is **not** sandbox-hosted, a fetched package carrying any `.py` is refused with a `501` `pipelex.deployment.custom_code_requires_sandbox`, whose `origin` is `package`. On a sandbox-hosted deployment, PipeFunc `.py` is accepted (the runner never imports it: validation reads its text, and each PipeFunc call runs in the isolated sandbox, which receives the files of the step's tree) — but a package declaring **Python structure classes** (`StructuredContent` subclasses) is refused with a `422` (`pipelex.deployment.python_structures_refused`), exactly as a tree sent with the request is: see [No Python structure classes](#sending-a-method-as-a-tree).
diff --git a/docs/building-methods/packages.md b/docs/building-methods/packages.md
index 081fed068..20223a3a7 100644
--- a/docs/building-methods/packages.md
+++ b/docs/building-methods/packages.md
@@ -45,7 +45,7 @@ pipes = ["compute_weighted_score"]
| `version` | Yes | Semantic version such as `1.0.0` or `2.1.3-beta.1` |
| `description` | Yes | Human-readable package description |
| `name` | No | Optional short method name |
-| `display_name` | No | Optional display label |
+| `display_name` | No | Optional human-readable name, such as `Slide Designer`; a run of the package fetched by its address records it in its provenance |
| `authors` | No | List of author names |
| `license` | No | License string such as `MIT` |
| `main_pipe` | No | Optional main pipe code |
@@ -130,7 +130,7 @@ steps = [
]
```
-An address-based reference resolves against the **installed methods** — `~/.mthds/methods/` (global), `.mthds/methods/` in the working directory (project-local), and a `.mthds/methods/` directory beside the bundle being loaded or in a directory above it — matched by the manifest's `address` + `name`, case-insensitively. When no installed method matches, Pipelex **fetches on miss**: it fetches the package by address — honoring an `@<tag>` pin, through the same grammar, bounds, and tags-only rules as a direct fetch — installs it into `~/.mthds/methods/`, records the fetch provenance (address, tag, and the commit SHA of what was actually cloned) in a `.provenance.json` sidecar beside the manifest, and loads it so resolution proceeds. The next load finds the installed copy and never touches the network again.
+An address-based reference resolves against the **installed methods** — `~/.mthds/methods/` (global), `.mthds/methods/` in the working directory (project-local), and a `.mthds/methods/` directory beside the bundle being loaded or in a directory above it — matched by the manifest's `address` + `name`, case-insensitively. When no installed method matches, Pipelex **fetches on miss**: it fetches the package by address — honoring an `@<tag>` pin, through the same grammar, bounds, and tags-only rules as a direct fetch — installs it into `~/.mthds/methods/`, records the fetch provenance (address, tag, the commit SHA of what was actually cloned, and the manifest's `display_name`) in a `.provenance.json` sidecar beside the manifest, and loads it so resolution proceeds. The next load finds the installed copy and never touches the network again.
A few behaviors worth knowing:
diff --git a/pipelex/cli/installed_methods.py b/pipelex/cli/installed_methods.py
index 7e7f289d5..208cb640a 100644
--- a/pipelex/cli/installed_methods.py
+++ b/pipelex/cli/installed_methods.py
@@ -25,7 +25,9 @@ GLOBAL_METHODS_DIR = Path.home() / ".mthds" / "methods"
PROJECT_METHODS_DIR = Path(".mthds") / "methods"
# Written beside METHODS.toml when a method is installed by fetch: the address, the tag when one
-# was named, and the commit SHA of what was actually cloned — tags can move, the SHA explains runs.
+# was named, the commit SHA of what was actually cloned — tags can move, the SHA explains runs —
+# and the display name the manifest declares. A sidecar written before the display name was
+# recorded still reads, its display name null.
PROVENANCE_FILENAME = ".provenance.json"
_STAGING_SUFFIX = ".staging"
@@ -35,7 +37,7 @@ class InstalledMethod(BaseModel):
"""An installed method discovered from the filesystem.
``provenance`` is set only for methods fetched by reference: the address, the tag when
- one was named, and the commit SHA of what was actually cloned.
+ one was named, the commit SHA of what was actually cloned, and the manifest's display name.
"""
name: str
diff --git a/pipelex/methods/fetch_on_miss.py b/pipelex/methods/fetch_on_miss.py
index 90bd56226..2bfbf7a51 100644
--- a/pipelex/methods/fetch_on_miss.py
+++ b/pipelex/methods/fetch_on_miss.py
@@ -134,7 +134,7 @@ def resolve_address_based_method(
lookup — the installed store is keyed by address alone). On a miss, when fetch-on-miss is
enabled and the address is fetchable, fetches the package by reference and installs it
into the installed-methods store, recording the fetch provenance (address, tag, commit
- SHA) beside the manifest.
+ SHA, and the manifest's display name) beside the manifest.
Args:
full_address: The address-based alias as written in the bundle, e.g.
diff --git a/pipelex/methods/fetching.py b/pipelex/methods/fetching.py
index 0b7e0d5a5..bf9236cb1 100644
--- a/pipelex/methods/fetching.py
+++ b/pipelex/methods/fetching.py
@@ -29,7 +29,7 @@ _SKIPPED_DIR_NAMES = {".git"}
class MethodProvenance(BaseModel):
- """What a fetched-method run must record: address, tag, and the resolved commit SHA."""
+ """What a fetched-method run must record: address, tag, the resolved commit SHA, and the name the package declares."""
model_config = ConfigDict(frozen=True)
@@ -43,6 +43,14 @@ class MethodProvenance(BaseModel):
"git objects, set by a runner that reads the package as a tree, and null otherwise."
),
)
+ # Defaulted, so a record written before the field existed, such as an installed copy's `.provenance.json`, still loads.
+ display_name: str | None = Field(
+ default=None,
+ description=(
+ "The package's human-readable name, the `display_name` its `METHODS.toml` manifest declares at the fetched commit, "
+ "for a host to name the run's method by, and null when the manifest declares none."
+ ),
+ )
class FetchedMethodPackage(BaseModel):
@@ -59,7 +67,12 @@ class FetchedMethodPackage(BaseModel):
@property
def provenance(self) -> MethodProvenance:
- return MethodProvenance(address=self.full_address, tag=self.ref.tag, commit_sha=self.commit_sha)
+ return MethodProvenance(
+ address=self.full_address,
+ tag=self.ref.tag,
+ commit_sha=self.commit_sha,
+ display_name=self.manifest.display_name,
+ )
CLONE_DIRECTORY_PLACEHOLDER = "<clone directory>"
diff --git a/tests/unit/pipelex/cli/test_install_method_package.py b/tests/unit/pipelex/cli/test_install_method_package.py
index ba66c6d87..886de0e6b 100644
--- a/tests/unit/pipelex/cli/test_install_method_package.py
+++ b/tests/unit/pipelex/cli/test_install_method_package.py
@@ -105,6 +105,36 @@ class TestInstallMethodPackage:
assert found.path == installed.path
assert found.provenance is None
+ @pytest.mark.parametrize(
+ "sidecar",
+ [
+ pytest.param(
+ b'{"address": "github.com/pipelex-tests/fom-methods/scoring", "tag": "v0.1.0", "commit_sha": "' + b"a" * 40 + b'"}',
+ id="before-tree-digest",
+ ),
+ pytest.param(
+ b'{"address": "github.com/pipelex-tests/fom-methods/scoring", "tag": "v0.1.0", "commit_sha": "'
+ + b"a" * 40
+ + b'", "tree_digest": null}',
+ id="before-display-name",
+ ),
+ ],
+ )
+ def test_discovery_reads_a_sidecar_an_earlier_release_wrote(self, tmp_path: Path, sidecar: bytes) -> None:
+ """A sidecar written before the provenance recorded a display name still reads, its display name null."""
+ package_dir = _make_package_dir(tmp_path)
+ methods_dir = tmp_path / "methods"
+ installed = install_method_package(package_dir=package_dir, name="scoring", provenance=PROVENANCE, methods_dir=methods_dir)
+ (installed.path / PROVENANCE_FILENAME).write_bytes(sidecar)
+
+ methods = discover_installed_methods(include_global=False, include_project=False, extra_search_dirs=[methods_dir])
+ found = find_method_by_full_address(FULL_ADDRESS, methods=methods)
+
+ assert found is not None
+ assert found.provenance is not None
+ assert found.provenance == PROVENANCE
+ assert found.provenance.display_name is None
+
def test_install_refuses_a_target_occupied_by_a_non_package(self, tmp_path: Path) -> None:
"""An existing target directory that is not a method package is never overwritten."""
package_dir = _make_package_dir(tmp_path)
diff --git a/tests/unit/pipelex/methods/test_fetch_on_miss.py b/tests/unit/pipelex/methods/test_fetch_on_miss.py
index a08d2e4b3..8b5ff6cbc 100644
--- a/tests/unit/pipelex/methods/test_fetch_on_miss.py
+++ b/tests/unit/pipelex/methods/test_fetch_on_miss.py
@@ -41,6 +41,8 @@ main_pipe = "compute"
pipes = ["compute"]
"""
+MANIFEST_WITH_DISPLAY_NAME = MANIFEST.replace('name = "scoring"\n', 'name = "scoring"\ndisplay_name = "Scoring Engine"\n', 1)
+
STRUCTURES_MODULE = """\
from pipelex.core.stuffs.structured_content import StructuredContent
@@ -50,15 +52,15 @@ class Invoice(StructuredContent):
"""
-def _make_package_dir(base: Path) -> Path:
+def _make_package_dir(base: Path, *, manifest: str = MANIFEST) -> Path:
package_dir = base / "fetched-pkg"
package_dir.mkdir(parents=True)
- (package_dir / MANIFEST_FILENAME).write_text(MANIFEST, encoding="utf-8")
+ (package_dir / MANIFEST_FILENAME).write_text(manifest, encoding="utf-8")
(package_dir / "scoring.mthds").write_text("# placeholder", encoding="utf-8")
return package_dir
-def _make_fetched(package_dir: Path, *, ref_str: str = f"{FULL_ADDRESS}@v0.1.0") -> FetchedMethodPackage:
+def _make_fetched(package_dir: Path, *, ref_str: str = f"{FULL_ADDRESS}@v0.1.0", manifest: str = MANIFEST) -> FetchedMethodPackage:
ref = parse_method_ref(ref_str)
return FetchedMethodPackage(
ref=ref,
@@ -66,7 +68,7 @@ def _make_fetched(package_dir: Path, *, ref_str: str = f"{FULL_ADDRESS}@v0.1.0")
commit_sha=COMMIT_SHA,
clone_dir=package_dir.parent,
package_dir=package_dir,
- manifest=parse_methods_toml(MANIFEST),
+ manifest=parse_methods_toml(manifest),
)
@@ -166,6 +168,30 @@ class TestResolveAddressBasedMethod:
assert resolved_again.path == resolved.path
assert fetch_mock.call_count == 1
+ @pytest.mark.parametrize(
+ ("manifest", "expected_display_name"),
+ [
+ pytest.param(MANIFEST_WITH_DISPLAY_NAME, "Scoring Engine", id="declared"),
+ pytest.param(MANIFEST, None, id="not-declared"),
+ ],
+ )
+ def test_installed_copy_records_the_manifest_display_name(
+ self, isolated_methods_dirs: Path, tmp_path: Path, mocker: MockerFixture, manifest: str, expected_display_name: str | None
+ ) -> None:
+ """The provenance sidecar of a fetched copy keeps the manifest's display name, which the next resolve reads back."""
+ mocker.patch("pipelex.methods.fetch_on_miss.is_method_fetch_on_miss_enabled", return_value=True)
+ package_dir = _make_package_dir(tmp_path, manifest=manifest)
+ fetch_mock = mocker.patch("pipelex.methods.fetch_on_miss.fetch_method_package", return_value=_make_fetched(package_dir, manifest=manifest))
+
+ resolved = resolve_address_based_method(full_address=f"{FULL_ADDRESS}@v0.1.0")
+ resolved_again = resolve_address_based_method(full_address=FULL_ADDRESS)
+
+ assert fetch_mock.call_count == 1
+ assert resolved.path == (isolated_methods_dirs / "scoring").resolve()
+ expected = MethodProvenance(address=FULL_ADDRESS, tag="v0.1.0", commit_sha=COMMIT_SHA, display_name=expected_display_name)
+ assert resolved.provenance == expected
+ assert resolved_again.provenance == expected
+
def test_bare_name_collision_across_addresses_is_a_loud_error(self, isolated_methods_dirs: Path, tmp_path: Path, mocker: MockerFixture) -> None:
"""Two packages sharing the bare name never silently load or overwrite each other — the collision names both addresses."""
package_dir = _make_package_dir(tmp_path)
diff --git a/tests/unit/pipelex/methods/test_fetching.py b/tests/unit/pipelex/methods/test_fetching.py
index 81bfe2bfa..edb9bf1e1 100644
--- a/tests/unit/pipelex/methods/test_fetching.py
+++ b/tests/unit/pipelex/methods/test_fetching.py
@@ -36,6 +36,8 @@ main_pipe = "extract"
pipes = ["extract"]
"""
+MANIFEST_WITH_DISPLAY_NAME = MANIFEST.replace('name = "documents"\n', 'name = "documents"\ndisplay_name = "Document Extractor"\n', 1)
+
STRUCTURES_MODULE = """\
from pipelex.core.stuffs.structured_content import StructuredContent
@@ -45,10 +47,10 @@ class Invoice(StructuredContent):
"""
-def _write_package(destination: Path) -> None:
+def _write_package(destination: Path, *, manifest: str = MANIFEST) -> None:
package_dir = destination / "methods" / "documents"
package_dir.mkdir(parents=True, exist_ok=True)
- (package_dir / "METHODS.toml").write_text(MANIFEST, encoding="utf-8")
+ (package_dir / "METHODS.toml").write_text(manifest, encoding="utf-8")
(package_dir / "documents.mthds").write_text("# placeholder", encoding="utf-8")
@@ -98,6 +100,36 @@ class TestFetchMethodPackage:
assert tag_check.call_count == 1
assert fetched.provenance.tag == "v0.2.0"
+ @pytest.mark.parametrize(
+ ("manifest", "expected_display_name"),
+ [
+ pytest.param(MANIFEST_WITH_DISPLAY_NAME, "Document Extractor", id="declared"),
+ pytest.param(MANIFEST, None, id="not-declared"),
+ ],
+ )
+ def test_provenance_carries_the_manifest_display_name(
+ self, mocker: MockerFixture, tmp_path: Path, manifest: str, expected_display_name: str | None
+ ) -> None:
+ """The provenance records the display name the fetched manifest declares, and null when it declares none."""
+
+ def fake_clone(*, clone_url: str, destination: Path) -> None:
+ assert clone_url == "https://github.com/Pipelex/methods.git"
+ _write_package(destination, manifest=manifest)
+
+ mocker.patch("pipelex.methods.fetching.clone_default_branch", side_effect=fake_clone)
+ mocker.patch("pipelex.methods.fetching.resolve_head_commit_sha", return_value=FAKE_SHA)
+
+ fetched = fetch_method_package(ref=parse_method_ref("github.com/Pipelex/methods/documents"), dest_dir=tmp_path)
+
+ assert fetched.manifest.display_name == expected_display_name
+ assert fetched.provenance.model_dump(mode="json") == {
+ "address": "github.com/Pipelex/methods/documents",
+ "tag": None,
+ "commit_sha": FAKE_SHA,
+ "tree_digest": None,
+ "display_name": expected_display_name,
+ }
+
def test_clone_failure_is_a_method_fetch_error(self, mocker: MockerFixture, tmp_path: Path) -> None:
"""A VCS failure surfaces as MethodFetchError naming the reference."""
mocker.patch("pipelex.methods.fetching.clone_default_branch", side_effect=VCSFetchError("connection refused"))