Skip to content

Build AI methods with your coding agent, then run them anywhere: as an MCP for chatbots, as a webapp for people, or via API for your software.

fix(docs): update deferred follow-ups and cleanup notes for cost reporting tests

Louis Choqueld115f52dc922
diff --git a/wip/registry/README.md b/wip/registry/README.md
index 499ff8ae3..9a06f5a93 100644
--- a/wip/registry/README.md
+++ b/wip/registry/README.md
@@ -12,7 +12,7 @@ Shipped in **PR #967** on `fix/For-API-update`, with the companion rename **#968
## Open follow-ups
-- [`deferred-followups.md`](deferred-followups.md) — the deliberate non-goals: the costs-only in-memory tracer skip (E1 deep half), cost-per-node correlation, the A1 `TraceContext.from_execution_config` factory, T5, T3 request-scoped tracing state, `--mock-inference` coverage, and the leftover `graph_id → run_id` rename + shelved `TraceContext` split.
+- [`deferred-followups.md`](deferred-followups.md) — the deliberate non-goals: the costs-only in-memory tracer skip (E1 deep half), cost-per-node correlation, the A1 `TraceContext.from_execution_config` factory, T5, T3 request-scoped tracing state, the costs-only event-log explicit-close cleanup (PR #977 re-review), `--mock-inference` coverage, and the leftover `graph_id → run_id` rename + shelved `TraceContext` split.
- [`cost-report-deferred-decisions.md`](cost-report-deferred-decisions.md) — two deferred **design decisions**: (#3) cost reporting is coupled to `tracing_config.is_enabled` (costs-on + tracing-off ⇒ no report); (#6) the agent CLI gates costs on the raw `costs` param while the main CLI gates on the resolved `is_generate_usage`. Both want a deliberate call.
- [`usage-reporting-without-cost.md`](usage-reporting-without-cost.md) — capability idea (low priority): make usage (token counts) a separately-requestable output, with cost as one view layered on top.
diff --git a/wip/registry/deferred-followups.md b/wip/registry/deferred-followups.md
index 504592e92..2b047b83c 100644
--- a/wip/registry/deferred-followups.md
+++ b/wip/registry/deferred-followups.md
@@ -22,6 +22,8 @@ Two further deferred items — both *design decisions* rather than follow-up wor
**Facet flagged on PR #967 (codex, P1) — activity usage writes can mutate a workflow-registered buffer.** In a *single-process* Temporal deployment (one worker hosting both `WfPipeRouter` workflows and the inference activities), the workflow registers its `BufferingEventLog` in the process-global `ReportingManager._event_log_contexts`. An inference activity in the same process then hits the fast path (`_emit_via_registered_context`) and appends to the workflow's in-memory buffer — an activity side effect that is **not** part of Temporal's deterministic replay (the activity *result* replays from history, but the buffer append does not), so on replay the later drain/flush can diverge or drop usage. **Why this is deferred, not patched on this PR:** (1) it is *pre-existing* — the fast path + per-context registration predate this work (the registry track kept the emit core unchanged; it only removed the submitter-side registry), so PR #967 neither introduced nor worsened it; (2) the **recommended split router+runner topology does not hit it** — activities run on the runner process, which never registered the workflow's buffer, so they correctly take the per-process activity-log fallback; (3) the clean fix is request-scoped tracing state (this T3 refactor) or making activities always bypass workflow-registered buffers — bolting Temporal-awareness (`activity.in_activity()`) into the generic, non-Temporal `ReportingManager` would be a layering violation; (4) Temporal is not shipped to prod. Pick this up with the T3 refactor.
+- **Costs-only event-log handle relies on `__del__`, not an explicit close (codex P2, PR #977 v0.32.0 release re-review).** In `--no-graph --costs` mode, `pipeline_run_setup` creates the `event_log` and registers it on the report delegate (`set_event_log`) but passes `event_log=None` to the tracer (D5: costs-only emits no graph events). So the tracer never owns it, and `GraphTracer.teardown()` — the only path that *explicitly* closes an event log — has nothing to close. On run end, `runner.py` (and the setup error-path) call `ReportingManager.clear_event_log`, which only `pop`s the `_event_log_contexts` entry; it does **not** close the log. The NDJSON file handle is therefore closed only by `NdjsonEventLog.__del__`. **This is *not* the unbounded FD leak the finding described:** `make_event_log` returns a *fresh* `NdjsonEventLog` per run, so once `clear_event_log` drops the delegate's reference (and the setup-local falls out of scope) the instance is refcount-collected and `__del__` closes the handle promptly under CPython — no descriptor growth across runs. The valid residual is that the costs-only path has no *explicit* close and leans on GC finalization. **Clean fix:** have `ReportingManager.clear_event_log` close the popped context's `event_log` before discarding it (`NdjsonEventLog.close()` is idempotent — in `--graph --costs` mode the tracer's `teardown` close and this close are both safe, the second being a no-op). Deferred off the v0.32.0 release branch (docs-only pass); land via normal `dev` flow. Closely related to **T3** above — the request-scoped `_event_log_contexts` refactor would subsume this.
+
## Pre-existing gaps surfaced during the PR #967 final review (flag, not fix-here)
These were found by the gstack `/review` adversarial pass (Codex + Claude). They are **pre-existing** (not introduced by the registry/cost work) and out of scope for this PR's purpose, so they are flagged here rather than expanding the PR. Per the repo "flag and fix existing bugs" rule, each is a candidate follow-up.
diff --git a/wip/runtime-bridge/README.md b/wip/runtime-bridge/README.md
index 1d57f57b5..c1fd82732 100644
--- a/wip/runtime-bridge/README.md
+++ b/wip/runtime-bridge/README.md
@@ -83,6 +83,7 @@ Carried forward, each recorded so review tooling doesn't re-raise them as new fi
- **[execution-mode-property-helpers-unused.md](execution-mode-property-helpers-unused.md)** — the three `PipelexExecutionMode` `@property` helpers are tested but not yet consumed by `bridge.py` (which inlines the equivalent checks). Wire them in or remove; no correctness impact.
- **[library-manager-pipe-source-map-concurrency.md](library-manager-pipe-source-map-concurrency.md)** — pre-existing `LibraryManager._pipe_source_map` is manager-level, not per-call scoped. Observability-only, Temporal not in prod.
- **[bridge-bootstrap-external-race.md](bridge-bootstrap-external-race.md)** — residual race against a *concurrent external* `Pipelex.make()` mid-setup; narrow/non-standard, bounded-wait fix deferred.
+- **[cost-reporting-test-cleanup-followups.md](cost-reporting-test-cleanup-followups.md)** — §3 (open, PR #977 re-review): the cross-worker cost assert (`assert_cross_worker_cost.py`) selects its CSV by globbing the *global* reports dir (lexicographically-latest), not the run being asserted, so a concurrent/earlier run's report can be compared against the current NDJSON. Dev/test helper only.
- **[distributed-cost-reporting-test-coverage.md](distributed-cost-reporting-test-coverage.md)** — audit (from a full `/temporal-e2e-validate` Mode 2 run) of cost-reporting test coverage in split-worker mode. No automated test asserts real-inference distributed cost numbers; the real-provider *capture* branch is asserted nowhere; cross-child sum and img-gen/extract/search usage are uncovered; plus two real risks (R2 retry over-count — documented/accepted; zero-token silent emit). Code-grounded map + prioritized test additions by layer, written as the input for a follow-up planning session.
## Remaining
diff --git a/wip/runtime-bridge/cost-reporting-test-cleanup-followups.md b/wip/runtime-bridge/cost-reporting-test-cleanup-followups.md
index 351ec5c0f..4791be30d 100644
--- a/wip/runtime-bridge/cost-reporting-test-cleanup-followups.md
+++ b/wip/runtime-bridge/cost-reporting-test-cleanup-followups.md
@@ -1,6 +1,6 @@
-# Cost-reporting test additions — two cleanup follow-ups
+# Cost-reporting test additions — cleanup follow-ups
-**Status:** 🧹 Optional cleanups, no correctness impact. Both items resolved. Surfaced by a code review of the distributed-cost-reporting test additions (see [`distributed-cost-reporting-test-coverage.md`](distributed-cost-reporting-test-coverage.md), "Resolution (as-built)"). The review found **no bugs**; these were cosmetic dead-code / weak-assertion items.
+**Status:** 🧹 Optional cleanups, no correctness impact. Items #1–#2 surfaced by a code review of the distributed-cost-reporting test additions (see [`distributed-cost-reporting-test-coverage.md`](distributed-cost-reporting-test-coverage.md), "Resolution (as-built)") and are resolved. Item #3 surfaced on the v0.32.0 release PR #977 and is deferred (dev/test helper only). The reviews found **no shipped-code bugs**; these are cosmetic dead-code / weak-assertion / test-helper-correctness items.
---
@@ -44,6 +44,20 @@ assert _FAKE_RUNNER_MODEL_NAME not in model_names, "Real inference must report a
---
+## 3. Global CSV selection in the cross-worker cost assert ⏭️ DEFERRED
+
+**File:** `.claude/skills/temporal-e2e-validate/scripts/assert_cross_worker_cost.py` (`_csv_token_totals`, ~lines 54-75)
+
+**Status:** ⏭️ Deferred — dev/test helper only (not shipped library code). Surfaced by **greptile (P2)** on the v0.32.0 release PR #977; left open on that PR for a fast-follow.
+
+`_csv_token_totals(reports_dir)` picks the cost-report CSV to compare against with `sorted(reports_dir.glob("cost_report*.csv"))[-1]` — the lexicographically-latest file in the **global** reports dir — even though the caller passes the run-specific `--run-dir` for the NDJSON side of the comparison. If another run already wrote a later-numbered `cost_report*.csv`, the assert compares the current run's NDJSON usage against a **stale CSV from a different run**, which can fail a good run or pass a bad one.
+
+This is the same helper as item #1, but a distinct concern: item #1 was about the *column* read inside a row (`nb_tokens_input` fallback, ~line 224); this is about *which file* is selected (~lines 54-75).
+
+**Why deferred, not fixed:** it's a `.claude/skills/` test helper, out of scope for the docs-only v0.32.0 release-branch pass. Real enough to fix on `dev`.
+
+**Fix:** tie CSV selection to the run being asserted — either point `--reports-dir` at a per-run subdir (e.g. `run_dir / "reports"`) and glob there, or filter `cost_report*.csv` by the run id derived from `run_dir.name`. Re-confirm where the runner writes `cost_report*.csv` relative to `--run-dir` before choosing.
+
## Verification
These files' tests are green as-is. After either change, re-run: