Skip to content

The fastest, cheapest, most efficient open-source agent harness. Run more than 500 agents on a $10 VPS.

19.9 KB
Raw Blame

Private Rust validation inventory and measurements

This inventory records the test selections before changing execution for #8063. Coverage replaces equivalent executions inside the private job. Feature sets are not treated as equivalent merely because they select the same crate.

Selection Earlier execution Private execution
openhuman product --lib --bins coverage nextest, background unit build coverage nextest, sequential
openhuman-cli product integration targets coverage nextest kind(test) same, fails empty selection
openhuman-embed, openhuman-tinyhumans product, all targets coverage libtest coverage nextest
openhuman-embed, openhuman-tinyhumans default, all targets omitted as presumed duplicates separate coverage nextest configurations
openhuman-rpc, openhuman-tui default, all targets coverage libtest coverage nextest
core no-default lib, existing gate filters plain libtest coverage nextest with equivalent substring unions
core no-default plus modules,mcp,e2e-test-support, gate filters serial plain libtest serial coverage nextest
Tauri own manifest/default features coverage libtest coverage nextest
CLI storage E2E targets, SQLite only plain libtest coverage nextest, same targets
core storage domain filters excluding cron module tests plain libtest coverage nextest, same inclusions/exclusions
core storage cron bus/documents tests serial plain libtest serial coverage nextest
RPC session-store,storage-sqlite, session_store:: plain libtest coverage nextest

The storage E2E list and domain filter list remain declared in lanes-plan.mjs. Product features remain in scripts/ci/product-features.txt; they are resolved per package, preserving existing feature forwarding. Checksum-pinned native test modules, coverage-presence checks and the 80% changed-line gate remain required. Doctests, clippy, rustdoc, examples and uninstrumented link checks remain separate because coverage/nextest does not replace them.

The Composio stack-budget regression has #![cfg(not(coverage))] and runs once through uninstrumented nextest in rust-lint. Instrumented integration expectations exclude that target and, outside Windows, the entirely #[cfg(windows)] sandbox target. Present binaries containing only ignored tests are checked against nextest's compiled inventory; all other missing or empty expected targets fail validation.

Full-workflow overlap audit

CI Full calls test-reusable.yml, whose core job already enumerates all 46 root Rust integration targets. The Rust E2E runner selects 35 of those same target names: tests/*_e2e.rs plus in_process_all. Job names do not establish that their test selections are distinct.

Executions or mode Overlap and configuration differences Decision
CI Full's direct Rust E2E job and its e2e-desktop call's nested rust-e2e-linux job Both invoke pnpm test:rust:e2e, selecting the same 35 targets with product features and the Node mock. The direct job pins its container digest and sets line-table debug information; the reusable job uses a moving image tag and inherits its own workflow's OPENHUMAN_NODE_ENABLED=0. Caller workflow environment is not automatically inherited by a reusable workflow. Retain: repeated selections, but runtime environments are not proven equivalent.
Reusable core integrations and Rust E2E All 35 E2E target names occur in the core job's 46-target inventory. The core job selects -p openhuman-cli with product features plus bin-tools; E2E selects the workspace with product features and starts an ambient Node mock. Individual tests may also start their own mocks or clear inherited backend settings. Retain: package, feature, mock and process setup differ; a shared target name does not prove equivalent execution.
mcp_registry_e2e The E2E runner explicitly adds --ignored --test-threads=1; ordinary integration execution skips these ignored cases. Preserve this distinct test mode.
tinymcp_storage_smoke_e2e The E2E runner adds e2e-test-support and serial execution. Preserve this distinct feature configuration.
tinydocs_linked_e2e The E2E runner adds non-product linked-modules. Preserve this distinct feature configuration.
Browser/desktop E2E and Windows secrets validation These exercise frontend or native host flows, Windows-specific code, or external-service configuration. Preserve these distinct contexts.

These full/reusable workflows do not invoke the private coverage lane. No execution above is removed on the assumption that private coverage replaces an equivalent selection in the same validation run. Further deduplication would require checking the resolved package features and normalizing the relevant runtime settings first.

The current local callers of test-reusable.yml are ci-full.yml and test.yml; the callers of e2e-reusable.yml are ci-full.yml and e2e.yml. Comments describing direct staging/production pretest callers are stale. The staging and production workflows remain untouched by this change.

Historical warm baseline

Run 39006804022, head a90ebba82e4bbfe4d454107620b52bf18092932a, completed its EX63 job on 2026-10-17 from 05:53:35 to 06:22:35 UTC: 29m00s, excluding approximately 18m15s queue time. This used parallel lanes and the previous sccache backend.

  • Cache: 5,373 Rust hits, zero misses. This is a warm baseline, not a cold run.
  • Core nextest: 10,168 unit tests and 447 integration tests passed; 10 and 19 skipped.
  • Remaining core coverage libtest records: 945 passed across 89 result records.
  • Tauri: 309 passed, one ignored. Gates-off: 488 passed. Storage: 2,549 passed.
  • Coverage: 1,554 eligible files present; changed-line report 86% over 125 measured changed lines (17 violations). Four LCOV reports were archived.
  • Core coverage check: 20m23s; unit execution 546.719s; integration 316.683s.
  • Cache save time is unavailable for this old backend.

These counts are executions, not deduplicated test identities. The historical commit and runner allocation differ from the implementation benchmark, so this is context rather than a controlled speed-up claim.

Measurement protocol

The initial diagnostic at 0b93ecf6bc1012fa06886c87b30ee0eebbd7e5f0 finished with a failing aggregate result in 117m49s: lanes took 108m24s and cache saving took 8m36s. Its resource limit changed mid-run, so these timings are not a controlled cold measurement. All eight main Rust configurations passed their 12,225 test executions; all 23 offline examples passed. The diagnostic exposed missing vendor tag history, the frontend lint heap limit, an unsupported llvm-cov show-env --features option, and conflicting auxiliary coverage flags. The seven auxiliary coverage configurations failed before running tests and must not be counted as covered merely because an LCOV report was produced.

A second diagnostic at e35e5c0240b61bc8cffaf857f8f826c4276ce8b6 failed after 107m03s when the 160 GiB scratch disk filled during the ordinary Tauri link. The subsequent coverage build and status writes reported ENOSPC; storage suites, Pester and cache saving were not reached. The 12,714 recorded executions across ten instrumented configurations and the plain Composio test passed, but this was a failed workflow and no warm run followed it.

Its closed artifacts showed per-process auxiliary coverage profiles and large ordinary debug sections. The corrected private plan bounds every coverage process to eight merging profile slots, removes only the completed release example graph, and uses line-tables-only for ordinary dev/test builds. Coverage keeps its existing debug level zero. Actual Cargo 1.96.1 checks confirmed dev and test file-and-line backtraces and assertions, the coverage override, and unchanged release artifacts. Release output includes reflinks, so its allocated size does not establish how much physical space cleanup recovers. These fixes require a new complete cold/warm pair; neither diagnostic establishes final performance.

The private workflow budget is now 180 minutes, including a bounded 15-minute cache save. The earlier 120-minute budget had almost expired before those seven configurations could execute; the provisional five-minute save limit was below the observed 8m36s save time. Final measurements must use the corrected source and unchanged resources throughout each run.

Run the complete selected lane plan twice at the same source revision and image, using fresh target outputs. Preserve package-manager caches, and identify whether the signed remote Rust cache already contains those compiler keys. Do not call a run cold merely because its target directory is new. Record Tinymold hit/miss and save time, test counts, coverage presence, linkage proof, each check's result and elapsed time. Preserve failed runs too. After merging, repeat through trusted-main workflow routing, since pre-merge runs can execute the old workflow.

Complete Linux manual validation

Both runs at 3a01ec0c54a54f30a14ab92cd5fdbf241d6e9316 independently passed with identical selected test identities. The measurement record contains exact counters, resource limits and archive digests. Both used the same published image, frozen injected rootfs/kernel, four vCPUs, 13 GiB RAM, 160 GiB scratch, 32 GiB guest swap and the production 58 GiB shared host limit. The measured contract used a 180-minute full-entry budget and 195-minute guest deadline; there were no additional per-unit CPU or memory caps.

Metric Cold Warm
Validation lane interval 8,631.081 s 4,134.076 s
Full manual entry, including cache save 8,869 s (2h27m49s) 4,172 s (1h09m32s)
Whole VM lifecycle 8,877.017 s 4,180.850 s
Rust test executions 15,572 15,572
Rust failures/errors 0 0
Local compiler-cache hits 2,286 2,717
Remote compiler-cache hits 0 3,801
Compiler-cache misses 3,830 33
Compiler-cache hit rate 37.4% 99.5%
Compiler-cache errors 0 0
Cache-save duration/result 206 s / exit 0 1 s / exit 0
Configuration Cold executions Warm executions
openhuman-lib-bins-nextest- 10,168 10,168
openhuman-embed 372 372
openhuman-rpc 154 154
openhuman-tinyhumans 328 328
openhuman-embed-default- 337 337
openhuman-tinyhumans-default- 328 328
openhuman-tui 91 91
openhuman-cli-integration-tests-nextest- 447 447
composio-stack-uninstrumented 1 1
gate-contract-tests 432 432
gate-contract-tests-features 56 56
tauri 309 309
storage-e2e 43 43
storage-lib-tests 2,486 2,486
cron-module-tests 13 13
storage-session-store-tests 7 7

Independent pair verification parsed the XML and individual identities against count JSON and the compiled CLI inventory: 42 expected CLI targets, 36 executed binaries and six present ignored-only binaries. All 16 configurations ran once; all 50 active checks passed sequentially, including ten dependency edges. Nine LCOV reports were nonempty, all 1,554 eligible source files were present, and actual ordinary and instrumented ELF files contained mold metadata. The retained changed-line gate exited zero with zero measured product lines; its percentage is not a measurement of these CI scripts. All 23 offline examples and Pester also passed.

The manual full-entry clock starts before cloning a local root Git bundle and includes fresh public vendor clones using the pinned shared checkout script, pinned action downloads, preflight, cache setup, all forced-area lanes, coverage/link checks and cache save. It excludes host preparation, VM boot and shutdown, GitHub step dispatch, the path-filter action and artifact upload. The whole-VM clock includes boot, entry smoke and shutdown. GitHub uses an authenticated root checkout of the PR merge ref instead of this local bundle. These values therefore describe the manual run rather than a GitHub job.

Cold means a new empty remote ActionCache namespace, fresh guest, target outputs and local Tinymold store. Production package caches are retained, and the shared CAS may already contain deduplicated blobs. The warm repeat retained that same namespace and package-cache disk. This is compiler-key cold, with observed production co-tenancy. Neighbors changed during the cold run and transient host memory reclaim occurred; periodic unit PSI avg10 samples reached 60.91%, with zero host OOM events. The warm observed unit PSI avg10 maximum was 4.56%. Historical and cold/warm elapsed differences do not isolate a cache-only causal effect.

The completed cold/warm scratch had 2,133,032,960 / 2,155,548,672 bytes (about 1.99 / 2.01 GiB) free. Optional live disk polling stopped early in cold and did not cover warm; these are closed footprints rather than peak headroom. Release example files were absent after scoped cleanup. Logical ordinary ELF sizes fell from 922,632,896 to 538,239,416 bytes for approvals and from 926,900,368 to 536,418,496 for Composio relative to the disk-full diagnostic. Reflinks make allocated subtree sizes upper bounds, so those figures do not establish physical reclaimed bytes. The remaining scratch margin is narrow.

Preflight guard added after measurement

After the measured revision, preflight gained a required LLVM-tool check and CARGO_LLVM_COV_SETUP=no. The paired results above retain their exact measured revision; the lane selections, build settings and compiler-cache policy are unchanged. Separate published-image verification reproduced the old guard accepting a missing LLVM-tool path, then confirmed the new guard rejects it before publishing environment. The real installed tools passed. Seven focused contract tests and the complete script suite passed: 772 tests, three platform skips, zero failures. No component installation or image modification was used for this guard verification.

Trusted-base rollout

The earlier GitHub run used main's 120-minute workflow and cancelled during Rust lint after core coverage passed. Budget prerequisite #8067 raises that finite limit to 180 minutes, matching the measured contract and the already configured 195-minute supervisor deadline. A fresh synchronize or ready-for-review event must follow its merge: reruns retain the original event revision. Verification of the new checkout/Tinymold action steps through trusted-main routing remains pending their rollout.