Garry's Opinionated OpenClaw/Hermes Agent Brain
Shared-skill lifecycle measurements
Run this opt-in workload on a quiet machine, separately from unit/E2E suites:
env -u DATABASE_URL bun --no-env-file scripts/shared-skills/lifecycle.ts \
--manifest=.context/shared-skills-lifecycle-pglite.json
For PostgreSQL, supply an explicitly test-shaped DATABASE_URL and add
--engine=postgres. The existing test-database guard applies, and each child
creates and drops its own database. The supplied database is never truncated.
The default workload measures 10, 100 and 1,000 skills, with three independent child processes per size, three authenticated members, 200 read samples per idle/loaded phase, and a 16 KiB changing asset. Each read sample fetches the current skill body and then its asset at that exact immutable revision, validating bytes, hashes, description and triggers. Members use real loopback HTTP MCP SDK clients and distinct explicit grants. There are no provider calls or native-harness claims.
Initial corpus files and sealed projections are seeded synthetic fixtures. Their
construction is reported separately and does not count as canonical publication
evidence. Every measured update uses authenticated put_skill, durable journal
admission, file-set recovery and the real canonical publisher. The publisher keeps
updating while the concurrent readers run, bounded to 256 updates. A cap that
ends pressure early cannot hide insufficient overlap.
The recorded phases include:
- Cold connection and concurrent join latency; each member then fetches and verifies every issued revision before acknowledging the complete batch.
- Warm compact-view checks and full membership sync, with unchanged-view checks.
- Changed metadata and assets, followed by polling without any change-notification subscription. Latency starts at the observed durable commit, not an assumed response timestamp.
- Idle and loaded body/asset reads; final convergence and verified fetch acknowledgments from every member.
- Per-phase engine API calls, tool argument/result JSON bytes, decoded asset bytes, RSS, sampled outstanding/intent/recovery bytes, and queue age. These are logical payload and API counts, not SQL wire counts or HTTP framing sizes.
- Exact recovery reservation bytes observed only after the outer journal transaction commits. The separate 25 ms sampler can miss short recovery windows; it is not presented as the exact maximum.
- Acknowledgment latency and throughput over the union of active acknowledgment intervals. Fetch acknowledgments never claim installation or native use.
The relative read gate reuses scripts/persistence/read-metrics.ts unchanged:
median loaded p99 must be at most 1.5× median idle p99 across three independent
runs, and each run needs at least 90% overlap with the union of in-flight
public mutation intervals. Intervals use invocation-to-terminal-receipt time,
matching the existing persistence gate; actual durable admission and commit
times are also recorded. Read/write errors, missing admission observations,
zero commits during reads, or incomplete samples cannot pass. This does not
replace the existing scripts/persistence/performance.ts gate.
This additional body/asset-RPC comparison is experimental and separate from the
existing memory-retrieval gate. Report its result per engine, including a failure
when correctness passes but relative latency does not. PGLite shares one backing
connection between catalog reads and bundle publication; a passing PostgreSQL
measurement does not establish the same latency behavior for PGLite. Neither a
smoke nor --informational may be presented as a passing full measurement.
The command writes the aggregate manifest plus uniquely named per-run JSON and full stdout/stderr logs beside it. It hashes the benchmark and exercised source files before and after each run; edits during measurement invalidate the run. No absolute latency, memory or throughput budget is invented by this workload.
Small smoke and helper tests
env -u DATABASE_URL bun --no-env-file scripts/shared-skills/lifecycle.ts \
--smoke --manifest=.context/shared-skills-lifecycle-smoke.json
bun test --timeout=60000 test/shared-skills-lifecycle-benchmark.test.ts
The smoke uses two skills, two members, four reads per phase and one independent
run. It verifies plumbing and invariants only and always reports
full_gate: false; a successful smoke is not a passing performance gate. Set
GBRAIN_TEST_SHARED_LIFECYCLE_SMOKE=1 to include it in the test-file invocation.
Custom sizes, members, queries, runs, asset bytes, maximum writes and warm checks
are available through the corresponding --key=value flags. They remain bounded
by the validator. --informational permits a measured relative-latency failure
to return zero, but never permits correctness or sample-validity errors. The
default full-gate identity still requires the documented default corpus and
measurement dimensions; custom or smoke runs cannot impersonate it.
Tests and CI
Shared-skill tests distinguish canonical publication, protocol delivery, installed
files and native harness use. test/shared-skills-transports.test.ts and
test/e2e/shared-skills-transports.test.ts use real HTTP authentication, OAuth
issuance and a new stdio process; they do not prove vendor-native activation.
test/persistence-skill-bundles.serial.test.ts and
test/persistence-skill-crash.slow.test.ts exercise typed file-set CAS and
independent-process publication/restoration kills on PGLite in the unit lane and
PostgreSQL through test/e2e/persistence-skill-bundles-postgres.test.ts.
The required shared-skills-compatibility CI job builds the pinned pre-feature
executable with scripts/build-shared-skills-baseline.sh and supplies
GBRAIN_TEST_OLD_BINARY to test/persistence-skill-old-binary.slow.test.ts.
An absent old executable is an explicit skip, never old-writer evidence.
test/shared-skills-catalog-performance.test.ts runs the reproducible
10/100/1,000-skill read benchmark when GBRAIN_TEST_SHARED_SKILLS_BENCHMARK=1;
its timings and database-call counts are recorded diagnostics, while identity
and catalog-size assertions are deterministic.
The shared-skills cases under evals/harness-instructions/ test interpretation
separately from executed calls and native sessions.
scripts/shared-skills/lifecycle.ts measures real authenticated HTTP enrollment,
revision/asset reads, publication, missed-notification polling, acknowledgments,
queue/recovery bytes, and concurrent read latency at 10/100/1,000 skills. Run it on
a quiet machine with the protocol and fixture boundaries above. Its additional body/asset latency
comparison is experimental; report each engine's measured result without
substituting it for the existing scripts/persistence/performance.ts memory-read
gate. The five pure accounting tests run normally; the small runtime smoke is
explicitly opt-in and never counts as a full performance pass.