The cross-harness, self-improving memory layer for AI agents. Join our community: https://discord.com/invite/zdNChS2fBu
Go agent-ai-cliagent-harnessesagent-securitybrowser-extensionciclaude-codecloudcodexcursordetection-engineeringjevknowledge-basememorymobile-device-managementobservabilitysecuritysecurity-information-and-event-managementsecurity-toolstelemetrytraces
feat(cli): add beacon endpoint harness2 sync|status and backfill Harness² rollouts
Claudea05499048ed4
diff --git a/CLAUDE.md b/CLAUDE.mdindex 72c7073b..fa290c2e 100644--- a/CLAUDE.md+++ b/CLAUDE.md@@ -56,6 +56,7 @@ Supported runtime surfaces today:- goose currently has only the adapter: `cli/beacon-hooks/cmd/goose.go` can map a manually written `hooks.json` that invokes `beacon-hooks --platform goose`, but there is no `beacon endpoint hooks install --harness goose`, no `DiscoverGoose`, and no automatic OTLP configuration. Keep README/docs from implying goose is generally installable until those pieces exist. goose uses HTTP OTLP (4318), its blocking hook stdout must be `{"decision":"allow"}` or `{"decision":"block"}`, and it exposes no approval decisions or tool output.- DeepSeek Harness (`dsh`) is installed through two user-level files: `$DSH_HOME/beacon-endpoint-hooks.json` and a row in `$DSH_HOME/cordis.patch.yml`; there is no project scope. The `@deepseek-ai/dsh-hooks-claude-code` bridge that row mounts is not shipped with the CLI (dev dependency only) and must be installed per profile with `@deepseek-ai/dsh-hook-protocol` at exactly the dsh version; a mount dsh cannot load is skipped at boot with only a warning. So `InstallDsh` first runs `hooks.InspectDshBridge` (`internal/endpoint/hooks/dsh_bridge.go`, read-only: the installation found from the `dsh` launcher on PATH, each `$DSH_HOME/profiles/*` resolved Node-style, and rows in any patch layer that mount a bridge by absolute path at Beacon's hooks file) and returns `*DshBridgeUnavailableError` without writing anything when nothing loads a matching bridge; explicit `--harness dsh` fails, `--all` and endpoint install skip with a warning. A healthy by-path mount (how Nix installs it; it must also resolve its declared dsh peers from its own location, because dsh's host-package interception applies only to profile plugins, else `peers_missing`) counts as installed and Beacon writes no row of its own (removing one it wrote earlier) so hooks never run twice. Beacon never installs the packages. `DshStatus.Active`/`Bridge`, discovery (misconfigured when inactive), doctor `dsh_hook_bridge`, and inventory (each profile's patch layer) surface it; tests build on-disk fixtures with `internal/endpoint/hooks/dshtest`. It also installs a Beacon-owned local skill at `$DSH_HOME/skills/beacon-endpoint/SKILL.md` and supports native session backfill from `$DSH_HOME/sessions` through `beacon endpoint dsh sync`, which fills assistant text/reasoning, tool failures, and token usage that the hook bridge does not expose. It rides the Claude-shaped envelope on the hook path, but `run_code` remains `tool.invoked` because nested commands fire their own hooks.- fx (vercel-labs/fx) telemetry read from the session store fx commits under `~/.fx/sessions/`, covering session start, prompts, tool calls, commands with exit codes and output, file reads/creates/edits with diffs, MCP tool calls, agent messages, history compaction, and per-turn token usage plus reported cost. fx exposes no third-party hook, plugin, or OTLP surface -- its lifecycle hooks are compiled into the binary -- so collection is a read of fx's own records, run by `beacon endpoint fx sync`. Every event carries `harness.collection_method=poll`; approvals are not synthesized (fx persists permission feedback text, not a per-call decision, matching the Cline and Pi posture) and no session-end event is invented, since fx sessions are resumable. Implementation lives in `cli/beacon/internal/fxsession` and `cli/beacon/cmd/endpoint_fx.go`.+- Harness² (google-research/harness2) rollouts, read from the executor event streams Harness² records under its runs directory (`$HARNESS2_RUNS_DIR`, `~/harness2/runs`, or a `--runs-dir` an earlier sync was given and the cursor file remembers): OpenCode `run --format json` and Codex `exec --json`, recognized by their first event rather than by file name. Harness² runs Codex under its own `CODEX_HOME` with `--ephemeral` and OpenCode from a source checkout, so Beacon's Codex/OpenCode integrations do not record these sessions. Events carry `harness.name=harness2`, `harness.collection_method=poll`, the executor's own session id, and the rollout's benchmark/task/label/run/trial/attempt/harness fingerprint under `raw.harness2`. Codex `turn.completed` usage is the thread's running total and is differenced per turn; Codex exec lines carry no times, so they take the file's mtime. No prompts (an argv argument, absent from both streams) and no approvals are recorded or synthesized. `testdata/recorded` holds streams recorded from the exact OpenCode and Codex releases Harness² pins; re-record them when it bumps either. Implementation lives in `cli/beacon/internal/harness2session` and `cli/beacon/cmd/endpoint_harness2.go`.- Claude Cowork admin-configured OpenTelemetry setup guidance and local validation.- OpenClaw Gateway uses a Beacon-managed plugin directory plus OpenClaw's own diagnostics OTLP guidance (`beacon endpoint integrations openclaw`). The plugin's `before_tool_call` hook is fail-closed, so the handler dispatches synchronously and returns without awaiting. Conversation hooks, including `llm_output`, require `plugins.entries.beacon-endpoint.hooks.allowConversationAccess`; without it, token and agent-message coverage is partial. Beacon reads OpenClaw JSON/JSON5 config for advice but does not rewrite it.- GitHub Copilot CLI uses MDM/customer-managed OTLP HTTP for live spans and approval-like activity, plus `beacon endpoint copilot sync` to backfill local session records from `~/.copilot/session-state`. Poll events carry assistant text, complete tool results, command/file details, and token usage with `harness.collection_method=poll`, plus the shutdown's session-wide `totalNanoAiu` differenced into `gen_ai.usage.cost_usd` once per shutdown (10^9 nano-AIU = one AI credit = US$0.01), only for sessions started on or after the 2026-06-01 AI-credit cutover (`cli/beacon/internal/copilotsession/cost.go`); approvals still come only from the live OTLP path and are not synthesized from persisted records.diff --git a/cli/beacon/cmd/endpoint_backfill.go b/cli/beacon/cmd/endpoint_backfill.goindex 8a7d0c21..bff65718 100644--- a/cli/beacon/cmd/endpoint_backfill.go+++ b/cli/beacon/cmd/endpoint_backfill.go@@ -22,6 +22,7 @@ import ("github.com/asymptote-labs/agent-beacon/cli/beacon/internal/factorysession""github.com/asymptote-labs/agent-beacon/cli/beacon/internal/fxsession""github.com/asymptote-labs/agent-beacon/cli/beacon/internal/groksession"+ "github.com/asymptote-labs/agent-beacon/cli/beacon/internal/harness2session""github.com/asymptote-labs/agent-beacon/cli/beacon/internal/hermessession""github.com/asymptote-labs/agent-beacon/cli/beacon/internal/openclawsession""github.com/asymptote-labs/agent-beacon/cli/beacon/internal/opencodesession"@@ -134,6 +135,10 @@ var endpointBackfillRunners = func() []endpointBackfillRunner {s, err := groksession.CollectOnce(groksession.CollectOptions{Write: true, UserMode: r.UserMode, LogPath: r.LogPath, StatePath: resolveGrokStatePath("", r.UserMode), Since: r.Since, Budget: r.Budget})return s.EventsEmitted, err}},+ {Name: "Harness²", Sync: "beacon endpoint harness2 sync", Run: func(r endpointBackfillRun) (int, error) {+ s, err := harness2session.CollectOnce(harness2session.CollectOptions{Write: true, UserMode: r.UserMode, LogPath: r.LogPath, StatePath: resolveHarness2StatePath("", r.UserMode), Since: r.Since, Budget: r.Budget})+ return s.EventsEmitted, err+ }},{Name: "OpenClaw Gateway", Sync: "beacon endpoint integrations openclaw sessions sync", Run: func(r endpointBackfillRun) (int, error) {s, err := openclawsession.CollectOnce(openclawsession.CollectOptions{Write: true, UserMode: r.UserMode, LogPath: r.LogPath, StatePath: resolveOpenClawSessionsStatePath("", r.UserMode), Since: r.Since, Budget: r.Budget})return s.EventsEmitted, errdiff --git a/cli/beacon/cmd/endpoint_harness2.go b/cli/beacon/cmd/endpoint_harness2.gonew file mode 100644index 00000000..5a74d6db--- /dev/null+++ b/cli/beacon/cmd/endpoint_harness2.go@@ -0,0 +1,297 @@+package cmd++import (+ "context"+ "encoding/json"+ "fmt"+ "path/filepath"+ "strings"+ "time"++ "github.com/asymptote-labs/agent-beacon/cli/beacon/internal/endpoint/lifecycle"+ "github.com/asymptote-labs/agent-beacon/cli/beacon/internal/harness2session"+ "github.com/spf13/cobra"+)++// `beacon endpoint harness2` collects telemetry from Harness² (google-research/harness2) rollouts.+//+// It is a command rather than an install step for the reason fx's is: there is nothing to install+// into. Harness² runs OpenCode and Codex under environments it builds itself -- Codex with its own+// CODEX_HOME and --ephemeral, OpenCode from a source checkout -- so Beacon's hooks and plugins for+// those agents are not what records these sessions. What Harness² keeps is each rollout's event+// stream under its runs directory, so collecting from it means reading those files.+//+// Everything it reads is already on this machine and nothing it does reaches the network.+var endpointHarness2Cmd = &cobra.Command{+ Use: "harness2",+ Short: "Collect telemetry from Harness² (google-research/harness2) rollouts",+ Long: `Collect endpoint telemetry from Harness², Google Research's framework for+recursive agent harnessing.++Harness² refines an OpenCode or Codex executor's harness by running it over+benchmark tasks many times. It records each rollout's event stream under its+runs directory ($HARNESS2_RUNS_DIR, or runs/ in its checkout), and Beacon reads+those streams into endpoint events with harness.name=harness2, keeping the+benchmark, task, run and executor of each rollout beside its events. Every event+is marked harness.collection_method=poll: Beacon sees a rollout once Harness²+recorded it, so nothing here can hold or deny a tool call.++Name a checkout's runs directory once with --runs-dir; later syncs, and the+install-time backfill, keep reading it. Reading is local and offline.`,+}++var endpointHarness2SyncCmd = &cobra.Command{+ Use: "sync",+ Short: "Read new Harness² rollout events into the runtime log",+ SilenceUsage: true,+ RunE: runEndpointHarness2Sync,+}++var endpointHarness2StatusCmd = &cobra.Command{+ Use: "status",+ Short: "Show Harness² rollouts on this machine and how much of each has been collected",+ SilenceUsage: true,+ RunE: runEndpointHarness2Status,+}++var endpointHarness2Opts struct {+ runsDirs []string+ statePath string+ logPath string+ print bool+ watch bool+ interval time.Duration+ rotateBytes int64+}++func init() {+ endpointCmd.AddCommand(endpointHarness2Cmd)+ endpointHarness2Cmd.AddCommand(endpointHarness2SyncCmd)+ endpointHarness2Cmd.AddCommand(endpointHarness2StatusCmd)++ for _, c := range []*cobra.Command{endpointHarness2SyncCmd, endpointHarness2StatusCmd} {+ f := c.Flags()+ f.StringArrayVar(&endpointHarness2Opts.runsDirs, "runs-dir", nil, "Harness² runs directory; repeatable (default $HARNESS2_RUNS_DIR or ~/harness2/runs, plus directories earlier syncs were given)")+ f.StringVar(&endpointHarness2Opts.statePath, "state", "", "Collector cursor file (default ~/.beacon/endpoint/state/harness2.json)")+ f.BoolVar(&endpointOpts.jsonOutput, "json", false, "Print the result as JSON")+ f.BoolVar(&endpointOpts.userMode, "user", true, "Use per-user endpoint paths")+ f.BoolVar(&endpointOpts.systemMode, "system", false, "Use system endpoint paths")+ }++ sync := endpointHarness2SyncCmd.Flags()+ sync.StringVar(&endpointHarness2Opts.logPath, "log-path", "", "Runtime JSONL log path (default resolved endpoint log)")+ sync.BoolVar(&endpointHarness2Opts.print, "print", false, "Print mapped events as JSON without writing them or advancing the cursor (dry run)")+ sync.BoolVar(&endpointHarness2Opts.watch, "watch", false, "Sweep continuously on --interval (default: one sweep then exit)")+ sync.DurationVar(&endpointHarness2Opts.interval, "interval", time.Minute, "Sweep interval for --watch")+ addSyncRotateBytesFlag(sync, &endpointHarness2Opts.rotateBytes)+}++func runEndpointHarness2Sync(cmd *cobra.Command, args []string) error {+ if err := checkSyncRotateBytes(endpointHarness2Opts.rotateBytes); err != nil {+ return err+ }+ userMode := endpointUserMode()+ opts := harness2session.CollectOptions{+ RotateBytes: endpointHarness2Opts.rotateBytes,+ RunsDirs: endpointHarness2Opts.runsDirs,+ Print: endpointHarness2Opts.print,+ Out: cmd.OutOrStdout(),+ Write: !endpointHarness2Opts.print,+ UserMode: userMode,+ }+ // --print is a dry run in both directions: it neither writes the runtime log nor advances the+ // cursor -- nor remembers a --runs-dir -- so running it changes nothing a later sweep would see.+ if !endpointHarness2Opts.print {+ opts.StatePath = resolveHarness2StatePath(endpointHarness2Opts.statePath, userMode)+ opts.LogPath = lifecycle.ResolveRuntimeLog(userMode, endpointHarness2Opts.logPath).EffectiveLogPath+ } else {+ // A dry run still reads the directories earlier syncs remembered, so it shows what a real+ // sweep would read; it only never writes the cursor file.+ opts.RunsDirs = harness2PrintRoots(endpointHarness2Opts.runsDirs, resolveHarness2StatePath(endpointHarness2Opts.statePath, userMode))+ }++ ctx := cmd.Context()+ if ctx == nil {+ ctx = context.Background()+ }+ if endpointHarness2Opts.watch && !endpointHarness2Opts.print {+ return watchHarness2(ctx, cmd, opts)+ }++ summary, err := harness2session.CollectOnce(opts)+ // Reported even when the sweep hit an error: a sweep that read nine rollouts and failed on the+ // tenth did nine rollouts' worth of work.+ reportHarness2Sweep(cmd, summary)+ return err+}++// harness2PrintRoots is the roots a dry run reads: the ones given, or the defaults and the ones a+// real sync remembered in the cursor file at statePath.+func harness2PrintRoots(runsDirs []string, statePath string) []string {+ if len(runsDirs) > 0 {+ return runsDirs+ }+ state, err := harness2session.LoadState(statePath)+ if err != nil {+ return nil+ }+ roots, err := harness2session.Roots(nil, state)+ if err != nil {+ return nil+ }+ return roots+}++func watchHarness2(ctx context.Context, cmd *cobra.Command, opts harness2session.CollectOptions) error {+ interval := endpointHarness2Opts.interval+ if interval < 5*time.Second {+ // A sweep walks every runs root, so a very short interval spends real work on directories+ // that hold thousands of finished rollouts.+ interval = 5 * time.Second+ }+ for {+ summary, err := harness2session.CollectOnce(opts)+ if err != nil {+ fmt.Fprintf(cmd.ErrOrStderr(), "Harness² sync error: %v\n", err)+ }+ reportHarness2Sweep(cmd, summary)+ select {+ case <-ctx.Done():+ return nil+ case <-time.After(interval):+ }+ }+}++func reportHarness2Sweep(cmd *cobra.Command, summary harness2session.Summary) {+ if endpointHarness2Opts.print {+ return+ }+ if endpointOpts.jsonOutput {+ _ = json.NewEncoder(cmd.OutOrStdout()).Encode(summary)+ return+ }+ fmt.Fprintf(cmd.OutOrStdout(), "Harness² sync: %d rollouts, %d changed, %d events, %d errors\n",+ summary.Trajectories, summary.FilesChanged, summary.EventsEmitted, summary.Errors)+ reportSyncRetention(cmd.OutOrStdout(), summary.Retention)+ if summary.Trajectories == 0 {+ fmt.Fprintf(cmd.OutOrStdout(), " no rollouts found in %s; name a checkout's runs directory with --runs-dir\n",+ strings.Join(summary.Roots, ", "))+ }+ if summary.PartialFiles > 0 {+ fmt.Fprintf(cmd.OutOrStdout(), " %d rollout(s) still being written; the rest is read on the next sweep\n", summary.PartialFiles)+ }+}++// harness2RolloutStatus is one trajectory as `status` reports it.+type harness2RolloutStatus struct {+ Path string `json:"path"`+ Executor string `json:"executor"`+ Benchmark string `json:"benchmark,omitempty"`+ Subset string `json:"subset,omitempty"`+ TaskID string `json:"task_id,omitempty"`+ Label string `json:"label,omitempty"`+ RunID string `json:"run_id,omitempty"`+ Trial string `json:"trial,omitempty"`+ SessionID string `json:"session_id,omitempty"`+ UpdatedAt string `json:"updated_at,omitempty"`+ SizeBytes int64 `json:"size_bytes"`+ Events int `json:"events"`+ // Collected means the file is unchanged since a sweep last read all of it.+ Collected bool `json:"collected"`+}++type harness2StatusReport struct {+ Roots []string `json:"roots"`+ Present bool `json:"present"`+ StatePath string `json:"state_path,omitempty"`+ Rollouts []harness2RolloutStatus `json:"rollouts"`+}++func runEndpointHarness2Status(cmd *cobra.Command, args []string) error {+ statePath := resolveHarness2StatePath(endpointHarness2Opts.statePath, endpointUserMode())+ state, err := harness2session.LoadState(statePath)+ if err != nil {+ return err+ }+ roots, err := harness2session.Roots(endpointHarness2Opts.runsDirs, state)+ if err != nil {+ return err+ }+ store := &harness2session.Store{Roots: roots}+ refs, err := store.List()+ if err != nil {+ return err+ }+ report := harness2StatusReport{+ Roots: roots,+ Present: store.Exists(),+ StatePath: statePath,+ Rollouts: make([]harness2RolloutStatus, 0, len(refs)),+ }+ for _, ref := range refs {+ status := harness2RolloutStatus{+ Path: ref.Path,+ Executor: ref.Format.Executor(),+ Benchmark: ref.Context.Benchmark,+ Subset: ref.Context.Subset,+ TaskID: ref.Context.TaskID,+ Label: ref.Context.Label,+ RunID: ref.Context.RunID,+ Trial: ref.Context.Trial,+ SizeBytes: ref.SizeBytes,+ }+ if ref.ModTimeUnixMS > 0 {+ status.UpdatedAt = time.UnixMilli(ref.ModTimeUnixMS).UTC().Format(time.RFC3339)+ }+ if cursor := state.Files[ref.Path]; cursor != nil {+ status.SessionID = cursor.SessionID+ status.Events = cursor.Events+ status.Collected = cursor.SizeBytes == ref.SizeBytes && cursor.ModTimeUnixMS == ref.ModTimeUnixMS+ }+ report.Rollouts = append(report.Rollouts, status)+ }++ if endpointOpts.jsonOutput {+ return json.NewEncoder(cmd.OutOrStdout()).Encode(report)+ }+ out := cmd.OutOrStdout()+ if !report.Present {+ fmt.Fprintf(out, "Harness² rollouts: none (no runs directory at %s); name one with --runs-dir\n", strings.Join(report.Roots, ", "))+ return nil+ }+ fmt.Fprintf(out, "Harness² rollouts: %d in %s\n", len(report.Rollouts), strings.Join(report.Roots, ", "))+ for _, rollout := range report.Rollouts {+ state := "pending"+ if rollout.Collected {+ state = "collected"+ }+ what := strings.Join(nonEmpty(rollout.Benchmark, rollout.Subset, rollout.TaskID, rollout.Label, rollout.RunID, rollout.Trial), " ")+ fmt.Fprintf(out, " %-9s %-8s %4d events %s %s\n", state, rollout.Executor, rollout.Events, what, rollout.Path)+ }+ return nil+}++func nonEmpty(values ...string) []string {+ out := make([]string, 0, len(values))+ for _, value := range values {+ if strings.TrimSpace(value) != "" {+ out = append(out, value)+ }+ }+ return out+}++// resolveHarness2StatePath always returns a path so the cursor survives between runs. A system-mode+// sweep keeps its cursor beside the system runtime log; everything else uses the per-user default.+func resolveHarness2StatePath(override string, userMode bool) string {+ if strings.TrimSpace(override) != "" {+ return override+ }+ if !userMode {+ if dir := filepath.Dir(lifecycle.ResolveRuntimeLog(false, "").EffectiveLogPath); dir != "" && dir != "." {+ return filepath.Join(dir, "harness2-state.json")+ }+ }+ return harness2session.DefaultStatePath()+}diff --git a/cli/beacon/cmd/endpoint_harness2_test.go b/cli/beacon/cmd/endpoint_harness2_test.gonew file mode 100644index 00000000..7b18c800--- /dev/null+++ b/cli/beacon/cmd/endpoint_harness2_test.go@@ -0,0 +1,212 @@+package cmd++import (+ "bytes"+ "encoding/json"+ "os"+ "path/filepath"+ "strings"+ "testing"++ "github.com/asymptote-labs/agent-beacon/cli/beacon/internal/harness2session"+ "github.com/asymptote-labs/agent-beacon/cli/beacon/internal/testenv"+ "github.com/spf13/cobra"+)++// harness2CommandFixture writes one LAB Codex rollout and one JobBench OpenCode rollout under a runs+// directory and points the command's flags at it, with the runtime log and cursor in temp+// directories so nothing touches the machine running the test.+type harness2CommandFixture struct {+ runsDir string+ statePath string+ logPath string+}++const harness2CodexStream = `{"type":"thread.started","thread_id":"th-cmd-1"}+{"type":"turn.started"}+{"type":"item.completed","item":{"id":"item_0","type":"command_execution","command":"bash -lc ls","aggregated_output":"a\n","exit_code":0,"status":"completed"}}+{"type":"item.completed","item":{"id":"item_1","type":"agent_message","text":"done"}}+{"type":"turn.completed","usage":{"input_tokens":10,"cached_input_tokens":4,"output_tokens":2,"reasoning_output_tokens":0}}+`++const harness2OpenCodeStream = `{"type":"step_start","timestamp":1793004900100,"sessionID":"ses_cmd_1","part":{"type":"step-start"}}+{"type":"text","timestamp":1793004900200,"sessionID":"ses_cmd_1","part":{"type":"text","text":"hello","time":{"start":1,"end":2}}}+`++func newHarness2CommandFixture(t *testing.T) harness2CommandFixture {+ t.Helper()+ testenv.SetHome(t, t.TempDir())+ t.Setenv("HARNESS2_RUNS_DIR", "")+ root := t.TempDir()+ f := harness2CommandFixture{+ runsDir: filepath.Join(root, "harness2", "runs"),+ statePath: filepath.Join(root, "state", "harness2.json"),+ logPath: filepath.Join(root, "logs", "runtime.jsonl"),+ }+ for rel, body := range map[string]string{+ "lab/results/cx/run-1/codex_session.jsonl": harness2CodexStream,+ "jb/dataset/p/task1/model_traj/base/p_task1_base_attempt1_x.jsonl": harness2OpenCodeStream,+ "jb/judged/scores.jsonl": `{"task":"p/task1","score":1}` + "\n",+ } {+ path := filepath.Join(f.runsDir, filepath.FromSlash(rel))+ if err := os.MkdirAll(filepath.Dir(path), 0o755); err != nil {+ t.Fatal(err)+ }+ if err := os.WriteFile(path, []byte(body), 0o644); err != nil {+ t.Fatal(err)+ }+ }++ // Command flags are package-level in this CLI, so each test sets the whole set it depends on.+ endpointHarness2Opts.runsDirs = []string{f.runsDir}+ endpointHarness2Opts.statePath = f.statePath+ endpointHarness2Opts.logPath = f.logPath+ endpointHarness2Opts.print = false+ endpointHarness2Opts.watch = false+ endpointOpts.jsonOutput = false+ endpointOpts.userMode = true+ endpointOpts.systemMode = false+ t.Cleanup(func() {+ endpointHarness2Opts.runsDirs = nil+ endpointHarness2Opts.statePath = ""+ endpointHarness2Opts.logPath = ""+ endpointHarness2Opts.print = false+ endpointOpts.jsonOutput = false+ })+ return f+}++func runHarness2(t *testing.T, cmd *cobra.Command, run func(*cobra.Command, []string) error) string {+ t.Helper()+ var out bytes.Buffer+ cmd.SetOut(&out)+ cmd.SetErr(&out)+ t.Cleanup(func() { cmd.SetOut(nil); cmd.SetErr(nil) })+ if err := run(cmd, nil); err != nil {+ t.Fatalf("%s: %v\n%s", cmd.Name(), err, out.String())+ }+ return out.String()+}++func readHarness2Log(t *testing.T, path string) []map[string]interface{} {+ t.Helper()+ data, err := os.ReadFile(path)+ if err != nil {+ t.Fatal(err)+ }+ var events []map[string]interface{}+ for _, line := range strings.Split(strings.TrimSpace(string(data)), "\n") {+ var ev map[string]interface{}+ if err := json.Unmarshal([]byte(line), &ev); err != nil {+ t.Fatalf("log line %q: %v", line, err)+ }+ events = append(events, ev)+ }+ return events+}++func TestEndpointHarness2SyncWritesTheRuntimeLog(t *testing.T) {+ f := newHarness2CommandFixture(t)+ out := runHarness2(t, endpointHarness2SyncCmd, runEndpointHarness2Sync)+ if !strings.Contains(out, "Harness² sync: 2 rollouts, 2 changed, 6 events, 0 errors") {+ t.Fatalf("output = %q", out)+ }+ events := readHarness2Log(t, f.logPath)+ if len(events) != 6 {+ t.Fatalf("log holds %d events, want 6", len(events))+ }+ for _, ev := range events {+ harness := ev["harness"].(map[string]interface{})+ if harness["name"] != "harness2" || harness["collection_method"] != "poll" {+ t.Fatalf("harness = %v", harness)+ }+ }++ // A second run writes nothing, and a run with no --runs-dir still reads the remembered one.+ endpointHarness2Opts.runsDirs = nil+ out = runHarness2(t, endpointHarness2SyncCmd, runEndpointHarness2Sync)+ if !strings.Contains(out, "2 rollouts, 0 changed, 0 events") {+ t.Fatalf("second run output = %q", out)+ }+ if len(readHarness2Log(t, f.logPath)) != 6 {+ t.Fatal("second run wrote events again")+ }+}++func TestEndpointHarness2SyncPrintLeavesNoTrace(t *testing.T) {+ f := newHarness2CommandFixture(t)+ endpointHarness2Opts.print = true+ out := runHarness2(t, endpointHarness2SyncCmd, runEndpointHarness2Sync)+ if lines := strings.Count(out, "\n"); lines != 6 {+ t.Fatalf("printed %d lines, want 6 events:\n%s", lines, out)+ }+ for _, path := range []string{f.statePath, f.logPath} {+ if _, err := os.Stat(path); !os.IsNotExist(err) {+ t.Fatalf("--print created %s (err=%v)", path, err)+ }+ }+}++// --print with no --runs-dir reads what a real sync would: the default checkout and the roots+// earlier syncs remembered.+func TestEndpointHarness2PrintReadsRememberedRoots(t *testing.T) {+ newHarness2CommandFixture(t)+ runHarness2(t, endpointHarness2SyncCmd, runEndpointHarness2Sync)+ endpointHarness2Opts.runsDirs = nil+ endpointHarness2Opts.print = true+ if out := runHarness2(t, endpointHarness2SyncCmd, runEndpointHarness2Sync); strings.Count(out, "\n") != 6 {+ t.Fatalf("print with remembered roots = %q", out)+ }+}++func TestEndpointHarness2StatusJSONDescribesEachRollout(t *testing.T) {+ f := newHarness2CommandFixture(t)+ runHarness2(t, endpointHarness2SyncCmd, runEndpointHarness2Sync)+ endpointOpts.jsonOutput = true+ out := runHarness2(t, endpointHarness2StatusCmd, runEndpointHarness2Status)+ var report harness2StatusReport+ if err := json.Unmarshal([]byte(out), &report); err != nil {+ t.Fatalf("status JSON %q: %v", out, err)+ }+ if !report.Present || len(report.Rollouts) != 2 || report.StatePath != f.statePath {+ t.Fatalf("report = %+v", report)+ }+ byExecutor := map[string]harness2RolloutStatus{}+ for _, r := range report.Rollouts {+ byExecutor[r.Executor] = r+ }+ if cx := byExecutor["codex"]; !cx.Collected || cx.Events != 4 || cx.SessionID != "th-cmd-1" || cx.Benchmark != "lab" || cx.RunID != "run-1" {+ t.Fatalf("codex rollout = %+v", cx)+ }+ if oc := byExecutor["opencode"]; !oc.Collected || oc.TaskID != "p/task1" || oc.Label != "base" {+ t.Fatalf("opencode rollout = %+v", oc)+ }+}++func TestEndpointHarness2OnAMachineWithoutHarness2IsAQuietNoOp(t *testing.T) {+ f := newHarness2CommandFixture(t)+ endpointHarness2Opts.runsDirs = []string{filepath.Join(t.TempDir(), "missing")}+ out := runHarness2(t, endpointHarness2SyncCmd, runEndpointHarness2Sync)+ if !strings.Contains(out, "0 rollouts") || !strings.Contains(out, "--runs-dir") {+ t.Fatalf("output = %q", out)+ }+ if _, err := os.Stat(f.logPath); !os.IsNotExist(err) {+ t.Fatal("a sync with nothing to read created the runtime log")+ }+ out = runHarness2(t, endpointHarness2StatusCmd, runEndpointHarness2Status)+ if !strings.Contains(out, "none") {+ t.Fatalf("status output = %q", out)+ }+}++func TestHarness2StatePathAlwaysResolvesSomewhereDurable(t *testing.T) {+ if got := resolveHarness2StatePath("/x/state.json", true); got != "/x/state.json" {+ t.Fatalf("override = %q", got)+ }+ if got := resolveHarness2StatePath("", true); got != harness2session.DefaultStatePath() {+ t.Fatalf("user default = %q", got)+ }+ if got := resolveHarness2StatePath("", false); filepath.Base(got) != "harness2-state.json" {+ t.Fatalf("system default = %q", got)+ }+}diff --git a/cli/beacon/cmd/session_sync_retention_test.go b/cli/beacon/cmd/session_sync_retention_test.goindex 8e518c3d..8d9a9e4f 100644--- a/cli/beacon/cmd/session_sync_retention_test.go+++ b/cli/beacon/cmd/session_sync_retention_test.go@@ -32,6 +32,7 @@ func TestEverySessionSyncRefusesARotateBytesBelowTheDefault(t *testing.T) {{"factory", endpointFactorySyncCmd, runEndpointFactorySync, &endpointFactoryOpts.rotateBytes},{"fx", endpointFxSyncCmd, runEndpointFxSync, &endpointFxOpts.rotateBytes},{"grok", endpointGrokSyncCmd, runEndpointGrokSync, &endpointGrokOpts.rotateBytes},+ {"harness2", endpointHarness2SyncCmd, runEndpointHarness2Sync, &endpointHarness2Opts.rotateBytes},{"hermes", endpointHermesSyncCmd, runEndpointHermesSync, &endpointHermesOpts.rotateBytes},{"openclaw sessions", endpointOpenClawSessionsSyncCmd, runEndpointOpenClawSessionsSync, &endpointOpenClawSessionsOpts.rotateBytes},{"opencode", endpointOpenCodeSyncCmd, runEndpointOpenCodeSync, &endpointOpenCodeOpts.rotateBytes},diff --git a/cli/beacon/internal/sessionfs/collectors_test.go b/cli/beacon/internal/sessionfs/collectors_test.goindex 2e287542..a517affd 100644--- a/cli/beacon/internal/sessionfs/collectors_test.go+++ b/cli/beacon/internal/sessionfs/collectors_test.go@@ -17,6 +17,7 @@ var backfillCollectors = []string{"claudesession", "codexsession", "cursorsession", "opencodesession", "copilotsession","clinesession", "factorysession", "pisession", "primesession", "hermessession","dshsession", "fxsession", "groksession", "openclawsession", "devinsession",+ "harness2session",}// directReads are the calls that would read a session store around the confinement.diff --git a/cli/beacon/internal/tokens/coverage.go b/cli/beacon/internal/tokens/coverage.goindex ae82c9ab..698de7d7 100644--- a/cli/beacon/internal/tokens/coverage.go+++ b/cli/beacon/internal/tokens/coverage.go@@ -87,9 +87,13 @@ var usageExpectation = map[string]struct {// `beacon endpoint hooks status --harness openclaw` is where that gap is surfaced. The// session-file sync reads the usage OpenClaw commits on each assistant message// (openclawsession/records.go), and needs no grant.- "openclaw_gateway": {ExpectReported, "plugin reports usage per model response when conversation access is granted; beacon endpoint integrations openclaw sessions sync reads it from session files"},- "vercel_fx": {ExpectReported, "session store carries cumulative usage and cost"},- "deepseek_harness": {ExpectReported, "native session backfill reports usage when DeepSeek persists it"},+ "openclaw_gateway": {ExpectReported, "plugin reports usage per model response when conversation access is granted; beacon endpoint integrations openclaw sessions sync reads it from session files"},+ "vercel_fx": {ExpectReported, "session store carries cumulative usage and cost"},+ "deepseek_harness": {ExpectReported, "native session backfill reports usage when DeepSeek persists it"},+ // Read from the executor's own stream: OpenCode's per-step usage and cost, and Codex's per-turn+ // thread totals, differenced. Neither stream names the model, so these tokens are unpriced+ // unless OpenCode reported a cost.+ "harness2": {ExpectReported, "rollout streams carry per-step (OpenCode) or per-turn (Codex) usage; OpenCode reports cost"},"copilot_cli": {ExpectReported, "session store carries output tokens, cumulative model usage, and AI-credit cost"},"asymptote_observe": {ExpectReported, "SDK spans carry semconv usage"},// Read from Gemini CLI's own gemini_cli.api_response log, which it emits for every modeldiff --git a/docs/cli/endpoint-harness2.mdx b/docs/cli/endpoint-harness2.mdxnew file mode 100644index 00000000..72b3613d--- /dev/null+++ b/docs/cli/endpoint-harness2.mdx@@ -0,0 +1,115 @@+---+title: "beacon endpoint harness2"+description: "Collect endpoint telemetry from Harness² (google-research/harness2) rollouts"+---++## Command overview++`beacon endpoint harness2` collects telemetry from [Harness²](/runtimes/harness2), Google Research's framework for recursive agent harnessing, by reading the executor event streams Harness² records under its runs directory and converting them into Beacon [endpoint events](/concepts/core-concepts#endpoint-event).++```bash title="Command syntax"+beacon endpoint harness2 [command]+```++It is a command rather than an install step because Harness² has nothing to install into. It runs OpenCode and Codex in environments it builds itself, so what records each rollout is the stream Harness² writes beside it, and collecting from that is an action rather than a configuration.++Every event is marked `harness.collection_method=poll` and `harness.name=harness2`. Beacon sees a rollout once Harness² recorded it, so nothing here can hold or deny a tool call.++Reading is local and offline. Nothing this command does reaches the network.++## Commands++| Command | Description |+|---------|-------------|+| `beacon endpoint harness2 sync` | Read new rollout events into the runtime log |+| `beacon endpoint harness2 status` | Show rollouts on this machine and how much of each has been collected |++## Flags++Both subcommands accept:++| Flag | Description |+|------|-------------|+| `--runs-dir <path>` | Harness² runs directory. Repeatable. Defaults to `$HARNESS2_RUNS_DIR`, or `~/harness2/runs`, plus every directory an earlier `sync` was given |+| `--state <path>` | Collector cursor file. Defaults to `~/.beacon/endpoint/state/harness2.json` |+| `--json` | Print the result as JSON |+| `--user` | Use per-user endpoint paths. Enabled by default |+| `--system` | Use system endpoint paths. The cursor moves beside the system runtime log |++`sync` also accepts:++| Flag | Description |+|------|-------------|+| `--log-path <path>` | Runtime JSONL log path. Defaults to the resolved endpoint log |+| `--print` | Print mapped events as JSON without writing them, advancing the cursor, or remembering a `--runs-dir` |+| `--watch` | Sweep continuously on `--interval` instead of sweeping once and exiting |+| `--interval <duration>` | Sweep interval for `--watch`. Defaults to `1m`, with a 5-second floor |+| `--rotate-bytes <bytes>` | Rotate the runtime log at this many bytes while this sync writes it. Defaults to 10485760, the size every Beacon writer uses; must not be smaller. See [large backfills and log rotation](/runtimes/claude-code#large-backfills-and-log-rotation) |++## Runs directories++Harness² writes to `runs/` inside its own checkout unless `HARNESS2_RUNS_DIR` says otherwise, and a checkout can be anywhere. Name it once:++```bash title="Read a checkout's runs directory"+beacon endpoint harness2 sync --runs-dir ~/src/harness2/runs+```++The directory is remembered in the cursor file, so later syncs with no `--runs-dir`, and the [install-time backfill](/cli/endpoint-install#session-backfill), keep reading it alongside the default. Each new value of `HARNESS2_RUNS_DIR`, which Harness² recommends when changing models or task selection, is a new directory to name.++## Sweeping++```bash title="Sweep once"+beacon endpoint harness2 sync+```++```bash title="Preview without writing anything"+beacon endpoint harness2 sync --print+```++```bash title="Sweep continuously while an experiment runs"+beacon endpoint harness2 sync --watch --interval 1m+```++Nothing sweeps on its own. Run `sync` after an experiment, under whatever scheduler the platform already uses, or leave `--watch` running during one.++```text+Harness² sync: 48 rollouts, 12 changed, 1934 events, 0 errors+ 2 rollout(s) still being written; the rest is read on the next sweep+```++## Status++```bash title="Show rollouts and collection progress"+beacon endpoint harness2 status+```++```text+Harness² rollouts: 3 in /Users/you/src/harness2/runs+ collected opencode 52 events jb accountant/task3 h2-final-1 /Users/you/src/harness2/runs/jb/dataset/...+ collected codex 38 events lab antitrust-competition__t07__h2-cand-dom-1 /Users/you/src/harness2/runs/lab/results/cx/...+ pending opencode 0 events wb office xlsx-budget__Ab3dE /Users/you/src/harness2/runs/wb/office/...+```++`collected` means the file is unchanged since a sweep last read all of it.++## Cursor behavior++The cursor keeps one entry per trajectory file: the last line whose events were all written, the file's size and modification time, and the session it held. A stream still being written is read up to its last complete line, and the rest on a later sweep. When Harness² re-rolls a run into the same path, the file names a new session and is read from its start.++| Mode | Default cursor path |+|------|--------------------|+| User | `~/.beacon/endpoint/state/harness2.json` |+| System | `harness2-state.json` beside the system runtime log |++Re-reading is safe: each event's id comes from its executor, session, line and kind, so the same line maps to the same event every time.++## Related++<Columns cols={2}>+ <Card title="Harness²" icon="flask" href="/runtimes/harness2">+ What Beacon records from Harness² rollouts, and the known gaps.+ </Card>+ <Card title="beacon endpoint install" icon="download" href="/cli/endpoint-install#session-backfill">+ The install-time backfill that runs every session collector once.+ </Card>+</Columns>diff --git a/docs/cli/endpoint-install.mdx b/docs/cli/endpoint-install.mdxindex 7f10d0af..397770dc 100644--- a/docs/cli/endpoint-install.mdx+++ b/docs/cli/endpoint-install.mdx@@ -62,7 +62,7 @@ runtime's own session store into the runtime log, so the [local dashboard](/cli/your recent sessions before any agent runs again. It is the same sweep each runtime's sync commandruns (`beacon endpoint claude sync`, `codex sync`, `cursor sync`, `opencode sync`, `copilot sync`,`cline sync`, `factory sync`, `pi sync`, `prime sync`, `hermes sync`, `devin sync`, `dsh sync`,-`fx sync`, `grok sync`, and `integrations openclaw sessions sync`), and its events carry+`fx sync`, `grok sync`, `harness2 sync`, and `integrations openclaw sessions sync`), and its events carry`harness.collection_method=poll` like theirs. It is local only and never touches the network.The backfill is bounded so a machine with a lot of history cannot crowd live telemetry out of thediff --git a/docs/cli/endpoint.mdx b/docs/cli/endpoint.mdxindex aacf877f..7c4f065c 100644--- a/docs/cli/endpoint.mdx+++ b/docs/cli/endpoint.mdx@@ -25,6 +25,7 @@ beacon endpoint [command]<span id="beacon-endpoint-config" /><span id="beacon-endpoint-onboarding" /><span id="beacon-endpoint-fx" />+<span id="beacon-endpoint-harness2" /><span id="beacon-endpoint-connect" /><span id="beacon-endpoint-disconnect" /><span id="beacon-endpoint-asymptote" />@@ -61,6 +62,9 @@ beacon endpoint [command]<Card title="beacon endpoint fx" icon="rotate" href="/cli/endpoint-fx">Collect telemetry from fx session records on this machine.</Card>+ <Card title="beacon endpoint harness2" icon="rotate" href="/cli/endpoint-harness2">+ Collect telemetry from Harness² rollouts recorded under its runs directory.+ </Card><Card title="beacon endpoint copilot" icon="rotate" href="/cli/endpoint-copilot">Collect telemetry from GitHub Copilot CLI session records.</Card>diff --git a/docs/cli/index.mdx b/docs/cli/index.mdxindex 2921a28b..6812d363 100644--- a/docs/cli/index.mdx+++ b/docs/cli/index.mdx@@ -183,6 +183,7 @@ Manage the local endpoint agent and inspect supported agent harness telemetry.| [`beacon endpoint copilot`](/cli/endpoint-copilot) | Collect endpoint telemetry from GitHub Copilot CLI session records on this machine. || [`beacon endpoint dsh`](/cli/endpoint-dsh) | Collect endpoint telemetry from DeepSeek Harness session records on this machine. || [`beacon endpoint fx`](/cli/endpoint-fx) | Collect endpoint telemetry from fx session records on this machine. |+| [`beacon endpoint harness2`](/cli/endpoint-harness2) | Collect endpoint telemetry from Harness² rollouts recorded under its runs directory. || [`beacon endpoint integrations openclaw`](/cli/openclaw) | Set up and validate OpenClaw Gateway OTLP export. || [`beacon endpoint integrations claude-cowork`](/cli/claude-cowork) | Set up and validate Claude Cowork OTLP export. |diff --git a/docs/cli/token-usage.mdx b/docs/cli/token-usage.mdxindex 269e4537..e36f3843 100644--- a/docs/cli/token-usage.mdx+++ b/docs/cli/token-usage.mdx@@ -52,6 +52,7 @@ from. A turn with no token attribution is a turn whose runtime never reported an| Factory Droid | input, output, cache read, cache creation, reasoning | Not reported in USD; Factory credits stay in `raw` | Yes, from `beacon endpoint factory sync` | Yes || DeepSeek Harness | input, output, cache read, cache creation | Not reported | Yes, from committed session records | Yes || fx (Vercel Labs) | input, output, cache read, cache creation | Reported (`cost_usd`) | Yes | Yes |+| Harness² | input, output, cache read, cache creation, reasoning | Reported (`cost_usd`) when the OpenCode executor reports it; not emitted by Codex. Neither stream names the model, so there is no list-price estimate | Yes, per rollout from `beacon endpoint harness2 sync` | Yes || GitHub Copilot CLI | input, output, cache read, cache creation, reasoning | Reported (`cost_usd`) for sessions started on or after 2026-06-01, from AI credits | Yes | Yes || OpenClaw Gateway | input, output, cache read, cache creation | Not reported on the hook | Yes, per model response | Yes || Gemini CLI | input, output, cache read, reasoning | Not emitted by Gemini CLI | Yes, per model response | When local user context is present |@@ -62,7 +63,7 @@ Hermes Agent and Factory Droid report usage only through their session-store synGemini CLI reports each model response on its `gemini_cli.api_response` log record and again on the `gemini_cli.token.usage` counter, under Gemini's own counts rather than the semconv names. Those counts are not disjoint: Gemini's input (`promptTokenCount`) already includes the cached content, its output (`candidatesTokenCount`) leaves out thinking tokens, and its tool-use prompt tokens are input that the prompt count does not hold. Beacon therefore records input as prompt minus cached plus tool-use prompt, cache read as the cached count, and output as candidates plus thoughts with the thoughts as its reasoning breakdown, so a response's Beacon total equals Gemini's own `total_token_count`. Gemini also restates input and output on a `gen_ai.client.inference.operation.details` record; Beacon reads usage only from `api_response`, so a response is counted once, and the counter is de-duplicated against the log because both carry Gemini's `session.id`. These shapes are pinned from Gemini CLI's telemetry source rather than from a recorded session.-Cline reports usage once per task rather than per model call, because Beacon's rollups sum `gen_ai.usage` across events and reporting at both granularities would count every token twice. fx reports cumulative totals, which Beacon differences into per-turn deltas for the same reason. GitHub Copilot CLI reports assistant message output tokens and cumulative shutdown model totals; Beacon records the message output directly, then differences shutdown totals for input, cache, reasoning, and any output not already counted. Cost comes from the shutdown's session-wide `totalNanoAiu`, differenced the same way and converted at one AI credit = 10^9 nano-AIU = US$0.01. It is recorded once per shutdown: on the model's usage event when the shutdown names one model, otherwise on one usage event with no model, so a multi-model session is not counted once per model. Sessions started before GitHub moved Copilot to AI credits on 2026-06-01 were billed per premium request, so they record no cost; the raw nano-AIU total stays in `raw.copilot`. Annual Copilot Pro and Pro+ subscribers stay on premium-request billing until their plan renews, and the session record does not say which plan applies, so for them the recorded cost is what the session would cost in AI credits rather than their bill.+Cline reports usage once per task rather than per model call, because Beacon's rollups sum `gen_ai.usage` across events and reporting at both granularities would count every token twice. fx reports cumulative totals, which Beacon differences into per-turn deltas for the same reason, and so does a Harness² Codex rollout, whose `turn.completed` carries the thread total. GitHub Copilot CLI reports assistant message output tokens and cumulative shutdown model totals; Beacon records the message output directly, then differences shutdown totals for input, cache, reasoning, and any output not already counted. Cost comes from the shutdown's session-wide `totalNanoAiu`, differenced the same way and converted at one AI credit = 10^9 nano-AIU = US$0.01. It is recorded once per shutdown: on the model's usage event when the shutdown names one model, otherwise on one usage event with no model, so a multi-model session is not counted once per model. Sessions started before GitHub moved Copilot to AI credits on 2026-06-01 were billed per premium request, so they record no cost; the raw nano-AIU total stays in `raw.copilot`. Annual Copilot Pro and Pro+ subscribers stay on premium-request billing until their plan renews, and the session record does not say which plan applies, so for them the recorded cost is what the session would cost in AI credits rather than their bill.**Collected through the generic OpenTelemetry path, not yet verified against captured payloads:**diff --git a/docs/docs.json b/docs/docs.jsonindex 219c2015..c4412f64 100644--- a/docs/docs.json+++ b/docs/docs.json@@ -171,6 +171,7 @@"runtimes/github-copilot-cli","runtimes/goose","runtimes/grok-build",+ "runtimes/harness2","runtimes/kimi-code","runtimes/kiro","runtimes/muse-code",@@ -337,6 +338,7 @@"cli/endpoint-devin","cli/endpoint-dsh","cli/endpoint-fx",+ "cli/endpoint-harness2","cli/endpoint-traces","cli/dashboard","cli/wazuh",diff --git a/docs/runtimes/harness2.mdx b/docs/runtimes/harness2.mdxnew file mode 100644index 00000000..0be719ae--- /dev/null+++ b/docs/runtimes/harness2.mdx@@ -0,0 +1,86 @@+---+title: "Harness²"+description: "Beacon support details for Harness² (google-research/harness2) rollout telemetry"+---++## Runtime overview++Beacon supports [Harness²](https://github.com/google-research/harness2), Google Research's framework for recursive agent harnessing, by reading the event stream of every executor rollout it records.++Harness² is not an agent itself. It refines an agent's *harness* (system prompt, guardrails, memory, tool guidance, subagents, skills, scripts and plugins) by running an executor, [OpenCode](/runtimes/opencode) or [Codex](/runtimes/codex-cli), over benchmark tasks (JobBench, LAB, WorkBuddy-Bench) with candidate harnesses, then composing a better harness from how those runs differed. One task at budget `k` is `4k` executor rollouts plus `3k` improvement sessions, so an experiment produces many agent sessions quickly, with unrestricted network access and, for Codex, its sandbox disabled.++```bash+beacon endpoint harness2 sync --runs-dir ~/src/harness2/runs+```++<Note>+Beacon's OpenCode and Codex integrations do not see these rollouts. Harness² runs Codex with its own `CODEX_HOME` and `--ephemeral`, so neither Beacon's user-level Codex settings nor `~/.codex/sessions` are involved, and it runs OpenCode from a source checkout with its event stream captured to a file. Reading those files is what records them.+</Note>++## Prerequisites++Before collecting Harness² telemetry, make sure:++- Harness² has run at least one rollout, so its runs directory exists (`runs/` in the checkout, or `$HARNESS2_RUNS_DIR`).+- `beacon endpoint install` has run so shared endpoint config and runtime log paths exist.+- Something runs [`beacon endpoint harness2 sync`](/cli/endpoint-harness2) after or during experiments. Nothing sweeps on its own.++## Collection path++Beacon walks the runs directory and recognizes a trajectory by its first event, not its name, so graded results, judge output and task inputs are left alone:++| Benchmark | Trajectory |+|-----------|------------|+| JobBench | `jb/dataset/<profession>/<task>/model_traj/<label>/<task>_<label>_attempt<N>_<stamp>.jsonl` |+| LAB | `lab/results/<oc\|cx>/<run-id>/opencode_session.jsonl` or `codex_session.jsonl` |+| WorkBuddy-Bench | `wb/<subset>/.../<trial>/agent/oc-output.txt` or `codex-output.jsonl` |++Two stream formats occur, one per executor: OpenCode's `opencode run --format json` events and Codex's `codex exec --json` thread events. Every event carries `harness.name=harness2` and `harness.collection_method=poll`; the session id is the executor's own (OpenCode's `sessionID`, Codex's thread id).++Each event also carries the rollout's place in the experiment under `raw.harness2`:++| Field | Meaning |+|-------|---------|+| `executor` | `opencode` or `codex` |+| `benchmark`, `subset` | Harness²'s benchmark code (`jb`, `lab`, `wb`) and WorkBuddy subset |+| `task_id`, `label`, `attempt` | JobBench task, harness label and retry |+| `run_id`, `trial` | LAB run id or the run's completion marker; WorkBuddy trial |+| `harness_fingerprint` | The digest Harness² records of the harness tree that produced the rollout, or `base` |+| `trajectory`, `line` | Where in the runs directory the event was read |++`raw` is removed by [Metadata-only](/log-forwarding/asymptote) forwarding, so a Metadata-only endpoint forwards the events without this context.++## Telemetry coverage++| Signal | OpenCode executor | Codex executor |+|--------|-------------------|----------------|+| Session start | Inferred from the stream's first event | `thread.started` |+| Agent messages and reasoning | Yes | Yes |+| Commands | `bash` with output and exit code | `command_execution` with output and exit code |+| File activity | Reads, writes and edits, with the diff OpenCode reports | Each path an `apply_patch` created, changed or deleted, without the diff |+| MCP tool calls | Tools whose name carries `mcp` | `mcp_tool_call` with server, tool, arguments and result |+| Other tools | Yes, with arguments and result | Web search and agent-collaboration calls |+| Errors | `session.error` | `turn.failed` and stream errors (high), item warnings (medium) |+| Token usage | Per step, with cost when OpenCode reports one | Per turn, differenced from Codex's running thread total |+| Prompts | Not recorded | Not recorded |+| Approvals | Not recorded | Not recorded |++## Known gaps++- **No prompts.** Harness² passes the task prompt to the executor as an argument, and neither stream repeats it.+- **No approvals.** Harness² runs Codex with approvals bypassed and OpenCode with a permission fence that rejects outside paths without asking, and neither stream reports a decision. Beacon does not synthesize one.+- **No per-step times for Codex.** `codex exec --json` carries no timestamps, so a Codex rollout's events take the file's modification time and keep their order.+- **No model name.** Neither stream names the model, so token reports cannot estimate a list price for these rollouts; OpenCode's reported cost is kept when it has one.+- **Improvement sessions.** The proposer and composer sessions Harness² runs between rollouts keep only their final answer on disk, not an event stream, so they are not collected.+- **Overlap with the OpenCode integration.** A JobBench OpenCode rollout uses OpenCode's default data directory, so with Beacon's OpenCode plugin or `beacon endpoint opencode sync` also active, the same session can appear under `opencode` as well.++## Related++<Columns cols={2}>+ <Card title="beacon endpoint harness2" icon="terminal" href="/cli/endpoint-harness2">+ Sync and status commands, flags, and cursor behavior.+ </Card>+ <Card title="Runtimes overview" icon="list" href="/runtimes">+ Every runtime Beacon supports and how each is collected.+ </Card>+</Columns>diff --git a/docs/runtimes/index.mdx b/docs/runtimes/index.mdxindex c6329da7..6c88b626 100644--- a/docs/runtimes/index.mdx+++ b/docs/runtimes/index.mdx@@ -30,6 +30,7 @@ These integrations are grouped by where the agent runs and how telemetry is coll| [Devin Desktop](/runtimes/devin-desktop) | Cascade/Windsurf hooks | Prompt, command, MCP tool, file read, and file write telemetry where Desktop exposes Cascade hook payloads || [Factory Droid](/runtimes/factory-droid) | OTLP HTTP, optional hooks, and poll of `~/.factory/sessions` through `beacon endpoint factory sync` | OTLP/hooks: session, prompt, write/edit/create tool use, stop, session-end, and available OTLP telemetry. Poll: historical session records committed by Factory || [fx (Vercel Labs)](/runtimes/vercel-fx) | Poll of fx's own session records under `~/.fx/sessions` through `beacon endpoint fx sync` | Session start, prompts, tool calls, commands with exit codes and output, file reads and edits with diffs, MCP tool calls, agent messages, history compaction, and token usage and cost. No approval or session-end telemetry, because fx persists neither |+| [Harness²](/runtimes/harness2) | Poll of the executor event streams Harness² records under its runs directory through `beacon endpoint harness2 sync` | Each OpenCode or Codex rollout's agent messages, reasoning, commands with output and exit code, file activity, MCP and other tool calls, errors, and token usage, tagged with the benchmark, task, run and harness fingerprint it belongs to. No prompts or approvals, because neither stream carries them || [Gemini CLI](/runtimes/gemini-cli) | Opt-in local OTLP | Prompts, tool calls, MCP activity, file operations, and approval-related events emitted through OTLP || [GitHub Copilot CLI](/runtimes/github-copilot-cli) | MDM-managed OTLP HTTP plus poll of `~/.copilot/session-state` through `beacon endpoint copilot sync` | OTLP: prompt, session, tool, and approval-like activity emitted through Copilot CLI spans. Poll: assistant text, complete tool results, commands, files, and token usage from committed session records || [goose](/runtimes/goose) | Adapter only today: manually written plugin hooks in `.agents/plugins/`, plus manually configured local OTLP export over HTTP | Hooks: session start and end, prompts, pre-tool and post-tool activity, failed tools, command lines, file creates and edits with diffs, and MCP tool calls. OTLP adds token usage, model and provider, response ids, and agent reasoning. No automatic installer/discovery yet, no tool output of any kind on the hook path, and no approval decisions because goose exposes none of those on a hook |@@ -120,6 +121,7 @@ MDM assets, meaning Jamf Pro Extension Attributes, Fleet osquery queries, and Ri| DeepSeek Harness | Install the hook bridge into each profile at your `dsh` version ([how](/runtimes/deepseek-harness#installing-the-hook-bridge)), then install hooks with `beacon endpoint hooks install --harness dsh`. To backfill committed native sessions, schedule `beacon endpoint dsh sync` in the logged-in user's context or leave `--watch` running. || Gemini CLI | The default `beacon endpoint install` detects Gemini and manages `~/.gemini/settings.json` for local OTLP collection. Use an explicit list such as `claude,codex,gemini` when automatic discovery is unavailable. || fx (Vercel Labs) | Nothing to configure in fx. Schedule [`beacon endpoint fx sync`](/cli/endpoint-fx) in the logged-in user's context, or leave `--watch` running, so committed sessions keep reaching the runtime log. Nothing sweeps on its own. |+| Harness² | Nothing to configure in Harness². Run [`beacon endpoint harness2 sync --runs-dir <checkout>/runs`](/cli/endpoint-harness2) once, then after or during experiments, or leave `--watch` running. Nothing sweeps on its own. || OpenCode | Install the managed plugin for live telemetry with `beacon endpoint hooks install --harness opencode`. To backfill committed local sessions, schedule [`beacon endpoint opencode sync`](/cli/endpoint-opencode) in the logged-in user's context or leave `--watch` running. || Pi | Install the managed extension for live telemetry with `beacon endpoint hooks install --harness pi`. To backfill committed local sessions, schedule [`beacon endpoint pi sync`](/cli/endpoint-pi) in the logged-in user's context or leave `--watch` running. || GitHub Copilot CLI | Configure launch environment through MDM or customer policy: `COPILOT_OTEL_ENABLED=true` and `OTEL_EXPORTER_OTLP_ENDPOINT=http://127.0.0.1:4318`. Beacon validates the endpoint but does not edit shell profiles or launch environments. To backfill committed sessions, schedule [`beacon endpoint copilot sync`](/cli/endpoint-copilot) in the logged-in user's context or leave `--watch` running. |