Skip to content

The cross-harness, self-improving memory layer for AI agents. Join our community: https://discord.com/invite/zdNChS2fBu

20.7 KB
Raw Blame

Beacon CLI End-To-End Release Validation Runbook

Goal

Validate a stable public Beacon release from the user-facing docs at https://docs.asymptotelabs.ai/cli, not a local build, and capture any install, docs, integration, or product bugs in beacon-installation-notes.md before changing code.

Use repo context only after runtime testing to decide whether each finding is docs-only, CLI behavior, hook behavior, collector/exporter behavior, packaging, or Elastic content-pack behavior.

Test Environment

Run on macOS with:

  • Homebrew available and online.
  • Docker Desktop or Docker Compose available for local Elastic validation.
  • A dedicated test macOS user or a workstation where changing local AI runtime config is acceptable.
  • Authenticated/installable runtimes for the full matrix: Cursor, Claude Code, Codex CLI, Gemini CLI, and OpenCode.
  • Repo checkout available locally. Set BEACON_REPO to that checkout path for writing notes and later researching fixes.
  • Integration validation is user-assisted for now. The agent prepares Beacon, asks the user to submit unique prompts in each runtime, then self-verifies the resulting events in runtime.jsonl and Elastic. Headless/noninteractive runtime commands can be used as an optional fast path only when local auth and CLI behavior are known to be reliable.

Before running commands, record environment evidence:

sw_vers
uname -m
brew --version
docker --version || true
docker compose version || true
which beacon || true
beacon version || true
which agent || true
agent --version || true
which claude || true
claude --version || true
which codex || true
codex --version || true
which gemini || true
gemini --version || true
which opencode || true
opencode --version || true

Safety And State Baseline

Create a timestamped test directory and preserve all user-owned config files Beacon may touch:

export BEACON_E2E_RUN="${HOME}/beacon-e2e-$(date +%Y%m%d-%H%M%S)"
export BEACON_REPO="${BEACON_REPO:-$(pwd)}"
mkdir -p "$BEACON_E2E_RUN/backups"

for path in \
  "$HOME/.beacon" \
  "$HOME/.claude/settings.json" \
  "$HOME/.codex/config.toml" \
  "$HOME/.gemini/settings.json" \
  "$HOME/.cursor/hooks.json" \
  "$HOME/.config/opencode/plugins/beacon.ts"; do
  if [ -e "$path" ]; then
    mkdir -p "$BEACON_E2E_RUN/backups/$(dirname "${path#$HOME/}")"
    cp -R "$path" "$BEACON_E2E_RUN/backups/${path#$HOME/}"
  fi
done

Do not restore backups, stop Beacon, stop Elastic, or uninstall anything automatically. The default outcome of this runbook is to preserve the tested Beacon state so the user can continue inspecting runtime logs, hook config, the local dashboard, and Kibana after validation.

Documentation Walkthrough

Read the docs exactly like a new user, starting from:

For every confusing step, mismatch, missing prerequisite, wrong command, broken link, or unclear success criterion, append a finding to beacon-installation-notes.md with:

## Finding: <short title>

- Area: docs | install | endpoint | cursor | claude | codex | opencode | elastic | packaging
- Severity: blocker | major | minor | polish
- Environment: <OS, arch, Beacon version, runtime version>
- Expected: <what a new user would expect from docs>
- Actual: <what happened>
- Reproduction: <commands or UI steps>
- Evidence: <key command output, log line, event excerpt, screenshot path if applicable>
- Suspected cause: <leave blank until repo research if unknown>
- Proposed fix: <leave blank until repo research if unknown>

Homebrew Install Validation

Validate the shipped Homebrew formula and installed artifacts:

brew tap asymptote-labs/tap
brew install beacon
brew info beacon
which beacon
beacon version
beacon --help
beacon endpoint --help

Acceptance criteria:

  • beacon version prints a released version, not a local dev build.
  • which beacon resolves to the Homebrew-installed binary.
  • Help output includes endpoint, endpoint hooks, and endpoint elastic command groups.

Managed Ingest Connect Smoke

Needs a browser signed in to the Asymptote dashboard as a member of an org with managed ingest enabled (Asymptote Test), and Vector 0.50+ on the machine (brew install beacon pulls in the beacon-vector keg on macOS; otherwise the signed .pkg or vector.dev).

  • beacon endpoint connect --no-browser prints an approval URL; approve it, and the command ends with "Connected to Asymptote as device …" and a running forwarder.
  • beacon endpoint status shows Asymptote managed ingest: connected … credential valid and ~/.beacon/endpoint/asymptote/vector.toml contains no bcn_device string.
  • Run a short agent session; within two minutes it appears on /dashboard, and the machine is listed among the endpoints there. The dashboard is a single page whose views are tabs, so there is no /dashboard/telemetry or /dashboard/endpoints.
  • Revoke the device on the dashboard; within a minute status says credential revoked.
  • beacon endpoint disconnect removes the forwarder and the asymptote/ directory.

Endpoint Quickstart Validation

Follow the quickstart exactly:

beacon endpoint install
beacon endpoint status
beacon endpoint status --json > "$BEACON_E2E_RUN/status-after-install.json"
beacon endpoint discover
beacon endpoint discover --json > "$BEACON_E2E_RUN/discover-after-install.json"

Validate local state:

test -f "$HOME/.beacon/endpoint/config.json"
test -f "$HOME/.beacon/endpoint/otelcol.yaml"
test -f "$HOME/.beacon/endpoint/logs/runtime.jsonl"
python3 -m json.tool < "$HOME/.beacon/endpoint/config.json" > "$BEACON_E2E_RUN/config.pretty.json"

Acceptance criteria:

  • Install exits successfully without sudo in user mode.
  • Status reports collector/config/log paths clearly.
  • ~/.beacon/endpoint/logs/runtime.jsonl exists.
  • The runtime log contains a Beacon endpoint event such as telemetry.enabled or another install/status event.
  • Claude and Codex configuration files are created or updated consistently with the docs.

Repo areas to research later if failures occur:

Cursor Integration Test

Install Cursor hooks, then ask the user to submit a unique prompt in Cursor:

beacon endpoint hooks install --harness cursor
beacon endpoint hooks status --harness cursor
beacon endpoint hooks status --harness cursor --json > "$BEACON_E2E_RUN/cursor-hooks-status.json"

User runtime action:

export BEACON_E2E_CURSOR_MARKER="Beacon E2E Cursor prompt $(date +%s)"
printf 'Please submit this prompt in Cursor: %s\n' "$BEACON_E2E_CURSOR_MARKER"

Optional headless fast path, only when Cursor CLI auth is known to work:

agent -p --output-format text "$BEACON_E2E_CURSOR_MARKER: answer with exactly the word beacon-ok." \
  > "$BEACON_E2E_RUN/cursor-headless.out" \
  2> "$BEACON_E2E_RUN/cursor-headless.err"

Validation:

rg "$BEACON_E2E_CURSOR_MARKER|cursor|prompt|beacon-ok" "$HOME/.beacon/endpoint/logs/runtime.jsonl" \
  > "$BEACON_E2E_RUN/cursor-runtime-events.txt"

Acceptance criteria:

  • Hook status reports Cursor installed.
  • The user confirms the prompt was submitted, or the optional headless command exits successfully.
  • Runtime log contains a Cursor prompt event with the marker.
  • Event has stable Beacon fields including vendor, product, schema_version, event, timestamp, and a Cursor/session context where available.
  • Cursor remains usable if hook telemetry fails; if the prompt succeeds but no event appears, record this as an integration or hook-coverage finding.

Repo areas to research later:

Claude Code Integration Test

Confirm Beacon configured Claude settings, then ask the user to submit a unique prompt in Claude Code:

python3 -m json.tool < "$HOME/.claude/settings.json" > "$BEACON_E2E_RUN/claude-settings.pretty.json" || true
beacon endpoint status --json > "$BEACON_E2E_RUN/status-before-claude.json"

Runtime action:

export BEACON_E2E_CLAUDE_MARKER="Beacon E2E Claude prompt $(date +%s)"
printf 'Please submit this prompt in Claude Code: %s\n' "$BEACON_E2E_CLAUDE_MARKER"

Optional headless fast path, only when Claude Code headless mode is known to work:

claude -p "$BEACON_E2E_CLAUDE_MARKER: answer with exactly the word beacon-ok." \
  --output-format json \
  > "$BEACON_E2E_RUN/claude-headless.json" \
  2> "$BEACON_E2E_RUN/claude-headless.err"

Validation:

rg "$BEACON_E2E_CLAUDE_MARKER|claude|beacon-ok" "$HOME/.beacon/endpoint/logs/runtime.jsonl" \
  > "$BEACON_E2E_RUN/claude-runtime-events.txt"

Acceptance criteria:

  • The user confirms the prompt was submitted, or the optional headless command exits successfully.
  • Runtime log receives Claude OTLP-derived events.
  • Prompt content appears when emitted by the configured Claude telemetry surface.
  • If no events appear, status output makes collector/log path issues diagnosable.

Repo areas to research later:

Codex Integration Test

Confirm Beacon configured Codex, then ask the user to submit a unique prompt in Codex:

cp "$HOME/.codex/config.toml" "$BEACON_E2E_RUN/codex-config.toml" || true
beacon endpoint status --json > "$BEACON_E2E_RUN/status-before-codex.json"

Runtime action:

export BEACON_E2E_CODEX_MARKER="Beacon E2E Codex prompt $(date +%s)"
printf 'Please submit this prompt in Codex: %s\n' "$BEACON_E2E_CODEX_MARKER"

Optional headless fast path, only when Codex noninteractive mode is known to work:

codex exec --json "$BEACON_E2E_CODEX_MARKER: answer with exactly the word beacon-ok." \
  > "$BEACON_E2E_RUN/codex-headless.jsonl" \
  2> "$BEACON_E2E_RUN/codex-headless.err"

Validation:

rg "codex|$BEACON_E2E_CODEX_MARKER|beacon-ok" "$HOME/.beacon/endpoint/logs/runtime.jsonl" \
  > "$BEACON_E2E_RUN/codex-runtime-events.txt"

Acceptance criteria:

  • The user confirms the prompt was submitted, or the optional noninteractive command exits successfully.
  • Runtime log receives Codex OTLP-derived events.
  • Prompt text appears when emitted by Codex; agent activity events should still appear if raw prompt text is absent.
  • Noisy internal transport spans should not dominate the log.

Repo areas to research later:

Gemini CLI Integration Test

Gemini CLI support is opt-in, so repair the endpoint with the Gemini harness included, then ask the user to submit a unique prompt in Gemini CLI:

beacon endpoint repair --harness claude,codex,gemini
python3 -m json.tool < "$HOME/.gemini/settings.json" > "$BEACON_E2E_RUN/gemini-settings.pretty.json" || true
beacon endpoint status --json > "$BEACON_E2E_RUN/status-before-gemini.json"

Runtime action:

export BEACON_E2E_GEMINI_MARKER="Beacon E2E Gemini prompt $(date +%s)"
printf 'Please submit this prompt in Gemini CLI: %s\n' "$BEACON_E2E_GEMINI_MARKER"

Optional headless fast path, only when Gemini CLI noninteractive mode and auth are known to work:

gemini --prompt "$BEACON_E2E_GEMINI_MARKER: answer with exactly the word beacon-ok." \
  > "$BEACON_E2E_RUN/gemini-headless.out" \
  2> "$BEACON_E2E_RUN/gemini-headless.err"

If the installed Gemini CLI uses a different noninteractive command or flag set, document the discrepancy and use the current recommended command from gemini --help.

Validation:

rg "gemini|$BEACON_E2E_GEMINI_MARKER|beacon-ok" "$HOME/.beacon/endpoint/logs/runtime.jsonl" \
  > "$BEACON_E2E_RUN/gemini-runtime-events.txt"

Acceptance criteria:

  • ~/.gemini/settings.json contains Beacon-managed local OTLP telemetry settings with target set to local, otlpProtocol set to grpc, useCollector set to true, and no outfile.
  • The user confirms the prompt was submitted, or the optional noninteractive command exits successfully.
  • Runtime log receives Gemini OTLP-derived events with harness.name=gemini_cli.
  • Prompt content appears when emitted by Gemini.
  • If no events appear, check for project Gemini settings or GEMINI_TELEMETRY_* environment variables overriding the user settings.

Repo areas to research later:

OpenCode Integration Test

Install OpenCode hooks, then ask the user to submit a unique prompt in OpenCode:

beacon endpoint hooks install --harness opencode
beacon endpoint hooks status --harness opencode
beacon endpoint hooks status --harness opencode --json > "$BEACON_E2E_RUN/opencode-hooks-status.json"

Runtime action:

export BEACON_E2E_OPENCODE_MARKER="Beacon E2E OpenCode prompt $(date +%s)"
printf 'Please submit this prompt in OpenCode: %s\n' "$BEACON_E2E_OPENCODE_MARKER"

Optional headless fast path, only when OpenCode noninteractive mode is known to work:

opencode run --format json "$BEACON_E2E_OPENCODE_MARKER: answer with exactly the word beacon-ok." \
  > "$BEACON_E2E_RUN/opencode-headless.json" \
  2> "$BEACON_E2E_RUN/opencode-headless.err"

If the installed OpenCode CLI uses a different noninteractive command or flag set, document the discrepancy and use the current recommended command from opencode --help.

Validation:

rg "OpenCode|opencode|$BEACON_E2E_OPENCODE_MARKER|beacon-ok" "$HOME/.beacon/endpoint/logs/runtime.jsonl" \
  > "$BEACON_E2E_RUN/opencode-runtime-events.txt"

Acceptance criteria:

  • Hook status reports OpenCode plugin installed.
  • The user confirms the prompt was submitted, or the optional noninteractive command exits successfully.
  • Runtime log contains OpenCode chat/session activity with the marker.
  • OpenCode remains usable if Beacon hook execution fails; if the prompt succeeds but no event appears, record this as an integration or hook-coverage finding.

Repo areas to research later:

Local Elastic Integration Test

Generate and run the local Elastic content pack from the Homebrew CLI:

beacon endpoint elastic install-pack --output "$BEACON_E2E_RUN/beacon-elastic-pack"
beacon endpoint elastic up --pack-dir "$BEACON_E2E_RUN/beacon-elastic-pack"

Wait for services, then validate Elasticsearch directly before using Kibana:

curl -fsS http://localhost:9200 >/dev/null
curl -fsS "http://localhost:9200/logs-beacon.endpoint-*/_search?q=beacon.product:endpoint-agent&size=5" > "$BEACON_E2E_RUN/elastic-search-product.json"
curl -fsS "http://localhost:9200/logs-beacon.endpoint-*/_search?q=beacon.prompt.text:%22Beacon%20E2E%22&size=20" > "$BEACON_E2E_RUN/elastic-search-marker.json"
curl -fsS "http://localhost:9200/logs-beacon.endpoint-*/_search?q=beacon.harness.name:cursor&size=5" > "$BEACON_E2E_RUN/elastic-search-cursor.json"

Kibana validation:

  • Open http://localhost:5601.
  • Go to Discover.
  • Select Beacon Endpoint Events.
  • Search for beacon.prompt.text:"Beacon E2E" and for each runtime field value: beacon.harness.name:cursor, beacon.harness.name:claude_code, beacon.harness.name:codex_cli, beacon.harness.name:opencode.
  • Save screenshots or notes if the data view is missing, fields are poorly named, timestamps are wrong, or events are hard to find.

Acceptance criteria:

  • elastic up creates or uses the content pack successfully.
  • Elasticsearch, Kibana, and Filebeat start on loopback ports.
  • Events from all successful runtime tests are searchable in Elasticsearch.
  • Kibana has a usable Beacon data view and Discover workflow.
  • Generated pack points Filebeat at the same runtime log Beacon writes.

Repo areas to research later:

Notes Review And Fix Planning

After all tests, review beacon-installation-notes.md and group findings:

  • Docs fixes: docs command, prerequisite, expected-output, broken-link, or troubleshooting updates.
  • CLI fixes: command flags, status clarity, install/repair behavior, error messages, or path mismatch.
  • Hook fixes: Cursor/OpenCode install/status, hook payload parsing, runtime resilience, or event content handling.
  • Exporter fixes: Claude/Codex OTLP normalization, filtering, field mapping, or event schema consistency.
  • Elastic fixes: generated pack, Docker Compose stack, Filebeat config, index templates, ingest pipeline, Kibana assets, or docs.

For each confirmed bug, research the smallest relevant code path and add or update focused tests before implementation. Use existing validation patterns from:

Optional Full Teardown

Only run teardown if the user explicitly asks to remove the test state. Preserve notes and evidence under $BEACON_E2E_RUN.

beacon endpoint elastic down --pack-dir "$BEACON_E2E_RUN/beacon-elastic-pack" || true
beacon endpoint uninstall --keep-logs || true

If the user also asks to roll back runtime configuration, restore the timestamped backups manually and verify each restored file before continuing. Do not restore backups by default.

Final Release Gate

Before considering fixes ready, run the focused tests for touched areas plus the standard release gates:

cd cli/beacon && go test ./...
cd cli/beacon && go test -race ./internal/endpoint/...
cd cli/beacon-hooks && go test ./...
cd collector-builder/exporter/beaconjsonexporter && go test ./...
sh packaging/macos/test-endpoint-scripts.sh

For any fix affecting install, hooks, collector packaging, or Elastic assets, rerun the affected slice of this runbook with a freshly installed Homebrew release or a release-candidate archive, and append the retest result to beacon-installation-notes.md.