The cross-harness, self-improving memory layer for AI agents. Join our community: https://discord.com/invite/zdNChS2fBu
beacon-sandbox
A testing tool for Beacon contributors. It starts a throwaway Linux machine in the cloud, installs your local Beacon build on it, runs a real Claude Code session inside it, and checks whether Beacon recorded what the agent actually did.
Use it when you have changed how Beacon captures telemetry and want proof it still works against a live agent, not just that the unit tests pass. Nothing runs on your own machine, and the sandbox is destroyed when the test finishes.
This is a tool for working on Beacon. It is not part of Beacon, and it is not in any release.
Full guide: Testing Beacon In A Sandbox
· Coding agents should read AGENTS.md.
Quick start
You need a Modal account for the throwaway machines (the free tier covers this) and an Anthropic API key with credit, since the test runs a real agent session. Each test costs a few cents.
pip install modal && modal token new
export ANTHROPIC_API_KEY=sk-ant-...
cd cli/beacon && make build-linux-amd64 # the Beacon being tested
cd ../../beacon-sandbox
go run ./cmd/beacon-sandbox doctor --fix # checks the setup
go run ./cmd/beacon-sandbox run --scenario s02-bash-command # one test, ~3 min
go run ./cmd/beacon-sandbox run # all of them, ~30 min
doctor checks everything the tool needs and prints the exact fix for anything missing, so it is
both the right first step and the right thing to rerun if a test behaves oddly.
Why markers instead of expected output
AI sessions are not repeatable — ask an agent to run a command twice and you get two slightly different sessions. So instead of comparing against fixed expected output, each test:
- puts a random one-off string in the prompt and checks it appears in Beacon's log, and
- asks the agent to write that string into a file.
If the file is there the agent definitely did the work, so a missing event is Beacon's problem rather than the model having declined the task. Without that second check, those two situations look identical.
Tests that leave no file behind (prompts, token counts, approvals) fall back to the agent's own success report for the same purpose.
One thing to know when reading results: a warning means a check could not run, not that it passed. Silence from this tool is meant to mean "checked and clean".
Commands
| Command | What it does |
|---|---|
doctor [--fix] [--json] [--provider NAME] |
Check the setup and say how to fix what is missing |
run [--scenario ID] [--repeat N] [--keep-sandbox] [--provider NAME] |
Start a sandbox, run a session, check results |
verify <run-dir> [--mutate MODE] |
Re-check a saved run. Free and instant |
diff <before> <after> |
Compare what two runs captured |
clean |
Delete saved run output |
verify is worth knowing about: every run saves its output under runs/, and verify re-checks
that output without starting a sandbox or calling the API. Use it while adjusting what a test
expects, rather than paying for another run.
Three ways to supply the API key, in the order the tool looks: --modal-secret NAME,
--api-key-command CMD, or ANTHROPIC_API_KEY.
Testing on Windows
Modal has no Windows machines, so Windows scenarios run on a GitHub Actions Windows runner instead — which is already a disposable machine with a real Service Control Manager and administrator rights. You dispatch the run, it happens there, and the artifacts come back and are judged on your machine by the same code that judges a Modal run:
go run ./cmd/beacon-sandbox doctor --provider github # checks gh, the workflow, the secret
go run ./cmd/beacon-sandbox run --provider github --scenario w00-probe
Two things to know. A dispatch runs the pushed ref, not your working tree, so uncommitted work
is not under test — doctor warns when that is the case, which is the Windows equivalent of the
stale-collector trap below. And the agent's API key comes from the workflow's windows-sandbox
environment secret rather than from your shell, so the artifact leak check has no value to search
for and reports unverified, exactly as --modal-secret does. The in-runner argv scan still runs,
because the key genuinely is present there.
Several dispatches can run at once without mixing up their results. Each sends a random
correlation_id that the workflow puts in the run's title, and each follows only the run that
carries its own id, so one maintainer's verdict is never built from another's artifacts. The
workflow reads that input from the dispatched branch, so a branch cut before it existed needs a
rebase onto main first.
Scenarios say which platform they are written for (platform: windows), defaulting to Linux, and
run reports how many it set aside rather than silently narrowing the suite.
--provider local is what the workflow itself uses: it runs scenarios on the machine it is already
on. It refuses to start unless that machine is a GitHub-hosted runner, because a scenario installs
Beacon and registers services for real — on a workstation that is damage, not a test. Override with
--allow-host-mutation only on a VM you are happy to lose.
Layout
credentials/ finds the Anthropic API key
sandbox/ the cloud-machine interface, and the Modal implementation
image/ the sandbox image: Linux, a pinned Claude Code, and your Beacon build
scenario/ the test file format
scenarios/ the tests themselves
runner/ runs one test and collects the output
check/ decides whether the collected output is correct
hostguard/ confirms your own machine was left alone
Checking that the checks work
A test that cannot fail is worse than no test, so you can deliberately damage a saved run and confirm the tool notices:
go test ./... # no accounts or cost needed
go run ./cmd/beacon-sandbox verify --mutate corrupt-line runs/<id>/ # a PASS must become a FAIL
Other options are drop-commands, drop-action:<action>, and plant-secret. Each should turn a
PASS into a FAIL.
Limitations
Modal only provides Linux x86 machines, so this cannot test the macOS build — launchd, the signed installer, and notarization are out of reach. Windows is covered by dispatching to a CI runner instead (see above), which has its own limitation: the run happens somewhere you cannot attach to, so a scenario that fails opaquely is debugged from the uploaded artifacts and the job log rather than interactively. It also confirms the thing it planted was recorded rather than proving nothing else was missed.
The i0* tests install Beacon for real and cover installation and service management; the s0*
tests use a temporary collector instead, the path Beacon uses in CI and cloud agents. i02 runs
inside a nested privileged container, since systemd only starts as PID 1 and Modal's own init holds
that slot, so it takes several minutes longer than the rest.
The full guide has the complete list.