Skip to content

The cross-harness, self-improving memory layer for AI agents. Join our community: https://discord.com/invite/zdNChS2fBu

test(sandbox): i04 proves the bundled Vector forwards Beacon's events

shukieshahffa0e476480b
diff --git a/beacon-sandbox/AGENTS.md b/beacon-sandbox/AGENTS.md
index 12f61c7f..bbd26066 100644
--- a/beacon-sandbox/AGENTS.md
+++ b/beacon-sandbox/AGENTS.md
@@ -66,6 +66,7 @@ real money on the user's account, so:
| Approval or permission handling | `--scenario s07-denied-tool` |
| `endpoint install`, config paths, service startup | `--scenario i01-install-supervised` |
| The systemd backend, unit files, Linux system mode | `--scenario i02-install-systemd` |
+| A Vector pack, the bundled Vector version, or its secrets file | `--scenario i04-vector-forwarding` |
| Anything Windows | `--provider github --scenario w00-probe` |
| Something broad, or preparing a PR | the whole suite (no `--scenario`) |
diff --git a/beacon-sandbox/check/forwarding.go b/beacon-sandbox/check/forwarding.go
new file mode 100644
index 00000000..e2193e84
--- /dev/null
+++ b/beacon-sandbox/check/forwarding.go
@@ -0,0 +1,259 @@
+package check
+
+import (
+ "bufio"
+ "encoding/json"
+ "errors"
+ "fmt"
+ "io/fs"
+ "os"
+ "regexp"
+ "sort"
+ "strings"
+)
+
+// DeviceKeyPattern matches an Asymptote device key, test or real.
+//
+// The leak checks search for the key's shape rather than its value, so they work offline too: the
+// forwarding probe generates the key inside the guest and nothing on the host ever holds it. No
+// Beacon event has any reason to contain this prefix, so any match is a disclosure.
+var DeviceKeyPattern = regexp.MustCompile(`bcn_device_[A-Za-z0-9_]{8,}`)
+
+// Forward is what the runner observed while the bundled Vector forwarded Beacon's runtime log to a
+// loopback stand-in for the ingest service.
+//
+// Ran, Err and VectorExe are strings for the same reason as Removal's fields: empty means the probe
+// did not make that observation, which must never read as a pass.
+type Forward struct {
+ // Asked is whether the scenario requested the probe. Without it, a run whose probe never started
+ // would look the same as a scenario that never asked.
+ Asked bool
+ Ran string
+ // Err is the first thing that went wrong while setting up or collecting.
+ Err string
+ // VectorExe is the executable of the running Vector process, from /proc/<pid>/exe.
+ VectorExe string
+ WantVectorExe string
+ // Canary is the run's canary, which the session's own events carry.
+ Canary string
+
+ // Received is the runtime NDJSON the stand-in accepted, and ReceivedErr why it could not be read.
+ Received Log
+ ReceivedErr error
+ // Requests is the stand-in's request log.
+ Requests []ForwardRequest
+ RequestsErr error
+ // Runtime is Beacon's runtime log as it stood after forwarding finished, to check the forwarded
+ // lines are Beacon's own bytes.
+ Runtime Log
+ RuntimeErr error
+ // Text is other collected output, such as Vector's own log, keyed by artifact name, searched
+ // for a device key.
+ Text map[string]string
+}
+
+// ForwardRequest is one request the stand-in served. It records whether the bearer key matched the
+// secrets file, never the header itself.
+type ForwardRequest struct {
+ Method string `json:"method"`
+ Path string `json:"path"`
+ AuthOK bool `json:"auth_ok"`
+ Lines int `json:"lines"`
+ Encoding string `json:"encoding"`
+}
+
+// ReadForwardRequests parses the stand-in's request log. A malformed line is an error: the log is
+// written by our own stand-in, so a bad line means the probe is broken.
+func ReadForwardRequests(path string) ([]ForwardRequest, error) {
+ f, err := os.Open(path)
+ if err != nil {
+ return nil, err
+ }
+ defer f.Close()
+ var out []ForwardRequest
+ sc := bufio.NewScanner(f)
+ for n := 1; sc.Scan(); n++ {
+ line := strings.TrimSpace(sc.Text())
+ if line == "" {
+ continue
+ }
+ var r ForwardRequest
+ if err := json.Unmarshal([]byte(line), &r); err != nil {
+ return out, fmt.Errorf("%s:%d: %w", path, n, err)
+ }
+ out = append(out, r)
+ }
+ return out, sc.Err()
+}
+
+// Asymptote's validation event, written by `beacon endpoint asymptote validate`.
+const (
+ asymptoteDestinationType = "asymptote"
+ asymptoteDestinationMode = "asymptote_managed_http"
+)
+
+func isAsymptoteValidation(e Event) bool {
+ if e.ParseErr != nil {
+ return false
+ }
+ typ, _ := e.Field("destination.type")
+ mode, _ := e.Field("destination.mode")
+ return typ == asymptoteDestinationType && mode == asymptoteDestinationMode
+}
+
+// Forwarding judges whether the bundled Vector delivered Beacon's events, intact and authenticated.
+//
+// Four things have to hold, and each is a separate way this has broken or could:
+//
+// - The Vector that ran is the one the packages install. The probe starts it by that path, so a
+// mismatch means the image or the probe is wrong, and the run would be testing some other
+// binary.
+// - Every request carried the device key from the secrets file. That is the secret backend
+// working; a wrong or missing key is exactly what a revoked-looking endpoint would send.
+// - The validation event and at least one event from the session arrived. The validation event
+// is written right after Vector starts; the session's events are written minutes later, so they
+// prove Vector kept tailing the file rather than shipping one batch and stopping.
+// - Every forwarded line is a line from Beacon's runtime log, byte for byte. Standard mode ships
+// retained content unmodified, so any difference is Vector or the pack altering events.
+//
+// No device key may appear in anything collected.
+func Forwarding(v *Verdict, f Forward) {
+ if !f.Asked {
+ return
+ }
+ fail := func(check, summary, why string, evidence ...string) {
+ v.Add(Finding{Check: check, Severity: SevFail, Summary: summary, Why: why, Evidence: cap5(evidence)})
+ }
+ if f.Ran != "true" {
+ fail("forwarding.probe_ran", "the forwarding probe did not run, so nothing is known about "+
+ "whether the bundled Vector forwards",
+ "An unrun probe is not a pass. The scenario exists to run Vector; a run that did not is a "+
+ "broken run.")
+ return
+ }
+ if f.Err != "" {
+ fail("forwarding.probe_ran", f.Err,
+ "The probe could not finish, so delivery is unproven. Check vector.out in the run directory.")
+ }
+
+ switch {
+ case f.VectorExe == "":
+ fail("forwarding.vector_path", "the running Vector's executable was not observed",
+ "Without it the run cannot say which Vector forwarded.")
+ case f.VectorExe != f.WantVectorExe:
+ fail("forwarding.vector_path",
+ fmt.Sprintf("Vector ran from %s, not %s", f.VectorExe, f.WantVectorExe),
+ "The probe is meant to exercise the Vector the Linux packages install. A different "+
+ "binary means the result says nothing about the bundled one.")
+ }
+
+ runtimePosts := 0
+ var badAuth []string
+ switch {
+ case f.RequestsErr != nil:
+ fail("forwarding.authenticated", "the stand-in's request log could not be read: "+f.RequestsErr.Error(),
+ "Without it there is no record that Vector sent the device key.")
+ default:
+ for i, r := range f.Requests {
+ if r.Method == "POST" && r.Path == "/v1/ingest/runtime" {
+ runtimePosts++
+ }
+ if !r.AuthOK {
+ badAuth = append(badAuth, fmt.Sprintf("request %d: %s %s", i+1, r.Method, r.Path))
+ }
+ }
+ if len(badAuth) > 0 {
+ fail("forwarding.authenticated",
+ fmt.Sprintf("%d request(s) did not carry the device key from the secrets file", len(badAuth)),
+ "Vector reads the key through its secret backend. A request without it is what a "+
+ "revoked device looks like to the ingest service, so every batch would be refused.",
+ badAuth...)
+ }
+ }
+
+ if f.ReceivedErr != nil && !errors.Is(f.ReceivedErr, fs.ErrNotExist) {
+ fail("forwarding.delivered", "the forwarded events could not be read: "+f.ReceivedErr.Error(), "")
+ }
+ var validation, session int
+ var unparseable, altered []string
+ runtimeLines := map[string]bool{}
+ for _, e := range f.Runtime.Events {
+ runtimeLines[e.Raw] = true
+ }
+ for _, e := range f.Received.Events {
+ loc := fmt.Sprintf("%s:%d", f.Received.Path, e.Line)
+ if e.ParseErr != nil {
+ unparseable = append(unparseable, loc)
+ continue
+ }
+ if isAsymptoteValidation(e) {
+ validation++
+ } else if f.Canary != "" && strings.Contains(e.Raw, f.Canary) {
+ session++
+ }
+ if f.RuntimeErr == nil && !runtimeLines[e.Raw] {
+ altered = append(altered, loc)
+ }
+ }
+ if len(unparseable) > 0 {
+ fail("forwarding.intact", fmt.Sprintf("%d forwarded line(s) are not valid JSON", len(unparseable)),
+ "Beacon writes one JSON object per line and Vector frames them by newline. A partial or "+
+ "merged line means the framing is wrong, and ingest would reject it.",
+ unparseable...)
+ }
+ switch {
+ case f.RuntimeErr != nil:
+ fail("forwarding.intact", "the runtime log could not be re-read after forwarding: "+f.RuntimeErr.Error(),
+ "Without it the forwarded lines cannot be compared with what Beacon wrote.")
+ case len(altered) > 0:
+ fail("forwarding.intact",
+ fmt.Sprintf("%d forwarded line(s) do not appear verbatim in Beacon's runtime log", len(altered)),
+ "Standard mode ships Beacon's lines unchanged. A line that differs was altered or "+
+ "invented between the file and the ingest service.",
+ altered...)
+ }
+ if validation == 0 {
+ fail("forwarding.delivered", "the validation event never reached the stand-in",
+ "`beacon endpoint asymptote validate` writes it right after Vector starts. If it did not "+
+ "arrive, Vector is not shipping the runtime log at all.")
+ }
+ if session == 0 {
+ fail("forwarding.delivered", "no event from the agent session reached the stand-in",
+ "The session's events carry this run's canary and are written minutes after Vector "+
+ "starts. Without them the run only shows that Vector shipped its first batch.")
+ }
+
+ var leaks []string
+ for _, l := range []Log{f.Received, f.Runtime} {
+ for _, e := range l.Events {
+ if DeviceKeyPattern.MatchString(e.Raw) {
+ leaks = append(leaks, fmt.Sprintf("%s:%d", l.Path, e.Line))
+ }
+ }
+ }
+ names := make([]string, 0, len(f.Text))
+ for name := range f.Text {
+ names = append(names, name)
+ }
+ sort.Strings(names)
+ for _, name := range names {
+ if DeviceKeyPattern.MatchString(f.Text[name]) {
+ leaks = append(leaks, name)
+ }
+ }
+ if len(leaks) > 0 {
+ fail("forwarding.no_device_key_leak", "a device key appears in collected output",
+ "The key's one home is the 0600 secrets file. Anywhere else it is a disclosure.",
+ leaks...)
+ }
+
+ if len(v.Failures()) == 0 {
+ v.Add(Finding{
+ Check: "forwarding.delivered",
+ Severity: SevInfo,
+ Summary: fmt.Sprintf("the bundled Vector (%s) forwarded %d event(s) in %d authenticated "+
+ "request(s): the validation event and %d from the session", f.VectorExe,
+ len(f.Received.Events), runtimePosts, session),
+ })
+ }
+}
diff --git a/beacon-sandbox/check/forwarding_test.go b/beacon-sandbox/check/forwarding_test.go
new file mode 100644
index 00000000..3dd22e6d
--- /dev/null
+++ b/beacon-sandbox/check/forwarding_test.go
@@ -0,0 +1,175 @@
+package check
+
+import (
+ "errors"
+ "os"
+ "path/filepath"
+ "strings"
+ "testing"
+)
+
+const (
+ fwdCanary = "BEACON_SANDBOX_fwd123"
+ fwdVector = "/opt/beacon/bin/vector"
+ fwdValidation = `{"vendor":"beacon","event":{"action":"agent.detected"},"destination":{"type":"asymptote","mode":"asymptote_managed_http"},"message":"Beacon endpoint Asymptote validation event"}`
+ fwdPrompt = `{"vendor":"beacon","event":{"action":"prompt.submitted"},"prompt":{"text":"echo BEACON_SANDBOX_fwd123"}}`
+ fwdOther = `{"vendor":"beacon","event":{"action":"token.usage"}}`
+)
+
+func logOf(t *testing.T, name string, lines ...string) Log {
+ t.Helper()
+ p := filepath.Join(t.TempDir(), name)
+ if err := os.WriteFile(p, []byte(strings.Join(lines, "\n")+"\n"), 0o600); err != nil {
+ t.Fatal(err)
+ }
+ l, err := ReadLog(p)
+ if err != nil {
+ t.Fatal(err)
+ }
+ return l
+}
+
+// healthyForward is a run where everything arrived: the validation event, a session event, every
+// request authenticated, and every forwarded line present in the runtime log.
+func healthyForward(t *testing.T) Forward {
+ return Forward{
+ Asked: true,
+ Ran: "true",
+ VectorExe: fwdVector,
+ WantVectorExe: fwdVector,
+ Canary: fwdCanary,
+ Received: logOf(t, "forwarded.ndjson", fwdValidation, fwdPrompt),
+ Requests: []ForwardRequest{
+ {Method: "GET", Path: "/v1/ingest/health", AuthOK: true},
+ {Method: "POST", Path: "/v1/ingest/runtime", AuthOK: true, Lines: 2, Encoding: "gzip"},
+ },
+ Runtime: logOf(t, "runtime.jsonl", fwdOther, fwdValidation, fwdPrompt),
+ Text: map[string]string{"vector.out": "INFO vector: Vector has started."},
+ }
+}
+
+func judgeForward(f Forward) Verdict {
+ v := Verdict{Scenario: "i04"}
+ Forwarding(&v, f)
+ v.Resolve()
+ return v
+}
+
+func failedChecks(v Verdict) string {
+ var names []string
+ for _, f := range v.Failures() {
+ names = append(names, f.Check+": "+f.Summary)
+ }
+ return strings.Join(names, "\n")
+}
+
+func TestForwardingPassesAHealthyRun(t *testing.T) {
+ v := judgeForward(healthyForward(t))
+ if v.Outcome != Pass {
+ t.Fatalf("a healthy forwarding run failed:\n%s", failedChecks(v))
+ }
+ if len(v.Findings) != 1 || v.Findings[0].Severity != SevInfo {
+ t.Fatalf("want one info finding summarising delivery, got %+v", v.Findings)
+ }
+}
+
+// A scenario that did not ask gets no finding at all, while one that asked and never ran fails.
+func TestForwardingDistinguishesNotAskedFromNotRun(t *testing.T) {
+ v := judgeForward(Forward{})
+ if len(v.Findings) != 0 {
+ t.Fatalf("a scenario that did not ask must produce nothing, got %+v", v.Findings)
+ }
+ v = judgeForward(Forward{Asked: true})
+ if v.Outcome != Fail || !strings.Contains(failedChecks(v), "forwarding.probe_ran") {
+ t.Fatalf("an asked-for probe that never ran must fail, got %s:\n%s", v.Outcome, failedChecks(v))
+ }
+}
+
+// Each requirement has to be able to fail on its own. A mutation that fails every check at once
+// would not show which one is doing the work.
+func TestForwardingFailsEachRequirementIndependently(t *testing.T) {
+ cases := []struct {
+ name string
+ mutate func(*Forward)
+ check string
+ }{
+ {"probe error", func(f *Forward) { f.Err = "vector exited before it was ready" }, "forwarding.probe_ran"},
+ {"vector not observed", func(f *Forward) { f.VectorExe = "" }, "forwarding.vector_path"},
+ {"another vector", func(f *Forward) { f.VectorExe = "/usr/bin/vector" }, "forwarding.vector_path"},
+ {"unauthenticated batch", func(f *Forward) { f.Requests[1].AuthOK = false }, "forwarding.authenticated"},
+ {"request log missing", func(f *Forward) { f.Requests, f.RequestsErr = nil, os.ErrNotExist }, "forwarding.authenticated"},
+ {"no validation event", func(f *Forward) {
+ f.Received = logOf(t, "forwarded.ndjson", fwdPrompt)
+ }, "forwarding.delivered"},
+ {"no session event", func(f *Forward) {
+ f.Received = logOf(t, "forwarded.ndjson", fwdValidation)
+ }, "forwarding.delivered"},
+ {"nothing received", func(f *Forward) {
+ f.Received, f.ReceivedErr = Log{}, os.ErrNotExist
+ }, "forwarding.delivered"},
+ {"altered line", func(f *Forward) {
+ f.Received = logOf(t, "forwarded.ndjson", fwdValidation,
+ strings.Replace(fwdPrompt, "prompt.submitted", "prompt.edited", 1))
+ }, "forwarding.intact"},
+ {"merged lines", func(f *Forward) {
+ f.Received = logOf(t, "forwarded.ndjson", fwdValidation, fwdPrompt[:40])
+ }, "forwarding.intact"},
+ {"runtime log unreadable", func(f *Forward) { f.RuntimeErr = errors.New("gone") }, "forwarding.intact"},
+ {"key in a forwarded event", func(f *Forward) {
+ leaked := strings.Replace(fwdPrompt, `"}}`, ` bcn_device_test_0123456789abcdef"}}`, 1)
+ f.Received = logOf(t, "forwarded.ndjson", fwdValidation, leaked)
+ f.Runtime = logOf(t, "runtime.jsonl", fwdValidation, leaked)
+ }, "forwarding.no_device_key_leak"},
+ {"key in vector's output", func(f *Forward) {
+ f.Text["vector.out"] = "token=bcn_device_test_0123456789abcdef"
+ }, "forwarding.no_device_key_leak"},
+ }
+ for _, tc := range cases {
+ t.Run(tc.name, func(t *testing.T) {
+ f := healthyForward(t)
+ tc.mutate(&f)
+ v := judgeForward(f)
+ if v.Outcome != Fail {
+ t.Fatalf("want FAIL, got %s", v.Outcome)
+ }
+ if got := failedChecks(v); !strings.Contains(got, tc.check) {
+ t.Fatalf("want a %s failure, got:\n%s", tc.check, got)
+ }
+ })
+ }
+}
+
+// The session's events are the evidence that Vector kept tailing; the validation event alone must
+// not count as one, even though the canary is not in it.
+func TestForwardingDoesNotCountTheValidationEventAsSession(t *testing.T) {
+ f := healthyForward(t)
+ f.Canary = "asymptote" // present in the validation event, absent from the prompt
+ v := judgeForward(f)
+ if !strings.Contains(failedChecks(v), "no event from the agent session") {
+ t.Fatalf("the validation event was counted as a session event:\n%s", failedChecks(v))
+ }
+}
+
+func TestReadForwardRequestsRejectsAMalformedLine(t *testing.T) {
+ p := filepath.Join(t.TempDir(), "requests.jsonl")
+ body := `{"method":"GET","path":"/v1/ingest/health","auth_ok":true}` + "\nnot json\n"
+ if err := os.WriteFile(p, []byte(body), 0o600); err != nil {
+ t.Fatal(err)
+ }
+ if _, err := ReadForwardRequests(p); err == nil {
+ t.Fatal("a malformed request log must be an error, since our own stand-in wrote it")
+ }
+}
+
+func TestDeviceKeyPatternMatchesTestAndRealKeys(t *testing.T) {
+ for _, s := range []string{"bcn_device_test_0123456789abcdef0123456789abcdef", "Bearer bcn_device_Ab12Cd34Ef56"} {
+ if !DeviceKeyPattern.MatchString(s) {
+ t.Errorf("%q should match", s)
+ }
+ }
+ for _, s := range []string{"device_key", "bcn_device_", "the bcn_device prefix"} {
+ if DeviceKeyPattern.MatchString(s) {
+ t.Errorf("%q should not match", s)
+ }
+ }
+}
diff --git a/beacon-sandbox/cmd/beacon-sandbox/main.go b/beacon-sandbox/cmd/beacon-sandbox/main.go
index 05117294..52c20fa0 100644
--- a/beacon-sandbox/cmd/beacon-sandbox/main.go
+++ b/beacon-sandbox/cmd/beacon-sandbox/main.go
@@ -22,6 +22,7 @@ import (
"github.com/asymptote-labs/agent-beacon/beacon-sandbox/check"
"github.com/asymptote-labs/agent-beacon/beacon-sandbox/credentials"
"github.com/asymptote-labs/agent-beacon/beacon-sandbox/dispatch"
+ "github.com/asymptote-labs/agent-beacon/beacon-sandbox/image"
"github.com/asymptote-labs/agent-beacon/beacon-sandbox/runner"
"github.com/asymptote-labs/agent-beacon/beacon-sandbox/sandbox"
"github.com/asymptote-labs/agent-beacon/beacon-sandbox/scenario"
@@ -428,10 +429,42 @@ func judge(sc scenario.Scenario, art runner.Artifacts, creds credentials.Resolve
check.Reinstall(&v, lifecycle)
check.UnprivilegedUninstall(&v, lifecycle)
check.FailedReinstallRollback(&v, lifecycle)
+ check.Forwarding(&v, forwardOf(sc, art))
v.Resolve()
return v, log
}
+// forwardOf gathers what the forwarding probe left behind. Everything comes from the run directory
+// and meta, never from the live probe, so `verify` re-judges a saved run the same way.
+func forwardOf(sc scenario.Scenario, art runner.Artifacts) check.Forward {
+ f := check.Forward{
+ Asked: sc.VerifiesVectorForwarding(),
+ Ran: art.Meta["forward_ran"],
+ Err: art.Meta["forward_error"],
+ VectorExe: art.Meta["forward_vector_exe"],
+ WantVectorExe: image.VectorPath,
+ Canary: art.Canary,
+ }
+ if !f.Asked {
+ return f
+ }
+ f.Received, f.ReceivedErr = check.ReadLog(filepath.Join(art.Dir, runner.ForwardedLogName))
+ f.Requests, f.RequestsErr = check.ReadForwardRequests(filepath.Join(art.Dir, runner.ForwardRequestsName))
+ f.Runtime, f.RuntimeErr = check.ReadLog(filepath.Join(art.Dir, runner.ForwardRuntimeName))
+ // Everything else collected, searched for a device key. Meta holds the probe's own output.
+ f.Text = map[string]string{}
+ for _, name := range []string{runner.ForwardVectorOutName, runner.ForwardRequestsName,
+ "runtime.jsonl", "claude-result.json"} {
+ if b, err := os.ReadFile(filepath.Join(art.Dir, name)); err == nil {
+ f.Text[name] = string(b)
+ }
+ }
+ if b, err := json.Marshal(art.Meta); err == nil {
+ f.Text["meta"] = string(b)
+ }
+ return f
+}
+
// safetyOf folds the out-of-band safety signals into a verdict.
//
// Shared by the normal and the run-failed paths so neither can quietly omit them: host escape and
@@ -588,7 +621,7 @@ func judgeRunDir(root, dir, mutate string) (check.Verdict, error) {
}
}
- art := runner.Artifacts{RuntimeLog: logPath, Meta: meta}
+ art := runner.Artifacts{Dir: dir, RuntimeLog: logPath, Meta: meta}
art.Canary = meta["canary"]
// Sentinel state is not re-derivable offline, so trust what the run recorded -- including
// the detail, which is the evidence a finding quotes. Dropping it made re-judged verdicts
diff --git a/beacon-sandbox/image/image.go b/beacon-sandbox/image/image.go
index 4654407b..5ef309bc 100644
--- a/beacon-sandbox/image/image.go
+++ b/beacon-sandbox/image/image.go
@@ -31,6 +31,20 @@ const (
// build. Empty string means "latest", which makes runs non-reproducible.
DefaultClaudeVersion = "2.1.220"
+ // VectorVersion and VectorSHA256 pin the Vector that WithVector installs, which is the release
+ // and build the Linux packages bundle. musl because it is the static build: the gnu build links
+ // glibc's loader and would not run on every host the packages support.
+ //
+ // 0.56.0 because it is the newest release that expands ${VAR} and ${VAR:-default} in config
+ // files by default, which every pack Beacon renders relies on. 0.57 stops expanding the
+ // defaulted form and 0.58 stops expanding either without an opt-in flag. Keep this in step with
+ // packaging/linux/fetch-vector.sh.
+ VectorVersion = "0.56.0"
+ VectorSHA256 = "8c114c5e9fd9646516f014d5d837690447cf0d4f43ba4a3746713bc0612b039b"
+ // VectorPath is where the Linux packages install Vector, and the first fixed location Beacon's
+ // Vector discovery tries after --vector-bin and BEACON_VECTOR_BIN.
+ VectorPath = BeaconDir + "/vector"
+
// DefaultBase is the only distro in the matrix today. M0 established that Modal
// offers no arch selection, so this is amd64.
DefaultBase = "ubuntu:24.04"
@@ -189,6 +203,10 @@ type Spec struct {
// a directory-backed fleet. See scenario.Install.NSSOnlyUser for why that distinction decides
// whether a system-mode install can configure anyone at all.
NSSOnlyUser bool
+
+ // WithVector installs the release Vector at VectorPath, where the Linux packages put it. See
+ // scenario.Install.VerifyVectorForwarding.
+ WithVector bool
}
// LinuxArtifacts are the binaries the sandbox needs, as host paths.
@@ -285,6 +303,10 @@ func Build(spec Spec, log func(string, ...any)) (sandbox.ImageSpec, error) {
AgentUser, WorkDir, WorkDir),
)
+ if spec.WithVector {
+ layers = append(layers, vectorLayer())
+ }
+
if spec.WithDocker {
// Group membership is set at image build time; adding it later would need a re-login to
// take effect, and the session runs as this user.
@@ -346,6 +368,35 @@ func accountLayers(nssOnly bool) []string {
}
}
+// vectorLayer downloads the release Vector and installs it where the Linux packages do.
+//
+// At image build because that is the only moment with unrestricted egress; a running sandbox can
+// reach nothing but the model API. Appended after the shared layers so every other scenario keeps
+// its cached image.
+//
+// Each check fails the build instead of a paid run. The archive must match the pinned hash, and
+// the release's own SHA256SUMS must list the same hash, so a typo in the pin is caught along with
+// a bad download. The binary must have no PT_INTERP program header, because the reason to ship the
+// musl build is that it is static; this is read with python3 since the image has no readelf. The
+// layer ends by running `vector --version`.
+func vectorLayer() string {
+ archive := fmt.Sprintf("vector-%s-x86_64-unknown-linux-musl.tar.gz", VectorVersion)
+ base := "https://packages.timber.io/vector/" + VectorVersion
+ noInterp := `python3 -c 'import struct,sys; f=open(sys.argv[1],"rb").read(); ` +
+ `o=struct.unpack_from("<Q",f,0x20)[0]; e,n=struct.unpack_from("<HH",f,0x36); ` +
+ `sys.exit(any(struct.unpack_from("<I",f,o+i*e)[0]==3 for i in range(n)))' ` + VectorPath
+ return "RUN set -e; cd /tmp; " +
+ fmt.Sprintf("curl -fsSLO %s/%s; ", base, archive) +
+ fmt.Sprintf("curl -fsSL %s/vector-%s-SHA256SUMS -o vector-sums; ", base, VectorVersion) +
+ fmt.Sprintf("echo '%s %s' | sha256sum -c -; ", VectorSHA256, archive) +
+ fmt.Sprintf("grep -q '^%s %s$' vector-sums; ", VectorSHA256, archive) +
+ fmt.Sprintf("mkdir vector-x && tar -xzf %s -C vector-x; ", archive) +
+ fmt.Sprintf(`install -m 0755 "$(find vector-x -path '*/bin/vector' -type f | head -1)" %s; `, VectorPath) +
+ fmt.Sprintf("rm -rf vector-x vector-sums %s; ", archive) +
+ noInterp + " || { echo 'vector is dynamically linked' >&2; exit 1; }; " +
+ VectorPath + " --version"
+}
+
// PostPushLayers are the commands that must run after the artifact files land. The
// provider snapshots after Put, so these run in the first launched instance instead.
func PostPushLayers() string {
diff --git a/beacon-sandbox/image/image_test.go b/beacon-sandbox/image/image_test.go
index e64f228a..9e406804 100644
--- a/beacon-sandbox/image/image_test.go
+++ b/beacon-sandbox/image/image_test.go
@@ -650,3 +650,44 @@ func TestStaleProvenanceFallsBackToTheMtimeComparison(t *testing.T) {
t.Errorf("a local rebuild from current sources is fresh, got: %s", why)
}
}
+
+// The Vector layer is what i04 exists to test, so a layer that installed the wrong build, or
+// skipped its checks, would make that scenario verify something the packages do not ship. It is
+// also appended only on request, so every other scenario keeps its cached image.
+func TestBuildInstallsTheBundledVectorOnlyWhenAsked(t *testing.T) {
+ spec, err := Build(Spec{RepoRoot: stubRepo(t)}, nil)
+ if err != nil {
+ t.Fatal(err)
+ }
+ if joined := strings.Join(spec.Layers, "\n"); strings.Contains(joined, VectorPath) {
+ t.Fatalf("Vector must not be installed unless a scenario asks for it:\n%s", joined)
+ }
+
+ spec, err = Build(Spec{RepoRoot: stubRepo(t), WithVector: true}, nil)
+ if err != nil {
+ t.Fatal(err)
+ }
+ var layer string
+ for _, l := range spec.Layers {
+ if strings.Contains(l, VectorPath) {
+ layer = l
+ }
+ }
+ if layer == "" {
+ t.Fatal("WithVector added no layer that installs Vector")
+ }
+ for _, want := range []string{
+ "vector-" + VectorVersion + "-x86_64-unknown-linux-musl.tar.gz",
+ VectorSHA256 + " vector-" + VectorVersion,
+ "sha256sum -c -",
+ "vector-" + VectorVersion + "-SHA256SUMS",
+ "dynamically linked",
+ } {
+ if !strings.Contains(layer, want) {
+ t.Errorf("the Vector layer is missing %q:\n%s", want, layer)
+ }
+ }
+ if !strings.HasSuffix(layer, VectorPath+" --version") {
+ t.Errorf("the layer must end by running Vector, so a bad binary fails the build:\n%s", layer)
+ }
+}
diff --git a/beacon-sandbox/runner/forward.go b/beacon-sandbox/runner/forward.go
new file mode 100644
index 00000000..ec0971b6
--- /dev/null
+++ b/beacon-sandbox/runner/forward.go
@@ -0,0 +1,284 @@
+package runner
+
+import (
+ "context"
+ "fmt"
+ "path/filepath"
+ "strconv"
+ "strings"
+ "time"
+
+ "github.com/asymptote-labs/agent-beacon/beacon-sandbox/image"
+ "github.com/asymptote-labs/agent-beacon/beacon-sandbox/sandbox"
+)
+
+// Guest paths for the forwarding probe. The secrets file is where a user-mode `beacon endpoint
+// connect` puts it, so Vector reads the key the way the managed forwarder does.
+const (
+ forwardBaseDir = image.AgentHome + "/.beacon/endpoint/asymptote"
+ forwardSecretsFile = forwardBaseDir + "/vector-secrets.json"
+ forwardDataDir = forwardBaseDir + "/vector-data"
+ forwardPackDir = forwardBaseDir + "/pack"
+ forwardVectorOut = forwardBaseDir + "/vector.out"
+ forwardIngestDir = "/tmp/beacon-ingest"
+ forwardStandInPath = "/tmp/beacon-ingest-standin.py"
+)
+
+// Artifact names in the run directory, read back by the offline judge.
+const (
+ ForwardedLogName = "forwarded-runtime.ndjson"
+ ForwardRequestsName = "forward-requests.jsonl"
+ ForwardVectorOutName = "vector.out"
+ // ForwardRuntimeName is a second copy of the runtime log taken after forwarding finished.
+ // runtime.jsonl is collected earlier, at quiescence, and Vector may ship a line written after
+ // that; comparing against the earlier copy would report such a line as invented.
+ ForwardRuntimeName = "runtime-after-forwarding.jsonl"
+)
+
+// forwardStandIn serves the three ingest routes Vector's Asymptote pack calls, on loopback.
+//
+// It checks the bearer token against the secrets file and records only whether it matched, so the
+// key never reaches its output. Runtime and inventory bodies are gunzipped and appended one line
+// per event, which is the NDJSON contract of the real service.
+const forwardStandIn = `import gzip, http.server, json, os, sys, threading
+
+out_dir, secrets_path = sys.argv[1], sys.argv[2]
+lock = threading.Lock()
+routes = {"/v1/ingest/runtime": "runtime.ndjson", "/v1/ingest/inventory": "inventory.ndjson"}
+
+def device_key():
+ try:
+ with open(secrets_path) as f:
+ return json.load(f).get("device_key", "")
+ except Exception:
+ return ""
+
+def append(name, text):
+ with lock, open(os.path.join(out_dir, name), "a") as f:
+ f.write(text)
+
+class Handler(http.server.BaseHTTPRequestHandler):
+ protocol_version = "HTTP/1.1"
+
+ def auth_ok(self):
+ key = device_key()
+ return bool(key) and self.headers.get("Authorization", "") == "Bearer " + key
+
+ def body(self):
+ if self.headers.get("Transfer-Encoding", "").lower() == "chunked":
+ data = b""
+ while True:
+ size = int(self.rfile.readline().strip() or b"0", 16)
+ if size == 0:
+ self.rfile.readline()
+ return data
+ data += self.rfile.read(size)
+ self.rfile.readline()
+ return self.rfile.read(int(self.headers.get("Content-Length") or 0))
+
+ def reply(self, code):
+ self.send_response(code)
+ self.send_header("Content-Length", "0")
+ self.end_headers()
+
+ def do_GET(self):
+ ok = self.auth_ok()
+ append("requests.jsonl", json.dumps({"method": self.command, "path": self.path, "auth_ok": ok}) + "\n")
+ self.reply(200 if ok else 401)
+
+ # Vector's http sink healthcheck sends HEAD, not GET.
+ do_HEAD = do_GET
+
+ def do_POST(self):
+ data = self.body()
+ encoding = self.headers.get("Content-Encoding", "")
+ if encoding == "gzip":
+ data = gzip.decompress(data)
+ lines = [l for l in data.decode("utf-8", "replace").split("\n") if l.strip()]
+ ok = self.auth_ok()
+ name = routes.get(self.path)
+ if ok and name:
+ append(name, "".join(l + "\n" for l in lines))
+ append("requests.jsonl", json.dumps({"method": "POST", "path": self.path, "auth_ok": ok,
+ "lines": len(lines), "encoding": encoding}) + "\n")
+ self.reply(401 if not ok else (200 if name else 404))
+
+ def log_message(self, *args):
+ pass
+
+server = http.server.ThreadingHTTPServer(("127.0.0.1", 0), Handler)
+with open(os.path.join(out_dir, "port.tmp"), "w") as f:
+ f.write(str(server.server_address[1]))
+os.rename(os.path.join(out_dir, "port.tmp"), os.path.join(out_dir, "port"))
+server.serve_forever()
+`
+
+// forwardSetup starts the stand-in and the bundled Vector, then writes the validation event.
+//
+// Runs after install and before the session, because the pack reads the runtime log from the end:
+// anything written before Vector starts is never shipped, which is the managed forwarder's
+// contract too.
+//
+// The device key is generated inside the guest with shell builtins and written straight to the
+// 0600 secrets file. Every script passed to Exec lands in a process's argv, so a key made on the
+// host would be disclosed by the harness itself. Nothing outside the guest ever holds it; the leak
+// checks search for its shape instead.
+func forwardSetup(ctx context.Context, g guest, agent sandbox.ExecOpts, logPath string,
+ art *Artifacts, logf func(string, ...any)) error {
+
+ art.Meta["forward_ran"] = "true"
+ fail := func(format string, a ...any) error {
+ err := fmt.Errorf(format, a...)
+ art.Meta["forward_error"] = oneLine(err.Error())
+ logf("forwarding: %v", err)
+ return err
+ }
+
+ start := fmt.Sprintf(`set -eu
+umask 077
+mkdir -p %[1]s %[2]s %[3]s
+chmod 0700 %[1]s
+key="bcn_device_test_$(od -An -tx1 -N16 /dev/urandom | tr -d ' \n')"
+printf '{"device_key": "%%s"}\n' "$key" > %[4]s
+unset key
+chmod 0600 %[4]s
+cat > %[5]s <<'PY'
+%[6]sPY
+rm -f %[3]s/port
+setsid nohup python3 %[5]s %[3]s %[4]s > %[3]s/standin.log 2>&1 < /dev/null &
+i=0
+while [ ! -s %[3]s/port ] && [ "$i" -lt 50 ]; do i=$((i+1)); sleep 0.2; done
+if [ ! -s %[3]s/port ]; then echo "the ingest stand-in did not start: $(tail -c 400 %[3]s/standin.log)"; exit 1; fi
+echo "FORWARD_PORT=$(cat %[3]s/port)"`,
+ forwardBaseDir, forwardDataDir, forwardIngestDir, forwardSecretsFile, forwardStandInPath, forwardStandIn)
+ r, err := g.Exec(ctx, start, agent)
+ if err != nil || r.ExitCode != 0 {
+ return fail("could not start the ingest stand-in (rc=%d): %v %s", r.ExitCode, err, oneLine(r.Stdout+r.Stderr))
+ }
+ port := ""
+ for _, line := range strings.Split(r.Stdout, "\n") {
+ if v, ok := strings.CutPrefix(strings.TrimSpace(line), "FORWARD_PORT="); ok {
+ port = v
+ }
+ }
+ if _, err := strconv.Atoi(port); err != nil {
+ return fail("the ingest stand-in reported no port: %s", oneLine(r.Stdout))
+ }
+ art.Meta["forward_port"] = port
+
+ env := fmt.Sprintf("BEACON_ASYMPTOTE_INGEST_URL=http://127.0.0.1:%s BEACON_ASYMPTOTE_SECRETS_FILE=%s "+
+ "BEACON_ASYMPTOTE_DATA_DIR=%s", port, forwardSecretsFile, forwardDataDir)
+ pack := fmt.Sprintf(`set -eu
+beacon endpoint asymptote install-pack --user --output %[1]s --log-path %[2]s
+export %[3]s
+%[4]s --version
+%[4]s validate --skip-healthchecks %[1]s/vector.toml`,
+ forwardPackDir, shq(logPath), env, image.VectorPath)
+ r, err = g.Exec(ctx, pack+" 2>&1", agent)
+ art.Meta["forward_pack_validate"] = oneLine(r.Stdout)
+ if err != nil || r.ExitCode != 0 {
+ return fail("vector rejected the Asymptote pack (rc=%d): %v %s", r.ExitCode, err, oneLine(r.Stdout))
+ }
+ for _, line := range strings.Split(r.Stdout, "\n") {
+ if line = strings.TrimSpace(line); strings.HasPrefix(line, "vector ") {
+ art.Meta["forward_vector_version"] = oneLine(line)
+ break
+ }
+ }
+
+ // Started by the packaged path, then identified from /proc: that is what the check compares
+ // against, rather than trusting the path this script typed.
+ //
+ // Ready means Vector's startup healthcheck reached the stand-in, which happens after its
+ // sources are built. The extra seconds cover the file source's first scan, which records the
+ // end of the log; a line written before that is not shipped.
+ run := fmt.Sprintf(`set -u
+export %[1]s
+setsid nohup %[2]s --config %[3]s/vector.toml > %[4]s 2>&1 < /dev/null &
+i=0
+while [ "$i" -lt 60 ]; do
+ pid="$(pgrep -n -x vector || true)"
+ if [ -n "$pid" ] && grep -q '"path": "/v1/ingest/health"' %[5]s/requests.jsonl 2>/dev/null; then break; fi
+ i=$((i+1)); sleep 0.5
+done
+pid="$(pgrep -n -x vector || true)"
+if [ -z "$pid" ]; then echo "vector exited: $(tail -c 600 %[4]s)"; exit 1; fi
+if ! grep -q '"path": "/v1/ingest/health"' %[5]s/requests.jsonl 2>/dev/null; then
+ echo "vector never called the health check: $(tail -c 600 %[4]s)"; exit 1
+fi
+sleep 3
+echo "FORWARD_VECTOR_EXE=$(readlink /proc/$pid/exe)"`,
+ env, image.VectorPath, forwardPackDir, forwardVectorOut, forwardIngestDir)
+ r, err = g.Exec(ctx, run, agent)
+ for _, line := range strings.Split(r.Stdout, "\n") {
+ if v, ok := strings.CutPrefix(strings.TrimSpace(line), "FORWARD_VECTOR_EXE="); ok {
+ art.Meta["forward_vector_exe"] = v
+ }
+ }
+ if err != nil || r.ExitCode != 0 {
+ return fail("vector did not start (rc=%d): %v %s", r.ExitCode, err, oneLine(r.Stdout))
+ }
+ logf("forwarding: vector %s running from %s, stand-in on 127.0.0.1:%s",
+ art.Meta["forward_vector_version"], art.Meta["forward_vector_exe"], port)
+
+ r, err = g.Exec(ctx, fmt.Sprintf("beacon endpoint asymptote validate --user --log-path %s 2>&1", shq(logPath)), agent)
+ art.Meta["forward_validate_output"] = oneLine(r.Stdout)
+ if err != nil || r.ExitCode != 0 {
+ return fail("beacon endpoint asymptote validate failed (rc=%d): %v %s", r.ExitCode, err, oneLine(r.Stdout))
+ }
+ return nil
+}
+
+// forwardCollect waits for the session's events to reach the stand-in, then brings its record
+// home.
+//
+// The pack batches for up to 60 seconds, so the wait is a poll for the two things the check needs,
+// bounded well past one batch. It is skipped when setup failed, since nothing is running to wait
+// for; whatever the stand-in and Vector wrote is still collected. Vector and the stand-in are
+// stopped afterwards; on a disposable sandbox that is tidiness, on a machine that outlives the run
+// it is required.
+func forwardCollect(ctx context.Context, g guest, agent sandbox.ExecOpts, logPath, canary, runDir string,
+ setupOK bool, art *Artifacts, logf func(string, ...any)) {
+
+ if !setupOK {
+ collectForwardArtifacts(ctx, g, agent, logPath, runDir, art, logf)
+ return
+ }
+ start := time.Now()
+ wait := fmt.Sprintf(`i=0
+f=%[1]s/runtime.ndjson
+while [ "$i" -lt 75 ]; do
+ if grep -q 'asymptote_managed_http' "$f" 2>/dev/null && grep -q -F %[2]s "$f" 2>/dev/null; then echo FORWARD_ARRIVED; break; fi
+ i=$((i+1)); sleep 2
+done
+if [ -n "$(pgrep -n -x vector || true)" ]; then echo VECTOR_ALIVE=true; else echo VECTOR_ALIVE=false; fi`,
+ forwardIngestDir, shq(canary))
+ r, err := g.Exec(ctx, wait, agent)
+ art.Meta["forward_wait_seconds"] = fmt.Sprintf("%.0f", time.Since(start).Seconds())
+ art.Meta["forward_arrived"] = fmt.Sprintf("%v", err == nil && strings.Contains(r.Stdout, "FORWARD_ARRIVED"))
+ art.Meta["forward_vector_alive"] = fmt.Sprintf("%v", strings.Contains(r.Stdout, "VECTOR_ALIVE=true"))
+ logf("forwarding: arrived=%s after %ss, vector alive=%s",
+ art.Meta["forward_arrived"], art.Meta["forward_wait_seconds"], art.Meta["forward_vector_alive"])
+
+ collectForwardArtifacts(ctx, g, agent, logPath, runDir, art, logf)
+}
+
+// collectForwardArtifacts stops Vector and the stand-in and copies their records home.
+func collectForwardArtifacts(ctx context.Context, g guest, agent sandbox.ExecOpts, logPath, runDir string,
+ art *Artifacts, logf func(string, ...any)) {
+
+ _, _ = g.Exec(ctx, "pkill -x vector; pkill -f "+shq(forwardStandInPath)+"; true", agent)
+
+ for remote, name := range map[string]string{
+ forwardIngestDir + "/runtime.ndjson": ForwardedLogName,
+ forwardIngestDir + "/requests.jsonl": ForwardRequestsName,
+ forwardVectorOut: ForwardVectorOutName,
+ logPath: ForwardRuntimeName,
+ } {
+ if err := g.Get(ctx, remote, filepath.Join(runDir, name)); err != nil {
+ art.Meta["forward_missing_"+strings.TrimSuffix(name, filepath.Ext(name))] = oneLine(err.Error())
+ logf("forwarding: could not collect %s: %v", remote, err)
+ }
+ }
+ art.ForwardedLog = filepath.Join(runDir, ForwardedLogName)
+}
diff --git a/beacon-sandbox/runner/runner.go b/beacon-sandbox/runner/runner.go
index bbfed751..00d09ac4 100644
--- a/beacon-sandbox/runner/runner.go
+++ b/beacon-sandbox/runner/runner.go
@@ -52,6 +52,9 @@ type Artifacts struct {
// ArgvCheckRan records whether the argv scan actually executed, so a skipped check is
// never mistaken for a passing one.
ArgvCheckRan bool
+ // ForwardedLog is what the ingest stand-in received from the bundled Vector, for scenarios
+ // that verify forwarding. The other forwarding artifacts sit beside it in Dir.
+ ForwardedLog string
// Meta records what produced this run.
Meta map[string]string
}
@@ -274,6 +277,7 @@ func Run(ctx context.Context, p sandbox.Provider, sc scenario.Scenario, opts Opt
PreserveEnv: true, PathPrepend: lay.PathPrepend, Timeout: sc.Timeout() + time.Minute,
}
var res sandbox.Result
+ forwardOK := false
if sc.Install != nil {
installedLog, err := doInstall(ctx, g, sc, privileged, agent, lay, sh, &art, logf)
// Uninstall afterwards when the sandbox is this machine, whether or not the install
@@ -295,6 +299,12 @@ func Run(ctx context.Context, p sandbox.Provider, sc scenario.Scenario, opts Opt
return art, err
}
logPath = installedLog
+ // Before the session, because the pack tails the log from the end: only lines written
+ // after Vector starts are shipped. A failed setup is recorded and the session still runs,
+ // so the capture half of the verdict is not lost with it.
+ if sc.VerifiesVectorForwarding() {
+ forwardOK = forwardSetup(ctx, g, agent, logPath, &art, logf) == nil
+ }
logf("running session against the installed endpoint (budget $%.2f, timeout %s)",
sc.Budget(), sc.Timeout())
res, err = g.Exec(ctx, sh.PlainSession(sessionDir, claudeFlags(sc, prompt, sh)), sessionOpts)
@@ -468,6 +478,10 @@ func Run(ctx context.Context, p sandbox.Provider, sc scenario.Scenario, opts Opt
art.RuntimeLog = localLog
logf("collected %s", localLog)
+ if sc.VerifiesVectorForwarding() {
+ forwardCollect(ctx, g, agent, logPath, canary, runDir, forwardOK, &art, logf)
+ }
+
// Reinstall before uninstall, for the same reason uninstall runs here at all: the log is already
// on the host, so restarting the collector cannot disturb a capture assertion.
if sc.VerifiesRestartOnReinstall() {
@@ -614,6 +628,7 @@ func imageSpecFor(p sandbox.Platform, sc scenario.Scenario, opts Options,
ClaudeVersion: opts.ClaudeVersion,
WithDocker: sc.NeedsRealSystemd(),
NSSOnlyUser: sc.NeedsNSSOnlyUser(),
+ WithVector: sc.VerifiesVectorForwarding(),
}, logf)
}
diff --git a/beacon-sandbox/runner/runner_test.go b/beacon-sandbox/runner/runner_test.go
index a321f1a0..86c5a23b 100644
--- a/beacon-sandbox/runner/runner_test.go
+++ b/beacon-sandbox/runner/runner_test.go
@@ -7,6 +7,7 @@ import (
"strings"
"testing"
+ "github.com/asymptote-labs/agent-beacon/beacon-sandbox/check"
"github.com/asymptote-labs/agent-beacon/beacon-sandbox/image"
"github.com/asymptote-labs/agent-beacon/beacon-sandbox/scenario"
)
@@ -260,3 +261,34 @@ func TestArgvSamplerEmitsWhetherItSawTheAgent(t *testing.T) {
}
}
}
+
+// The sampler's device-key matcher has to find a key on a command line and must not fire on the
+// forwarding probe's own script, which spells the key's prefix while generating it. Run through
+// the host's awk, since a pattern this tool only ever reads as a string proves nothing.
+func TestArgvSamplerDeviceKeyMatcher(t *testing.T) {
+ awk, err := exec.LookPath("awk")
+ if err != nil {
+ t.Skip("no awk on this host")
+ }
+ if !strings.Contains(posixShell{}.ArgvSampler(), "/"+deviceKeyAwkPattern+"/") {
+ t.Fatal("the sampler does not use deviceKeyAwkPattern")
+ }
+ cases := map[string]bool{
+ "vector --token bcn_device_test_0123456789abcdef": true,
+ "curl -H Authorization: Bearer bcn_device_Ab12Cd34Ef56": true,
+ `sh -c key="bcn_device_test_$(od -An -tx1 -N16 /dev/urandom)"`: false,
+ `printf '{"device_key": "%s"}\n' "$key"`: false,
+ "/opt/beacon/bin/vector --config /home/agent/.beacon/pack/vector.toml": false,
+ }
+ for line, want := range cases {
+ cmd := exec.Command(awk, "/"+deviceKeyAwkPattern+"/ {found=1} END {exit !found}")
+ cmd.Stdin = strings.NewReader(line + "\n")
+ got := cmd.Run() == nil
+ if got != want {
+ t.Errorf("awk matcher on %q = %v, want %v", line, got, want)
+ }
+ if re := check.DeviceKeyPattern.MatchString(line); re != want {
+ t.Errorf("check.DeviceKeyPattern on %q = %v, want %v; the two must agree", line, re, want)
+ }
+ }
+}
diff --git a/beacon-sandbox/runner/shell.go b/beacon-sandbox/runner/shell.go
index fdf66f21..d654e7af 100644
--- a/beacon-sandbox/runner/shell.go
+++ b/beacon-sandbox/runner/shell.go
@@ -142,6 +142,10 @@ func (posixShell) ArgvVerdictPath() string { return "/tmp/beacon-sandbox-argv-ve
// used `grep -qF "$KEY"`, which the shell expands into grep's argv, so the `ps` in the same
// pipeline saw the grep process and reported a leak the tool had created. awk reads the key
// from ENVIRON instead, and the sampler's own processes are skipped as a second guard.
+//
+// It also searches for an Asymptote device key by shape. The forwarding probe generates that key
+// inside the guest, so no process outside it knows the value to search for; see
+// deviceKeyAwkPattern.
func (s posixShell) ArgvSampler() string {
return fmt.Sprintf(`set -u
rm -f %[1]s
@@ -158,6 +162,11 @@ scan() {
index($0,k){found=1}
END{exit !found}'
}
+scan_device() {
+ awk '/awk|ARGV_SAMPLER_SELF/ {next}
+ /%[3]s/ {found=1}
+ END{exit !found}'
+}
n=0
saw_agent=0
# Bounded so a leaked sandbox cannot spin forever; the instance is destroyed long before this.
@@ -168,7 +177,11 @@ while [ "$n" -lt 900 ]; do
# process holding the key is not evidence of anything.
case "$procs" in *claude*) saw_agent=1 ;; esac
if printf '%%s\n' "$procs" | scan; then
- echo "ARGV_LEAK samples=$n saw_agent=$saw_agent via=ps" > "$VERDICT"
+ echo "ARGV_LEAK samples=$n saw_agent=$saw_agent via=ps what=api_key" > "$VERDICT"
+ exit 0
+ fi
+ if printf '%%s\n' "$procs" | scan_device; then
+ echo "ARGV_LEAK samples=$n saw_agent=$saw_agent via=ps what=device_key" > "$VERDICT"
exit 0
fi
for f in /proc/[0-9]*/cmdline; do
@@ -176,7 +189,11 @@ while [ "$n" -lt 900 ]; do
cmd="$(tr '\0' '\n' < "$f" 2>/dev/null || true)"
case "$cmd" in *claude*) saw_agent=1 ;; esac
if printf '%%s\n' "$cmd" | scan; then
- echo "ARGV_LEAK samples=$n saw_agent=$saw_agent via=proc" > "$VERDICT"
+ echo "ARGV_LEAK samples=$n saw_agent=$saw_agent via=proc what=api_key" > "$VERDICT"
+ exit 0
+ fi
+ if printf '%%s\n' "$cmd" | scan_device; then
+ echo "ARGV_LEAK samples=$n saw_agent=$saw_agent via=proc what=device_key" > "$VERDICT"
exit 0
fi
done
@@ -186,9 +203,16 @@ done
SAMPLER
chmod +x %[2]s
nohup %[2]s %[1]s > /dev/null 2>&1 &
-echo ARGV_SAMPLER_STARTED`, s.ArgvVerdictPath(), "/tmp/beacon-sandbox-argv-sampler.sh")
+echo ARGV_SAMPLER_STARTED`, s.ArgvVerdictPath(), "/tmp/beacon-sandbox-argv-sampler.sh", deviceKeyAwkPattern)
}
+// deviceKeyAwkPattern is check.DeviceKeyPattern spelled for awk: "bcn_device_" and at least eight
+// key characters. The eight are written out because mawk, Ubuntu's default awk, has not always
+// supported interval expressions. The minimum length is what keeps the forwarding probe's own
+// script from matching: it contains `bcn_device_test_$(od ...)`, five key characters and then `$`.
+const deviceKeyAwkPattern = `bcn_device_[A-Za-z0-9_][A-Za-z0-9_][A-Za-z0-9_][A-Za-z0-9_]` +
+ `[A-Za-z0-9_][A-Za-z0-9_][A-Za-z0-9_][A-Za-z0-9_]`
+
// ---------------------------------------------------------------------------
// PowerShell
// ---------------------------------------------------------------------------
diff --git a/beacon-sandbox/scenario/scenario.go b/beacon-sandbox/scenario/scenario.go
index 818f33b4..ccd9e9ab 100644
--- a/beacon-sandbox/scenario/scenario.go
+++ b/beacon-sandbox/scenario/scenario.go
@@ -119,6 +119,15 @@ type Install struct {
// Only the account's *visibility* changes; the username, home directory and uid are otherwise
// identical, so everything a scenario asserts stays comparable to the local-account lanes.
NSSOnlyUser bool `yaml:"nss_only_user,omitempty"`
+ // VerifyVectorForwarding runs the bundled Vector against Beacon's Asymptote pack and checks
+ // what reached a loopback ingest stand-in.
+ //
+ // The Linux packages ship Vector at /opt/beacon/bin/vector, and it is the engine behind Beacon
+ // Managed and six customer-managed destinations. Nothing else in the suite runs it, so a pack
+ // Vector rejects, a secret backend that sends the wrong key, or a binary that does not start
+ // on a stripped host would all ship unnoticed. The stand-in serves the ingest contract on
+ // 127.0.0.1, so the run needs no account and no egress beyond the model API.
+ VerifyVectorForwarding bool `yaml:"verify_vector_forwarding,omitempty"`
}
// Platform selects the guest OS family a scenario is written for.
@@ -213,6 +222,11 @@ func (s Scenario) Validate() error {
return fmt.Errorf("%s: platform %q must be linux or windows", s.ID, s.Platform)
}
if s.TargetPlatform() == PlatformWindows {
+ // Vector is installed by the Linux image. The Windows runner is provisioned by CI and has
+ // no bundled Vector to run.
+ if s.VerifiesVectorForwarding() {
+ return fmt.Errorf("%s: verify_vector_forwarding is Linux-only", s.ID)
+ }
// systemd does not exist on Windows, and the nested-container recipe that provides it is
// a Linux arrangement. A scenario asking for both would be arranged as neither and would
// then quietly verify whatever backend happened to be selected.
@@ -257,6 +271,13 @@ func (s Scenario) Validate() error {
return fmt.Errorf("%s: nss_only_user requires mode=system; a user-mode install "+
"resolves $HOME and never looks the account up by name", s.ID)
}
+ // The probe runs Vector by hand as the agent user, against the user-mode log and secrets
+ // file. System mode keeps both under root-owned paths, and the nested lane runs the
+ // endpoint somewhere the stand-in is not, so either would need a different probe.
+ if s.Install.VerifyVectorForwarding && (s.Install.Mode == "system" || s.Install.NeedsRealSystemd) {
+ return fmt.Errorf("%s: verify_vector_forwarding needs a user-mode install outside the "+
+ "nested systemd lane, because the probe runs Vector as the agent user", s.ID)
+ }
// A user-mode uninstall needs no privileges, so there is no privilege failure to catch
// and the probe would assert something that cannot happen.
if s.Install.VerifyUnprivilegedUninstallFails && s.Install.Mode != "system" {
@@ -458,6 +479,11 @@ func (s Scenario) NeedsNSSOnlyUser() bool {
return s.Install != nil && s.Install.NSSOnlyUser
}
+// VerifiesVectorForwarding reports whether the bundled Vector must forward to a loopback stand-in.
+func (s Scenario) VerifiesVectorForwarding() bool {
+ return s.Install != nil && s.Install.VerifyVectorForwarding
+}
+
// NeedsRealSystemd reports whether this scenario must run somewhere systemd is genuinely PID 1.
func (s Scenario) NeedsRealSystemd() bool {
return s.Install != nil && s.Install.NeedsRealSystemd
diff --git a/beacon-sandbox/scenario/scenario_test.go b/beacon-sandbox/scenario/scenario_test.go
index 867c64fe..78fe800e 100644
--- a/beacon-sandbox/scenario/scenario_test.go
+++ b/beacon-sandbox/scenario/scenario_test.go
@@ -436,3 +436,32 @@ func TestNeedsRealSystemdRejectsContradictions(t *testing.T) {
t.Errorf("system+systemd is the intended combination: %v", err)
}
}
+
+// The forwarding probe runs Vector by hand as the agent user against the user-mode log and secrets
+// file, so it only means something in that arrangement.
+func TestVerifyVectorForwardingRejectsArrangementsTheProbeCannotRun(t *testing.T) {
+ base := func(in *Install) Scenario {
+ return Scenario{ID: "i", Prompt: "do a thing", Install: in,
+ Expect: []Expect{{Action: "prompt.submitted", Why: "baseline"}}}
+ }
+ for _, in := range []*Install{
+ {VerifyVectorForwarding: true, Mode: "system", Service: "none"},
+ {VerifyVectorForwarding: true, NeedsRealSystemd: true, Mode: "system", Service: "systemd"},
+ } {
+ if err := base(in).Validate(); err == nil {
+ t.Errorf("%+v cannot run the forwarding probe and must be rejected", in)
+ }
+ }
+ win := base(&Install{VerifyVectorForwarding: true, Mode: "user", Service: "none"})
+ win.Platform = PlatformWindows
+ if err := win.Validate(); err == nil {
+ t.Error("a Windows scenario has no bundled Vector and must be rejected")
+ }
+ ok := base(&Install{VerifyVectorForwarding: true, Mode: "user", Service: "none"})
+ if err := ok.Validate(); err != nil {
+ t.Errorf("user mode with no service manager is the intended arrangement: %v", err)
+ }
+ if !ok.VerifiesVectorForwarding() {
+ t.Error("VerifiesVectorForwarding should report the request")
+ }
+}
diff --git a/beacon-sandbox/scenarios/i04-vector-forwarding.yaml b/beacon-sandbox/scenarios/i04-vector-forwarding.yaml
new file mode 100644
index 00000000..4b0964ba
--- /dev/null
+++ b/beacon-sandbox/scenarios/i04-vector-forwarding.yaml
@@ -0,0 +1,37 @@
+id: i04-vector-forwarding
+# Does the Vector that the Linux packages bundle actually forward Beacon's events?
+#
+# The .deb, .rpm and Linux archives ship Vector at /opt/beacon/bin/vector. It runs Beacon Managed
+# forwarding and six customer-managed destinations, and nothing else in the suite starts it. So a
+# pack that Vector rejects, a secret backend that sends the wrong key, or a Vector build that will
+# not start on the target host would all go unnoticed.
+#
+# The image installs that Vector build at that path. After a user-mode install the probe renders
+# Beacon's Asymptote pack, writes a throwaway device key to the 0600 secrets file, and runs Vector
+# against a loopback stand-in for the ingest service. `beacon endpoint asymptote validate` writes
+# the first event, and the session writes the rest. The check wants both at the stand-in,
+# authenticated and byte-identical to the runtime log.
+#
+# User mode in the gVisor lane on purpose: there is no systemd here, so `beacon endpoint connect`
+# and the forwarder service are out of reach. Vector runs by hand as the agent user, which is
+# what the forwarder service does minus the service.
+install:
+ mode: user
+ service: none
+ expect_status_running: true
+ expect_service_kind: none
+ verify_vector_forwarding: true
+
+prompt: >-
+ Run exactly this shell command and report its output: echo {{canary}} | tee {{sentinel}}
+sentinel: /home/agent/work/i04.sentinel
+budget_usd: 1.0
+timeout_seconds: 420
+
+expect:
+ - action: prompt.submitted
+ contains: ["{{canary}}"]
+ fields: ["prompt.text", "session.id", "harness.name"]
+ why: >-
+ The session's events are what the forwarding check looks for at the stand-in. If they never
+ reached the runtime log, nothing downstream can be judged.
diff --git a/docs/contributing/beacon-sandbox.mdx b/docs/contributing/beacon-sandbox.mdx
index 8ddf27e9..6eb2b15f 100644
--- a/docs/contributing/beacon-sandbox.mdx
+++ b/docs/contributing/beacon-sandbox.mdx
@@ -205,6 +205,7 @@ go run ./cmd/beacon-sandbox run # all of them, ~30 min
| `s07-denied-tool` | Approvals and denials |
| `i01-install-supervised` | `beacon endpoint install`, then a session against the installed collector |
| `i02-install-systemd` | A system-mode install managed by systemd, then a session against it |
+| `i04-vector-forwarding` | The Vector the Linux packages ship, forwarding a session's events to a local stand-in for Beacon's ingest service |
Use one test while you are iterating. Run all of them before opening a pull request.