The cross-harness, self-improving memory layer for AI agents. Join our community: https://discord.com/invite/zdNChS2fBu
Go agent-ai-cliagent-harnessesagent-securitybrowser-extensionciclaude-codecloudcodexcursordetection-engineeringjevknowledge-basememorymobile-device-managementobservabilitysecuritysecurity-information-and-event-managementsecurity-toolstelemetrytraces
Stop partial forwarders after failed first connect
Justin D'Souza5eaa06006458
diff --git a/CLAUDE.md b/CLAUDE.mdindex ad21800c..74cfb3aa 100644--- a/CLAUDE.md+++ b/CLAUDE.md@@ -496,7 +496,7 @@ unpacked through Chrome's developer mode.- Keep `gen_ai.usage` as the single canonical token-usage representation: normalize all runtime token telemetry (span attributes, metric datapoints, legacy `llm.usage.*` aliases) into it, mirror OTel GenAI semconv JSON names exactly, and never add parallel or per-harness token fields. `gen_ai.usage.cost_usd` carries runtime-reported cost only; do not derive cost from local pricing tables.- When adding a new signal, include stable identifiers/counts/hashes alongside any retained raw content, and route raw fields through redaction, sanitization, truncation, and event-size controls.- Keep the dashboard read-only. It should inspect local status and JSONL events but must not mutate endpoint configuration or telemetry.-- The Asymptote device key has exactly one home, `<BaseDir>/asymptote/vector-secrets.json`, read by Vector's `secret` file backend. Never copy it into `config.json`, `vector.toml`, a service unit, an environment variable, a log line, or a `--json` result; `HasSecretDestinations` is unchanged because the key is not in `config.json`. `internal/auth` is flow-agnostic (PKCE, loopback callback server with an injected exchange function, browser opener, `PostJSON`); new browser flows reuse it rather than adding a second callback server. `connect` must fail before opening a browser when Vector or a service manager is missing, or when Vector rejects the rendered config, so no key is minted or rotated that cannot be used; enrollment rotates a connected device's key in place, so every local check runs ahead of it; the steps that can only fail after the exchange (storing the key, writing the config and unit, starting the service) are covered by the `asymptote/connect-pending` marker, written as soon as the key arrives and removed only when connect finishes, so status reports an incomplete connect instead of the previous connection.+- The Asymptote device key has exactly one home, `<BaseDir>/asymptote/vector-secrets.json`, read by Vector's `secret` file backend. Never copy it into `config.json`, `vector.toml`, a service unit, an environment variable, a log line, or a `--json` result; `HasSecretDestinations` is unchanged because the key is not in `config.json`. `internal/auth` is flow-agnostic (PKCE, loopback callback server with an injected exchange function, browser opener, `PostJSON`); new browser flows reuse it rather than adding a second callback server. `connect` must fail before opening a browser when Vector or a service manager is missing, or when Vector rejects the rendered config, so no key is minted or rotated that cannot be used; enrollment rotates a connected device's key in place, so every local check runs ahead of it; the steps that can only fail after the exchange (storing the key, writing the config and unit, starting the service) are covered by the `asymptote/connect-pending` marker, written as soon as the key arrives and removed only when connect finishes, so status reports an incomplete connect instead of the previous connection. A failed first connect stops and removes any partial forwarder before returning, which keeps a later local-only onboarding choice local.- Keep the release readiness guidance in `README.md` up to date when install, packaging, collector, or dashboard behavior changes.## CI Expectationsdiff --git a/cli/beacon/internal/endpoint/asymptote/connect.go b/cli/beacon/internal/endpoint/asymptote/connect.goindex e3c3e679..9aec47ac 100644--- a/cli/beacon/internal/endpoint/asymptote/connect.go+++ b/cli/beacon/internal/endpoint/asymptote/connect.go@@ -104,6 +104,19 @@ func Connect(ctx context.Context, opts ConnectOptions) (_ *ConnectResult, err erif manager == nil {manager = service.ForwarderManager{UserMode: opts.UserMode}}+ firstConnectForwarderTouched := false+ defer func() {+ // A first connect can fail after writing or starting its forwarder but before+ // the endpoint records a complete connection. Stop that partial forwarder so+ // a later onboarding retry may safely choose local-only telemetry.+ if err == nil || !keyIssued || reconnect || !firstConnectForwarderTouched {+ return+ }+ if unloadErr := manager.Unload(); unloadErr != nil {+ err = errors.Join(err, fmt.Errorf("could not stop the incomplete forwarder: %w", unloadErr))+ }+ manager.RemoveUnits()+ }()if !manager.Supported() {return nil, errors.New(manager.UnsupportedReason())}@@ -188,7 +201,7 @@ func Connect(ctx context.Context, opts ConnectOptions) (_ *ConnectResult, err er// marker itself cannot be written, the secrets file still holds the rotated-out key, which// the status credential check reports as revoked. A marker left by an earlier attempt// stays until a connect finishes, since that attempt's rotation still stands.- if err := markConnectPending(opts.UserMode, now()); err != nil {+ if err := markConnectPending(opts.UserMode, now(), reconnect); err != nil {return nil, fmt.Errorf("could not record that connect is in progress: %w", err)}if opts.AccountEnroll != nil {@@ -244,6 +257,7 @@ func Connect(ctx context.Context, opts ConnectOptions) (_ *ConnectResult, err erif err := writeFileAtomic(configPath, []byte(rendered), 0o644); err != nil {return nil, err}+ firstConnectForwarderTouched = !reconnectunitPath, err := manager.WriteUnit(vector.Path, configPath)if err != nil {return nil, errdiff --git a/cli/beacon/internal/endpoint/asymptote/enrollment.go b/cli/beacon/internal/endpoint/asymptote/enrollment.goindex 4316a8e9..e009bf73 100644--- a/cli/beacon/internal/endpoint/asymptote/enrollment.go+++ b/cli/beacon/internal/endpoint/asymptote/enrollment.go@@ -24,7 +24,7 @@ const (// ConnectPendingFileName marks a connect that has received a key from the server and not// yet finished. On a re-connect that key replaced the one the running forwarder holds, so// from then until every later step has succeeded the endpoint must not be reported as- // connected. It holds only a timestamp.+ // connected. It holds only a timestamp and whether this was a re-connect.ConnectPendingFileName = "connect-pending"SecretsFileName = "vector-secrets.json"VectorConfigName = "vector.toml"@@ -52,12 +52,33 @@ func ConnectIncomplete(userMode bool) bool {return err == nil}+type connectPending struct {+ StartedAt time.Time `json:"started_at"`+ Reconnect bool `json:"reconnect"`+}+// markConnectPending records that a connect has received a key and not yet finished.-func markConnectPending(userMode bool, at time.Time) error {+func markConnectPending(userMode bool, at time.Time, reconnect bool) error {if err := ensureDir(userMode); err != nil {return err}- return writeFileAtomic(ConnectPendingPath(userMode), []byte(at.UTC().Format(time.RFC3339)+"\n"), 0o600)+ data, err := json.Marshal(connectPending{StartedAt: at.UTC(), Reconnect: reconnect})+ if err != nil {+ return err+ }+ return writeFileAtomic(ConnectPendingPath(userMode), append(data, '\n'), 0o600)+}++func loadConnectPending(userMode bool) (connectPending, error) {+ data, err := os.ReadFile(ConnectPendingPath(userMode))+ if err != nil {+ return connectPending{}, err+ }+ var pending connectPending+ if err := json.Unmarshal(data, &pending); err != nil {+ return connectPending{}, fmt.Errorf("pending connect marker %s is not valid JSON: %w", ConnectPendingPath(userMode), err)+ }+ return pending, nil}// clearConnectPending removes the marker once a connect has finished.diff --git a/cli/beacon/internal/endpoint/asymptote/reconnect_test.go b/cli/beacon/internal/endpoint/asymptote/reconnect_test.goindex a0fc5e57..35800b16 100644--- a/cli/beacon/internal/endpoint/asymptote/reconnect_test.go+++ b/cli/beacon/internal/endpoint/asymptote/reconnect_test.go@@ -77,6 +77,13 @@ func assertConnectIncomplete(t *testing.T, connectErr error) {if !ConnectIncomplete(true) {t.Fatal("ConnectIncomplete must be true after a failure that followed the key exchange")}+ pending, err := loadConnectPending(true)+ if err != nil {+ t.Fatalf("read pending marker: %v", err)+ }+ if !pending.Reconnect {+ t.Fatal("a failed re-connect must be recorded as a re-connect")+ }status := Status(true, StatusOptions{SkipCredentialCheck: true})if status.Enabled || !status.ConnectIncomplete {t.Fatalf("status must not report an incomplete connect as connected: %+v", status)@@ -303,6 +310,44 @@ func TestFirstConnectFailingAfterApprovalIsReportedIncomplete(t *testing.T) {}}+func TestFirstConnectFailureAfterStartingForwarderStopsIt(t *testing.T) {+ t.Setenv("HOME", t.TempDir())+ isolateVectorDiscovery(t)+ fd := newFakeDashboard(t)+ fwd := &fakeForwarder{supported: true}++ // Make the final config.json update fail, after the forwarder has been written,+ // loaded and enrollment.json has been saved.+ cfgPath := endpointconfig.ConfigPath(true)+ if err := os.MkdirAll(filepath.Join(cfgPath, "blocker"), 0o700); err != nil {+ t.Fatal(err)+ }+ _, err := Connect(context.Background(), connectOptions(t, fd, fwd, fakeVector(t, "0.56.0", 0)))+ if err == nil {+ t.Fatal("expected the final config update to fail")+ }+ if fwd.loads != 1 || fwd.unloads != 1 || fwd.removed != 1 {+ t.Fatalf("a partial first-connect forwarder must be stopped and removed: loads=%d unloads=%d removed=%d", fwd.loads, fwd.unloads, fwd.removed)+ }+ pending, pendingErr := loadConnectPending(true)+ if pendingErr != nil {+ t.Fatalf("read pending marker: %v", pendingErr)+ }+ if pending.Reconnect {+ t.Fatal("a failed first connect must not be recorded as a re-connect")+ }+ status := Status(true, StatusOptions{SkipCredentialCheck: true})+ if status.Enabled || !status.ConnectIncomplete {+ t.Fatalf("status after partial first connect = %+v", status)+ }+ if strings.Contains(status.Message, "re-connect") || strings.Contains(status.Message, "rotated") {+ t.Fatalf("first-connect status must not claim an earlier key was rotated: %q", status.Message)+ }+ if !strings.Contains(status.Message, "partial forwarder was stopped") {+ t.Fatalf("first-connect status must say the partial forwarder was stopped: %q", status.Message)+ }+}+func TestReconnectAwaitingApprovalStillReportsTheWorkingConnection(t *testing.T) {fd, fwd := connectedEndpoint(t, fakeVector(t, "0.56.0", 0))// Until the server issues a key nothing has been rotated. A connect interrupted herediff --git a/cli/beacon/internal/endpoint/asymptote/status.go b/cli/beacon/internal/endpoint/asymptote/status.goindex ce28d2bf..1fd0f782 100644--- a/cli/beacon/internal/endpoint/asymptote/status.go+++ b/cli/beacon/internal/endpoint/asymptote/status.go@@ -67,9 +67,16 @@ func Status(userMode bool, opts StatusOptions) ManagedIngestStatus {}if incomplete {privacyMode, _ := managedprivacy.Normalize(enrollment.PrivacyMode)- // The record and config are the previous connection's, and so is the key prefix,- // which the server has rotated out; the key file may already hold the new one, so- // a credential check would say valid while the running forwarder cannot upload.+ pending, pendingErr := loadConnectPending(userMode)+ message := "connect incomplete: this device was approved but connect did not finish; the partial forwarder was stopped; run `beacon endpoint connect` again"+ if pendingErr != nil {+ message = fmt.Sprintf("connect incomplete: could not read its state: %v; run `beacon endpoint connect` again", pendingErr)+ } else if pending.Reconnect {+ // The record and config are the previous connection's, and so is the key+ // prefix, which the server has rotated out; the key file may already hold+ // the new one, so a credential check would be misleading here.+ message = fmt.Sprintf("re-connect incomplete: the server rotated device %s's key but connect did not finish, so the forwarder cannot upload; run `beacon endpoint connect` again", enrollment.DeviceID)+ }return ManagedIngestStatus{Enabled: false,ConnectIncomplete: true,@@ -78,7 +85,7 @@ func Status(userMode bool, opts StatusOptions) ManagedIngestStatus {OrganizationName: enrollment.OrganizationName,PrivacyMode: privacyMode,Forwarder: ForwarderStatus(userMode),- Message: fmt.Sprintf("re-connect incomplete: the server rotated device %s's key but connect did not finish, so the forwarder cannot upload; run `beacon endpoint connect` again", enrollment.DeviceID),+ Message: message,}}if !Connected(userMode) {diff --git a/docs/cli/endpoint-connect.mdx b/docs/cli/endpoint-connect.mdxindex 9d81c02f..f5871c1c 100644--- a/docs/cli/endpoint-connect.mdx+++ b/docs/cli/endpoint-connect.mdx@@ -39,7 +39,7 @@ A first connect that fails at `vector validate` or at loading the service savesRe-running `connect` on a connected machine reuses its install id, so the server rotates the existing device's key in place instead of creating a second device. Because that rotation invalidates the key the running forwarder holds, the new config is validated before enrollment: a re-run whose config Vector rejects leaves the forwarder, its key, and its buffer untouched. Under launchd the new forwarder is loaded only after the old one has exited (Vector drains in-flight requests first), and if the new instance has not started within the wait, `connect` fails and points at `launchctl print` for the job (v1.3.2).-Some steps can only fail after the server has issued the key: storing it, validating the config that carries the server's ingest URL, writing the service unit, starting the service, and recording the enrollment. `connect` writes `asymptote/connect-pending` as soon as the key arrives and removes it only when every step has succeeded. If a step in between fails, `connect` says the connect is incomplete and names the retry, and `beacon endpoint status` reports Beacon Managed as not connected with `connect_incomplete: true` in `--json`, because on a re-connect the forwarder still running holds the key the server just rotated out. The previous enrollment record and config are not treated as a working connection. Run `beacon endpoint connect` again once the cause is fixed. An approval that is cancelled or expires before a key is issued rotates nothing and leaves an existing connection as it was.+Some steps can only fail after the server has issued the key: storing it, validating the config that carries the server's ingest URL, writing the service unit, starting the service, and recording the enrollment. `connect` writes `asymptote/connect-pending` as soon as the key arrives and removes it only when every step has succeeded. If a step in between fails, `connect` says the connect is incomplete and names the retry, and `beacon endpoint status` reports Beacon Managed as not connected with `connect_incomplete: true` in `--json`. On a re-connect, the forwarder still running holds the key the server just rotated out, so the previous enrollment record and config are not treated as a working connection. On a first connect, Beacon stops and removes any partial forwarder before returning the error, so a later onboarding retry can safely select local-only telemetry. Run `beacon endpoint connect` again once the cause is fixed. An approval that is cancelled or expires before a key is issued rotates nothing and leaves an existing connection as it was.## Privacy modes@@ -63,7 +63,7 @@ are not uploaded under the new one.|------|-----------|---------------------|---------------------|| Enrollment record | `~/.beacon/endpoint/asymptote/enrollment.json` (0600) | `/Library/Application Support/Beacon/Endpoint/asymptote/enrollment.json` | `/etc/beacon/endpoint/asymptote/enrollment.json` || Install id | `~/.beacon/endpoint/asymptote/install-id` (0600) | `…/Endpoint/asymptote/install-id` | `/etc/beacon/endpoint/asymptote/install-id` |-| Pending-connect marker (only while a connect is unfinished) | `~/.beacon/endpoint/asymptote/connect-pending` (0600, timestamp only) | `…/Endpoint/asymptote/connect-pending` | `/etc/beacon/endpoint/asymptote/connect-pending` |+| Pending-connect marker (only while a connect is unfinished) | `~/.beacon/endpoint/asymptote/connect-pending` (0600, timestamp and reconnect flag only) | `…/Endpoint/asymptote/connect-pending` | `/etc/beacon/endpoint/asymptote/connect-pending` || Device key | `~/.beacon/endpoint/asymptote/vector-secrets.json` (0600) | `…/Endpoint/asymptote/vector-secrets.json` | `/etc/beacon/endpoint/asymptote/vector-secrets.json` || Vector config | `~/.beacon/endpoint/asymptote/vector.toml` (0644, no secret) | `…/Endpoint/asymptote/vector.toml` | `/etc/beacon/endpoint/asymptote/vector.toml` || Buffer and checkpoints | `~/.beacon/endpoint/asymptote/vector-data/` | `…/Endpoint/asymptote/vector-data/` | `/etc/beacon/endpoint/asymptote/vector-data/` |