Skip to content

The cross-harness, self-improving memory layer for AI agents. Join our community: https://discord.com/invite/zdNChS2fBu

Report a connect that fails after the key exchange as incomplete

Claudeed5a2fee7fac
diff --git a/CLAUDE.md b/CLAUDE.md
index 9b18b2b0..ad21800c 100644
--- a/CLAUDE.md
+++ b/CLAUDE.md
@@ -496,7 +496,7 @@ unpacked through Chrome's developer mode.
- Keep `gen_ai.usage` as the single canonical token-usage representation: normalize all runtime token telemetry (span attributes, metric datapoints, legacy `llm.usage.*` aliases) into it, mirror OTel GenAI semconv JSON names exactly, and never add parallel or per-harness token fields. `gen_ai.usage.cost_usd` carries runtime-reported cost only; do not derive cost from local pricing tables.
- When adding a new signal, include stable identifiers/counts/hashes alongside any retained raw content, and route raw fields through redaction, sanitization, truncation, and event-size controls.
- Keep the dashboard read-only. It should inspect local status and JSONL events but must not mutate endpoint configuration or telemetry.
-- The Asymptote device key has exactly one home, `<BaseDir>/asymptote/vector-secrets.json`, read by Vector's `secret` file backend. Never copy it into `config.json`, `vector.toml`, a service unit, an environment variable, a log line, or a `--json` result; `HasSecretDestinations` is unchanged because the key is not in `config.json`. `internal/auth` is flow-agnostic (PKCE, loopback callback server with an injected exchange function, browser opener, `PostJSON`); new browser flows reuse it rather than adding a second callback server. `connect` must fail before opening a browser when Vector or a service manager is missing, or when Vector rejects the rendered config, so no key is minted or rotated that cannot be used; enrollment rotates a connected device's key in place, so every local check runs ahead of it.
+- The Asymptote device key has exactly one home, `<BaseDir>/asymptote/vector-secrets.json`, read by Vector's `secret` file backend. Never copy it into `config.json`, `vector.toml`, a service unit, an environment variable, a log line, or a `--json` result; `HasSecretDestinations` is unchanged because the key is not in `config.json`. `internal/auth` is flow-agnostic (PKCE, loopback callback server with an injected exchange function, browser opener, `PostJSON`); new browser flows reuse it rather than adding a second callback server. `connect` must fail before opening a browser when Vector or a service manager is missing, or when Vector rejects the rendered config, so no key is minted or rotated that cannot be used; enrollment rotates a connected device's key in place, so every local check runs ahead of it; the steps that can only fail after the exchange (storing the key, writing the config and unit, starting the service) are covered by the `asymptote/connect-pending` marker, written as soon as the key arrives and removed only when connect finishes, so status reports an incomplete connect instead of the previous connection.
- Keep the release readiness guidance in `README.md` up to date when install, packaging, collector, or dashboard behavior changes.
## CI Expectations
diff --git a/cli/beacon/internal/endpoint/asymptote/connect.go b/cli/beacon/internal/endpoint/asymptote/connect.go
index 50f38820..e3c3e679 100644
--- a/cli/beacon/internal/endpoint/asymptote/connect.go
+++ b/cli/beacon/internal/endpoint/asymptote/connect.go
@@ -75,8 +75,11 @@ type ConnectResult struct {
// A re-run on an enrolled machine reuses its install id, which rotates the key in place on the
// server rather than creating a second device. That rotation is why validation comes first:
// once the server has rotated the key, a forwarder still running with the old one is failing
-// every upload, so nothing that can be checked locally may fail after that point.
-func Connect(ctx context.Context, opts ConnectOptions) (*ConnectResult, error) {
+// every upload, so nothing that can be checked locally may fail after that point. What cannot
+// be checked ahead (storing the key, writing the config and unit, starting the service) is
+// covered by a pending marker written as soon as the key arrives and removed only when every step has
+// succeeded: until then status reports the connect as incomplete, not as the old connection.
+func Connect(ctx context.Context, opts ConnectOptions) (_ *ConnectResult, err error) {
out := opts.Out
if out == nil {
out = io.Discard
@@ -85,6 +88,14 @@ func Connect(ctx context.Context, opts ConnectOptions) (*ConnectResult, error) {
if now == nil {
now = time.Now
}
+ // keyIssued is set once the server has handed over a key. Any failure after that point
+ // is reported as an incomplete connect with the retry command.
+ keyIssued, reconnect := false, false
+ defer func() {
+ if err != nil && keyIssued {
+ err = incompleteConnectError(reconnect, err)
+ }
+ }()
privacyMode, err := managedprivacy.Normalize(opts.PrivacyMode)
if err != nil {
return nil, err
@@ -106,6 +117,7 @@ func Connect(ctx context.Context, opts ConnectOptions) (*ConnectResult, error) {
if err != nil && !errors.Is(err, ErrNotEnrolled) {
return nil, err
}
+ reconnect = previous != nil
installID := opts.InstallID
if installID == "" && previous != nil {
installID = previous.InstallID
@@ -167,6 +179,18 @@ func Connect(ctx context.Context, opts ConnectOptions) (*ConnectResult, error) {
if err != nil {
return nil, err
}
+ keyIssued = true
+ // The server has issued the key; on a re-connect it has rotated the one the running
+ // forwarder holds. Mark the connect pending before anything else can fail, and remove the
+ // marker only once every step below has succeeded. It is written here rather than before
+ // enrollment so an approval that is cancelled, expires, or is interrupted with Ctrl-C
+ // (which ends the process without unwinding) never flags a working connection. If the
+ // marker itself cannot be written, the secrets file still holds the rotated-out key, which
+ // the status credential check reports as revoked. A marker left by an earlier attempt
+ // stays until a connect finishes, since that attempt's rotation still stands.
+ if err := markConnectPending(opts.UserMode, now()); err != nil {
+ return nil, fmt.Errorf("could not record that connect is in progress: %w", err)
+ }
if opts.AccountEnroll != nil {
fmt.Fprintf(out, "Account authorized device enrollment for %s (device %s, key %s)\n", displayOrg(result), result.DeviceID, result.KeyPrefix)
} else {
@@ -261,6 +285,9 @@ func Connect(ctx context.Context, opts ConnectOptions) (*ConnectResult, error) {
}); err != nil {
return nil, err
}
+ if err := clearConnectPending(opts.UserMode); err != nil {
+ return nil, fmt.Errorf("could not record that connect finished: %w", err)
+ }
return &ConnectResult{
Enrollment: enrollment,
VectorConfig: configPath,
@@ -272,6 +299,15 @@ func Connect(ctx context.Context, opts ConnectOptions) (*ConnectResult, error) {
}, nil
}
+// incompleteConnectError says that connect stopped after the key exchange and names the retry.
+// The cause stays last and wrapped so the step that failed is still visible to callers.
+func incompleteConnectError(reconnect bool, err error) error {
+ if reconnect {
+ return fmt.Errorf("re-connect incomplete: the server has already rotated this device's key, so the forwarder cannot upload until connect finishes; fix the cause and run `beacon endpoint connect` again: %w", err)
+ }
+ return fmt.Errorf("connect incomplete after the device was approved; fix the cause and run `beacon endpoint connect` again: %w", err)
+}
+
// DisconnectOptions drives Disconnect.
type DisconnectOptions struct {
UserMode bool
diff --git a/cli/beacon/internal/endpoint/asymptote/connect_test.go b/cli/beacon/internal/endpoint/asymptote/connect_test.go
index 9f93fb3c..c971d0e7 100644
--- a/cli/beacon/internal/endpoint/asymptote/connect_test.go
+++ b/cli/beacon/internal/endpoint/asymptote/connect_test.go
@@ -26,6 +26,7 @@ type fakeForwarder struct {
unloads int
removed int
loadErr error
+ writeErr error
}
func (f *fakeForwarder) Supported() bool { return f.supported }
@@ -34,6 +35,9 @@ func (f *fakeForwarder) Label() string { return "fake.forwarder" }
func (f *fakeForwarder) UnitPath() (string, error) { return f.unitPath, nil }
func (f *fakeForwarder) WriteUnit(vectorBin, configPath string) (string, error) {
f.written = append(f.written, vectorBin+" --config "+configPath)
+ if f.writeErr != nil {
+ return "", f.writeErr
+ }
if f.unitPath != "" {
if err := os.WriteFile(f.unitPath, []byte("unit"), 0o644); err != nil {
return "", err
diff --git a/cli/beacon/internal/endpoint/asymptote/enroll_test.go b/cli/beacon/internal/endpoint/asymptote/enroll_test.go
index f46fd91f..22cd3ff5 100644
--- a/cli/beacon/internal/endpoint/asymptote/enroll_test.go
+++ b/cli/beacon/internal/endpoint/asymptote/enroll_test.go
@@ -26,6 +26,9 @@ type fakeDashboard struct {
server *httptest.Server
initStatus int
exchangeFail string
+ // deviceKey overrides the key the exchange mints, so a test can tell a rotated key
+ // from the one it replaced.
+ deviceKey string
}
func newFakeDashboard(t *testing.T) *fakeDashboard {
@@ -62,9 +65,13 @@ func newFakeDashboard(t *testing.T) *fakeDashboard {
if ingest == "" {
ingest = "https://ingest.example.test"
}
+ key := fd.deviceKey
+ if key == "" {
+ key = "bcn_device_abcdefgh_" + strings.Repeat("k", 43)
+ }
_ = json.NewEncoder(w).Encode(map[string]any{
"device_id": "dev-1",
- "device_key": "bcn_device_abcdefgh_" + strings.Repeat("k", 43),
+ "device_key": key,
"key_prefix": "bcn_device_abcdefgh",
"ingest_url": ingest,
"organization_id": "org-1",
diff --git a/cli/beacon/internal/endpoint/asymptote/enrollment.go b/cli/beacon/internal/endpoint/asymptote/enrollment.go
index 3e2c85c7..4316a8e9 100644
--- a/cli/beacon/internal/endpoint/asymptote/enrollment.go
+++ b/cli/beacon/internal/endpoint/asymptote/enrollment.go
@@ -21,9 +21,14 @@ const (
// connect that fails after approval (validate, unit, load) retries with the same id and
// the server rotates the device's key instead of registering a second device.
InstallIDFileName = "install-id"
- SecretsFileName = "vector-secrets.json"
- VectorConfigName = "vector.toml"
- DataDirName = "vector-data"
+ // ConnectPendingFileName marks a connect that has received a key from the server and not
+ // yet finished. On a re-connect that key replaced the one the running forwarder holds, so
+ // from then until every later step has succeeded the endpoint must not be reported as
+ // connected. It holds only a timestamp.
+ ConnectPendingFileName = "connect-pending"
+ SecretsFileName = "vector-secrets.json"
+ VectorConfigName = "vector.toml"
+ DataDirName = "vector-data"
)
// Dir is the managed-ingest state directory for the selected endpoint mode.
@@ -35,6 +40,33 @@ func SecretsPath(userMode bool) string { return filepath.Join(Dir(userMode)
func VectorConfigPath(userMode bool) string { return filepath.Join(Dir(userMode), VectorConfigName) }
func DataDir(userMode bool) string { return filepath.Join(Dir(userMode), DataDirName) }
func InstallIDPath(userMode bool) string { return filepath.Join(Dir(userMode), InstallIDFileName) }
+func ConnectPendingPath(userMode bool) string {
+ return filepath.Join(Dir(userMode), ConnectPendingFileName)
+}
+
+// ConnectIncomplete reports whether a connect got as far as the key exchange and then failed
+// before finishing. On a re-connect the server has already rotated the device key by then, so
+// the forwarder still running with the previous key cannot upload until connect is run again.
+func ConnectIncomplete(userMode bool) bool {
+ _, err := os.Stat(ConnectPendingPath(userMode))
+ return err == nil
+}
+
+// markConnectPending records that a connect has received a key and not yet finished.
+func markConnectPending(userMode bool, at time.Time) error {
+ if err := ensureDir(userMode); err != nil {
+ return err
+ }
+ return writeFileAtomic(ConnectPendingPath(userMode), []byte(at.UTC().Format(time.RFC3339)+"\n"), 0o600)
+}
+
+// clearConnectPending removes the marker once a connect has finished.
+func clearConnectPending(userMode bool) error {
+ if err := os.Remove(ConnectPendingPath(userMode)); err != nil && !os.IsNotExist(err) {
+ return err
+ }
+ return nil
+}
// ReadInstallID returns the pinned install id, or "" when none has been written.
func ReadInstallID(userMode bool) string {
@@ -54,16 +86,21 @@ func WriteInstallID(userMode bool, installID string) error {
}
// Connected reports whether this endpoint is enrolled and has its forwarder configured: both
-// enrollment.json and vector.toml exist. Either alone is not a connection. disconnect
-// --keep-credentials leaves the record behind without a config, and a connect that failed at
-// vector validate or at loading the service leaves the config behind without a record; a
-// machine in either state must be offered connect again and reported as not connected.
+// enrollment.json and vector.toml exist and no connect is left incomplete. Either file alone is
+// not a connection. disconnect --keep-credentials leaves the record behind without a config,
+// and a first connect that failed at vector validate or at loading the service leaves the
+// config behind without a record. A re-connect that failed after the key exchange leaves both
+// files from the previous connection, but the server has rotated the key that connection's
+// forwarder holds, so the pending marker decides. A machine in any of these states must be
+// offered connect again and reported as not connected.
func Connected(userMode bool) bool {
if _, err := os.Stat(VectorConfigPath(userMode)); err != nil {
return false
}
- _, err := os.Stat(EnrollmentPath(userMode))
- return err == nil
+ if _, err := os.Stat(EnrollmentPath(userMode)); err != nil {
+ return false
+ }
+ return !ConnectIncomplete(userMode)
}
// Enrollment is the non-secret record of a device's enrollment. Everything `beacon endpoint
@@ -156,7 +193,8 @@ func RemoveState(userMode bool, keepCredentials bool) error {
}
return nil
}
- for _, path := range []string{VectorConfigPath(userMode), DataDir(userMode)} {
+ // The forwarder is gone, so a pending connect has nothing left running on a dead key.
+ for _, path := range []string{VectorConfigPath(userMode), DataDir(userMode), ConnectPendingPath(userMode)} {
if err := os.RemoveAll(path); err != nil && !os.IsNotExist(err) {
return err
}
diff --git a/cli/beacon/internal/endpoint/asymptote/reconnect_test.go b/cli/beacon/internal/endpoint/asymptote/reconnect_test.go
new file mode 100644
index 00000000..a0fc5e57
--- /dev/null
+++ b/cli/beacon/internal/endpoint/asymptote/reconnect_test.go
@@ -0,0 +1,324 @@
+package asymptote
+
+import (
+ "context"
+ "encoding/json"
+ "errors"
+ "os"
+ "path/filepath"
+ "strings"
+ "testing"
+
+ endpointconfig "github.com/asymptote-labs/agent-beacon/cli/beacon/internal/endpoint/config"
+)
+
+// Issue #450: a re-connect exchanges the code first, and the server rotates the device key in
+// place at that moment. Every step after the exchange that fails used to leave the previous
+// enrollment record and vector.toml on disk, so status said "connected" while the running
+// forwarder held the key the server had just rotated out. These tests pin that a failure
+// after the exchange is always reported as an incomplete connect, never as connected.
+
+var (
+ originalDeviceKey = "bcn_device_abcdefgh_" + strings.Repeat("k", 43)
+ rotatedDeviceKey = "bcn_device_rotated1_" + strings.Repeat("r", 43)
+)
+
+const movedIngestHost = "moved.example.test"
+
+// fakeVectorRejecting answers validate with exit 78 only for a config that mentions reject,
+// so the pre-enrollment preflight (which renders the previous ingest URL) passes and the
+// validate of the config carrying the server's new ingest URL fails.
+func fakeVectorRejecting(t *testing.T, reject string) string {
+ t.Helper()
+ path := fakeVector(t, "0.56.0", 0)
+ script := "#!/bin/sh\ncase \"$1\" in\n --version) echo \"vector 0.56.0 (aarch64-apple-darwin test)\";;\n" +
+ " validate) if grep -q '" + reject + "' \"$3\"; then echo 'ingest host refused' >&2; exit 78; fi; exit 0;;\n" +
+ " *) exit 0;;\nesac\n"
+ if err := os.WriteFile(path, []byte(script), 0o755); err != nil {
+ t.Fatal(err)
+ }
+ return path
+}
+
+// connectedEndpoint performs a successful first connect and arranges for every later exchange
+// to rotate the key, as the server does for a known install id.
+func connectedEndpoint(t *testing.T, vector string) (*fakeDashboard, *fakeForwarder) {
+ t.Helper()
+ t.Setenv("HOME", t.TempDir())
+ isolateVectorDiscovery(t)
+ fd := newFakeDashboard(t)
+ fwd := &fakeForwarder{supported: true}
+ if _, err := Connect(context.Background(), connectOptions(t, fd, fwd, vector)); err != nil {
+ t.Fatalf("first connect: %v", err)
+ }
+ if !Connected(true) || ConnectIncomplete(true) {
+ t.Fatal("a successful connect must be connected and not incomplete")
+ }
+ if key, _ := ReadDeviceKey(true); key != originalDeviceKey {
+ t.Fatalf("first connect stored an unexpected key")
+ }
+ fd.mu.Lock()
+ fd.deviceKey = rotatedDeviceKey
+ fd.mu.Unlock()
+ return fd, fwd
+}
+
+// assertConnectIncomplete is the contract after any failure that follows the exchange: the
+// endpoint is not reported as connected, status says the connect did not finish and names the
+// retry command, and no device key leaks into the error, the marker, or status JSON.
+func assertConnectIncomplete(t *testing.T, connectErr error) {
+ t.Helper()
+ if connectErr == nil {
+ t.Fatal("connect should have failed")
+ }
+ if Connected(true) {
+ t.Fatal("an endpoint whose connect failed after the key exchange must not be reported as connected")
+ }
+ if !ConnectIncomplete(true) {
+ t.Fatal("ConnectIncomplete must be true after a failure that followed the key exchange")
+ }
+ status := Status(true, StatusOptions{SkipCredentialCheck: true})
+ if status.Enabled || !status.ConnectIncomplete {
+ t.Fatalf("status must not report an incomplete connect as connected: %+v", status)
+ }
+ if !strings.Contains(status.Message, "incomplete") || !strings.Contains(status.Message, "beacon endpoint connect") {
+ t.Fatalf("status message must say the connect is incomplete and name the retry command: %q", status.Message)
+ }
+ msg := connectErr.Error()
+ if !strings.Contains(msg, "incomplete") || !strings.Contains(msg, "beacon endpoint connect") {
+ t.Fatalf("connect error must say the connect is incomplete and name the retry command: %v", connectErr)
+ }
+ marker, err := os.ReadFile(ConnectPendingPath(true))
+ if err != nil {
+ t.Fatalf("pending marker: %v", err)
+ }
+ if info, _ := os.Stat(ConnectPendingPath(true)); info.Mode().Perm() != 0o600 {
+ t.Fatalf("pending marker mode = %o, want 600", info.Mode().Perm())
+ }
+ for _, text := range []string{msg, string(marker), string(mustJSON(t, status))} {
+ for _, key := range []string{originalDeviceKey, rotatedDeviceKey} {
+ if strings.Contains(text, key) {
+ t.Fatalf("a device key leaked outside the secrets file: %s", text)
+ }
+ }
+ }
+}
+
+func TestReconnectThatCannotStoreTheRotatedKeyIsReportedIncomplete(t *testing.T) {
+ fd, fwd := connectedEndpoint(t, fakeVector(t, "0.56.0", 0))
+ // A directory where the secrets file belongs makes the atomic rename fail, the same way
+ // a full or read-only disk would, after the server has already rotated the key.
+ if err := os.Remove(SecretsPath(true)); err != nil {
+ t.Fatal(err)
+ }
+ if err := os.MkdirAll(filepath.Join(SecretsPath(true), "blocker"), 0o700); err != nil {
+ t.Fatal(err)
+ }
+ _, err := Connect(context.Background(), connectOptions(t, fd, fwd, fakeVector(t, "0.56.0", 0)))
+ assertConnectIncomplete(t, err)
+ if !strings.Contains(err.Error(), "could not store the device key") {
+ t.Fatalf("error should name the failed step: %v", err)
+ }
+ if fwd.loads != 1 {
+ t.Fatalf("the forwarder must not be reloaded without a stored key: loads=%d", fwd.loads)
+ }
+}
+
+func TestReconnectWhoseNewIngestURLVectorRejectsIsReportedIncomplete(t *testing.T) {
+ vector := fakeVectorRejecting(t, movedIngestHost)
+ fd, fwd := connectedEndpoint(t, vector)
+ fd.mu.Lock()
+ fd.ingestURL = "https://" + movedIngestHost
+ fd.mu.Unlock()
+
+ _, err := Connect(context.Background(), connectOptions(t, fd, fwd, vector))
+ assertConnectIncomplete(t, err)
+ if !strings.Contains(err.Error(), "vector validate failed") {
+ t.Fatalf("error should carry Vector's refusal: %v", err)
+ }
+ // This is exactly the state from the issue: the secrets file already holds the rotated
+ // key, so a credential check alone would say "valid", while the running forwarder was
+ // never restarted and still uploads with the key the server rotated out.
+ if key, _ := ReadDeviceKey(true); key != rotatedDeviceKey {
+ t.Fatal("the rotated key should be the one stored")
+ }
+ if fwd.loads != 1 || len(fwd.written) != 1 {
+ t.Fatalf("a rejected config must not reach the service manager: loads=%d written=%d", fwd.loads, len(fwd.written))
+ }
+}
+
+func TestReconnectThatCannotWriteTheForwarderUnitIsReportedIncomplete(t *testing.T) {
+ fd, fwd := connectedEndpoint(t, fakeVector(t, "0.56.0", 0))
+ fwd.writeErr = errors.New("unit directory is read-only")
+ _, err := Connect(context.Background(), connectOptions(t, fd, fwd, fakeVector(t, "0.56.0", 0)))
+ assertConnectIncomplete(t, err)
+ if !strings.Contains(err.Error(), "unit directory is read-only") {
+ t.Fatalf("error should carry the service manager's reason: %v", err)
+ }
+}
+
+func TestReconnectThatCannotStartTheForwarderIsReportedIncomplete(t *testing.T) {
+ fd, fwd := connectedEndpoint(t, fakeVector(t, "0.56.0", 0))
+ fwd.loadErr = errors.New("bootstrap failed: 5: Input/output error")
+ _, err := Connect(context.Background(), connectOptions(t, fd, fwd, fakeVector(t, "0.56.0", 0)))
+ assertConnectIncomplete(t, err)
+ if !strings.Contains(err.Error(), "could not be started") {
+ t.Fatalf("error should keep the load failure wording: %v", err)
+ }
+}
+
+func TestReconnectThatCannotRecordItsConfigIsReportedIncomplete(t *testing.T) {
+ fd, fwd := connectedEndpoint(t, fakeVector(t, "0.56.0", 0))
+ // config.json unreadable as a file: the last step, recording managed_ingest, fails.
+ cfgPath := endpointconfig.ConfigPath(true)
+ if err := os.Remove(cfgPath); err != nil {
+ t.Fatal(err)
+ }
+ if err := os.MkdirAll(filepath.Join(cfgPath, "blocker"), 0o700); err != nil {
+ t.Fatal(err)
+ }
+ _, err := Connect(context.Background(), connectOptions(t, fd, fwd, fakeVector(t, "0.56.0", 0)))
+ assertConnectIncomplete(t, err)
+}
+
+func TestRetryAfterAnIncompleteReconnectReportsConnected(t *testing.T) {
+ fd, fwd := connectedEndpoint(t, fakeVector(t, "0.56.0", 0))
+ fwd.loadErr = errors.New("bootstrap failed")
+ _, err := Connect(context.Background(), connectOptions(t, fd, fwd, fakeVector(t, "0.56.0", 0)))
+ assertConnectIncomplete(t, err)
+
+ fwd.loadErr = nil
+ result, err := Connect(context.Background(), connectOptions(t, fd, fwd, fakeVector(t, "0.56.0", 0)))
+ if err != nil {
+ t.Fatalf("retry: %v", err)
+ }
+ if !result.ReEnrolled || !Connected(true) || ConnectIncomplete(true) {
+ t.Fatalf("a successful retry must be connected: connected=%t incomplete=%t", Connected(true), ConnectIncomplete(true))
+ }
+ if _, err := os.Stat(ConnectPendingPath(true)); !os.IsNotExist(err) {
+ t.Fatalf("the pending marker must be removed once connect finishes: %v", err)
+ }
+ status := Status(true, StatusOptions{SkipCredentialCheck: true})
+ if !status.Enabled || status.ConnectIncomplete || status.Message != "" {
+ t.Fatalf("status after a successful retry = %+v", status)
+ }
+}
+
+func TestCancelledReconnectLeavesAWorkingConnectionConnected(t *testing.T) {
+ fd, fwd := connectedEndpoint(t, fakeVector(t, "0.56.0", 0))
+ // The approval never completes: no key was issued, so nothing was rotated and the
+ // running forwarder is still good.
+ fd.mu.Lock()
+ fd.exchangeFail = "enrollment session expired"
+ fd.mu.Unlock()
+ _, err := Connect(context.Background(), connectOptions(t, fd, fwd, fakeVector(t, "0.56.0", 0)))
+ if err == nil {
+ t.Fatal("expected the exchange failure")
+ }
+ if strings.Contains(err.Error(), "incomplete") {
+ t.Fatalf("a connect that never received a key must not claim the connection is broken: %v", err)
+ }
+ if !Connected(true) || ConnectIncomplete(true) {
+ t.Fatalf("a cancelled re-connect must leave the working connection connected: connected=%t incomplete=%t", Connected(true), ConnectIncomplete(true))
+ }
+ if key, _ := ReadDeviceKey(true); key != originalDeviceKey {
+ t.Fatal("a cancelled re-connect must not touch the stored key")
+ }
+ if status := Status(true, StatusOptions{SkipCredentialCheck: true}); !status.Enabled || status.ConnectIncomplete {
+ t.Fatalf("status after a cancelled re-connect = %+v", status)
+ }
+}
+
+func TestCancelledRetryKeepsAnEarlierIncompleteReconnectFlagged(t *testing.T) {
+ fd, fwd := connectedEndpoint(t, fakeVector(t, "0.56.0", 0))
+ fwd.loadErr = errors.New("bootstrap failed")
+ _, err := Connect(context.Background(), connectOptions(t, fd, fwd, fakeVector(t, "0.56.0", 0)))
+ assertConnectIncomplete(t, err)
+
+ // A later attempt that is cancelled before a key is issued changes nothing: the earlier
+ // rotation still left the forwarder on a dead key.
+ fwd.loadErr = nil
+ fd.mu.Lock()
+ fd.exchangeFail = "enrollment session expired"
+ fd.mu.Unlock()
+ if _, err := Connect(context.Background(), connectOptions(t, fd, fwd, fakeVector(t, "0.56.0", 0))); err == nil {
+ t.Fatal("expected the exchange failure")
+ }
+ if Connected(true) || !ConnectIncomplete(true) {
+ t.Fatalf("an earlier incomplete re-connect must stay flagged: connected=%t incomplete=%t", Connected(true), ConnectIncomplete(true))
+ }
+}
+
+func TestConnectRejectedBeforeTheExchangeIsNotFlaggedIncomplete(t *testing.T) {
+ fd, fwd := connectedEndpoint(t, fakeVector(t, "0.56.0", 0))
+ // The preflight fails before anything is sent to the server, so no key was rotated.
+ if _, err := Connect(context.Background(), connectOptions(t, fd, fwd, fakeVector(t, "0.56.0", 78))); err == nil {
+ t.Fatal("expected the preflight to fail")
+ }
+ if !Connected(true) || ConnectIncomplete(true) {
+ t.Fatalf("a preflight failure must leave the connection as it was: connected=%t incomplete=%t", Connected(true), ConnectIncomplete(true))
+ }
+}
+
+func TestDisconnectClearsAnIncompleteConnect(t *testing.T) {
+ fd, fwd := connectedEndpoint(t, fakeVector(t, "0.56.0", 0))
+ fwd.loadErr = errors.New("bootstrap failed")
+ _, err := Connect(context.Background(), connectOptions(t, fd, fwd, fakeVector(t, "0.56.0", 0)))
+ assertConnectIncomplete(t, err)
+
+ if err := Disconnect(DisconnectOptions{UserMode: true, Forwarder: fwd, KeepCredentials: true}); err != nil {
+ t.Fatal(err)
+ }
+ if ConnectIncomplete(true) {
+ t.Fatal("disconnect --keep-credentials removed the forwarder, so there is no incomplete connect left to report")
+ }
+ status := Status(true, StatusOptions{SkipCredentialCheck: true})
+ if status.Enabled || status.ConnectIncomplete || !strings.Contains(status.Message, "credentials for device dev-1 kept") {
+ t.Fatalf("status after disconnect --keep-credentials = %+v", status)
+ }
+}
+
+func TestFirstConnectFailingAfterApprovalIsReportedIncomplete(t *testing.T) {
+ t.Setenv("HOME", t.TempDir())
+ isolateVectorDiscovery(t)
+ fd := newFakeDashboard(t)
+ fwd := &fakeForwarder{supported: true, loadErr: errors.New("bootstrap failed")}
+ _, err := Connect(context.Background(), connectOptions(t, fd, fwd, fakeVector(t, "0.56.0", 0)))
+ if err == nil || !strings.Contains(err.Error(), "incomplete") || !strings.Contains(err.Error(), "beacon endpoint connect") {
+ t.Fatalf("a first connect that fails after approval must say so and name the retry: %v", err)
+ }
+ if Connected(true) || !ConnectIncomplete(true) {
+ t.Fatalf("connected=%t incomplete=%t", Connected(true), ConnectIncomplete(true))
+ }
+ status := Status(true, StatusOptions{SkipCredentialCheck: true})
+ if status.Enabled || !status.ConnectIncomplete || !strings.Contains(status.Message, "beacon endpoint connect") {
+ t.Fatalf("status after a first connect that failed after approval = %+v", status)
+ }
+ var encoded map[string]any
+ if err := json.Unmarshal(mustJSON(t, status), &encoded); err != nil {
+ t.Fatal(err)
+ }
+ if encoded["connect_incomplete"] != true {
+ t.Fatalf("status JSON must expose connect_incomplete: %v", encoded)
+ }
+}
+
+func TestReconnectAwaitingApprovalStillReportsTheWorkingConnection(t *testing.T) {
+ fd, fwd := connectedEndpoint(t, fakeVector(t, "0.56.0", 0))
+ // Until the server issues a key nothing has been rotated. A connect interrupted here
+ // (Ctrl-C ends the process without unwinding) must leave no trace that flags the working
+ // connection, so the state seen while the browser approval is pending is the final word.
+ opts := connectOptions(t, fd, fwd, fakeVector(t, "0.56.0", 0))
+ approve := opts.Enroll.OpenBrowser
+ var connectedWhileWaiting, incompleteWhileWaiting bool
+ opts.Enroll.OpenBrowser = func(url string) error {
+ connectedWhileWaiting, incompleteWhileWaiting = Connected(true), ConnectIncomplete(true)
+ return approve(url)
+ }
+ if _, err := Connect(context.Background(), opts); err != nil {
+ t.Fatal(err)
+ }
+ if !connectedWhileWaiting || incompleteWhileWaiting {
+ t.Fatalf("before a key is issued the endpoint must still read as connected: connected=%t incomplete=%t", connectedWhileWaiting, incompleteWhileWaiting)
+ }
+}
diff --git a/cli/beacon/internal/endpoint/asymptote/status.go b/cli/beacon/internal/endpoint/asymptote/status.go
index 19025421..ce28d2bf 100644
--- a/cli/beacon/internal/endpoint/asymptote/status.go
+++ b/cli/beacon/internal/endpoint/asymptote/status.go
@@ -37,6 +37,10 @@ type ManagedIngestStatus struct {
CredentialMessage string `json:"credential_message,omitempty"`
BufferBytes int64 `json:"buffer_bytes"`
Message string `json:"message,omitempty"`
+ // ConnectIncomplete is true when a connect received a key from the server and then
+ // failed before finishing. Enabled is false then: on a re-connect the server has rotated
+ // the key the running forwarder holds, so it cannot upload until connect is run again.
+ ConnectIncomplete bool `json:"connect_incomplete,omitempty"`
}
// StatusOptions tunes Status; zero values are the production defaults.
@@ -49,14 +53,34 @@ type StatusOptions struct {
// Status describes the managed-ingest state of this endpoint. It never reads the device key
// into the caller's structures: the key is used for one HEAD-equivalent GET and discarded.
func Status(userMode bool, opts StatusOptions) ManagedIngestStatus {
+ incomplete := ConnectIncomplete(userMode)
enrollment, err := LoadEnrollment(userMode)
if err != nil {
- status := ManagedIngestStatus{Enabled: false}
- if !errors.Is(err, ErrNotEnrolled) {
+ status := ManagedIngestStatus{Enabled: false, ConnectIncomplete: incomplete}
+ switch {
+ case !errors.Is(err, ErrNotEnrolled):
status.Message = err.Error()
+ case incomplete:
+ status.Message = "connect incomplete: this device was approved but connect did not finish; run `beacon endpoint connect` again"
}
return status
}
+ if incomplete {
+ privacyMode, _ := managedprivacy.Normalize(enrollment.PrivacyMode)
+ // The record and config are the previous connection's, and so is the key prefix,
+ // which the server has rotated out; the key file may already hold the new one, so
+ // a credential check would say valid while the running forwarder cannot upload.
+ return ManagedIngestStatus{
+ Enabled: false,
+ ConnectIncomplete: true,
+ DeviceID: enrollment.DeviceID,
+ OrganizationID: enrollment.OrganizationID,
+ OrganizationName: enrollment.OrganizationName,
+ PrivacyMode: privacyMode,
+ Forwarder: ForwarderStatus(userMode),
+ Message: fmt.Sprintf("re-connect incomplete: the server rotated device %s's key but connect did not finish, so the forwarder cannot upload; run `beacon endpoint connect` again", enrollment.DeviceID),
+ }
+ }
if !Connected(userMode) {
privacyMode, _ := managedprivacy.Normalize(enrollment.PrivacyMode)
// disconnect --keep-credentials leaves the record and key for a later connect.
diff --git a/docs/cli/endpoint-connect.mdx b/docs/cli/endpoint-connect.mdx
index 3f92d64f..9d81c02f 100644
--- a/docs/cli/endpoint-connect.mdx
+++ b/docs/cli/endpoint-connect.mdx
@@ -35,10 +35,12 @@ beacon endpoint disconnect [flags]
5. Renders `asymptote/vector.toml` with the selected privacy mode, runs `vector validate` on the file the service will run, installs a service unit, and starts it.
6. Records the non-secret enrollment, including privacy mode, in `asymptote/enrollment.json` and `config.json`'s `managed_ingest` block.
-A connect that fails at `vector validate` or at loading the service saves no enrollment record, so the endpoint is not connected and `beacon endpoint disconnect` cleans up what was written. After `disconnect --keep-credentials`, status reports not connected and names the kept device id.
+A first connect that fails at `vector validate` or at loading the service saves no enrollment record, so the endpoint is not connected and `beacon endpoint disconnect` cleans up what was written. After `disconnect --keep-credentials`, status reports not connected and names the kept device id.
Re-running `connect` on a connected machine reuses its install id, so the server rotates the existing device's key in place instead of creating a second device. Because that rotation invalidates the key the running forwarder holds, the new config is validated before enrollment: a re-run whose config Vector rejects leaves the forwarder, its key, and its buffer untouched. Under launchd the new forwarder is loaded only after the old one has exited (Vector drains in-flight requests first), and if the new instance has not started within the wait, `connect` fails and points at `launchctl print` for the job (v1.3.2).
+Some steps can only fail after the server has issued the key: storing it, validating the config that carries the server's ingest URL, writing the service unit, starting the service, and recording the enrollment. `connect` writes `asymptote/connect-pending` as soon as the key arrives and removes it only when every step has succeeded. If a step in between fails, `connect` says the connect is incomplete and names the retry, and `beacon endpoint status` reports Beacon Managed as not connected with `connect_incomplete: true` in `--json`, because on a re-connect the forwarder still running holds the key the server just rotated out. The previous enrollment record and config are not treated as a working connection. Run `beacon endpoint connect` again once the cause is fixed. An approval that is cancelled or expires before a key is issued rotates nothing and leaves an existing connection as it was.
+
## Privacy modes
The onboarding wizard records one of two modes. `connect` uses that choice unless
@@ -61,6 +63,7 @@ are not uploaded under the new one.
|------|-----------|---------------------|---------------------|
| Enrollment record | `~/.beacon/endpoint/asymptote/enrollment.json` (0600) | `/Library/Application Support/Beacon/Endpoint/asymptote/enrollment.json` | `/etc/beacon/endpoint/asymptote/enrollment.json` |
| Install id | `~/.beacon/endpoint/asymptote/install-id` (0600) | `…/Endpoint/asymptote/install-id` | `/etc/beacon/endpoint/asymptote/install-id` |
+| Pending-connect marker (only while a connect is unfinished) | `~/.beacon/endpoint/asymptote/connect-pending` (0600, timestamp only) | `…/Endpoint/asymptote/connect-pending` | `/etc/beacon/endpoint/asymptote/connect-pending` |
| Device key | `~/.beacon/endpoint/asymptote/vector-secrets.json` (0600) | `…/Endpoint/asymptote/vector-secrets.json` | `/etc/beacon/endpoint/asymptote/vector-secrets.json` |
| Vector config | `~/.beacon/endpoint/asymptote/vector.toml` (0644, no secret) | `…/Endpoint/asymptote/vector.toml` | `/etc/beacon/endpoint/asymptote/vector.toml` |
| Buffer and checkpoints | `~/.beacon/endpoint/asymptote/vector-data/` | `…/Endpoint/asymptote/vector-data/` | `/etc/beacon/endpoint/asymptote/vector-data/` |
diff --git a/docs/cli/endpoint-status.mdx b/docs/cli/endpoint-status.mdx
index 732b00b5..a610b342 100644
--- a/docs/cli/endpoint-status.mdx
+++ b/docs/cli/endpoint-status.mdx
@@ -61,7 +61,7 @@ Status includes:
- Endpoint diagnostics
- Destination configuration in JSON output, including Splunk HEC or Falcon LogScale HEC endpoint, index, source, and sourcetype when configured
- Last Beacon event in the runtime log, when present
-- Beacon Managed: whether this endpoint is connected, organization and device id, active privacy mode, forwarder service state, whether the device key is still accepted (`valid`, `revoked`, or `unknown` when offline), and disk-buffer size. In `--json` this is the `managed_ingest` object; account and device keys are never included. The line reports the connection for the install mode you asked about (`--user` or `--system`), even when the collector view switches to the system install because it holds the OTLP ports.
+- Beacon Managed: whether this endpoint is connected (a connect that failed after the server issued the device key is reported as not connected, with `connect_incomplete: true` and the retry command), organization and device id, active privacy mode, forwarder service state, whether the device key is still accepted (`valid`, `revoked`, or `unknown` when offline), and disk-buffer size. In `--json` this is the `managed_ingest` object; account and device keys are never included. The line reports the connection for the install mode you asked about (`--user` or `--system`), even when the collector view switches to the system install because it holds the OTLP ports.
- Inventory heartbeat: whether the scheduled inventory job is enabled and loaded, its interval, and when it last wrote. In `--json` this is the `inventory_heartbeat` object (`enabled`, `scheduled`, `interval`, `job`, `last_emitted_at`, `log_path`), reported for the requested install mode like managed ingest.
When Splunk HEC is configured, `--json` includes a `destinations.splunk_hec` object with `configured: true` and the non-secret destination settings. When Falcon LogScale HEC is configured, `--json` includes `destinations.falcon_hec` with the same non-secret fields. HEC tokens are not printed. File-based destinations such as Wazuh, Elastic, Datadog, Sumo Logic, Rapid7, Microsoft Sentinel, and AWS S3 validate against the configured runtime log path and their generated shipper configuration.