The Open Framework for Building Agents
test(eve): reproduce automation bypass secret in production sign-in callback URLs (#3978)
roboeve[bot]70f75eda233c
diff --git a/.changeset/connection-callback-no-bypass-secret.md b/.changeset/connection-callback-no-bypass-secret.mdnew file mode 100644index 000000000..254ac6b5a--- /dev/null+++ b/.changeset/connection-callback-no-bypass-secret.md@@ -0,0 +1,5 @@+---+"eve": patch+---++Connection sign-in callback URLs no longer include `VERCEL_AUTOMATION_BYPASS_SECRET`, so the secret stays out of the links shown to users and the `redirect_uri` sent to providers. On a deployment behind Deployment Protection, the person signing in now passes protection on their own, for example through Vercel Authentication or a Deployment Protection Exception.diff --git a/docs/connections/overview.mdx b/docs/connections/overview.mdxindex fbce413e8..0e54392fd 100644--- a/docs/connections/overview.mdx+++ b/docs/connections/overview.mdx@@ -254,12 +254,13 @@ App-scoped Connect auth is non-interactive. eve asks Vercel Connect for an app t### Troubleshooting Vercel Connect auth-| Symptom | What it means | Fix |-| -------------------------------------------- | --------------------------------------------------------------------------------------------------------------------------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------- |-| `reason: "principal_required"` | A user-scoped connection ran without an authenticated user on the active session. | Return `principalType: "user"` from the channel's route auth, or use `connect({ connector, principalType: "app" })` if the Connect credential should be shared. |-| `authorization.required` appears but no UI | eve parked the turn for OAuth, but the channel or frontend is not rendering the challenge. | Render the challenge from the stream event and continue the same session after the callback. |-| OAuth works locally but fails after deploy | The project may not be linked to the Connect client, or the deployed runtime may not have the expected Vercel OIDC/project scope. | Run Connect setup from the consuming project directory, link the project, deploy again, and verify the connector UID in `connect("...")`. |-| A scheduled or internal run needs user OAuth | Schedules and runtime callers do not automatically carry an end-user principal. | Dispatch through a user-authenticated channel when work is user-owned, or use app-scoped auth for agent-owned background work. |+| Symptom | What it means | Fix |+| ------------------------------------------------- | --------------------------------------------------------------------------------------------------------------------------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------- |+| `reason: "principal_required"` | A user-scoped connection ran without an authenticated user on the active session. | Return `principalType: "user"` from the channel's route auth, or use `connect({ connector, principalType: "app" })` if the Connect credential should be shared. |+| `authorization.required` appears but no UI | eve parked the turn for OAuth, but the channel or frontend is not rendering the challenge. | Render the challenge from the stream event and continue the same session after the callback. |+| OAuth works locally but fails after deploy | The project may not be linked to the Connect client, or the deployed runtime may not have the expected Vercel OIDC/project scope. | Run Connect setup from the consuming project directory, link the project, deploy again, and verify the connector UID in `connect("...")`. |+| A scheduled or internal run needs user OAuth | Schedules and runtime callers do not automatically carry an end-user principal. | Dispatch through a user-authenticated channel when work is user-owned, or use app-scoped auth for agent-owned background work. |+| Sign-in ends on a Vercel login or protection page | The deployment is behind Deployment Protection, and sign-in callback URLs never carry the automation bypass secret. | Have the user sign in to Vercel with access to the project, or add a Deployment Protection Exception for the domain. |## Self-hosted interactive OAuth@@ -316,6 +317,8 @@ export default defineMcpClientConnection({Drop `resume` when the provider keeps flow state server-side, so nothing has to cross the step boundary.+The callback URL never carries `VERCEL_AUTOMATION_BYPASS_SECRET`, because the user's browser and the provider both see it. On a deployment behind [Deployment Protection](https://vercel.com/docs/deployment-protection), the person signing in must pass protection on their own, for example through Vercel Authentication or a [Deployment Protection Exception](https://vercel.com/docs/deployment-protection/methods-to-bypass-deployment-protection/deployment-protection-exceptions).+`displayName` is presentation-only. The connection's resolved name still keysthe callback URL, while eve scopes token caching and authorization completionto the opaque resolved instance identity. You can also set `displayName` on thediff --git a/e2e/fixtures/agent-subagents/evals/remote-direct-authorization.eval.ts b/e2e/fixtures/agent-subagents/evals/remote-direct-authorization.eval.tsindex dff2614ea..3a9bcc869 100644--- a/e2e/fixtures/agent-subagents/evals/remote-direct-authorization.eval.ts+++ b/e2e/fixtures/agent-subagents/evals/remote-direct-authorization.eval.ts@@ -31,7 +31,7 @@ export default defineEval({});const callback = new URL(required.data.webhookUrl);callback.searchParams.set("code", "direct-release-code");- const response = await fetch(callback);+ const response = await t.target.fetch(`${callback.pathname}${callback.search}`);if (!response.ok) throw new Error(`Direct authorization callback returned ${response.status}.`);const turn = await resumed.result();turn.expectOk();diff --git a/e2e/fixtures/agent-subagents/evals/remote-nested-authorization.eval.ts b/e2e/fixtures/agent-subagents/evals/remote-nested-authorization.eval.tsindex 459a9bc02..af0beea1b 100644--- a/e2e/fixtures/agent-subagents/evals/remote-nested-authorization.eval.ts+++ b/e2e/fixtures/agent-subagents/evals/remote-nested-authorization.eval.ts@@ -28,7 +28,7 @@ export default defineEval({});const callback = new URL(required.data.webhookUrl);callback.searchParams.set("code", AUTHORIZATION_CODE);- const response = await fetch(callback);+ const response = await t.target.fetch(`${callback.pathname}${callback.search}`);if (!response.ok) throw new Error(`Nested authorization callback returned ${response.status}.`);const turn = await resumed.result();diff --git a/e2e/fixtures/agent-tools-hitl/evals/auth/authorization-principal-after-other-speaker.eval.ts b/e2e/fixtures/agent-tools-hitl/evals/auth/authorization-principal-after-other-speaker.eval.tsindex a29dd0d5f..8fc962e20 100644--- a/e2e/fixtures/agent-tools-hitl/evals/auth/authorization-principal-after-other-speaker.eval.ts+++ b/e2e/fixtures/agent-tools-hitl/evals/auth/authorization-principal-after-other-speaker.eval.ts@@ -34,7 +34,8 @@ export default defineEval({"Do not call any tools. Reply with exactly BOB-STATUS-OK.",as(BOB),);- const callback = await fetch(new URL(required.data.authorization.url));+ const callbackUrl = new URL(required.data.authorization.url);+ const callback = await t.target.fetch(`${callbackUrl.pathname}${callbackUrl.search}`);if (!callback.ok) throw new Error(`Fixture sign-in callback failed (${callback.status}).`);const completed = await resumed.result();diff --git a/e2e/fixtures/agent-tools-hitl/evals/hitl/authorizations/approved-call-needs-authorization.eval.ts b/e2e/fixtures/agent-tools-hitl/evals/hitl/authorizations/approved-call-needs-authorization.eval.tsindex 8b001060f..83eab1db8 100644--- a/e2e/fixtures/agent-tools-hitl/evals/hitl/authorizations/approved-call-needs-authorization.eval.ts+++ b/e2e/fixtures/agent-tools-hitl/evals/hitl/authorizations/approved-call-needs-authorization.eval.ts@@ -36,7 +36,7 @@ export default defineEval({const authorization = authorizationFrom(signing);const startIndex = session.state.streamIndex;- await completeAuthorization(authorization.url);+ await completeAuthorization(t, authorization.url);const resumed = (await follow(t, session, startIndex)).expectOk();resumed.event("authorization.completed", {count: 1,diff --git a/e2e/fixtures/agent-tools-hitl/evals/hitl/authorizations/authorization-holds-then-resumes-as-requester.eval.ts b/e2e/fixtures/agent-tools-hitl/evals/hitl/authorizations/authorization-holds-then-resumes-as-requester.eval.tsindex efba7b644..3ce7496d2 100644--- a/e2e/fixtures/agent-tools-hitl/evals/hitl/authorizations/authorization-holds-then-resumes-as-requester.eval.ts+++ b/e2e/fixtures/agent-tools-hitl/evals/hitl/authorizations/authorization-holds-then-resumes-as-requester.eval.ts@@ -31,7 +31,7 @@ export default defineEval({const authorization = authorizationFrom(held);const startIndex = session.state.streamIndex;- await completeAuthorization(authorization.url);+ await completeAuthorization(t, authorization.url);const resumed = (await follow(t, session, startIndex)).expectOk();resumed.notEvent("turn.started");resumed.event("authorization.completed", {diff --git a/e2e/fixtures/agent-tools-hitl/evals/hitl/authorizations/newer-attempt-supersedes.eval.ts b/e2e/fixtures/agent-tools-hitl/evals/hitl/authorizations/newer-attempt-supersedes.eval.tsindex 495c21382..067549be4 100644--- a/e2e/fixtures/agent-tools-hitl/evals/hitl/authorizations/newer-attempt-supersedes.eval.ts+++ b/e2e/fixtures/agent-tools-hitl/evals/hitl/authorizations/newer-attempt-supersedes.eval.ts@@ -41,7 +41,7 @@ export default defineEval({throw new Error("Expected a new authorization attempt.");const startIndex = session.state.streamIndex;- await completeAuthorization(newer.url);+ await completeAuthorization(t, newer.url);const resumed = (await follow(t, session, startIndex)).expectOk();resumed.event("authorization.completed", {count: 1,diff --git a/e2e/fixtures/agent-tools-hitl/evals/hitl/authorized-response-oauth-cancel.eval.ts b/e2e/fixtures/agent-tools-hitl/evals/hitl/authorized-response-oauth-cancel.eval.tsindex ca2905bf2..497aa921b 100644--- a/e2e/fixtures/agent-tools-hitl/evals/hitl/authorized-response-oauth-cancel.eval.ts+++ b/e2e/fixtures/agent-tools-hitl/evals/hitl/authorized-response-oauth-cancel.eval.ts@@ -40,7 +40,7 @@ export default defineEval({throw new Error("Expected candidate OAuth URL.");}const callbackUrl = new URL(required.data.authorization.url);- const callback = await fetch(callbackUrl);+ const callback = await t.target.fetch(`${callbackUrl.pathname}${callbackUrl.search}`);if (!callback.ok)throw new Error(`Late fixture OAuth callback failed (${String(callback.status)}).`);diff --git a/e2e/fixtures/agent-tools-hitl/evals/hitl/authorized-response-oauth-nonblocking.eval.ts b/e2e/fixtures/agent-tools-hitl/evals/hitl/authorized-response-oauth-nonblocking.eval.tsindex 91f362d76..cd36074ed 100644--- a/e2e/fixtures/agent-tools-hitl/evals/hitl/authorized-response-oauth-nonblocking.eval.ts+++ b/e2e/fixtures/agent-tools-hitl/evals/hitl/authorized-response-oauth-nonblocking.eval.ts@@ -49,7 +49,7 @@ export default defineEval({}const callbackUrl = new URL(required.data.authorization.url);const callbackTurn = t.target.watchTurn(held.sessionId, { startIndex: resumeIndex });- const callback = await fetch(callbackUrl);+ const callback = await t.target.fetch(`${callbackUrl.pathname}${callbackUrl.search}`);if (!callback.ok)throw new Error(`Fixture OAuth callback failed (${String(callback.status)}).`);const resumed = await callbackTurn.result();diff --git a/e2e/fixtures/agent-tools-hitl/evals/hitl/authorized-response-oauth.eval.ts b/e2e/fixtures/agent-tools-hitl/evals/hitl/authorized-response-oauth.eval.tsindex e08849251..587cf875a 100644--- a/e2e/fixtures/agent-tools-hitl/evals/hitl/authorized-response-oauth.eval.ts+++ b/e2e/fixtures/agent-tools-hitl/evals/hitl/authorized-response-oauth.eval.ts@@ -39,7 +39,7 @@ export default defineEval({const resumedTurn = t.target.watchTurn(held.sessionId, {startIndex: held.session.state.streamIndex,});- const callback = await fetch(callbackUrl);+ const callback = await t.target.fetch(`${callbackUrl.pathname}${callbackUrl.search}`);if (!callback.ok) {throw new Error(`Fixture OAuth callback failed (${String(callback.status)}): ${await callback.text()}`,diff --git a/e2e/fixtures/agent-tools-hitl/evals/hitl/helpers.ts b/e2e/fixtures/agent-tools-hitl/evals/hitl/helpers.tsindex 7291abc30..5d1d42be6 100644--- a/e2e/fixtures/agent-tools-hitl/evals/hitl/helpers.ts+++ b/e2e/fixtures/agent-tools-hitl/evals/hitl/helpers.ts@@ -120,9 +120,9 @@ export function authorizationFrom(turn: Pick<EveEvalTurn, "events">): {return { attemptId: required.data.attemptId, url: new URL(required.data.authorization.url) };}-/** Completes a fixture authorization the way a browser would: an unauthenticated GET. */-export async function completeAuthorization(url: URL) {- const callback = await fetch(url);+/** Completes a fixture authorization the way a browser would: a GET on the callback URL. */+export async function completeAuthorization(t: EveEvalContext, url: URL) {+ const callback = await t.target.fetch(`${url.pathname}${url.search}`);if (!callback.ok) throw new Error(`Fixture authorization callback failed (${callback.status}).`);}diff --git a/e2e/fixtures/agent-tools-hitl/evals/hitl/response-policies/responder-authorization-then-settles.eval.ts b/e2e/fixtures/agent-tools-hitl/evals/hitl/response-policies/responder-authorization-then-settles.eval.tsindex 597b6a9f7..612bd7a5c 100644--- a/e2e/fixtures/agent-tools-hitl/evals/hitl/response-policies/responder-authorization-then-settles.eval.ts+++ b/e2e/fixtures/agent-tools-hitl/evals/hitl/response-policies/responder-authorization-then-settles.eval.ts@@ -45,7 +45,7 @@ export default defineEval({const authorization = authorizationFrom(held);const startIndex = session.state.streamIndex;- await completeAuthorization(authorization.url);+ await completeAuthorization(t, authorization.url);const resumed = (await follow(t, session, startIndex)).expectOk();resumed.notEvent("turn.started");resumed.eventOrder([diff --git a/e2e/fixtures/agent-workflow-tools/evals/agent-probe.shared.ts b/e2e/fixtures/agent-workflow-tools/evals/agent-probe.shared.tsindex f6790ad20..0dbb72cbe 100644--- a/e2e/fixtures/agent-workflow-tools/evals/agent-probe.shared.ts+++ b/e2e/fixtures/agent-workflow-tools/evals/agent-probe.shared.ts@@ -101,7 +101,7 @@ async function completeSignIn(}const callbackUrl = toCallbackUrl(required.data.authorization?.url);const resumed = watchNext(t, live.session);- const response = await fetch(callbackUrl);+ const response = await t.target.fetch(`${callbackUrl.pathname}${callbackUrl.search}`);if (!response.ok) {throw new Error(`Authorization callback failed (${response.status}).`);}@@ -152,7 +152,7 @@ export async function runRejectedStepAuth(t: EveEvalContext): Promise<void> {);requireAuthorizationOutcome(turn, "failed");- const repeatedCallback = await fetch(callbackUrl);+ const repeatedCallback = await t.target.fetch(`${callbackUrl.pathname}${callbackUrl.search}`);if (repeatedCallback.status !== 404) {throw new Error("The completed authorization callback must be disposed.");}diff --git a/e2e/fixtures/agent-workflow-tools/evals/private-catalog.sign-in.eval.ts b/e2e/fixtures/agent-workflow-tools/evals/private-catalog.sign-in.eval.tsindex 107ae0962..9d7c412fe 100644--- a/e2e/fixtures/agent-workflow-tools/evals/private-catalog.sign-in.eval.ts+++ b/e2e/fixtures/agent-workflow-tools/evals/private-catalog.sign-in.eval.ts@@ -53,9 +53,8 @@ export default defineEval({const resumed = t.target.watchTurn(session.sessionId, {startIndex: session.state.streamIndex,});- const response = await fetch(- fixtureAuthorizationCallback(t.target.url, required.data.authorization?.url),- );+ const callback = fixtureAuthorizationCallback(t.target.url, required.data.authorization?.url);+ const response = await t.target.fetch(`${callback.pathname}${callback.search}`);if (!response.ok) throw new Error(`Authorization callback failed (${response.status}).`);const completed = await resumed.result();diff --git a/e2e/fixtures/agent-workflow-tools/evals/public-catalog.lazy-authorization.eval.ts b/e2e/fixtures/agent-workflow-tools/evals/public-catalog.lazy-authorization.eval.tsindex 7f938cec9..123d9515d 100644--- a/e2e/fixtures/agent-workflow-tools/evals/public-catalog.lazy-authorization.eval.ts+++ b/e2e/fixtures/agent-workflow-tools/evals/public-catalog.lazy-authorization.eval.ts@@ -68,7 +68,7 @@ export default defineEval({const resumed = t.target.watchTurn(session.sessionId, {startIndex: session.state.streamIndex,});- const response = await fetch(callback);+ const response = await t.target.fetch(`${callback.pathname}${callback.search}`);if (!response.ok) {throw new Error(`Authorization callback failed (${response.status}).`);}diff --git a/packages/eve/src/execution/workflow-callback-url.ts b/packages/eve/src/execution/workflow-callback-url.tsindex 0a061b4cb..38ee096ef 100644--- a/packages/eve/src/execution/workflow-callback-url.ts+++ b/packages/eve/src/execution/workflow-callback-url.ts@@ -50,17 +50,30 @@ export function resolveWorkflowCallbackBaseUrl(metadataUrl: string): string {}/**- * Builds a framework-owned callback URL from a resolved callback base URL.+ * Builds a framework-owned callback URL that a browser or third party sees,+ * such as the connection sign-in redirect handed to an IdP.** `callbackPath` is appended to the full base URL rather than resolved* against it: the base may carry a public route prefix path, which* `new URL(path, base)` would drop for absolute paths.*/-export function createWorkflowCallbackUrl(baseUrl: string, callbackPath: string): string {+export function createBrowserCallbackUrl(baseUrl: string, callbackPath: string): string {const callback = new URL(callbackPath, "http://eve.local");const url = new URL(baseUrl);url.pathname = joinEveRoutePath(url.pathname, callback.pathname);url.search = callback.search;+ return url.toString();+}++/**+ * Builds a framework-owned callback URL for server-to-server delivery.+ *+ * Unlike {@link createBrowserCallbackUrl}, it carries the deployment's+ * Protection Bypass for Automation secret so the caller can reach a+ * protected deployment.+ */+export function createWorkflowCallbackUrl(baseUrl: string, callbackPath: string): string {+ const url = new URL(createBrowserCallbackUrl(baseUrl, callbackPath));// https://vercel.com/docs/deployment-protection/methods-to-bypass-deployment-protection/protection-bypass-automationconst bypassSecret = process.env.VERCEL_AUTOMATION_BYPASS_SECRET?.trim();diff --git a/packages/eve/src/harness/authorization.test.ts b/packages/eve/src/harness/authorization.test.tsindex 3653c3933..d3bcbddef 100644--- a/packages/eve/src/harness/authorization.test.ts+++ b/packages/eve/src/harness/authorization.test.ts@@ -2,6 +2,7 @@ import { afterEach, describe, expect, it, vi } from "vitest";import { ContextContainer, contextStorage } from "#context/container.js";import { SessionIdKey } from "#context/keys.js";+import { resolveWorkflowCallbackBaseUrl } from "#execution/workflow-callback-url.js";import {CallbackBaseUrlKey,consumeAuthorizationResult,@@ -22,14 +23,27 @@ afterEach(() => {});describe("authorization callback URLs", () => {- it("includes the Vercel automation bypass query when configured", () => {+ it("keeps the automation bypass secret out of sign-in callback URLs", () => {vi.stubEnv("VERCEL_AUTOMATION_BYPASS_SECRET", "secret value");const ctx = new ContextContainer();ctx.set(CallbackBaseUrlKey, "https://agent.example.com");ctx.set(SessionIdKey, "session-1");expect(contextStorage.run(ctx, () => getHookUrl("linear", "attempt-1"))).toBe(- "https://agent.example.com/eve/v1/connections/linear/callback/attempt-1/eve%3Ainbox%3Av1%3Aeve%3Asession%3Asession-1%3Ainbox?x-vercel-protection-bypass=secret+value",+ "https://agent.example.com/eve/v1/connections/linear/callback/attempt-1/eve%3Ainbox%3Av1%3Aeve%3Asession%3Asession-1%3Ainbox",+ );+ });++ it("keeps the automation bypass secret out of production sign-in callback URLs", () => {+ vi.stubEnv("VERCEL_AUTOMATION_BYPASS_SECRET", "secret value");+ vi.stubEnv("VERCEL_ENV", "production");+ vi.stubEnv("VERCEL_PROJECT_PRODUCTION_URL", "agent.example.com");+ const ctx = new ContextContainer();+ ctx.set(CallbackBaseUrlKey, resolveWorkflowCallbackBaseUrl("https://agent-abc123.vercel.app"));+ ctx.set(SessionIdKey, "session-1");++ expect(contextStorage.run(ctx, () => getHookUrl("linear", "attempt-1"))).toBe(+ "https://agent.example.com/eve/v1/connections/linear/callback/attempt-1/eve%3Ainbox%3Av1%3Aeve%3Asession%3Asession-1%3Ainbox",);});});diff --git a/packages/eve/src/harness/authorization.ts b/packages/eve/src/harness/authorization.tsindex 552314665..e4a078071 100644--- a/packages/eve/src/harness/authorization.ts+++ b/packages/eve/src/harness/authorization.ts@@ -38,7 +38,7 @@ import {import { loadContext } from "#context/container.js";import { ContextKey } from "#context/key.js";import { SessionIdKey, type SessionAuthContext } from "#context/keys.js";-import { createWorkflowCallbackUrl } from "#execution/workflow-callback-url.js";+import { createBrowserCallbackUrl } from "#execution/workflow-callback-url.js";import type { ConnectionAuthorizationChallenge } from "#connections/errors.js";import type { AuthorizationCallback, ConnectionPrincipal } from "#shared/connection-types.js";import type { JsonValue } from "#shared/json.js";@@ -190,6 +190,9 @@ export function consumeAuthorizationResult(* It is independent of the continuation token, so channel aliasing mid-turn* does not invalidate the callback URL.*+ * The URL reaches the user's browser and the IdP, so it never carries the+ * deployment's protection bypass secret.+ ** Returns `undefined` if no callback address is available.*/export function getHookUrl(name: string, attemptId: string): string | undefined {@@ -198,7 +201,7 @@ export function getHookUrl(name: string, attemptId: string): string | undefinedconst baseUrl = ctx.get(CallbackBaseUrlKey);const token = ctx.get(AuthorizationHookKey) ?? (sessionId ? authHookToken(sessionId) : undefined);if (!token || !baseUrl) return undefined;- return createWorkflowCallbackUrl(+ return createBrowserCallbackUrl(baseUrl,createEveConnectionCallbackRoutePath(name, attemptId, token),);