Skip to content

armgit / eve

Public Imported from GitHub
0

The Open Framework for Building Agents

test(eve): reproduce automation bypass secret in production sign-in callback URLs (#3978)

roboeve[bot]70f75eda233c
diff --git a/.changeset/connection-callback-no-bypass-secret.md b/.changeset/connection-callback-no-bypass-secret.md
new file mode 100644
index 000000000..254ac6b5a
--- /dev/null
+++ b/.changeset/connection-callback-no-bypass-secret.md
@@ -0,0 +1,5 @@
+---
+"eve": patch
+---
+
+Connection sign-in callback URLs no longer include `VERCEL_AUTOMATION_BYPASS_SECRET`, so the secret stays out of the links shown to users and the `redirect_uri` sent to providers. On a deployment behind Deployment Protection, the person signing in now passes protection on their own, for example through Vercel Authentication or a Deployment Protection Exception.
diff --git a/docs/connections/overview.mdx b/docs/connections/overview.mdx
index fbce413e8..0e54392fd 100644
--- a/docs/connections/overview.mdx
+++ b/docs/connections/overview.mdx
@@ -254,12 +254,13 @@ App-scoped Connect auth is non-interactive. eve asks Vercel Connect for an app t
### Troubleshooting Vercel Connect auth
-| Symptom | What it means | Fix |
-| -------------------------------------------- | --------------------------------------------------------------------------------------------------------------------------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------- |
-| `reason: "principal_required"` | A user-scoped connection ran without an authenticated user on the active session. | Return `principalType: "user"` from the channel's route auth, or use `connect({ connector, principalType: "app" })` if the Connect credential should be shared. |
-| `authorization.required` appears but no UI | eve parked the turn for OAuth, but the channel or frontend is not rendering the challenge. | Render the challenge from the stream event and continue the same session after the callback. |
-| OAuth works locally but fails after deploy | The project may not be linked to the Connect client, or the deployed runtime may not have the expected Vercel OIDC/project scope. | Run Connect setup from the consuming project directory, link the project, deploy again, and verify the connector UID in `connect("...")`. |
-| A scheduled or internal run needs user OAuth | Schedules and runtime callers do not automatically carry an end-user principal. | Dispatch through a user-authenticated channel when work is user-owned, or use app-scoped auth for agent-owned background work. |
+| Symptom | What it means | Fix |
+| ------------------------------------------------- | --------------------------------------------------------------------------------------------------------------------------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------- |
+| `reason: "principal_required"` | A user-scoped connection ran without an authenticated user on the active session. | Return `principalType: "user"` from the channel's route auth, or use `connect({ connector, principalType: "app" })` if the Connect credential should be shared. |
+| `authorization.required` appears but no UI | eve parked the turn for OAuth, but the channel or frontend is not rendering the challenge. | Render the challenge from the stream event and continue the same session after the callback. |
+| OAuth works locally but fails after deploy | The project may not be linked to the Connect client, or the deployed runtime may not have the expected Vercel OIDC/project scope. | Run Connect setup from the consuming project directory, link the project, deploy again, and verify the connector UID in `connect("...")`. |
+| A scheduled or internal run needs user OAuth | Schedules and runtime callers do not automatically carry an end-user principal. | Dispatch through a user-authenticated channel when work is user-owned, or use app-scoped auth for agent-owned background work. |
+| Sign-in ends on a Vercel login or protection page | The deployment is behind Deployment Protection, and sign-in callback URLs never carry the automation bypass secret. | Have the user sign in to Vercel with access to the project, or add a Deployment Protection Exception for the domain. |
## Self-hosted interactive OAuth
@@ -316,6 +317,8 @@ export default defineMcpClientConnection({
Drop `resume` when the provider keeps flow state server-side, so nothing has to cross the step boundary.
+The callback URL never carries `VERCEL_AUTOMATION_BYPASS_SECRET`, because the user's browser and the provider both see it. On a deployment behind [Deployment Protection](https://vercel.com/docs/deployment-protection), the person signing in must pass protection on their own, for example through Vercel Authentication or a [Deployment Protection Exception](https://vercel.com/docs/deployment-protection/methods-to-bypass-deployment-protection/deployment-protection-exceptions).
+
`displayName` is presentation-only. The connection's resolved name still keys
the callback URL, while eve scopes token caching and authorization completion
to the opaque resolved instance identity. You can also set `displayName` on the
diff --git a/e2e/fixtures/agent-subagents/evals/remote-direct-authorization.eval.ts b/e2e/fixtures/agent-subagents/evals/remote-direct-authorization.eval.ts
index dff2614ea..3a9bcc869 100644
--- a/e2e/fixtures/agent-subagents/evals/remote-direct-authorization.eval.ts
+++ b/e2e/fixtures/agent-subagents/evals/remote-direct-authorization.eval.ts
@@ -31,7 +31,7 @@ export default defineEval({
});
const callback = new URL(required.data.webhookUrl);
callback.searchParams.set("code", "direct-release-code");
- const response = await fetch(callback);
+ const response = await t.target.fetch(`${callback.pathname}${callback.search}`);
if (!response.ok) throw new Error(`Direct authorization callback returned ${response.status}.`);
const turn = await resumed.result();
turn.expectOk();
diff --git a/e2e/fixtures/agent-subagents/evals/remote-nested-authorization.eval.ts b/e2e/fixtures/agent-subagents/evals/remote-nested-authorization.eval.ts
index 459a9bc02..af0beea1b 100644
--- a/e2e/fixtures/agent-subagents/evals/remote-nested-authorization.eval.ts
+++ b/e2e/fixtures/agent-subagents/evals/remote-nested-authorization.eval.ts
@@ -28,7 +28,7 @@ export default defineEval({
});
const callback = new URL(required.data.webhookUrl);
callback.searchParams.set("code", AUTHORIZATION_CODE);
- const response = await fetch(callback);
+ const response = await t.target.fetch(`${callback.pathname}${callback.search}`);
if (!response.ok) throw new Error(`Nested authorization callback returned ${response.status}.`);
const turn = await resumed.result();
diff --git a/e2e/fixtures/agent-tools-hitl/evals/auth/authorization-principal-after-other-speaker.eval.ts b/e2e/fixtures/agent-tools-hitl/evals/auth/authorization-principal-after-other-speaker.eval.ts
index a29dd0d5f..8fc962e20 100644
--- a/e2e/fixtures/agent-tools-hitl/evals/auth/authorization-principal-after-other-speaker.eval.ts
+++ b/e2e/fixtures/agent-tools-hitl/evals/auth/authorization-principal-after-other-speaker.eval.ts
@@ -34,7 +34,8 @@ export default defineEval({
"Do not call any tools. Reply with exactly BOB-STATUS-OK.",
as(BOB),
);
- const callback = await fetch(new URL(required.data.authorization.url));
+ const callbackUrl = new URL(required.data.authorization.url);
+ const callback = await t.target.fetch(`${callbackUrl.pathname}${callbackUrl.search}`);
if (!callback.ok) throw new Error(`Fixture sign-in callback failed (${callback.status}).`);
const completed = await resumed.result();
diff --git a/e2e/fixtures/agent-tools-hitl/evals/hitl/authorizations/approved-call-needs-authorization.eval.ts b/e2e/fixtures/agent-tools-hitl/evals/hitl/authorizations/approved-call-needs-authorization.eval.ts
index 8b001060f..83eab1db8 100644
--- a/e2e/fixtures/agent-tools-hitl/evals/hitl/authorizations/approved-call-needs-authorization.eval.ts
+++ b/e2e/fixtures/agent-tools-hitl/evals/hitl/authorizations/approved-call-needs-authorization.eval.ts
@@ -36,7 +36,7 @@ export default defineEval({
const authorization = authorizationFrom(signing);
const startIndex = session.state.streamIndex;
- await completeAuthorization(authorization.url);
+ await completeAuthorization(t, authorization.url);
const resumed = (await follow(t, session, startIndex)).expectOk();
resumed.event("authorization.completed", {
count: 1,
diff --git a/e2e/fixtures/agent-tools-hitl/evals/hitl/authorizations/authorization-holds-then-resumes-as-requester.eval.ts b/e2e/fixtures/agent-tools-hitl/evals/hitl/authorizations/authorization-holds-then-resumes-as-requester.eval.ts
index efba7b644..3ce7496d2 100644
--- a/e2e/fixtures/agent-tools-hitl/evals/hitl/authorizations/authorization-holds-then-resumes-as-requester.eval.ts
+++ b/e2e/fixtures/agent-tools-hitl/evals/hitl/authorizations/authorization-holds-then-resumes-as-requester.eval.ts
@@ -31,7 +31,7 @@ export default defineEval({
const authorization = authorizationFrom(held);
const startIndex = session.state.streamIndex;
- await completeAuthorization(authorization.url);
+ await completeAuthorization(t, authorization.url);
const resumed = (await follow(t, session, startIndex)).expectOk();
resumed.notEvent("turn.started");
resumed.event("authorization.completed", {
diff --git a/e2e/fixtures/agent-tools-hitl/evals/hitl/authorizations/newer-attempt-supersedes.eval.ts b/e2e/fixtures/agent-tools-hitl/evals/hitl/authorizations/newer-attempt-supersedes.eval.ts
index 495c21382..067549be4 100644
--- a/e2e/fixtures/agent-tools-hitl/evals/hitl/authorizations/newer-attempt-supersedes.eval.ts
+++ b/e2e/fixtures/agent-tools-hitl/evals/hitl/authorizations/newer-attempt-supersedes.eval.ts
@@ -41,7 +41,7 @@ export default defineEval({
throw new Error("Expected a new authorization attempt.");
const startIndex = session.state.streamIndex;
- await completeAuthorization(newer.url);
+ await completeAuthorization(t, newer.url);
const resumed = (await follow(t, session, startIndex)).expectOk();
resumed.event("authorization.completed", {
count: 1,
diff --git a/e2e/fixtures/agent-tools-hitl/evals/hitl/authorized-response-oauth-cancel.eval.ts b/e2e/fixtures/agent-tools-hitl/evals/hitl/authorized-response-oauth-cancel.eval.ts
index ca2905bf2..497aa921b 100644
--- a/e2e/fixtures/agent-tools-hitl/evals/hitl/authorized-response-oauth-cancel.eval.ts
+++ b/e2e/fixtures/agent-tools-hitl/evals/hitl/authorized-response-oauth-cancel.eval.ts
@@ -40,7 +40,7 @@ export default defineEval({
throw new Error("Expected candidate OAuth URL.");
}
const callbackUrl = new URL(required.data.authorization.url);
- const callback = await fetch(callbackUrl);
+ const callback = await t.target.fetch(`${callbackUrl.pathname}${callbackUrl.search}`);
if (!callback.ok)
throw new Error(`Late fixture OAuth callback failed (${String(callback.status)}).`);
diff --git a/e2e/fixtures/agent-tools-hitl/evals/hitl/authorized-response-oauth-nonblocking.eval.ts b/e2e/fixtures/agent-tools-hitl/evals/hitl/authorized-response-oauth-nonblocking.eval.ts
index 91f362d76..cd36074ed 100644
--- a/e2e/fixtures/agent-tools-hitl/evals/hitl/authorized-response-oauth-nonblocking.eval.ts
+++ b/e2e/fixtures/agent-tools-hitl/evals/hitl/authorized-response-oauth-nonblocking.eval.ts
@@ -49,7 +49,7 @@ export default defineEval({
}
const callbackUrl = new URL(required.data.authorization.url);
const callbackTurn = t.target.watchTurn(held.sessionId, { startIndex: resumeIndex });
- const callback = await fetch(callbackUrl);
+ const callback = await t.target.fetch(`${callbackUrl.pathname}${callbackUrl.search}`);
if (!callback.ok)
throw new Error(`Fixture OAuth callback failed (${String(callback.status)}).`);
const resumed = await callbackTurn.result();
diff --git a/e2e/fixtures/agent-tools-hitl/evals/hitl/authorized-response-oauth.eval.ts b/e2e/fixtures/agent-tools-hitl/evals/hitl/authorized-response-oauth.eval.ts
index e08849251..587cf875a 100644
--- a/e2e/fixtures/agent-tools-hitl/evals/hitl/authorized-response-oauth.eval.ts
+++ b/e2e/fixtures/agent-tools-hitl/evals/hitl/authorized-response-oauth.eval.ts
@@ -39,7 +39,7 @@ export default defineEval({
const resumedTurn = t.target.watchTurn(held.sessionId, {
startIndex: held.session.state.streamIndex,
});
- const callback = await fetch(callbackUrl);
+ const callback = await t.target.fetch(`${callbackUrl.pathname}${callbackUrl.search}`);
if (!callback.ok) {
throw new Error(
`Fixture OAuth callback failed (${String(callback.status)}): ${await callback.text()}`,
diff --git a/e2e/fixtures/agent-tools-hitl/evals/hitl/helpers.ts b/e2e/fixtures/agent-tools-hitl/evals/hitl/helpers.ts
index 7291abc30..5d1d42be6 100644
--- a/e2e/fixtures/agent-tools-hitl/evals/hitl/helpers.ts
+++ b/e2e/fixtures/agent-tools-hitl/evals/hitl/helpers.ts
@@ -120,9 +120,9 @@ export function authorizationFrom(turn: Pick<EveEvalTurn, "events">): {
return { attemptId: required.data.attemptId, url: new URL(required.data.authorization.url) };
}
-/** Completes a fixture authorization the way a browser would: an unauthenticated GET. */
-export async function completeAuthorization(url: URL) {
- const callback = await fetch(url);
+/** Completes a fixture authorization the way a browser would: a GET on the callback URL. */
+export async function completeAuthorization(t: EveEvalContext, url: URL) {
+ const callback = await t.target.fetch(`${url.pathname}${url.search}`);
if (!callback.ok) throw new Error(`Fixture authorization callback failed (${callback.status}).`);
}
diff --git a/e2e/fixtures/agent-tools-hitl/evals/hitl/response-policies/responder-authorization-then-settles.eval.ts b/e2e/fixtures/agent-tools-hitl/evals/hitl/response-policies/responder-authorization-then-settles.eval.ts
index 597b6a9f7..612bd7a5c 100644
--- a/e2e/fixtures/agent-tools-hitl/evals/hitl/response-policies/responder-authorization-then-settles.eval.ts
+++ b/e2e/fixtures/agent-tools-hitl/evals/hitl/response-policies/responder-authorization-then-settles.eval.ts
@@ -45,7 +45,7 @@ export default defineEval({
const authorization = authorizationFrom(held);
const startIndex = session.state.streamIndex;
- await completeAuthorization(authorization.url);
+ await completeAuthorization(t, authorization.url);
const resumed = (await follow(t, session, startIndex)).expectOk();
resumed.notEvent("turn.started");
resumed.eventOrder([
diff --git a/e2e/fixtures/agent-workflow-tools/evals/agent-probe.shared.ts b/e2e/fixtures/agent-workflow-tools/evals/agent-probe.shared.ts
index f6790ad20..0dbb72cbe 100644
--- a/e2e/fixtures/agent-workflow-tools/evals/agent-probe.shared.ts
+++ b/e2e/fixtures/agent-workflow-tools/evals/agent-probe.shared.ts
@@ -101,7 +101,7 @@ async function completeSignIn(
}
const callbackUrl = toCallbackUrl(required.data.authorization?.url);
const resumed = watchNext(t, live.session);
- const response = await fetch(callbackUrl);
+ const response = await t.target.fetch(`${callbackUrl.pathname}${callbackUrl.search}`);
if (!response.ok) {
throw new Error(`Authorization callback failed (${response.status}).`);
}
@@ -152,7 +152,7 @@ export async function runRejectedStepAuth(t: EveEvalContext): Promise<void> {
);
requireAuthorizationOutcome(turn, "failed");
- const repeatedCallback = await fetch(callbackUrl);
+ const repeatedCallback = await t.target.fetch(`${callbackUrl.pathname}${callbackUrl.search}`);
if (repeatedCallback.status !== 404) {
throw new Error("The completed authorization callback must be disposed.");
}
diff --git a/e2e/fixtures/agent-workflow-tools/evals/private-catalog.sign-in.eval.ts b/e2e/fixtures/agent-workflow-tools/evals/private-catalog.sign-in.eval.ts
index 107ae0962..9d7c412fe 100644
--- a/e2e/fixtures/agent-workflow-tools/evals/private-catalog.sign-in.eval.ts
+++ b/e2e/fixtures/agent-workflow-tools/evals/private-catalog.sign-in.eval.ts
@@ -53,9 +53,8 @@ export default defineEval({
const resumed = t.target.watchTurn(session.sessionId, {
startIndex: session.state.streamIndex,
});
- const response = await fetch(
- fixtureAuthorizationCallback(t.target.url, required.data.authorization?.url),
- );
+ const callback = fixtureAuthorizationCallback(t.target.url, required.data.authorization?.url);
+ const response = await t.target.fetch(`${callback.pathname}${callback.search}`);
if (!response.ok) throw new Error(`Authorization callback failed (${response.status}).`);
const completed = await resumed.result();
diff --git a/e2e/fixtures/agent-workflow-tools/evals/public-catalog.lazy-authorization.eval.ts b/e2e/fixtures/agent-workflow-tools/evals/public-catalog.lazy-authorization.eval.ts
index 7f938cec9..123d9515d 100644
--- a/e2e/fixtures/agent-workflow-tools/evals/public-catalog.lazy-authorization.eval.ts
+++ b/e2e/fixtures/agent-workflow-tools/evals/public-catalog.lazy-authorization.eval.ts
@@ -68,7 +68,7 @@ export default defineEval({
const resumed = t.target.watchTurn(session.sessionId, {
startIndex: session.state.streamIndex,
});
- const response = await fetch(callback);
+ const response = await t.target.fetch(`${callback.pathname}${callback.search}`);
if (!response.ok) {
throw new Error(`Authorization callback failed (${response.status}).`);
}
diff --git a/packages/eve/src/execution/workflow-callback-url.ts b/packages/eve/src/execution/workflow-callback-url.ts
index 0a061b4cb..38ee096ef 100644
--- a/packages/eve/src/execution/workflow-callback-url.ts
+++ b/packages/eve/src/execution/workflow-callback-url.ts
@@ -50,17 +50,30 @@ export function resolveWorkflowCallbackBaseUrl(metadataUrl: string): string {
}
/**
- * Builds a framework-owned callback URL from a resolved callback base URL.
+ * Builds a framework-owned callback URL that a browser or third party sees,
+ * such as the connection sign-in redirect handed to an IdP.
*
* `callbackPath` is appended to the full base URL rather than resolved
* against it: the base may carry a public route prefix path, which
* `new URL(path, base)` would drop for absolute paths.
*/
-export function createWorkflowCallbackUrl(baseUrl: string, callbackPath: string): string {
+export function createBrowserCallbackUrl(baseUrl: string, callbackPath: string): string {
const callback = new URL(callbackPath, "http://eve.local");
const url = new URL(baseUrl);
url.pathname = joinEveRoutePath(url.pathname, callback.pathname);
url.search = callback.search;
+ return url.toString();
+}
+
+/**
+ * Builds a framework-owned callback URL for server-to-server delivery.
+ *
+ * Unlike {@link createBrowserCallbackUrl}, it carries the deployment's
+ * Protection Bypass for Automation secret so the caller can reach a
+ * protected deployment.
+ */
+export function createWorkflowCallbackUrl(baseUrl: string, callbackPath: string): string {
+ const url = new URL(createBrowserCallbackUrl(baseUrl, callbackPath));
// https://vercel.com/docs/deployment-protection/methods-to-bypass-deployment-protection/protection-bypass-automation
const bypassSecret = process.env.VERCEL_AUTOMATION_BYPASS_SECRET?.trim();
diff --git a/packages/eve/src/harness/authorization.test.ts b/packages/eve/src/harness/authorization.test.ts
index 3653c3933..d3bcbddef 100644
--- a/packages/eve/src/harness/authorization.test.ts
+++ b/packages/eve/src/harness/authorization.test.ts
@@ -2,6 +2,7 @@ import { afterEach, describe, expect, it, vi } from "vitest";
import { ContextContainer, contextStorage } from "#context/container.js";
import { SessionIdKey } from "#context/keys.js";
+import { resolveWorkflowCallbackBaseUrl } from "#execution/workflow-callback-url.js";
import {
CallbackBaseUrlKey,
consumeAuthorizationResult,
@@ -22,14 +23,27 @@ afterEach(() => {
});
describe("authorization callback URLs", () => {
- it("includes the Vercel automation bypass query when configured", () => {
+ it("keeps the automation bypass secret out of sign-in callback URLs", () => {
vi.stubEnv("VERCEL_AUTOMATION_BYPASS_SECRET", "secret value");
const ctx = new ContextContainer();
ctx.set(CallbackBaseUrlKey, "https://agent.example.com");
ctx.set(SessionIdKey, "session-1");
expect(contextStorage.run(ctx, () => getHookUrl("linear", "attempt-1"))).toBe(
- "https://agent.example.com/eve/v1/connections/linear/callback/attempt-1/eve%3Ainbox%3Av1%3Aeve%3Asession%3Asession-1%3Ainbox?x-vercel-protection-bypass=secret+value",
+ "https://agent.example.com/eve/v1/connections/linear/callback/attempt-1/eve%3Ainbox%3Av1%3Aeve%3Asession%3Asession-1%3Ainbox",
+ );
+ });
+
+ it("keeps the automation bypass secret out of production sign-in callback URLs", () => {
+ vi.stubEnv("VERCEL_AUTOMATION_BYPASS_SECRET", "secret value");
+ vi.stubEnv("VERCEL_ENV", "production");
+ vi.stubEnv("VERCEL_PROJECT_PRODUCTION_URL", "agent.example.com");
+ const ctx = new ContextContainer();
+ ctx.set(CallbackBaseUrlKey, resolveWorkflowCallbackBaseUrl("https://agent-abc123.vercel.app"));
+ ctx.set(SessionIdKey, "session-1");
+
+ expect(contextStorage.run(ctx, () => getHookUrl("linear", "attempt-1"))).toBe(
+ "https://agent.example.com/eve/v1/connections/linear/callback/attempt-1/eve%3Ainbox%3Av1%3Aeve%3Asession%3Asession-1%3Ainbox",
);
});
});
diff --git a/packages/eve/src/harness/authorization.ts b/packages/eve/src/harness/authorization.ts
index 552314665..e4a078071 100644
--- a/packages/eve/src/harness/authorization.ts
+++ b/packages/eve/src/harness/authorization.ts
@@ -38,7 +38,7 @@ import {
import { loadContext } from "#context/container.js";
import { ContextKey } from "#context/key.js";
import { SessionIdKey, type SessionAuthContext } from "#context/keys.js";
-import { createWorkflowCallbackUrl } from "#execution/workflow-callback-url.js";
+import { createBrowserCallbackUrl } from "#execution/workflow-callback-url.js";
import type { ConnectionAuthorizationChallenge } from "#connections/errors.js";
import type { AuthorizationCallback, ConnectionPrincipal } from "#shared/connection-types.js";
import type { JsonValue } from "#shared/json.js";
@@ -190,6 +190,9 @@ export function consumeAuthorizationResult(
* It is independent of the continuation token, so channel aliasing mid-turn
* does not invalidate the callback URL.
*
+ * The URL reaches the user's browser and the IdP, so it never carries the
+ * deployment's protection bypass secret.
+ *
* Returns `undefined` if no callback address is available.
*/
export function getHookUrl(name: string, attemptId: string): string | undefined {
@@ -198,7 +201,7 @@ export function getHookUrl(name: string, attemptId: string): string | undefined
const baseUrl = ctx.get(CallbackBaseUrlKey);
const token = ctx.get(AuthorizationHookKey) ?? (sessionId ? authHookToken(sessionId) : undefined);
if (!token || !baseUrl) return undefined;
- return createWorkflowCallbackUrl(
+ return createBrowserCallbackUrl(
baseUrl,
createEveConnectionCallbackRoutePath(name, attemptId, token),
);