Skip to content

Open-source manufacturing ERP, MES and QMS. Quoting, MRP, inventory, shop floor, quality and lot/serial traceability on one Postgres schema, with a REST API and MCP server. Self-host or use Carbon Cloud.

fix(webhooks): guard event webhook delivery against private addresses (#1974)

Brad Barbin4f67c697b178
diff --git a/.claude/rules/workflow-actions.md b/.claude/rules/workflow-actions.md
index 8fbb6ae0e0..84a2da493e 100644
--- a/.claude/rules/workflow-actions.md
+++ b/.claude/rules/workflow-actions.md
@@ -328,21 +328,28 @@ covered completely: a literal cannot rebind, because there is no name to re-reso
The two layers are what make the guard whole, so don't drop the pre-check as
"redundant".
-## The two older webhook systems are untouched
+## The other two webhook systems
-Three things in this repo are called "webhook". Only the first is new:
+Three things in this repo are called "webhook":
| | What | Where |
|---|---|---|
| **Workflow action** | Outbound, SSRF-guarded, https-only | `packages/jobs/src/workflows/actions/webhook.ts` |
-| Event-system `WEBHOOK` handler | Outbound `axios.post` on `carbon/event-webhook` to a subscription's configured URL with its configured headers — **no signing, no URL guard** | `packages/jobs/src/inngest/functions/events/webhook.ts` |
+| Event-system `WEBHOOK` handler | Outbound `axios.post` on `carbon/event-webhook` to a subscription's configured URL, signed (`Carbon-Signature`), SSRF-guarded, http or https | `packages/jobs/src/inngest/functions/events/webhook.ts` |
| Integration webhooks | **Inbound** routes for Jira, Linear, Paperless Parts, Stripe and Xero | `apps/erp/app/routes/api+/webhook.{jira,linear,paperless-parts,stripe,xero}*.ts` |
-They share no code with the workflow action and were deliberately not changed.
-Do not "unify" them: the event-system handler's contract is an existing customer
-integration, and hardening it is its own decision with its own migration. The
-matcher's subscription reconciler is already careful never to touch a company's
-`WEBHOOK` rows (see `workflow-matcher.md`).
+The event-system handler shares only the guard with the workflow action, through
+`checkWebhookUrl` (the private-network check of `checkOutboundUrl`, but http and URL
+credentials still allowed, since its customer contract never forbade them) and
+`outboundHttpAgent` / `outboundHttpsAgent` (`guardedLookup` for node's http client),
+both in `url-guard.ts`. `webhookTransport` (`events/webhook-body.ts`) holds its axios
+options: those agents, `maxRedirects: 0` (a 3xx fails the attempt) and `proxy: false`
+(through an `HTTP_PROXY` the guard would check the proxy, not the target). A refused
+address counts one error and throws `NonRetriableError`; a failed lookup is
+`retryable` and takes the ordinary retry path. Do not go further and
+"unify" the two: the event handler's contract (http, signing, retries, the 30 s
+timeout) is an existing customer integration. The matcher's subscription reconciler
+is careful never to touch a company's `WEBHOOK` rows (see `workflow-matcher.md`).
## Gotchas
diff --git a/apps/erp/app/modules/settings/settings.models.ts b/apps/erp/app/modules/settings/settings.models.ts
index 859f362f19..6960129f6d 100644
--- a/apps/erp/app/modules/settings/settings.models.ts
+++ b/apps/erp/app/modules/settings/settings.models.ts
@@ -533,7 +533,12 @@ export const webhookValidator = z
id: zfd.text(z.string().optional()),
name: z.string().trim().min(1, { message: "Name is required" }),
table: z.string().min(1, { message: "Table is required" }),
- url: z.string().url({ message: "Must be a valid URL" }),
+ url: z
+ .string()
+ .url({ message: "Must be a valid URL" })
+ .refine((url) => /^https?:\/\//i.test(url), {
+ message: "Must be an http or https URL"
+ }),
onInsert: zfd.checkbox(),
onUpdate: zfd.checkbox(),
onDelete: zfd.checkbox(),
diff --git a/docs/content/docs/building/webhooks.mdx b/docs/content/docs/building/webhooks.mdx
index a4e8c6b728..204736b834 100644
--- a/docs/content/docs/building/webhooks.mdx
+++ b/docs/content/docs/building/webhooks.mdx
@@ -49,7 +49,7 @@ Create and manage webhooks under **Settings → Webhooks**. A webhook has:
- **Name:** a label, unique within your company.
- **Table:** the one table to watch.
-- **URL:** where the `POST` is sent; must be a valid URL.
+- **URL:** where the `POST` is sent; must be an `http` or `https` URL that resolves to a public address. Carbon refuses to deliver to loopback, private-network, link-local and cloud metadata addresses.
- **Triggers:** any of insert, update, delete (at least one).
- **Active:** webhooks are on by default; switch off to pause delivery.
- **Signing secret:** reveal or roll it from the webhook's edit drawer (see below).
@@ -67,6 +67,8 @@ Delivery is **at least once**. If the endpoint does not return a success status
The success and error counters record one outcome per change, not per attempt, so a retried-then-delivered event counts once as a success.
+Carbon does not follow redirects. A `3xx` response is a failed attempt, so configure the final URL. A delivery to a refused address is not retried; it counts once as an error.
+
## Verifying deliveries
Each webhook has its own **signing secret** (`whsec_…`). Every delivery carries a `Carbon-Signature` header. With the secret `whsec_example`, a body of `{}` sent at `1760000000` is signed:
diff --git a/packages/jobs/src/inngest/functions/events/webhook-body.ts b/packages/jobs/src/inngest/functions/events/webhook-body.ts
index 24853154e1..b7ea4176a3 100644
--- a/packages/jobs/src/inngest/functions/events/webhook-body.ts
+++ b/packages/jobs/src/inngest/functions/events/webhook-body.ts
@@ -4,6 +4,10 @@
import { createHmac } from "node:crypto";
import { z } from "zod";
+import {
+ outboundHttpAgent,
+ outboundHttpsAgent
+} from "../../../workflows/actions/url-guard.ts";
// Separate from webhook.ts so the contract below is testable — the handler
// imports client.server, which validates the server env at module load.
@@ -84,6 +88,22 @@ export function toWebhookBody(
*/
export const WEBHOOK_SIGNATURE_HEADER = "Carbon-Signature";
+/**
+ * The axios options every delivery uses. The agents run the SSRF guard at
+ * connect time, so the address `checkWebhookUrl` approved is the address
+ * dialled. Redirects are off: following one would leave the guard behind, so a
+ * 3xx fails the attempt. So is any proxy from the environment (`HTTP_PROXY`):
+ * through one, the socket — and the guard — would reach the proxy, and the proxy
+ * the target. The timeout is PUBLIC contract (30 s, see the docs).
+ */
+export const webhookTransport = {
+ timeout: 30_000,
+ maxRedirects: 0,
+ proxy: false as const,
+ httpAgent: outboundHttpAgent,
+ httpsAgent: outboundHttpsAgent
+};
+
/**
* Sign a delivery: `t=<unix seconds>,v1=<hex HMAC-SHA256>`, where the HMAC is
* keyed by the webhook's signing secret (the whole `whsec_…` string) and runs
diff --git a/packages/jobs/src/inngest/functions/events/webhook-transport.test.ts b/packages/jobs/src/inngest/functions/events/webhook-transport.test.ts
new file mode 100644
index 0000000000..5901962150
--- /dev/null
+++ b/packages/jobs/src/inngest/functions/events/webhook-transport.test.ts
@@ -0,0 +1,86 @@
+// SPDX-License-Identifier: AGPL-3.0-only
+// Carbon (github.com/crbnos/carbon). Modified or adapted versions of this file,
+// including ports, remain AGPLv3; serving them over a network requires releasing their source.
+
+import http from "node:http";
+import type { AddressInfo } from "node:net";
+import axios from "axios";
+import { afterEach, describe, expect, it } from "vitest";
+import { webhookTransport } from "./webhook-body.ts";
+
+const servers: http.Server[] = [];
+
+function listen(handler: http.RequestListener): Promise<number> {
+ const server = http.createServer(handler);
+ servers.push(server);
+ return new Promise((resolve) =>
+ server.listen(0, "127.0.0.1", () =>
+ resolve((server.address() as AddressInfo).port)
+ )
+ );
+}
+
+afterEach(async () => {
+ await Promise.all(
+ servers.splice(0).map((s) => new Promise((r) => s.close(r)))
+ );
+});
+
+describe("webhookTransport", () => {
+ it("refuses at connect time a name that resolves to loopback", async () => {
+ let received = 0;
+ const port = await listen((_req, res) => {
+ received++;
+ res.end("{}");
+ });
+
+ await expect(
+ axios.post(`http://localhost:${port}/hook`, "{}", webhookTransport)
+ ).rejects.toThrow("That address is inside a private network.");
+ expect(received).toBe(0);
+ });
+
+ it("ignores a proxy from the environment", async () => {
+ let proxied = 0;
+ const proxy = await listen((_req, res) => {
+ proxied++;
+ res.end("{}");
+ });
+ const saved = process.env.HTTP_PROXY;
+ process.env.HTTP_PROXY = `http://127.0.0.1:${proxy}`;
+ try {
+ await expect(
+ axios.post("http://localhost:1/hook", "{}", webhookTransport)
+ ).rejects.toThrow("That address is inside a private network.");
+ } finally {
+ if (saved === undefined) delete process.env.HTTP_PROXY;
+ else process.env.HTTP_PROXY = saved;
+ }
+ expect(proxied).toBe(0);
+ });
+
+ it("does not follow a redirect", async () => {
+ let reached = 0;
+ const target = await listen((_req, res) => {
+ reached++;
+ res.end("{}");
+ });
+ const bounce = await listen((_req, res) => {
+ res.writeHead(302, {
+ location: `http://127.0.0.1:${target}/latest/meta-data/`
+ });
+ res.end();
+ });
+
+ // The agents are swapped out only so the test can reach its own loopback
+ // listener; the redirect setting under test is the transport's.
+ await expect(
+ axios.post(`http://127.0.0.1:${bounce}/bounce`, "{}", {
+ ...webhookTransport,
+ httpAgent: undefined,
+ httpsAgent: undefined
+ })
+ ).rejects.toMatchObject({ response: { status: 302 } });
+ expect(reached).toBe(0);
+ });
+});
diff --git a/packages/jobs/src/inngest/functions/events/webhook.ts b/packages/jobs/src/inngest/functions/events/webhook.ts
index 05312f17fb..a463ab838e 100644
--- a/packages/jobs/src/inngest/functions/events/webhook.ts
+++ b/packages/jobs/src/inngest/functions/events/webhook.ts
@@ -6,18 +6,42 @@ import { getCarbonServiceRole } from "@carbon/auth/client.server";
import { companyHasFeature } from "@carbon/ee/plan.server";
import { getWebhookSigningSecret } from "@carbon/ee/webhooks.server";
import axios from "axios";
+import { NonRetriableError } from "inngest";
import { getJobDatabaseClient } from "../../../db.ts";
+import { checkWebhookUrl } from "../../../workflows/actions/url-guard.ts";
import { inngest } from "../../client.ts";
import {
signWebhookBody,
toWebhookBody,
WEBHOOK_SIGNATURE_HEADER,
- webhookPayloadSchema
+ webhookPayloadSchema,
+ webhookTransport
} from "./webhook-body.ts";
const RETRIES = 3;
-// A documented part of the webhook contract (docs: building/webhooks).
-const WEBHOOK_TIMEOUT_MS = 30_000;
+
+/** Best effort: a counter that fails to move is logged, never fails the delivery. */
+async function countOutcome(
+ webhookId: string,
+ outcome: "success" | "error",
+ logger: {
+ error: (message: string, properties: Record<string, unknown>) => void;
+ }
+) {
+ const { error } = await getCarbonServiceRole().rpc(
+ outcome === "success"
+ ? "increment_webhook_success"
+ : "increment_webhook_error",
+ { webhook_id: webhookId }
+ );
+ if (error) {
+ logger.error("Could not count the {outcome} of webhook {webhookId}", {
+ outcome,
+ webhookId,
+ error: error.message
+ });
+ }
+}
export const webhookFunction = inngest.createFunction(
{
@@ -84,33 +108,41 @@ export const webhookFunction = inngest.createFunction(
);
return;
}
+
+ // A webhook may not reach into Carbon's own network (loopback, private
+ // ranges, cloud metadata). A retry would be refused the same way, so a
+ // refusal counts one failure and stops. A failed lookup is retried.
+ const verdict = await checkWebhookUrl(payload.url);
+ if (!verdict.ok && !verdict.retryable) {
+ logger.warn(
+ `Webhook ${webhookId ?? "<unknown>"} refused: ${verdict.reason}`
+ );
+ if (webhookId) await countOutcome(webhookId, "error", logger);
+ throw new NonRetriableError(verdict.reason);
+ }
+
const timestamp = Math.floor(Date.now() / 1000);
try {
+ if (!verdict.ok) throw new Error(verdict.reason);
await axios.post(payload.url, raw, {
headers: {
"Content-Type": "application/json",
...payload.config.headers,
[WEBHOOK_SIGNATURE_HEADER]: signWebhookBody(secret, raw, timestamp)
},
- timeout: WEBHOOK_TIMEOUT_MS
+ ...webhookTransport
});
} catch (err) {
// Count one failure per event, not per attempt — otherwise retries
// inflate errorCount 4x for a single undelivered event.
if (webhookId && attempt >= RETRIES) {
- await getCarbonServiceRole().rpc("increment_webhook_error", {
- webhook_id: webhookId
- });
+ await countOutcome(webhookId, "error", logger);
}
throw err;
}
- if (webhookId) {
- await getCarbonServiceRole().rpc("increment_webhook_success", {
- webhook_id: webhookId
- });
- }
+ if (webhookId) await countOutcome(webhookId, "success", logger);
});
}
);
diff --git a/packages/jobs/src/workflows/actions/url-guard.test.ts b/packages/jobs/src/workflows/actions/url-guard.test.ts
index 3e64e13460..78bff380d7 100644
--- a/packages/jobs/src/workflows/actions/url-guard.test.ts
+++ b/packages/jobs/src/workflows/actions/url-guard.test.ts
@@ -4,7 +4,7 @@
import dns from "node:dns";
import { afterEach, describe, expect, it, vi } from "vitest";
-import { checkOutboundUrl, guardedLookup } from "./url-guard";
+import { checkOutboundUrl, checkWebhookUrl, guardedLookup } from "./url-guard";
/** Stubbed so no test performs real DNS. */
function resolvesTo(...addresses: { address: string; family: number }[]) {
@@ -172,6 +172,58 @@ function lookup(
});
}
+describe("checkWebhookUrl", () => {
+ it("accepts http, which event webhooks have always allowed", async () => {
+ resolvesTo({ address: "93.184.216.34", family: 4 });
+ expect(await checkWebhookUrl("http://example.com/hook")).toEqual({
+ ok: true,
+ url: new URL("http://example.com/hook")
+ });
+ });
+
+ it("accepts credentials in the address", async () => {
+ resolvesTo({ address: "93.184.216.34", family: 4 });
+ expect((await checkWebhookUrl("https://u:[email protected]/hook")).ok).toBe(
+ true
+ );
+ });
+
+ it("refuses a scheme other than http or https", async () => {
+ expect(await checkWebhookUrl("ftp://example.com/hook")).toEqual({
+ ok: false,
+ reason: "Only http and https addresses are allowed.",
+ retryable: false
+ });
+ });
+
+ it.each([
+ ["http://127.0.0.1:8080/admin", "127.0.0.1"],
+ ["http://10.0.0.5/internal", "10.0.0.5"],
+ ["http://169.254.169.254/latest/meta-data/", "169.254.169.254"]
+ ])("refuses %s", async (url, address) => {
+ resolvesTo({ address, family: 4 });
+ expect(await checkWebhookUrl(url)).toEqual({
+ ok: false,
+ reason: "That address is inside a private network.",
+ retryable: false
+ });
+ });
+
+ it("refuses a public-looking name that resolves inward", async () => {
+ resolvesTo({ address: "192.168.1.10", family: 4 });
+ expect((await checkWebhookUrl("https://hooks.example.com")).ok).toBe(false);
+ });
+
+ it("marks a failed lookup retryable, since DNS can recover", async () => {
+ vi.spyOn(dns.promises, "lookup").mockRejectedValue(new Error("EAI_AGAIN"));
+ expect(await checkWebhookUrl("https://hooks.example.com")).toEqual({
+ ok: false,
+ reason: "That address could not be found.",
+ retryable: true
+ });
+ });
+});
+
describe("guardedLookup", () => {
it("passes a public address through as a single answer", async () => {
lookupResolvesTo({ address: "93.184.216.34", family: 4 });
diff --git a/packages/jobs/src/workflows/actions/url-guard.ts b/packages/jobs/src/workflows/actions/url-guard.ts
index c01148e814..b098a33c30 100644
--- a/packages/jobs/src/workflows/actions/url-guard.ts
+++ b/packages/jobs/src/workflows/actions/url-guard.ts
@@ -3,12 +3,15 @@
// including ports, remain AGPLv3; serving them over a network requires releasing their source.
import dns from "node:dns";
+import http from "node:http";
+import https from "node:https";
import { Agent } from "undici";
export type UrlVerdict = { ok: true; url: URL } | { ok: false; reason: string };
const NOT_A_URL = "That is not a valid web address.";
const NOT_HTTPS = "Only https addresses are allowed.";
+const NOT_HTTP = "Only http and https addresses are allowed.";
const NOT_FOUND = "That address could not be found.";
const PRIVATE = "That address is inside a private network.";
const HAS_CREDENTIALS =
@@ -72,6 +75,45 @@ export async function checkOutboundUrl(raw: string): Promise<UrlVerdict> {
return { ok: false, reason: HAS_CREDENTIALS };
}
+ return checkResolvedHost(url);
+}
+
+/**
+ * A webhook verdict also says whether a refusal is worth another attempt: a
+ * lookup can fail for a moment, a private address does not stop being one.
+ */
+export type WebhookUrlVerdict =
+ | { ok: true; url: URL }
+ | { ok: false; reason: string; retryable: boolean };
+
+/**
+ * The guard for an event webhook (Settings → Webhooks). Looser than
+ * `checkOutboundUrl` on two points, because those webhooks predate the guard and
+ * their contract never forbade either: http is allowed, and so are credentials in
+ * the URL (the person who typed them sees them in settings). The private-network
+ * check is the same.
+ */
+export async function checkWebhookUrl(raw: string): Promise<WebhookUrlVerdict> {
+ let url: URL;
+ try {
+ url = new URL(raw);
+ } catch {
+ return { ok: false, reason: NOT_A_URL, retryable: false };
+ }
+
+ if (url.protocol !== "https:" && url.protocol !== "http:") {
+ return { ok: false, reason: NOT_HTTP, retryable: false };
+ }
+ if (url.hostname.length === 0) {
+ return { ok: false, reason: NOT_A_URL, retryable: false };
+ }
+
+ const verdict = await checkResolvedHost(url);
+ if (verdict.ok) return verdict;
+ return { ...verdict, retryable: verdict.reason === NOT_FOUND };
+}
+
+async function checkResolvedHost(url: URL): Promise<UrlVerdict> {
let addresses: { address: string; family: number }[];
try {
addresses = await dns.promises.lookup(url.hostname, { all: true });
@@ -123,3 +165,11 @@ export function guardedLookup(
export const outboundDispatcher = new Agent({
connect: { lookup: guardedLookup }
});
+
+/**
+ * The same guard for node's http client (axios), as `httpAgent` / `httpsAgent`.
+ * As with the dispatcher, a literal IP skips the lookup, so `checkWebhookUrl`
+ * must still run first.
+ */
+export const outboundHttpAgent = new http.Agent({ lookup: guardedLookup });
+export const outboundHttpsAgent = new https.Agent({ lookup: guardedLookup });