Open-source manufacturing ERP, MES and QMS. Quoting, MRP, inventory, shop floor, quality and lot/serial traceability on one Postgres schema, with a REST API and MCP server. Self-host or use Carbon Cloud.
TypeScript ai-agentsbill-of-materialserpinventory-managementleanmanufacturingmanufacturing-execution-systemmcp-servermesmrpopen-source-erppostgresqlproduction-planningqmsquality-management-systemreact-routerself-hostedsupabasetraceabilitytypescript
fix(webhooks): guard event webhook delivery against private addresses (#1974)
Brad Barbin4f67c697b178
diff --git a/.claude/rules/workflow-actions.md b/.claude/rules/workflow-actions.mdindex 8fbb6ae0e0..84a2da493e 100644--- a/.claude/rules/workflow-actions.md+++ b/.claude/rules/workflow-actions.md@@ -328,21 +328,28 @@ covered completely: a literal cannot rebind, because there is no name to re-resoThe two layers are what make the guard whole, so don't drop the pre-check as"redundant".-## The two older webhook systems are untouched+## The other two webhook systems-Three things in this repo are called "webhook". Only the first is new:+Three things in this repo are called "webhook":| | What | Where ||---|---|---|| **Workflow action** | Outbound, SSRF-guarded, https-only | `packages/jobs/src/workflows/actions/webhook.ts` |-| Event-system `WEBHOOK` handler | Outbound `axios.post` on `carbon/event-webhook` to a subscription's configured URL with its configured headers — **no signing, no URL guard** | `packages/jobs/src/inngest/functions/events/webhook.ts` |+| Event-system `WEBHOOK` handler | Outbound `axios.post` on `carbon/event-webhook` to a subscription's configured URL, signed (`Carbon-Signature`), SSRF-guarded, http or https | `packages/jobs/src/inngest/functions/events/webhook.ts` || Integration webhooks | **Inbound** routes for Jira, Linear, Paperless Parts, Stripe and Xero | `apps/erp/app/routes/api+/webhook.{jira,linear,paperless-parts,stripe,xero}*.ts` |-They share no code with the workflow action and were deliberately not changed.-Do not "unify" them: the event-system handler's contract is an existing customer-integration, and hardening it is its own decision with its own migration. The-matcher's subscription reconciler is already careful never to touch a company's-`WEBHOOK` rows (see `workflow-matcher.md`).+The event-system handler shares only the guard with the workflow action, through+`checkWebhookUrl` (the private-network check of `checkOutboundUrl`, but http and URL+credentials still allowed, since its customer contract never forbade them) and+`outboundHttpAgent` / `outboundHttpsAgent` (`guardedLookup` for node's http client),+both in `url-guard.ts`. `webhookTransport` (`events/webhook-body.ts`) holds its axios+options: those agents, `maxRedirects: 0` (a 3xx fails the attempt) and `proxy: false`+(through an `HTTP_PROXY` the guard would check the proxy, not the target). A refused+address counts one error and throws `NonRetriableError`; a failed lookup is+`retryable` and takes the ordinary retry path. Do not go further and+"unify" the two: the event handler's contract (http, signing, retries, the 30 s+timeout) is an existing customer integration. The matcher's subscription reconciler+is careful never to touch a company's `WEBHOOK` rows (see `workflow-matcher.md`).## Gotchasdiff --git a/apps/erp/app/modules/settings/settings.models.ts b/apps/erp/app/modules/settings/settings.models.tsindex 859f362f19..6960129f6d 100644--- a/apps/erp/app/modules/settings/settings.models.ts+++ b/apps/erp/app/modules/settings/settings.models.ts@@ -533,7 +533,12 @@ export const webhookValidator = zid: zfd.text(z.string().optional()),name: z.string().trim().min(1, { message: "Name is required" }),table: z.string().min(1, { message: "Table is required" }),- url: z.string().url({ message: "Must be a valid URL" }),+ url: z+ .string()+ .url({ message: "Must be a valid URL" })+ .refine((url) => /^https?:\/\//i.test(url), {+ message: "Must be an http or https URL"+ }),onInsert: zfd.checkbox(),onUpdate: zfd.checkbox(),onDelete: zfd.checkbox(),diff --git a/docs/content/docs/building/webhooks.mdx b/docs/content/docs/building/webhooks.mdxindex a4e8c6b728..204736b834 100644--- a/docs/content/docs/building/webhooks.mdx+++ b/docs/content/docs/building/webhooks.mdx@@ -49,7 +49,7 @@ Create and manage webhooks under **Settings → Webhooks**. A webhook has:- **Name:** a label, unique within your company.- **Table:** the one table to watch.-- **URL:** where the `POST` is sent; must be a valid URL.+- **URL:** where the `POST` is sent; must be an `http` or `https` URL that resolves to a public address. Carbon refuses to deliver to loopback, private-network, link-local and cloud metadata addresses.- **Triggers:** any of insert, update, delete (at least one).- **Active:** webhooks are on by default; switch off to pause delivery.- **Signing secret:** reveal or roll it from the webhook's edit drawer (see below).@@ -67,6 +67,8 @@ Delivery is **at least once**. If the endpoint does not return a success statusThe success and error counters record one outcome per change, not per attempt, so a retried-then-delivered event counts once as a success.+Carbon does not follow redirects. A `3xx` response is a failed attempt, so configure the final URL. A delivery to a refused address is not retried; it counts once as an error.+## Verifying deliveriesEach webhook has its own **signing secret** (`whsec_…`). Every delivery carries a `Carbon-Signature` header. With the secret `whsec_example`, a body of `{}` sent at `1760000000` is signed:diff --git a/packages/jobs/src/inngest/functions/events/webhook-body.ts b/packages/jobs/src/inngest/functions/events/webhook-body.tsindex 24853154e1..b7ea4176a3 100644--- a/packages/jobs/src/inngest/functions/events/webhook-body.ts+++ b/packages/jobs/src/inngest/functions/events/webhook-body.ts@@ -4,6 +4,10 @@import { createHmac } from "node:crypto";import { z } from "zod";+import {+ outboundHttpAgent,+ outboundHttpsAgent+} from "../../../workflows/actions/url-guard.ts";// Separate from webhook.ts so the contract below is testable — the handler// imports client.server, which validates the server env at module load.@@ -84,6 +88,22 @@ export function toWebhookBody(*/export const WEBHOOK_SIGNATURE_HEADER = "Carbon-Signature";+/**+ * The axios options every delivery uses. The agents run the SSRF guard at+ * connect time, so the address `checkWebhookUrl` approved is the address+ * dialled. Redirects are off: following one would leave the guard behind, so a+ * 3xx fails the attempt. So is any proxy from the environment (`HTTP_PROXY`):+ * through one, the socket — and the guard — would reach the proxy, and the proxy+ * the target. The timeout is PUBLIC contract (30 s, see the docs).+ */+export const webhookTransport = {+ timeout: 30_000,+ maxRedirects: 0,+ proxy: false as const,+ httpAgent: outboundHttpAgent,+ httpsAgent: outboundHttpsAgent+};+/*** Sign a delivery: `t=<unix seconds>,v1=<hex HMAC-SHA256>`, where the HMAC is* keyed by the webhook's signing secret (the whole `whsec_…` string) and runsdiff --git a/packages/jobs/src/inngest/functions/events/webhook-transport.test.ts b/packages/jobs/src/inngest/functions/events/webhook-transport.test.tsnew file mode 100644index 0000000000..5901962150--- /dev/null+++ b/packages/jobs/src/inngest/functions/events/webhook-transport.test.ts@@ -0,0 +1,86 @@+// SPDX-License-Identifier: AGPL-3.0-only+// Carbon (github.com/crbnos/carbon). Modified or adapted versions of this file,+// including ports, remain AGPLv3; serving them over a network requires releasing their source.++import http from "node:http";+import type { AddressInfo } from "node:net";+import axios from "axios";+import { afterEach, describe, expect, it } from "vitest";+import { webhookTransport } from "./webhook-body.ts";++const servers: http.Server[] = [];++function listen(handler: http.RequestListener): Promise<number> {+ const server = http.createServer(handler);+ servers.push(server);+ return new Promise((resolve) =>+ server.listen(0, "127.0.0.1", () =>+ resolve((server.address() as AddressInfo).port)+ )+ );+}++afterEach(async () => {+ await Promise.all(+ servers.splice(0).map((s) => new Promise((r) => s.close(r)))+ );+});++describe("webhookTransport", () => {+ it("refuses at connect time a name that resolves to loopback", async () => {+ let received = 0;+ const port = await listen((_req, res) => {+ received++;+ res.end("{}");+ });++ await expect(+ axios.post(`http://localhost:${port}/hook`, "{}", webhookTransport)+ ).rejects.toThrow("That address is inside a private network.");+ expect(received).toBe(0);+ });++ it("ignores a proxy from the environment", async () => {+ let proxied = 0;+ const proxy = await listen((_req, res) => {+ proxied++;+ res.end("{}");+ });+ const saved = process.env.HTTP_PROXY;+ process.env.HTTP_PROXY = `http://127.0.0.1:${proxy}`;+ try {+ await expect(+ axios.post("http://localhost:1/hook", "{}", webhookTransport)+ ).rejects.toThrow("That address is inside a private network.");+ } finally {+ if (saved === undefined) delete process.env.HTTP_PROXY;+ else process.env.HTTP_PROXY = saved;+ }+ expect(proxied).toBe(0);+ });++ it("does not follow a redirect", async () => {+ let reached = 0;+ const target = await listen((_req, res) => {+ reached++;+ res.end("{}");+ });+ const bounce = await listen((_req, res) => {+ res.writeHead(302, {+ location: `http://127.0.0.1:${target}/latest/meta-data/`+ });+ res.end();+ });++ // The agents are swapped out only so the test can reach its own loopback+ // listener; the redirect setting under test is the transport's.+ await expect(+ axios.post(`http://127.0.0.1:${bounce}/bounce`, "{}", {+ ...webhookTransport,+ httpAgent: undefined,+ httpsAgent: undefined+ })+ ).rejects.toMatchObject({ response: { status: 302 } });+ expect(reached).toBe(0);+ });+});diff --git a/packages/jobs/src/inngest/functions/events/webhook.ts b/packages/jobs/src/inngest/functions/events/webhook.tsindex 05312f17fb..a463ab838e 100644--- a/packages/jobs/src/inngest/functions/events/webhook.ts+++ b/packages/jobs/src/inngest/functions/events/webhook.ts@@ -6,18 +6,42 @@ import { getCarbonServiceRole } from "@carbon/auth/client.server";import { companyHasFeature } from "@carbon/ee/plan.server";import { getWebhookSigningSecret } from "@carbon/ee/webhooks.server";import axios from "axios";+import { NonRetriableError } from "inngest";import { getJobDatabaseClient } from "../../../db.ts";+import { checkWebhookUrl } from "../../../workflows/actions/url-guard.ts";import { inngest } from "../../client.ts";import {signWebhookBody,toWebhookBody,WEBHOOK_SIGNATURE_HEADER,- webhookPayloadSchema+ webhookPayloadSchema,+ webhookTransport} from "./webhook-body.ts";const RETRIES = 3;-// A documented part of the webhook contract (docs: building/webhooks).-const WEBHOOK_TIMEOUT_MS = 30_000;++/** Best effort: a counter that fails to move is logged, never fails the delivery. */+async function countOutcome(+ webhookId: string,+ outcome: "success" | "error",+ logger: {+ error: (message: string, properties: Record<string, unknown>) => void;+ }+) {+ const { error } = await getCarbonServiceRole().rpc(+ outcome === "success"+ ? "increment_webhook_success"+ : "increment_webhook_error",+ { webhook_id: webhookId }+ );+ if (error) {+ logger.error("Could not count the {outcome} of webhook {webhookId}", {+ outcome,+ webhookId,+ error: error.message+ });+ }+}export const webhookFunction = inngest.createFunction({@@ -84,33 +108,41 @@ export const webhookFunction = inngest.createFunction();return;}++ // A webhook may not reach into Carbon's own network (loopback, private+ // ranges, cloud metadata). A retry would be refused the same way, so a+ // refusal counts one failure and stops. A failed lookup is retried.+ const verdict = await checkWebhookUrl(payload.url);+ if (!verdict.ok && !verdict.retryable) {+ logger.warn(+ `Webhook ${webhookId ?? "<unknown>"} refused: ${verdict.reason}`+ );+ if (webhookId) await countOutcome(webhookId, "error", logger);+ throw new NonRetriableError(verdict.reason);+ }+const timestamp = Math.floor(Date.now() / 1000);try {+ if (!verdict.ok) throw new Error(verdict.reason);await axios.post(payload.url, raw, {headers: {"Content-Type": "application/json",...payload.config.headers,[WEBHOOK_SIGNATURE_HEADER]: signWebhookBody(secret, raw, timestamp)},- timeout: WEBHOOK_TIMEOUT_MS+ ...webhookTransport});} catch (err) {// Count one failure per event, not per attempt — otherwise retries// inflate errorCount 4x for a single undelivered event.if (webhookId && attempt >= RETRIES) {- await getCarbonServiceRole().rpc("increment_webhook_error", {- webhook_id: webhookId- });+ await countOutcome(webhookId, "error", logger);}throw err;}- if (webhookId) {- await getCarbonServiceRole().rpc("increment_webhook_success", {- webhook_id: webhookId- });- }+ if (webhookId) await countOutcome(webhookId, "success", logger);});});diff --git a/packages/jobs/src/workflows/actions/url-guard.test.ts b/packages/jobs/src/workflows/actions/url-guard.test.tsindex 3e64e13460..78bff380d7 100644--- a/packages/jobs/src/workflows/actions/url-guard.test.ts+++ b/packages/jobs/src/workflows/actions/url-guard.test.ts@@ -4,7 +4,7 @@import dns from "node:dns";import { afterEach, describe, expect, it, vi } from "vitest";-import { checkOutboundUrl, guardedLookup } from "./url-guard";+import { checkOutboundUrl, checkWebhookUrl, guardedLookup } from "./url-guard";/** Stubbed so no test performs real DNS. */function resolvesTo(...addresses: { address: string; family: number }[]) {@@ -172,6 +172,58 @@ function lookup(});}+describe("checkWebhookUrl", () => {+ it("accepts http, which event webhooks have always allowed", async () => {+ resolvesTo({ address: "93.184.216.34", family: 4 });+ expect(await checkWebhookUrl("http://example.com/hook")).toEqual({+ ok: true,+ url: new URL("http://example.com/hook")+ });+ });++ it("accepts credentials in the address", async () => {+ resolvesTo({ address: "93.184.216.34", family: 4 });+ expect((await checkWebhookUrl("https://u:[email protected]/hook")).ok).toBe(+ true+ );+ });++ it("refuses a scheme other than http or https", async () => {+ expect(await checkWebhookUrl("ftp://example.com/hook")).toEqual({+ ok: false,+ reason: "Only http and https addresses are allowed.",+ retryable: false+ });+ });++ it.each([+ ["http://127.0.0.1:8080/admin", "127.0.0.1"],+ ["http://10.0.0.5/internal", "10.0.0.5"],+ ["http://169.254.169.254/latest/meta-data/", "169.254.169.254"]+ ])("refuses %s", async (url, address) => {+ resolvesTo({ address, family: 4 });+ expect(await checkWebhookUrl(url)).toEqual({+ ok: false,+ reason: "That address is inside a private network.",+ retryable: false+ });+ });++ it("refuses a public-looking name that resolves inward", async () => {+ resolvesTo({ address: "192.168.1.10", family: 4 });+ expect((await checkWebhookUrl("https://hooks.example.com")).ok).toBe(false);+ });++ it("marks a failed lookup retryable, since DNS can recover", async () => {+ vi.spyOn(dns.promises, "lookup").mockRejectedValue(new Error("EAI_AGAIN"));+ expect(await checkWebhookUrl("https://hooks.example.com")).toEqual({+ ok: false,+ reason: "That address could not be found.",+ retryable: true+ });+ });+});+describe("guardedLookup", () => {it("passes a public address through as a single answer", async () => {lookupResolvesTo({ address: "93.184.216.34", family: 4 });diff --git a/packages/jobs/src/workflows/actions/url-guard.ts b/packages/jobs/src/workflows/actions/url-guard.tsindex c01148e814..b098a33c30 100644--- a/packages/jobs/src/workflows/actions/url-guard.ts+++ b/packages/jobs/src/workflows/actions/url-guard.ts@@ -3,12 +3,15 @@// including ports, remain AGPLv3; serving them over a network requires releasing their source.import dns from "node:dns";+import http from "node:http";+import https from "node:https";import { Agent } from "undici";export type UrlVerdict = { ok: true; url: URL } | { ok: false; reason: string };const NOT_A_URL = "That is not a valid web address.";const NOT_HTTPS = "Only https addresses are allowed.";+const NOT_HTTP = "Only http and https addresses are allowed.";const NOT_FOUND = "That address could not be found.";const PRIVATE = "That address is inside a private network.";const HAS_CREDENTIALS =@@ -72,6 +75,45 @@ export async function checkOutboundUrl(raw: string): Promise<UrlVerdict> {return { ok: false, reason: HAS_CREDENTIALS };}+ return checkResolvedHost(url);+}++/**+ * A webhook verdict also says whether a refusal is worth another attempt: a+ * lookup can fail for a moment, a private address does not stop being one.+ */+export type WebhookUrlVerdict =+ | { ok: true; url: URL }+ | { ok: false; reason: string; retryable: boolean };++/**+ * The guard for an event webhook (Settings → Webhooks). Looser than+ * `checkOutboundUrl` on two points, because those webhooks predate the guard and+ * their contract never forbade either: http is allowed, and so are credentials in+ * the URL (the person who typed them sees them in settings). The private-network+ * check is the same.+ */+export async function checkWebhookUrl(raw: string): Promise<WebhookUrlVerdict> {+ let url: URL;+ try {+ url = new URL(raw);+ } catch {+ return { ok: false, reason: NOT_A_URL, retryable: false };+ }++ if (url.protocol !== "https:" && url.protocol !== "http:") {+ return { ok: false, reason: NOT_HTTP, retryable: false };+ }+ if (url.hostname.length === 0) {+ return { ok: false, reason: NOT_A_URL, retryable: false };+ }++ const verdict = await checkResolvedHost(url);+ if (verdict.ok) return verdict;+ return { ...verdict, retryable: verdict.reason === NOT_FOUND };+}++async function checkResolvedHost(url: URL): Promise<UrlVerdict> {let addresses: { address: string; family: number }[];try {addresses = await dns.promises.lookup(url.hostname, { all: true });@@ -123,3 +165,11 @@ export function guardedLookup(export const outboundDispatcher = new Agent({connect: { lookup: guardedLookup }});++/**+ * The same guard for node's http client (axios), as `httpAgent` / `httpsAgent`.+ * As with the dispatcher, a literal IP skips the lookup, so `checkWebhookUrl`+ * must still run first.+ */+export const outboundHttpAgent = new http.Agent({ lookup: guardedLookup });+export const outboundHttpsAgent = new https.Agent({ lookup: guardedLookup });