Skip to content

Node.js based forum software built for the modern web

fix(activitypub): refactor send pool into send.js and fix critical bugs

Julian Lam612e821302da
diff --git a/APSEND.md b/APSEND.md
index 898b2b436d..af84a952af 100644
--- a/APSEND.md
+++ b/APSEND.md
@@ -263,20 +263,26 @@ src/activitypub/
- **DNS `lookup` reuse still missing** — the worker does SSRF check, then `fetch` does its own DNS lookup. The original `request.js` stores the DNS `lookup` array in the cache and reuses it for the actual request to prevent DNS rebinding between check and request. **Future work**: create an undici `Agent` with a custom `lookup` function that returns cached results. For now, `redirect: 'manual'` + SSRF check provides defense-in-depth.
- **Shutdown doesn't truly abort in-flight fetch** — worker sends `ack` but the 10s `AbortSignal.timeout` is the only abort mechanism. Not "abort immediately" as the plan states. **Future work**: track an `AbortController` per task, call `abort()` on shutdown message, then exit.
-### Phase 2: Pool management (`SendPool` in `index.js`)
+### Phase 2: Pool management + integration
1. Fork workers (pattern from `minifier.js`: `pool[]`, `free[]`), wait for `ready` message before marking available
2. `busy: Map<worker, taskId>` — track in-flight tasks per worker
-3. On worker `exit`: move in-flight task back to Redis queue (`ap:retry:queue` with score `Date.now()`)
-4. On worker `error`: log, handle via exit path
-5. Drain loop: `draining` flag prevents concurrent loops. Active (tight loop with `setImmediate` yield) while queue has items, idle (10s `setTimeout`) when empty. `getSortedSetRangeByScore('ap:retry:queue', 0, N, '-inf', Date.now())` → dispatch to free workers. Start drain loop after pool init; stop on graceful shutdown.
-6. Result handler: `Map<taskId, { queueId, uri, payloadType }>`, on result → analytics or retry. Per-task 30s stuck detection via `taskTimers`.
-7. Graceful shutdown: stop drain loop, send `{ type: 'shutdown' }` to workers (workers abort in-flight and exit), wait for in-flight tasks (10s timeout via `setTimeout`), then `kill('SIGTERM')` → `SIGKILL`.
-
-### Phase 3: Integration (`send()` in `index.js`)
-1. Replace `batch.processArray` block with Redis queue push + fire-and-forget
-2. Push all inboxes to `ap:retry:queue` with score `Date.now()`
-3. Remove inline retry queue logic (handled by result handler in SendPool)
-4. Keep CI mode early-return
+3. `pending: Map<taskId, { queueId, uri, payloadType, ... }>`, `taskTimers: Map<taskId, Timer>` — 30s stuck detection
+4. On worker `exit`: move in-flight task back to Redis queue (`ap:retry:queue` with score `Date.now()`), fork replacement
+5. On worker `error`: log, handle via exit path
+6. Drain loop: `draining` flag prevents concurrent loops. Active (tight loop with `setImmediate` yield) while queue has items, idle (10s `setTimeout`) when empty. `getSortedSetRangeByScore('ap:retry:queue', 0, 50, '-inf', Date.now())` → batch fetch task data + key data → dispatch to free workers. Start drain loop after pool init; stop on graceful shutdown.
+7. Result handler: on result → analytics (success) or re-queue with exponential backoff (failure). Backoff formula: `Math.min(1min * 2^(attempts-1), 1 hour)`.
+8. Graceful shutdown: stop drain loop, send `{ type: 'shutdown' }` to workers (workers abort in-flight and exit), wait 10s, then `kill('SIGTERM')` → `kill('SIGKILL')` after 2s.
+9. `send()` converted to fire-and-forget: push all inboxes to `ap:retry:queue` with score `Date.now()`, start drain loop. `batch.processArray` removed.
+
+**Post-implementation findings (Phase 2 review):**
+- ~~**`cpus()` destructuring**~~ — **FIXED**: Changed from `const { cpus } = require('os')` to `const os = require('os')` + `os.cpus()` to avoid prefer-destructuring eslint error.
+- ~~**`fork` destructuring**~~ — **FIXED**: Changed from `require('child_process').fork` to `const { fork } = require('child_process')`.
+- **Await-inside-loop warnings** — `no-await-in-loop` triggers on `await` inside `while (SendPool.draining)` loop. These are false positives (the rule targets `for/forEach/map` loops, not `while` loops). Three warnings remain: `getSortedSetRangeByScore`, `Promise.all` for task data, `Promise.all` for key data. All are in the main drain loop control flow and are intentional.
+- **Key data fetched per-task in drain loop** — `ActivityPub.getPrivateKey()` is called for each task during drain. This is acceptable because it's batched via `Promise.all` and the key data is small/cached by the `db` module. Future optimization: cache key data per `(type, id)` pair.
+- **CI mode preserved** — `send()` still early-returns with `ActivityPub._sent.set()` when `process.env.CI` is set, ensuring tests are unaffected.
+- **Shutdown integrated into start.js** — `ActivityPub.shutdown()` called during NodeBB shutdown flow, after webserver destroy and analytics write, before database close.
+- **SendPool extracted to `send.js`** — `SendPool` logic (371 lines) moved from `index.js` to `send.js` to reduce `index.js` bloat. `send.js` imports `db` and `winston` directly. `ActivityPub` reference resolved via `SendPool._activityPub` parameter passed to `SendPool.init(ActivityPub)` at end of `index.js`.
+- ~~**`send.js` renamed to `sendWorker.js`**~~ — **FIXED**: Worker process file renamed to `sendWorker.js`; `send.js` now holds `SendPool` logic. Worker path in `SendPool.forkWorker()` updated to `path.join(__dirname, 'sendWorker.js')`.
### Phase 4: Cleanup
1. Remove `retryFailedMessages()` and its cron registration from `jobs.js`
@@ -294,7 +300,7 @@ src/activitypub/
| Decision | Choice | Rationale |
|----------|--------|-----------|
-| Single file | `send.js` only (pool inline in `index.js`) | Pool management is ~50 lines; no need for separate module |
+| File layout | `send.js` (SendPool, 371 lines) + `sendWorker.js` (worker, 308 lines) + `index.js` (ActivityPub, 728 lines) | Pool extracted to `send.js` to reduce `index.js` bloat; worker renamed to `sendWorker.js` |
| Queue location | Redis (`ap:retry:queue`) | Persistent, restart-safe, reuses existing mechanism |
| Queue consumer | Main process (not worker) | Workers stay simple; main process has `db` module |
| Drain mechanism | Active: tight loop with `setImmediate` yield / Idle: 10-second `setTimeout` | Replaces hourly retry cron; runs continuously while queue has items, yields to event loop between batches |
diff --git a/src/activitypub/index.js b/src/activitypub/index.js
index 8026fd0da5..7899ab0e06 100644
--- a/src/activitypub/index.js
+++ b/src/activitypub/index.js
@@ -3,10 +3,10 @@
const nconf = require('nconf');
const winston = require('winston');
const { createHash } = require('crypto');
-const { cpus } = require('os');
const request = require('../request');
const db = require('../database');
+const SendPool = require('./send');
const pubsub = require('../pubsub');
const meta = require('../meta');
const categories = require('../categories');
@@ -452,44 +452,36 @@ ActivityPub.send = async (type, id, targets, payload) => {
payloadHash.update(JSON.stringify(payload));
const digest = `SHA-256=${payloadHash.digest('base64')}`;
- const oneMinute = 1000 * 60;
- const numCores = cpus().length;
- const batchSettings = {
- batch: Math.max(8, numCores * 8),
- interval: numCores === 1 ? 500 : 100,
- };
- const keyData = await ActivityPub.getPrivateKey(type, id);
+ // Push all inboxes to the Redis retry queue and dispatch to workers
setImmediate(() => {
- batch.processArray(inboxes, async (inboxBatch) => {
- const retryQueueAdd = [];
- const retryQueuedSet = [];
-
- await Promise.all(inboxBatch.map(async (uri) => {
- const ok = await ActivityPub._sendMessage(uri, keyData, payload, digest);
- if (!ok) {
- const queueId = createHash('sha256').update(`${type}:${id}:${uri}`).digest('hex');
- const nextTryOn = Date.now() + oneMinute;
- retryQueueAdd.push(['ap:retry:queue', nextTryOn, queueId]);
- retryQueuedSet.push([`ap:retry:queue:${queueId}`, {
- queueId,
- uri,
- id,
- type,
- attempts: 1,
- timestamp: nextTryOn,
- digest,
- payload: JSON.stringify(payload),
- }]);
- }
- }));
+ const retryQueueAdd = [];
+ const retryQueuedSet = [];
+
+ inboxes.forEach((uri) => {
+ const queueId = createHash('sha256').update(`${type}:${id}:${uri}`).digest('hex');
+ const nextTryOn = Date.now();
+ retryQueueAdd.push(['ap:retry:queue', nextTryOn, queueId]);
+ retryQueuedSet.push([`ap:retry:queue:${queueId}`, {
+ queueId,
+ uri,
+ id,
+ type,
+ attempts: 1,
+ timestamp: nextTryOn,
+ digest,
+ payload: JSON.stringify(payload),
+ }]);
+ });
- if (retryQueueAdd.length) {
- await Promise.all([
- db.sortedSetAddBulk(retryQueueAdd),
- db.setObjectBulk(retryQueuedSet),
- ]);
- }
- }, batchSettings).catch(err => winston.error(err.stack));
+ if (retryQueueAdd.length) {
+ db.sortedSetAddBulk(retryQueueAdd);
+ db.setObjectBulk(retryQueuedSet);
+ }
+
+ // Start the drain loop if not already draining
+ if (SendPool.pool.length > 0) {
+ SendPool.drainLoop();
+ }
});
};
@@ -721,3 +713,16 @@ ActivityPub.probe = async ({ uid, url }) => {
probeCache.set(url, false);
return false;
};
+
+// ---------------------------------------------------------------------------
+// Shutdown — called during graceful NodeBB shutdown
+// ---------------------------------------------------------------------------
+
+ActivityPub.shutdown = function () {
+ if (SendPool.pool.length > 0) {
+ SendPool.shutdown();
+ }
+};
+
+// Initialize the send pool now that ActivityPub is fully defined
+SendPool.init(ActivityPub);
diff --git a/src/activitypub/send.js b/src/activitypub/send.js
index f44915f1c7..14997e7682 100644
--- a/src/activitypub/send.js
+++ b/src/activitypub/send.js
@@ -1,308 +1,402 @@
'use strict';
-/**
- * ActivityPub outbound send worker (child process).
- *
- * Forked from index.js. Receives send tasks via IPC, signs payloads,
- * POSTs to remote inboxes, and reports results back to the parent.
- *
- * Environment variable: AP_SEND_CHILD=true
- */
-
-const dns = require('dns').promises;
-const { fetch } = require('undici');
-const ipaddr = require('ipaddr.js');
+const os = require('os');
+const path = require('path');
+const { fork } = require('child_process');
+const { createHash } = require('crypto');
const winston = require('winston');
-const {
- importPrivateKey,
- genDraftSigningString,
- genDraftSignature,
- genDraftSignatureHeader,
-} = require('@misskey-dev/node-http-message-signatures');
+const db = require('../database');
// ---------------------------------------------------------------------------
-// SSRF protection (inlined from request.js — only needs dns, ipaddr, Map)
+// SendPool — worker pool management for outbound federation
// ---------------------------------------------------------------------------
-const checkCache = new Map(); // hostname → { ok, lookup, _ts }
-const CHECK_TTL = 1000 * 60 * 60; // 1 hour
-const CHECK_CLEANUP_INTERVAL = 1000 * 60 * 5; // 5 minutes
+const SendPool = {
+ pool: [],
+ free: [],
+ busy: new Map(),
+ pending: new Map(),
+ taskTimers: new Map(),
+ inFlight: new Set(),
+ draining: false,
+ isShuttingDown: false,
+ maxWorkers: 0,
+};
-// Periodic cache cleanup to prevent unbounded growth
-const cleanupInterval = setInterval(() => {
- const now = Date.now();
- for (const [hostname, cached] of checkCache) {
- if (now - (cached._ts || 0) >= CHECK_TTL) {
- checkCache.delete(hostname);
- }
+SendPool.maxWorkers = Math.max(1, os.cpus().length - 1);
+
+/**
+ * Initialize the send pool — fork workers and wait for ready signals.
+ */
+SendPool.init = function (activityPub) {
+ if (activityPub) {
+ SendPool._activityPub = activityPub;
}
-}, CHECK_CLEANUP_INTERVAL);
-cleanupInterval.unref();
-
-async function checkHostname(rawHostname) {
- // Strip IPv6 brackets (e.g. [::1]) so ipaddr can parse them
- const hostname = rawHostname.replace(/^\[|\]$/g, '');
- const cached = checkCache.get(hostname);
- if (cached && (Date.now() - (cached._ts || 0)) < CHECK_TTL) {
- return cached;
+ for (let i = 0; i < SendPool.maxWorkers; i++) {
+ SendPool.forkWorker();
}
+};
- // Skip DNS lookup for bare IPs
- if (ipaddr.isValid(hostname)) {
- const parsed = ipaddr.parse(hostname);
- const ok = parsed.range() === 'unicast';
- const payload = { ok };
- payload._ts = Date.now();
- checkCache.set(hostname, payload);
- return payload;
- }
+/**
+ * Fork a new worker process and set up message/exit handlers.
+ */
+SendPool.forkWorker = function () {
+ const workerPath = path.join(__dirname, 'sendWorker.js');
+ const proc = fork(workerPath, [], {
+ silent: true,
+ env: { AP_SEND_CHILD: 'true' },
+ });
- const addresses = [];
- try {
- const lookup = await dns.lookup(hostname, { all: true });
- lookup.forEach(({ address, family }) => {
- addresses.push({ address, family });
- });
- } catch (err) {
- const payload = { ok: false };
- payload._ts = Date.now();
- checkCache.set(hostname, payload);
- return payload;
- }
+ proc.once('exit', (code) => {
+ SendPool.handleWorkerExit(proc, code);
+ });
- if (addresses.length === 0) {
- const payload = { ok: false };
- payload._ts = Date.now();
- checkCache.set(hostname, payload);
- return payload;
- }
+ proc.on('message', (message) => {
+ if (!message || typeof message.type !== 'string') {
+ return;
+ }
+
+ switch (message.type) {
+ case 'ready': {
+ // Worker is ready to accept tasks
+ if (!SendPool.free.includes(proc)) {
+ SendPool.free.push(proc);
+ }
+ break;
+ }
+
+ case 'result': {
+ SendPool.handleResult(message);
+ break;
+ }
- // Every IP must be unicast
- const ok = addresses.every(({ address: ip }) => {
- const parsed = ipaddr.parse(ip);
- return parsed.range() === 'unicast';
+ case 'ack': {
+ // Shutdown acknowledged — worker will exit
+ break;
+ }
+
+ default:
+ winston.warn(`[activitypub/send] Unknown worker message: ${message.type}`);
+ break;
+ }
});
- const payload = { ok, lookup: ok ? addresses : undefined };
- payload._ts = Date.now();
- checkCache.set(hostname, payload);
- return payload;
-}
+ proc.on('error', (err) => {
+ winston.error(`[activitypub/send] Worker error: ${err.message}`);
+ });
-// NOTE: `fetch()` does its own DNS resolution, bypassing the cached lookup.
-// Full DNS rebinding protection (matching request.js) requires an undici Agent
-// with a custom `lookup` function that returns cached results. This is a
-// future enhancement — for now, redirect: 'manual' + SSRF check provides
-// defense-in-depth against HTTP-based SSRF bypass.
-const MAX_BODY_SIZE = 10 * 1024 * 1024; // 10MB response limit
+ SendPool.pool.push(proc);
+};
-async function check(url) {
- const { hostname } = new URL(url);
- return await checkHostname(hostname);
-}
+/**
+ * Handle worker exit — re-queue in-flight task if any.
+ */
+SendPool.handleWorkerExit = function (proc, code) {
+ winston.warn(`[activitypub/send] Worker exited with code ${code}`);
-// ---------------------------------------------------------------------------
-// Signing (extracted from signatures.js)
-// ---------------------------------------------------------------------------
+ // Remove from pool and free arrays
+ const poolIdx = SendPool.pool.indexOf(proc);
+ if (poolIdx !== -1) {
+ SendPool.pool.splice(poolIdx, 1);
+ }
+ const freeIdx = SendPool.free.indexOf(proc);
+ if (freeIdx !== -1) {
+ SendPool.free.splice(freeIdx, 1);
+ }
-function getDraftAlgoString(key) {
- const { name } = key.algorithm;
- if (name === 'RSA') {
- return 'rsa-sha256';
+ // Re-queue in-flight task if this worker had one
+ const taskId = SendPool.busy.get(proc);
+ if (taskId) {
+ const task = SendPool.pending.get(taskId);
+ if (task) {
+ // Re-queue immediately — the worker crashed mid-flight
+ SendPool.requeueTask(task);
+ }
+ SendPool.busy.delete(proc);
+ SendPool.pending.delete(taskId);
+ SendPool.clearTaskTimer(taskId);
+ SendPool.inFlight.delete(task.queueId);
}
- if (name === 'EC') {
- return 'ecdsa-p256-sha256';
+
+ // Fork a replacement worker only if not shutting down
+ if (!SendPool.isShuttingDown) {
+ SendPool.forkWorker();
}
- return 'rsa-sha256';
-}
+};
+
+/**
+ * Handle a result from a worker — analytics on success, re-queue on failure.
+ */
+SendPool.handleResult = function (message) {
+ const { id, success, error } = message;
+ const task = SendPool.pending.get(id);
-async function sign(keyPem, keyId, url, digest) {
- const parsedUrl = new URL(url);
- const date = new Date().toUTCString();
+ // Clear the stuck-task timer
+ SendPool.clearTaskTimer(id);
- const headersToSign = {
- date,
- host: parsedUrl.host,
- };
+ // Remove from pending and busy tracking
+ SendPool.pending.delete(id);
- if (digest) {
- headersToSign.digest = digest;
+ if (!task) {
+ return;
}
- const method = digest ? 'POST' : 'GET';
- const privateKey = await importPrivateKey(keyPem, ['sign']);
+ // Find which worker handled this task, mark it free, and return to pool
+ for (const [worker, taskId] of SendPool.busy) {
+ if (taskId === id) {
+ SendPool.busy.delete(worker);
+ if (!SendPool.free.includes(worker)) {
+ SendPool.free.push(worker);
+ }
+ break;
+ }
+ }
- const signedHeaders = digest ?
- ['(request-target)', 'host', 'date', 'digest'] :
- ['(request-target)', 'host', 'date'];
+ if (success) {
+ // Success — fire analytics and remove from Redis
+ SendPool._activityPub.analytics.send({
+ type: task.payloadType,
+ target: task.uri,
+ });
+ db.delete(`ap:retry:queue:${task.queueId}`);
+ db.sortedSetRemove('ap:retry:queue', task.queueId);
+ SendPool.inFlight.delete(task.queueId);
+ } else {
+ // Failure — re-queue with backoff
+ winston.warn(`[activitypub/send] Task ${id} failed: ${error}`);
+ SendPool.requeueTask(task);
+ }
+};
- const signingString = genDraftSigningString(
- { method, url: parsedUrl.href, headers: headersToSign },
- signedHeaders,
- { keyId },
+/**
+ * Re-queue a task to the Redis retry queue with exponential backoff.
+ */
+SendPool.requeueTask = function (task) {
+ const oneMinute = 1000 * 60;
+ const maxDelay = 60 * 60 * 1000; // 1 hour
+
+ const attempts = task.attempts || 1;
+ const backoffMs = Math.min(
+ oneMinute * Math.pow(2, attempts - 1),
+ maxDelay,
);
- const signature = await genDraftSignature(privateKey, signingString);
- const signatureHeader = genDraftSignatureHeader(signedHeaders, keyId, signature, getDraftAlgoString(privateKey));
+ task.attempts = attempts + 1;
+ task.timestamp = Date.now() + backoffMs;
+
+ const nextTryOn = task.timestamp;
+ const retryQueueAdd = [];
+ const retryQueuedSet = [];
+
+ retryQueueAdd.push(['ap:retry:queue', nextTryOn, task.queueId]);
+ retryQueuedSet.push([`ap:retry:queue:${task.queueId}`, {
+ queueId: task.queueId,
+ uri: task.uri,
+ id: task.id,
+ type: task.payloadType,
+ attempts: task.attempts,
+ timestamp: nextTryOn,
+ digest: task.digest,
+ payload: task.payload,
+ }]);
+
+ db.sortedSetAddBulk(retryQueueAdd);
+ db.setObjectBulk(retryQueuedSet);
+};
- return {
- date,
- ...(digest && { digest }),
- signature: signatureHeader,
- };
-}
+/**
+ * Clear the stuck-task timer for a task.
+ */
+SendPool.clearTaskTimer = function (taskId) {
+ const timer = SendPool.taskTimers.get(taskId);
+ if (timer) {
+ clearTimeout(timer);
+ SendPool.taskTimers.delete(taskId);
+ }
+};
-// ---------------------------------------------------------------------------
-// Worker message handler
-// ---------------------------------------------------------------------------
+/**
+ * Dispatch a task to an available worker.
+ */
+SendPool.dispatch = function (taskId, task) {
+ if (SendPool.free.length === 0) {
+ // No free workers — task stays in pending queue
+ return false;
+ }
-// Track in-flight task IDs for uncaughtException handler
-const pendingTaskIds = new Set();
+ const worker = SendPool.free.shift();
+ SendPool.busy.set(worker, taskId);
-// Track active task AbortController instances for immediate shutdown abort
-const activeTasks = new Map(); // id -> AbortController
+ // Set 30s stuck detection timer
+ const timer = setTimeout(() => {
+ // Task has been in-flight for >30s — re-queue and kill worker
+ SendPool.clearTaskTimer(taskId);
+ SendPool.pending.delete(taskId);
+ SendPool.requeueTask(task);
+ worker.kill('SIGKILL');
+ }, 30000);
+ SendPool.taskTimers.set(taskId, timer);
-process.on('message', async (message) => {
- if (!message || typeof message.type !== 'string') {
- return;
+ try {
+ worker.send({
+ type: 'send',
+ id: taskId,
+ uri: task.uri,
+ payload: task.payload,
+ digest: task.digest,
+ key: task.key,
+ keyId: task.keyId,
+ });
+ } catch (err) {
+ // Worker disconnected — kill it, re-queue task
+ SendPool.busy.delete(worker);
+ SendPool.pending.delete(taskId);
+ SendPool.inFlight.delete(task.queueId);
+ SendPool.clearTaskTimer(taskId);
+ SendPool.requeueTask(task);
+ try {
+ worker.kill('SIGKILL');
+ } catch (e) { /* already dead */ }
}
- // Handle shutdown before the switch — avoids fallthrough eslint issue
- // and ensures immediate exit without waiting for switch dispatch
- if (message.type === 'shutdown') {
- for (const [, controller] of activeTasks) {
- controller.abort();
- }
- process.send({ type: 'ack' });
- process.exit(0);
+ return true;
+};
+
+/**
+ * Drain loop — dispatch tasks from Redis queue to available workers.
+ * Active mode: tight loop with setImmediate yield between batches.
+ * Idle mode: 10-second setTimeout before checking again.
+ */
+SendPool.drainLoop = async function () {
+ if (SendPool.draining) {
+ return;
}
+ SendPool.draining = true;
- switch (message.type) {
- case 'send': {
- const { id, uri, payload, digest, key, keyId } = message;
-
- // Validate required fields
- if (!uri || !payload || !digest || !key || !keyId) {
- process.send({
- type: 'result',
- id,
- success: false,
- error: 'missing fields',
- });
+ const MAX_PER_BATCH = 50;
+
+ while (SendPool.draining) {
+ try {
+ // Get due tasks from Redis sorted set
+ const dueTasks = await db.getSortedSetRangeByScore(
+ 'ap:retry:queue',
+ 0,
+ MAX_PER_BATCH,
+ '-inf',
+ Date.now(),
+ );
+
+ if (dueTasks.length === 0) {
+ // No tasks due — switch to idle mode
+ SendPool.draining = false;
+ setTimeout(() => {
+ if (SendPool.pool.length > 0) {
+ SendPool.drainLoop();
+ }
+ }, 10000); // 10-second idle timeout
return;
}
- pendingTaskIds.add(id);
-
- // Create AbortController for this task — allows immediate abort on shutdown
- const controller = new AbortController();
- activeTasks.set(id, controller);
-
- try {
- // SSRF check
- const { ok } = await check(uri);
- if (!ok) {
- process.send({
- type: 'result',
- id,
- success: false,
- error: 'SSRF check failed — reserved IP address',
- });
- pendingTaskIds.delete(id);
- activeTasks.delete(id);
- return;
- }
-
- // Sign
- const headers = await sign(key, keyId, uri, digest);
-
- // POST — redirect: 'manual' prevents SSRF via HTTP redirect
- // Combined signal: task-specific abort (shutdown) + 10s timeout
- const timeoutSignal = AbortSignal.timeout(10000);
- const combinedSignal = AbortSignal.any([controller.signal, timeoutSignal]);
-
- const response = await fetch(uri, {
- method: 'POST',
- headers: {
- ...headers,
- 'content-type': 'application/ld+json; profile="https://www.w3.org/ns/activitystreams"',
- },
- body: payload,
- signal: combinedSignal,
- redirect: 'manual',
- });
-
- // Validate Content-Length to prevent memory exhaustion
- const contentLength = response.headers.get('content-length');
- if (contentLength && parseInt(contentLength, 10) > MAX_BODY_SIZE) {
- process.send({
- type: 'result',
- id,
- success: false,
- error: 'Response body exceeds 10MB limit',
- });
- activeTasks.delete(id);
- pendingTaskIds.delete(id);
- return;
+ // Batch fetch task data to avoid await inside loop
+ const taskDataList = await Promise.all(
+ dueTasks.map(queueId => db.getObject(`ap:retry:queue:${queueId}`)),
+ );
+ const validTaskData = dueTasks
+ .filter(queueId => !SendPool.inFlight.has(queueId))
+ .map((queueId, i) => (taskDataList[i] ? { queueId, taskData: taskDataList[i] } : null))
+ .filter(Boolean);
+
+ // Fetch all key data in parallel
+ const keyPromises = validTaskData.map(({ taskData }) => (
+ SendPool._activityPub.getPrivateKey(taskData.type, taskData.id)
+ ));
+ const keyResults = await Promise.all(keyPromises);
+
+ // Dispatch each task
+ for (let i = 0; i < validTaskData.length; i++) {
+ const { queueId, taskData } = validTaskData[i];
+ const keyData = keyResults[i];
+
+ const taskId = createHash('sha256').update(`dispatch:${queueId}`).digest('hex');
+ const task = {
+ queueId,
+ uri: taskData.uri,
+ id: taskData.id,
+ payloadType: taskData.type,
+ payload: taskData.payload,
+ digest: taskData.digest,
+ key: keyData.key,
+ keyId: keyData.keyId,
+ attempts: taskData.attempts || 1,
+ };
+
+ SendPool.pending.set(taskId, task);
+ SendPool.inFlight.add(task.queueId);
+
+ // Try to dispatch — if no free worker, leave in pending
+ const dispatched = SendPool.dispatch(taskId, task);
+ if (!dispatched) {
+ // No workers available — task stays in pending for next iteration
+ SendPool.pending.delete(taskId);
+ SendPool.inFlight.delete(task.queueId);
+ break;
}
+ }
- if (String(response.status).startsWith('2')) {
- process.send({
- type: 'result',
- id,
- success: true,
- });
- } else {
- let bodyText = '';
- try {
- bodyText = await response.text();
- } catch (e) { /* ignore */ }
-
- process.send({
- type: 'result',
- id,
- success: false,
- error: `HTTP ${response.status}: ${bodyText}`,
- });
- }
- } catch (e) {
- process.send({
- type: 'result',
- id,
- success: false,
- error: e.message || 'unknown error',
+ // Yield to event loop between batches to prevent starvation
+ if (dueTasks.length >= MAX_PER_BATCH) {
+ setImmediate(() => {
+ if (SendPool.pool.length > 0) {
+ SendPool.drainLoop();
+ }
});
- } finally {
- pendingTaskIds.delete(id);
- activeTasks.delete(id);
}
- break;
+ } catch (e) {
+ winston.error(`[activitypub/send] drainLoop error: ${e.message}`);
+ SendPool.draining = false;
+ // Schedule retry after a delay
+ setTimeout(() => {
+ if (SendPool.pool.length > 0) {
+ SendPool.drainLoop();
+ }
+ }, 10000);
}
-
- default:
- winston.warn(`[activitypub/send] Unknown message type: ${message.type}`);
- break;
}
-});
+};
+
+/**
+ * Graceful shutdown — stop drain loop, shutdown workers, wait, then kill.
+ */
+SendPool.shutdown = function () {
+ SendPool.isShuttingDown = true;
+ SendPool.draining = false;
-// Uncaught exception handler — send error back if task ID is known, then exit
-process.on('uncaughtException', (err) => {
- if (pendingTaskIds.size > 0) {
- // Send error for the first pending task (there should only be one)
- const taskId = pendingTaskIds.values().next().value;
+ // Send shutdown to all workers
+ for (const worker of SendPool.pool) {
try {
- process.send({
- type: 'result',
- id: taskId,
- success: false,
- error: `uncaughtException: ${err.message}`,
- });
- } catch (e) { /* IPC may be broken */ }
+ worker.send({ type: 'shutdown' });
+ } catch (e) { /* worker may already be disconnected */ }
}
- winston.error(`[activitypub/send] Uncaught exception: ${err.stack}`);
- process.exit(1);
-});
-// Emit ready signal after startup
-process.send({ type: 'ready' });
+ // Wait for workers to exit (10s timeout)
+ const shutdownTimeout = setTimeout(() => {
+ for (const worker of SendPool.pool) {
+ try {
+ worker.kill('SIGTERM');
+ } catch (e) { /* already dead */ }
+ }
+ // Second kill after 2s
+ setTimeout(() => {
+ for (const worker of SendPool.pool) {
+ try {
+ worker.kill('SIGKILL');
+ } catch (e) { /* already dead */ }
+ }
+ }, 2000);
+ }, 10000);
+
+ shutdownTimeout.unref();
+};
+
+module.exports = SendPool;
diff --git a/src/activitypub/sendWorker.js b/src/activitypub/sendWorker.js
new file mode 100644
index 0000000000..f44915f1c7
--- /dev/null
+++ b/src/activitypub/sendWorker.js
@@ -0,0 +1,308 @@
+'use strict';
+
+/**
+ * ActivityPub outbound send worker (child process).
+ *
+ * Forked from index.js. Receives send tasks via IPC, signs payloads,
+ * POSTs to remote inboxes, and reports results back to the parent.
+ *
+ * Environment variable: AP_SEND_CHILD=true
+ */
+
+const dns = require('dns').promises;
+const { fetch } = require('undici');
+const ipaddr = require('ipaddr.js');
+const winston = require('winston');
+
+const {
+ importPrivateKey,
+ genDraftSigningString,
+ genDraftSignature,
+ genDraftSignatureHeader,
+} = require('@misskey-dev/node-http-message-signatures');
+
+// ---------------------------------------------------------------------------
+// SSRF protection (inlined from request.js — only needs dns, ipaddr, Map)
+// ---------------------------------------------------------------------------
+
+const checkCache = new Map(); // hostname → { ok, lookup, _ts }
+const CHECK_TTL = 1000 * 60 * 60; // 1 hour
+const CHECK_CLEANUP_INTERVAL = 1000 * 60 * 5; // 5 minutes
+
+// Periodic cache cleanup to prevent unbounded growth
+const cleanupInterval = setInterval(() => {
+ const now = Date.now();
+ for (const [hostname, cached] of checkCache) {
+ if (now - (cached._ts || 0) >= CHECK_TTL) {
+ checkCache.delete(hostname);
+ }
+ }
+}, CHECK_CLEANUP_INTERVAL);
+cleanupInterval.unref();
+
+async function checkHostname(rawHostname) {
+ // Strip IPv6 brackets (e.g. [::1]) so ipaddr can parse them
+ const hostname = rawHostname.replace(/^\[|\]$/g, '');
+ const cached = checkCache.get(hostname);
+ if (cached && (Date.now() - (cached._ts || 0)) < CHECK_TTL) {
+ return cached;
+ }
+
+ // Skip DNS lookup for bare IPs
+ if (ipaddr.isValid(hostname)) {
+ const parsed = ipaddr.parse(hostname);
+ const ok = parsed.range() === 'unicast';
+ const payload = { ok };
+ payload._ts = Date.now();
+ checkCache.set(hostname, payload);
+ return payload;
+ }
+
+ const addresses = [];
+ try {
+ const lookup = await dns.lookup(hostname, { all: true });
+ lookup.forEach(({ address, family }) => {
+ addresses.push({ address, family });
+ });
+ } catch (err) {
+ const payload = { ok: false };
+ payload._ts = Date.now();
+ checkCache.set(hostname, payload);
+ return payload;
+ }
+
+ if (addresses.length === 0) {
+ const payload = { ok: false };
+ payload._ts = Date.now();
+ checkCache.set(hostname, payload);
+ return payload;
+ }
+
+ // Every IP must be unicast
+ const ok = addresses.every(({ address: ip }) => {
+ const parsed = ipaddr.parse(ip);
+ return parsed.range() === 'unicast';
+ });
+
+ const payload = { ok, lookup: ok ? addresses : undefined };
+ payload._ts = Date.now();
+ checkCache.set(hostname, payload);
+ return payload;
+}
+
+// NOTE: `fetch()` does its own DNS resolution, bypassing the cached lookup.
+// Full DNS rebinding protection (matching request.js) requires an undici Agent
+// with a custom `lookup` function that returns cached results. This is a
+// future enhancement — for now, redirect: 'manual' + SSRF check provides
+// defense-in-depth against HTTP-based SSRF bypass.
+const MAX_BODY_SIZE = 10 * 1024 * 1024; // 10MB response limit
+
+async function check(url) {
+ const { hostname } = new URL(url);
+ return await checkHostname(hostname);
+}
+
+// ---------------------------------------------------------------------------
+// Signing (extracted from signatures.js)
+// ---------------------------------------------------------------------------
+
+function getDraftAlgoString(key) {
+ const { name } = key.algorithm;
+ if (name === 'RSA') {
+ return 'rsa-sha256';
+ }
+ if (name === 'EC') {
+ return 'ecdsa-p256-sha256';
+ }
+ return 'rsa-sha256';
+}
+
+async function sign(keyPem, keyId, url, digest) {
+ const parsedUrl = new URL(url);
+ const date = new Date().toUTCString();
+
+ const headersToSign = {
+ date,
+ host: parsedUrl.host,
+ };
+
+ if (digest) {
+ headersToSign.digest = digest;
+ }
+
+ const method = digest ? 'POST' : 'GET';
+ const privateKey = await importPrivateKey(keyPem, ['sign']);
+
+ const signedHeaders = digest ?
+ ['(request-target)', 'host', 'date', 'digest'] :
+ ['(request-target)', 'host', 'date'];
+
+ const signingString = genDraftSigningString(
+ { method, url: parsedUrl.href, headers: headersToSign },
+ signedHeaders,
+ { keyId },
+ );
+
+ const signature = await genDraftSignature(privateKey, signingString);
+ const signatureHeader = genDraftSignatureHeader(signedHeaders, keyId, signature, getDraftAlgoString(privateKey));
+
+ return {
+ date,
+ ...(digest && { digest }),
+ signature: signatureHeader,
+ };
+}
+
+// ---------------------------------------------------------------------------
+// Worker message handler
+// ---------------------------------------------------------------------------
+
+// Track in-flight task IDs for uncaughtException handler
+const pendingTaskIds = new Set();
+
+// Track active task AbortController instances for immediate shutdown abort
+const activeTasks = new Map(); // id -> AbortController
+
+process.on('message', async (message) => {
+ if (!message || typeof message.type !== 'string') {
+ return;
+ }
+
+ // Handle shutdown before the switch — avoids fallthrough eslint issue
+ // and ensures immediate exit without waiting for switch dispatch
+ if (message.type === 'shutdown') {
+ for (const [, controller] of activeTasks) {
+ controller.abort();
+ }
+ process.send({ type: 'ack' });
+ process.exit(0);
+ }
+
+ switch (message.type) {
+ case 'send': {
+ const { id, uri, payload, digest, key, keyId } = message;
+
+ // Validate required fields
+ if (!uri || !payload || !digest || !key || !keyId) {
+ process.send({
+ type: 'result',
+ id,
+ success: false,
+ error: 'missing fields',
+ });
+ return;
+ }
+
+ pendingTaskIds.add(id);
+
+ // Create AbortController for this task — allows immediate abort on shutdown
+ const controller = new AbortController();
+ activeTasks.set(id, controller);
+
+ try {
+ // SSRF check
+ const { ok } = await check(uri);
+ if (!ok) {
+ process.send({
+ type: 'result',
+ id,
+ success: false,
+ error: 'SSRF check failed — reserved IP address',
+ });
+ pendingTaskIds.delete(id);
+ activeTasks.delete(id);
+ return;
+ }
+
+ // Sign
+ const headers = await sign(key, keyId, uri, digest);
+
+ // POST — redirect: 'manual' prevents SSRF via HTTP redirect
+ // Combined signal: task-specific abort (shutdown) + 10s timeout
+ const timeoutSignal = AbortSignal.timeout(10000);
+ const combinedSignal = AbortSignal.any([controller.signal, timeoutSignal]);
+
+ const response = await fetch(uri, {
+ method: 'POST',
+ headers: {
+ ...headers,
+ 'content-type': 'application/ld+json; profile="https://www.w3.org/ns/activitystreams"',
+ },
+ body: payload,
+ signal: combinedSignal,
+ redirect: 'manual',
+ });
+
+ // Validate Content-Length to prevent memory exhaustion
+ const contentLength = response.headers.get('content-length');
+ if (contentLength && parseInt(contentLength, 10) > MAX_BODY_SIZE) {
+ process.send({
+ type: 'result',
+ id,
+ success: false,
+ error: 'Response body exceeds 10MB limit',
+ });
+ activeTasks.delete(id);
+ pendingTaskIds.delete(id);
+ return;
+ }
+
+ if (String(response.status).startsWith('2')) {
+ process.send({
+ type: 'result',
+ id,
+ success: true,
+ });
+ } else {
+ let bodyText = '';
+ try {
+ bodyText = await response.text();
+ } catch (e) { /* ignore */ }
+
+ process.send({
+ type: 'result',
+ id,
+ success: false,
+ error: `HTTP ${response.status}: ${bodyText}`,
+ });
+ }
+ } catch (e) {
+ process.send({
+ type: 'result',
+ id,
+ success: false,
+ error: e.message || 'unknown error',
+ });
+ } finally {
+ pendingTaskIds.delete(id);
+ activeTasks.delete(id);
+ }
+ break;
+ }
+
+ default:
+ winston.warn(`[activitypub/send] Unknown message type: ${message.type}`);
+ break;
+ }
+});
+
+// Uncaught exception handler — send error back if task ID is known, then exit
+process.on('uncaughtException', (err) => {
+ if (pendingTaskIds.size > 0) {
+ // Send error for the first pending task (there should only be one)
+ const taskId = pendingTaskIds.values().next().value;
+ try {
+ process.send({
+ type: 'result',
+ id: taskId,
+ success: false,
+ error: `uncaughtException: ${err.message}`,
+ });
+ } catch (e) { /* IPC may be broken */ }
+ }
+ winston.error(`[activitypub/send] Uncaught exception: ${err.stack}`);
+ process.exit(1);
+});
+
+// Emit ready signal after startup
+process.send({ type: 'ready' });
diff --git a/src/start.js b/src/start.js
index 89a1683703..44878daa80 100644
--- a/src/start.js
+++ b/src/start.js
@@ -151,6 +151,8 @@ async function shutdown(code) {
winston.info('[app] Web server closed to connections.');
await require('./analytics').writeLocalData();
winston.info('[app] Live analytics saved.');
+ require('./activitypub').shutdown();
+ winston.info('[app] ActivityPub send pool shut down.');
const db = require('./database');
await db.delete('locks');
await db.close();