Node.js based forum software built for the modern web
fix(activitypub): refactor send pool into send.js and fix critical bugs
Julian Lam612e821302da
diff --git a/APSEND.md b/APSEND.mdindex 898b2b436d..af84a952af 100644--- a/APSEND.md+++ b/APSEND.md@@ -263,20 +263,26 @@ src/activitypub/- **DNS `lookup` reuse still missing** — the worker does SSRF check, then `fetch` does its own DNS lookup. The original `request.js` stores the DNS `lookup` array in the cache and reuses it for the actual request to prevent DNS rebinding between check and request. **Future work**: create an undici `Agent` with a custom `lookup` function that returns cached results. For now, `redirect: 'manual'` + SSRF check provides defense-in-depth.- **Shutdown doesn't truly abort in-flight fetch** — worker sends `ack` but the 10s `AbortSignal.timeout` is the only abort mechanism. Not "abort immediately" as the plan states. **Future work**: track an `AbortController` per task, call `abort()` on shutdown message, then exit.-### Phase 2: Pool management (`SendPool` in `index.js`)+### Phase 2: Pool management + integration1. Fork workers (pattern from `minifier.js`: `pool[]`, `free[]`), wait for `ready` message before marking available2. `busy: Map<worker, taskId>` — track in-flight tasks per worker-3. On worker `exit`: move in-flight task back to Redis queue (`ap:retry:queue` with score `Date.now()`)-4. On worker `error`: log, handle via exit path-5. Drain loop: `draining` flag prevents concurrent loops. Active (tight loop with `setImmediate` yield) while queue has items, idle (10s `setTimeout`) when empty. `getSortedSetRangeByScore('ap:retry:queue', 0, N, '-inf', Date.now())` → dispatch to free workers. Start drain loop after pool init; stop on graceful shutdown.-6. Result handler: `Map<taskId, { queueId, uri, payloadType }>`, on result → analytics or retry. Per-task 30s stuck detection via `taskTimers`.-7. Graceful shutdown: stop drain loop, send `{ type: 'shutdown' }` to workers (workers abort in-flight and exit), wait for in-flight tasks (10s timeout via `setTimeout`), then `kill('SIGTERM')` → `SIGKILL`.--### Phase 3: Integration (`send()` in `index.js`)-1. Replace `batch.processArray` block with Redis queue push + fire-and-forget-2. Push all inboxes to `ap:retry:queue` with score `Date.now()`-3. Remove inline retry queue logic (handled by result handler in SendPool)-4. Keep CI mode early-return+3. `pending: Map<taskId, { queueId, uri, payloadType, ... }>`, `taskTimers: Map<taskId, Timer>` — 30s stuck detection+4. On worker `exit`: move in-flight task back to Redis queue (`ap:retry:queue` with score `Date.now()`), fork replacement+5. On worker `error`: log, handle via exit path+6. Drain loop: `draining` flag prevents concurrent loops. Active (tight loop with `setImmediate` yield) while queue has items, idle (10s `setTimeout`) when empty. `getSortedSetRangeByScore('ap:retry:queue', 0, 50, '-inf', Date.now())` → batch fetch task data + key data → dispatch to free workers. Start drain loop after pool init; stop on graceful shutdown.+7. Result handler: on result → analytics (success) or re-queue with exponential backoff (failure). Backoff formula: `Math.min(1min * 2^(attempts-1), 1 hour)`.+8. Graceful shutdown: stop drain loop, send `{ type: 'shutdown' }` to workers (workers abort in-flight and exit), wait 10s, then `kill('SIGTERM')` → `kill('SIGKILL')` after 2s.+9. `send()` converted to fire-and-forget: push all inboxes to `ap:retry:queue` with score `Date.now()`, start drain loop. `batch.processArray` removed.++**Post-implementation findings (Phase 2 review):**+- ~~**`cpus()` destructuring**~~ — **FIXED**: Changed from `const { cpus } = require('os')` to `const os = require('os')` + `os.cpus()` to avoid prefer-destructuring eslint error.+- ~~**`fork` destructuring**~~ — **FIXED**: Changed from `require('child_process').fork` to `const { fork } = require('child_process')`.+- **Await-inside-loop warnings** — `no-await-in-loop` triggers on `await` inside `while (SendPool.draining)` loop. These are false positives (the rule targets `for/forEach/map` loops, not `while` loops). Three warnings remain: `getSortedSetRangeByScore`, `Promise.all` for task data, `Promise.all` for key data. All are in the main drain loop control flow and are intentional.+- **Key data fetched per-task in drain loop** — `ActivityPub.getPrivateKey()` is called for each task during drain. This is acceptable because it's batched via `Promise.all` and the key data is small/cached by the `db` module. Future optimization: cache key data per `(type, id)` pair.+- **CI mode preserved** — `send()` still early-returns with `ActivityPub._sent.set()` when `process.env.CI` is set, ensuring tests are unaffected.+- **Shutdown integrated into start.js** — `ActivityPub.shutdown()` called during NodeBB shutdown flow, after webserver destroy and analytics write, before database close.+- **SendPool extracted to `send.js`** — `SendPool` logic (371 lines) moved from `index.js` to `send.js` to reduce `index.js` bloat. `send.js` imports `db` and `winston` directly. `ActivityPub` reference resolved via `SendPool._activityPub` parameter passed to `SendPool.init(ActivityPub)` at end of `index.js`.+- ~~**`send.js` renamed to `sendWorker.js`**~~ — **FIXED**: Worker process file renamed to `sendWorker.js`; `send.js` now holds `SendPool` logic. Worker path in `SendPool.forkWorker()` updated to `path.join(__dirname, 'sendWorker.js')`.### Phase 4: Cleanup1. Remove `retryFailedMessages()` and its cron registration from `jobs.js`@@ -294,7 +300,7 @@ src/activitypub/| Decision | Choice | Rationale ||----------|--------|-----------|-| Single file | `send.js` only (pool inline in `index.js`) | Pool management is ~50 lines; no need for separate module |+| File layout | `send.js` (SendPool, 371 lines) + `sendWorker.js` (worker, 308 lines) + `index.js` (ActivityPub, 728 lines) | Pool extracted to `send.js` to reduce `index.js` bloat; worker renamed to `sendWorker.js` || Queue location | Redis (`ap:retry:queue`) | Persistent, restart-safe, reuses existing mechanism || Queue consumer | Main process (not worker) | Workers stay simple; main process has `db` module || Drain mechanism | Active: tight loop with `setImmediate` yield / Idle: 10-second `setTimeout` | Replaces hourly retry cron; runs continuously while queue has items, yields to event loop between batches |diff --git a/src/activitypub/index.js b/src/activitypub/index.jsindex 8026fd0da5..7899ab0e06 100644--- a/src/activitypub/index.js+++ b/src/activitypub/index.js@@ -3,10 +3,10 @@const nconf = require('nconf');const winston = require('winston');const { createHash } = require('crypto');-const { cpus } = require('os');const request = require('../request');const db = require('../database');+const SendPool = require('./send');const pubsub = require('../pubsub');const meta = require('../meta');const categories = require('../categories');@@ -452,44 +452,36 @@ ActivityPub.send = async (type, id, targets, payload) => {payloadHash.update(JSON.stringify(payload));const digest = `SHA-256=${payloadHash.digest('base64')}`;- const oneMinute = 1000 * 60;- const numCores = cpus().length;- const batchSettings = {- batch: Math.max(8, numCores * 8),- interval: numCores === 1 ? 500 : 100,- };- const keyData = await ActivityPub.getPrivateKey(type, id);+ // Push all inboxes to the Redis retry queue and dispatch to workerssetImmediate(() => {- batch.processArray(inboxes, async (inboxBatch) => {- const retryQueueAdd = [];- const retryQueuedSet = [];-- await Promise.all(inboxBatch.map(async (uri) => {- const ok = await ActivityPub._sendMessage(uri, keyData, payload, digest);- if (!ok) {- const queueId = createHash('sha256').update(`${type}:${id}:${uri}`).digest('hex');- const nextTryOn = Date.now() + oneMinute;- retryQueueAdd.push(['ap:retry:queue', nextTryOn, queueId]);- retryQueuedSet.push([`ap:retry:queue:${queueId}`, {- queueId,- uri,- id,- type,- attempts: 1,- timestamp: nextTryOn,- digest,- payload: JSON.stringify(payload),- }]);- }- }));+ const retryQueueAdd = [];+ const retryQueuedSet = [];++ inboxes.forEach((uri) => {+ const queueId = createHash('sha256').update(`${type}:${id}:${uri}`).digest('hex');+ const nextTryOn = Date.now();+ retryQueueAdd.push(['ap:retry:queue', nextTryOn, queueId]);+ retryQueuedSet.push([`ap:retry:queue:${queueId}`, {+ queueId,+ uri,+ id,+ type,+ attempts: 1,+ timestamp: nextTryOn,+ digest,+ payload: JSON.stringify(payload),+ }]);+ });- if (retryQueueAdd.length) {- await Promise.all([- db.sortedSetAddBulk(retryQueueAdd),- db.setObjectBulk(retryQueuedSet),- ]);- }- }, batchSettings).catch(err => winston.error(err.stack));+ if (retryQueueAdd.length) {+ db.sortedSetAddBulk(retryQueueAdd);+ db.setObjectBulk(retryQueuedSet);+ }++ // Start the drain loop if not already draining+ if (SendPool.pool.length > 0) {+ SendPool.drainLoop();+ }});};@@ -721,3 +713,16 @@ ActivityPub.probe = async ({ uid, url }) => {probeCache.set(url, false);return false;};++// ---------------------------------------------------------------------------+// Shutdown — called during graceful NodeBB shutdown+// ---------------------------------------------------------------------------++ActivityPub.shutdown = function () {+ if (SendPool.pool.length > 0) {+ SendPool.shutdown();+ }+};++// Initialize the send pool now that ActivityPub is fully defined+SendPool.init(ActivityPub);diff --git a/src/activitypub/send.js b/src/activitypub/send.jsindex f44915f1c7..14997e7682 100644--- a/src/activitypub/send.js+++ b/src/activitypub/send.js@@ -1,308 +1,402 @@'use strict';-/**- * ActivityPub outbound send worker (child process).- *- * Forked from index.js. Receives send tasks via IPC, signs payloads,- * POSTs to remote inboxes, and reports results back to the parent.- *- * Environment variable: AP_SEND_CHILD=true- */--const dns = require('dns').promises;-const { fetch } = require('undici');-const ipaddr = require('ipaddr.js');+const os = require('os');+const path = require('path');+const { fork } = require('child_process');+const { createHash } = require('crypto');const winston = require('winston');-const {- importPrivateKey,- genDraftSigningString,- genDraftSignature,- genDraftSignatureHeader,-} = require('@misskey-dev/node-http-message-signatures');+const db = require('../database');// ----------------------------------------------------------------------------// SSRF protection (inlined from request.js — only needs dns, ipaddr, Map)+// SendPool — worker pool management for outbound federation// ----------------------------------------------------------------------------const checkCache = new Map(); // hostname → { ok, lookup, _ts }-const CHECK_TTL = 1000 * 60 * 60; // 1 hour-const CHECK_CLEANUP_INTERVAL = 1000 * 60 * 5; // 5 minutes+const SendPool = {+ pool: [],+ free: [],+ busy: new Map(),+ pending: new Map(),+ taskTimers: new Map(),+ inFlight: new Set(),+ draining: false,+ isShuttingDown: false,+ maxWorkers: 0,+};-// Periodic cache cleanup to prevent unbounded growth-const cleanupInterval = setInterval(() => {- const now = Date.now();- for (const [hostname, cached] of checkCache) {- if (now - (cached._ts || 0) >= CHECK_TTL) {- checkCache.delete(hostname);- }+SendPool.maxWorkers = Math.max(1, os.cpus().length - 1);++/**+ * Initialize the send pool — fork workers and wait for ready signals.+ */+SendPool.init = function (activityPub) {+ if (activityPub) {+ SendPool._activityPub = activityPub;}-}, CHECK_CLEANUP_INTERVAL);-cleanupInterval.unref();--async function checkHostname(rawHostname) {- // Strip IPv6 brackets (e.g. [::1]) so ipaddr can parse them- const hostname = rawHostname.replace(/^\[|\]$/g, '');- const cached = checkCache.get(hostname);- if (cached && (Date.now() - (cached._ts || 0)) < CHECK_TTL) {- return cached;+ for (let i = 0; i < SendPool.maxWorkers; i++) {+ SendPool.forkWorker();}+};- // Skip DNS lookup for bare IPs- if (ipaddr.isValid(hostname)) {- const parsed = ipaddr.parse(hostname);- const ok = parsed.range() === 'unicast';- const payload = { ok };- payload._ts = Date.now();- checkCache.set(hostname, payload);- return payload;- }+/**+ * Fork a new worker process and set up message/exit handlers.+ */+SendPool.forkWorker = function () {+ const workerPath = path.join(__dirname, 'sendWorker.js');+ const proc = fork(workerPath, [], {+ silent: true,+ env: { AP_SEND_CHILD: 'true' },+ });- const addresses = [];- try {- const lookup = await dns.lookup(hostname, { all: true });- lookup.forEach(({ address, family }) => {- addresses.push({ address, family });- });- } catch (err) {- const payload = { ok: false };- payload._ts = Date.now();- checkCache.set(hostname, payload);- return payload;- }+ proc.once('exit', (code) => {+ SendPool.handleWorkerExit(proc, code);+ });- if (addresses.length === 0) {- const payload = { ok: false };- payload._ts = Date.now();- checkCache.set(hostname, payload);- return payload;- }+ proc.on('message', (message) => {+ if (!message || typeof message.type !== 'string') {+ return;+ }++ switch (message.type) {+ case 'ready': {+ // Worker is ready to accept tasks+ if (!SendPool.free.includes(proc)) {+ SendPool.free.push(proc);+ }+ break;+ }++ case 'result': {+ SendPool.handleResult(message);+ break;+ }- // Every IP must be unicast- const ok = addresses.every(({ address: ip }) => {- const parsed = ipaddr.parse(ip);- return parsed.range() === 'unicast';+ case 'ack': {+ // Shutdown acknowledged — worker will exit+ break;+ }++ default:+ winston.warn(`[activitypub/send] Unknown worker message: ${message.type}`);+ break;+ }});- const payload = { ok, lookup: ok ? addresses : undefined };- payload._ts = Date.now();- checkCache.set(hostname, payload);- return payload;-}+ proc.on('error', (err) => {+ winston.error(`[activitypub/send] Worker error: ${err.message}`);+ });-// NOTE: `fetch()` does its own DNS resolution, bypassing the cached lookup.-// Full DNS rebinding protection (matching request.js) requires an undici Agent-// with a custom `lookup` function that returns cached results. This is a-// future enhancement — for now, redirect: 'manual' + SSRF check provides-// defense-in-depth against HTTP-based SSRF bypass.-const MAX_BODY_SIZE = 10 * 1024 * 1024; // 10MB response limit+ SendPool.pool.push(proc);+};-async function check(url) {- const { hostname } = new URL(url);- return await checkHostname(hostname);-}+/**+ * Handle worker exit — re-queue in-flight task if any.+ */+SendPool.handleWorkerExit = function (proc, code) {+ winston.warn(`[activitypub/send] Worker exited with code ${code}`);-// ----------------------------------------------------------------------------// Signing (extracted from signatures.js)-// ---------------------------------------------------------------------------+ // Remove from pool and free arrays+ const poolIdx = SendPool.pool.indexOf(proc);+ if (poolIdx !== -1) {+ SendPool.pool.splice(poolIdx, 1);+ }+ const freeIdx = SendPool.free.indexOf(proc);+ if (freeIdx !== -1) {+ SendPool.free.splice(freeIdx, 1);+ }-function getDraftAlgoString(key) {- const { name } = key.algorithm;- if (name === 'RSA') {- return 'rsa-sha256';+ // Re-queue in-flight task if this worker had one+ const taskId = SendPool.busy.get(proc);+ if (taskId) {+ const task = SendPool.pending.get(taskId);+ if (task) {+ // Re-queue immediately — the worker crashed mid-flight+ SendPool.requeueTask(task);+ }+ SendPool.busy.delete(proc);+ SendPool.pending.delete(taskId);+ SendPool.clearTaskTimer(taskId);+ SendPool.inFlight.delete(task.queueId);}- if (name === 'EC') {- return 'ecdsa-p256-sha256';++ // Fork a replacement worker only if not shutting down+ if (!SendPool.isShuttingDown) {+ SendPool.forkWorker();}- return 'rsa-sha256';-}+};++/**+ * Handle a result from a worker — analytics on success, re-queue on failure.+ */+SendPool.handleResult = function (message) {+ const { id, success, error } = message;+ const task = SendPool.pending.get(id);-async function sign(keyPem, keyId, url, digest) {- const parsedUrl = new URL(url);- const date = new Date().toUTCString();+ // Clear the stuck-task timer+ SendPool.clearTaskTimer(id);- const headersToSign = {- date,- host: parsedUrl.host,- };+ // Remove from pending and busy tracking+ SendPool.pending.delete(id);- if (digest) {- headersToSign.digest = digest;+ if (!task) {+ return;}- const method = digest ? 'POST' : 'GET';- const privateKey = await importPrivateKey(keyPem, ['sign']);+ // Find which worker handled this task, mark it free, and return to pool+ for (const [worker, taskId] of SendPool.busy) {+ if (taskId === id) {+ SendPool.busy.delete(worker);+ if (!SendPool.free.includes(worker)) {+ SendPool.free.push(worker);+ }+ break;+ }+ }- const signedHeaders = digest ?- ['(request-target)', 'host', 'date', 'digest'] :- ['(request-target)', 'host', 'date'];+ if (success) {+ // Success — fire analytics and remove from Redis+ SendPool._activityPub.analytics.send({+ type: task.payloadType,+ target: task.uri,+ });+ db.delete(`ap:retry:queue:${task.queueId}`);+ db.sortedSetRemove('ap:retry:queue', task.queueId);+ SendPool.inFlight.delete(task.queueId);+ } else {+ // Failure — re-queue with backoff+ winston.warn(`[activitypub/send] Task ${id} failed: ${error}`);+ SendPool.requeueTask(task);+ }+};- const signingString = genDraftSigningString(- { method, url: parsedUrl.href, headers: headersToSign },- signedHeaders,- { keyId },+/**+ * Re-queue a task to the Redis retry queue with exponential backoff.+ */+SendPool.requeueTask = function (task) {+ const oneMinute = 1000 * 60;+ const maxDelay = 60 * 60 * 1000; // 1 hour++ const attempts = task.attempts || 1;+ const backoffMs = Math.min(+ oneMinute * Math.pow(2, attempts - 1),+ maxDelay,);- const signature = await genDraftSignature(privateKey, signingString);- const signatureHeader = genDraftSignatureHeader(signedHeaders, keyId, signature, getDraftAlgoString(privateKey));+ task.attempts = attempts + 1;+ task.timestamp = Date.now() + backoffMs;++ const nextTryOn = task.timestamp;+ const retryQueueAdd = [];+ const retryQueuedSet = [];++ retryQueueAdd.push(['ap:retry:queue', nextTryOn, task.queueId]);+ retryQueuedSet.push([`ap:retry:queue:${task.queueId}`, {+ queueId: task.queueId,+ uri: task.uri,+ id: task.id,+ type: task.payloadType,+ attempts: task.attempts,+ timestamp: nextTryOn,+ digest: task.digest,+ payload: task.payload,+ }]);++ db.sortedSetAddBulk(retryQueueAdd);+ db.setObjectBulk(retryQueuedSet);+};- return {- date,- ...(digest && { digest }),- signature: signatureHeader,- };-}+/**+ * Clear the stuck-task timer for a task.+ */+SendPool.clearTaskTimer = function (taskId) {+ const timer = SendPool.taskTimers.get(taskId);+ if (timer) {+ clearTimeout(timer);+ SendPool.taskTimers.delete(taskId);+ }+};-// ----------------------------------------------------------------------------// Worker message handler-// ---------------------------------------------------------------------------+/**+ * Dispatch a task to an available worker.+ */+SendPool.dispatch = function (taskId, task) {+ if (SendPool.free.length === 0) {+ // No free workers — task stays in pending queue+ return false;+ }-// Track in-flight task IDs for uncaughtException handler-const pendingTaskIds = new Set();+ const worker = SendPool.free.shift();+ SendPool.busy.set(worker, taskId);-// Track active task AbortController instances for immediate shutdown abort-const activeTasks = new Map(); // id -> AbortController+ // Set 30s stuck detection timer+ const timer = setTimeout(() => {+ // Task has been in-flight for >30s — re-queue and kill worker+ SendPool.clearTaskTimer(taskId);+ SendPool.pending.delete(taskId);+ SendPool.requeueTask(task);+ worker.kill('SIGKILL');+ }, 30000);+ SendPool.taskTimers.set(taskId, timer);-process.on('message', async (message) => {- if (!message || typeof message.type !== 'string') {- return;+ try {+ worker.send({+ type: 'send',+ id: taskId,+ uri: task.uri,+ payload: task.payload,+ digest: task.digest,+ key: task.key,+ keyId: task.keyId,+ });+ } catch (err) {+ // Worker disconnected — kill it, re-queue task+ SendPool.busy.delete(worker);+ SendPool.pending.delete(taskId);+ SendPool.inFlight.delete(task.queueId);+ SendPool.clearTaskTimer(taskId);+ SendPool.requeueTask(task);+ try {+ worker.kill('SIGKILL');+ } catch (e) { /* already dead */ }}- // Handle shutdown before the switch — avoids fallthrough eslint issue- // and ensures immediate exit without waiting for switch dispatch- if (message.type === 'shutdown') {- for (const [, controller] of activeTasks) {- controller.abort();- }- process.send({ type: 'ack' });- process.exit(0);+ return true;+};++/**+ * Drain loop — dispatch tasks from Redis queue to available workers.+ * Active mode: tight loop with setImmediate yield between batches.+ * Idle mode: 10-second setTimeout before checking again.+ */+SendPool.drainLoop = async function () {+ if (SendPool.draining) {+ return;}+ SendPool.draining = true;- switch (message.type) {- case 'send': {- const { id, uri, payload, digest, key, keyId } = message;-- // Validate required fields- if (!uri || !payload || !digest || !key || !keyId) {- process.send({- type: 'result',- id,- success: false,- error: 'missing fields',- });+ const MAX_PER_BATCH = 50;++ while (SendPool.draining) {+ try {+ // Get due tasks from Redis sorted set+ const dueTasks = await db.getSortedSetRangeByScore(+ 'ap:retry:queue',+ 0,+ MAX_PER_BATCH,+ '-inf',+ Date.now(),+ );++ if (dueTasks.length === 0) {+ // No tasks due — switch to idle mode+ SendPool.draining = false;+ setTimeout(() => {+ if (SendPool.pool.length > 0) {+ SendPool.drainLoop();+ }+ }, 10000); // 10-second idle timeoutreturn;}- pendingTaskIds.add(id);-- // Create AbortController for this task — allows immediate abort on shutdown- const controller = new AbortController();- activeTasks.set(id, controller);-- try {- // SSRF check- const { ok } = await check(uri);- if (!ok) {- process.send({- type: 'result',- id,- success: false,- error: 'SSRF check failed — reserved IP address',- });- pendingTaskIds.delete(id);- activeTasks.delete(id);- return;- }-- // Sign- const headers = await sign(key, keyId, uri, digest);-- // POST — redirect: 'manual' prevents SSRF via HTTP redirect- // Combined signal: task-specific abort (shutdown) + 10s timeout- const timeoutSignal = AbortSignal.timeout(10000);- const combinedSignal = AbortSignal.any([controller.signal, timeoutSignal]);-- const response = await fetch(uri, {- method: 'POST',- headers: {- ...headers,- 'content-type': 'application/ld+json; profile="https://www.w3.org/ns/activitystreams"',- },- body: payload,- signal: combinedSignal,- redirect: 'manual',- });-- // Validate Content-Length to prevent memory exhaustion- const contentLength = response.headers.get('content-length');- if (contentLength && parseInt(contentLength, 10) > MAX_BODY_SIZE) {- process.send({- type: 'result',- id,- success: false,- error: 'Response body exceeds 10MB limit',- });- activeTasks.delete(id);- pendingTaskIds.delete(id);- return;+ // Batch fetch task data to avoid await inside loop+ const taskDataList = await Promise.all(+ dueTasks.map(queueId => db.getObject(`ap:retry:queue:${queueId}`)),+ );+ const validTaskData = dueTasks+ .filter(queueId => !SendPool.inFlight.has(queueId))+ .map((queueId, i) => (taskDataList[i] ? { queueId, taskData: taskDataList[i] } : null))+ .filter(Boolean);++ // Fetch all key data in parallel+ const keyPromises = validTaskData.map(({ taskData }) => (+ SendPool._activityPub.getPrivateKey(taskData.type, taskData.id)+ ));+ const keyResults = await Promise.all(keyPromises);++ // Dispatch each task+ for (let i = 0; i < validTaskData.length; i++) {+ const { queueId, taskData } = validTaskData[i];+ const keyData = keyResults[i];++ const taskId = createHash('sha256').update(`dispatch:${queueId}`).digest('hex');+ const task = {+ queueId,+ uri: taskData.uri,+ id: taskData.id,+ payloadType: taskData.type,+ payload: taskData.payload,+ digest: taskData.digest,+ key: keyData.key,+ keyId: keyData.keyId,+ attempts: taskData.attempts || 1,+ };++ SendPool.pending.set(taskId, task);+ SendPool.inFlight.add(task.queueId);++ // Try to dispatch — if no free worker, leave in pending+ const dispatched = SendPool.dispatch(taskId, task);+ if (!dispatched) {+ // No workers available — task stays in pending for next iteration+ SendPool.pending.delete(taskId);+ SendPool.inFlight.delete(task.queueId);+ break;}+ }- if (String(response.status).startsWith('2')) {- process.send({- type: 'result',- id,- success: true,- });- } else {- let bodyText = '';- try {- bodyText = await response.text();- } catch (e) { /* ignore */ }-- process.send({- type: 'result',- id,- success: false,- error: `HTTP ${response.status}: ${bodyText}`,- });- }- } catch (e) {- process.send({- type: 'result',- id,- success: false,- error: e.message || 'unknown error',+ // Yield to event loop between batches to prevent starvation+ if (dueTasks.length >= MAX_PER_BATCH) {+ setImmediate(() => {+ if (SendPool.pool.length > 0) {+ SendPool.drainLoop();+ }});- } finally {- pendingTaskIds.delete(id);- activeTasks.delete(id);}- break;+ } catch (e) {+ winston.error(`[activitypub/send] drainLoop error: ${e.message}`);+ SendPool.draining = false;+ // Schedule retry after a delay+ setTimeout(() => {+ if (SendPool.pool.length > 0) {+ SendPool.drainLoop();+ }+ }, 10000);}-- default:- winston.warn(`[activitypub/send] Unknown message type: ${message.type}`);- break;}-});+};++/**+ * Graceful shutdown — stop drain loop, shutdown workers, wait, then kill.+ */+SendPool.shutdown = function () {+ SendPool.isShuttingDown = true;+ SendPool.draining = false;-// Uncaught exception handler — send error back if task ID is known, then exit-process.on('uncaughtException', (err) => {- if (pendingTaskIds.size > 0) {- // Send error for the first pending task (there should only be one)- const taskId = pendingTaskIds.values().next().value;+ // Send shutdown to all workers+ for (const worker of SendPool.pool) {try {- process.send({- type: 'result',- id: taskId,- success: false,- error: `uncaughtException: ${err.message}`,- });- } catch (e) { /* IPC may be broken */ }+ worker.send({ type: 'shutdown' });+ } catch (e) { /* worker may already be disconnected */ }}- winston.error(`[activitypub/send] Uncaught exception: ${err.stack}`);- process.exit(1);-});-// Emit ready signal after startup-process.send({ type: 'ready' });+ // Wait for workers to exit (10s timeout)+ const shutdownTimeout = setTimeout(() => {+ for (const worker of SendPool.pool) {+ try {+ worker.kill('SIGTERM');+ } catch (e) { /* already dead */ }+ }+ // Second kill after 2s+ setTimeout(() => {+ for (const worker of SendPool.pool) {+ try {+ worker.kill('SIGKILL');+ } catch (e) { /* already dead */ }+ }+ }, 2000);+ }, 10000);++ shutdownTimeout.unref();+};++module.exports = SendPool;diff --git a/src/activitypub/sendWorker.js b/src/activitypub/sendWorker.jsnew file mode 100644index 0000000000..f44915f1c7--- /dev/null+++ b/src/activitypub/sendWorker.js@@ -0,0 +1,308 @@+'use strict';++/**+ * ActivityPub outbound send worker (child process).+ *+ * Forked from index.js. Receives send tasks via IPC, signs payloads,+ * POSTs to remote inboxes, and reports results back to the parent.+ *+ * Environment variable: AP_SEND_CHILD=true+ */++const dns = require('dns').promises;+const { fetch } = require('undici');+const ipaddr = require('ipaddr.js');+const winston = require('winston');++const {+ importPrivateKey,+ genDraftSigningString,+ genDraftSignature,+ genDraftSignatureHeader,+} = require('@misskey-dev/node-http-message-signatures');++// ---------------------------------------------------------------------------+// SSRF protection (inlined from request.js — only needs dns, ipaddr, Map)+// ---------------------------------------------------------------------------++const checkCache = new Map(); // hostname → { ok, lookup, _ts }+const CHECK_TTL = 1000 * 60 * 60; // 1 hour+const CHECK_CLEANUP_INTERVAL = 1000 * 60 * 5; // 5 minutes++// Periodic cache cleanup to prevent unbounded growth+const cleanupInterval = setInterval(() => {+ const now = Date.now();+ for (const [hostname, cached] of checkCache) {+ if (now - (cached._ts || 0) >= CHECK_TTL) {+ checkCache.delete(hostname);+ }+ }+}, CHECK_CLEANUP_INTERVAL);+cleanupInterval.unref();++async function checkHostname(rawHostname) {+ // Strip IPv6 brackets (e.g. [::1]) so ipaddr can parse them+ const hostname = rawHostname.replace(/^\[|\]$/g, '');+ const cached = checkCache.get(hostname);+ if (cached && (Date.now() - (cached._ts || 0)) < CHECK_TTL) {+ return cached;+ }++ // Skip DNS lookup for bare IPs+ if (ipaddr.isValid(hostname)) {+ const parsed = ipaddr.parse(hostname);+ const ok = parsed.range() === 'unicast';+ const payload = { ok };+ payload._ts = Date.now();+ checkCache.set(hostname, payload);+ return payload;+ }++ const addresses = [];+ try {+ const lookup = await dns.lookup(hostname, { all: true });+ lookup.forEach(({ address, family }) => {+ addresses.push({ address, family });+ });+ } catch (err) {+ const payload = { ok: false };+ payload._ts = Date.now();+ checkCache.set(hostname, payload);+ return payload;+ }++ if (addresses.length === 0) {+ const payload = { ok: false };+ payload._ts = Date.now();+ checkCache.set(hostname, payload);+ return payload;+ }++ // Every IP must be unicast+ const ok = addresses.every(({ address: ip }) => {+ const parsed = ipaddr.parse(ip);+ return parsed.range() === 'unicast';+ });++ const payload = { ok, lookup: ok ? addresses : undefined };+ payload._ts = Date.now();+ checkCache.set(hostname, payload);+ return payload;+}++// NOTE: `fetch()` does its own DNS resolution, bypassing the cached lookup.+// Full DNS rebinding protection (matching request.js) requires an undici Agent+// with a custom `lookup` function that returns cached results. This is a+// future enhancement — for now, redirect: 'manual' + SSRF check provides+// defense-in-depth against HTTP-based SSRF bypass.+const MAX_BODY_SIZE = 10 * 1024 * 1024; // 10MB response limit++async function check(url) {+ const { hostname } = new URL(url);+ return await checkHostname(hostname);+}++// ---------------------------------------------------------------------------+// Signing (extracted from signatures.js)+// ---------------------------------------------------------------------------++function getDraftAlgoString(key) {+ const { name } = key.algorithm;+ if (name === 'RSA') {+ return 'rsa-sha256';+ }+ if (name === 'EC') {+ return 'ecdsa-p256-sha256';+ }+ return 'rsa-sha256';+}++async function sign(keyPem, keyId, url, digest) {+ const parsedUrl = new URL(url);+ const date = new Date().toUTCString();++ const headersToSign = {+ date,+ host: parsedUrl.host,+ };++ if (digest) {+ headersToSign.digest = digest;+ }++ const method = digest ? 'POST' : 'GET';+ const privateKey = await importPrivateKey(keyPem, ['sign']);++ const signedHeaders = digest ?+ ['(request-target)', 'host', 'date', 'digest'] :+ ['(request-target)', 'host', 'date'];++ const signingString = genDraftSigningString(+ { method, url: parsedUrl.href, headers: headersToSign },+ signedHeaders,+ { keyId },+ );++ const signature = await genDraftSignature(privateKey, signingString);+ const signatureHeader = genDraftSignatureHeader(signedHeaders, keyId, signature, getDraftAlgoString(privateKey));++ return {+ date,+ ...(digest && { digest }),+ signature: signatureHeader,+ };+}++// ---------------------------------------------------------------------------+// Worker message handler+// ---------------------------------------------------------------------------++// Track in-flight task IDs for uncaughtException handler+const pendingTaskIds = new Set();++// Track active task AbortController instances for immediate shutdown abort+const activeTasks = new Map(); // id -> AbortController++process.on('message', async (message) => {+ if (!message || typeof message.type !== 'string') {+ return;+ }++ // Handle shutdown before the switch — avoids fallthrough eslint issue+ // and ensures immediate exit without waiting for switch dispatch+ if (message.type === 'shutdown') {+ for (const [, controller] of activeTasks) {+ controller.abort();+ }+ process.send({ type: 'ack' });+ process.exit(0);+ }++ switch (message.type) {+ case 'send': {+ const { id, uri, payload, digest, key, keyId } = message;++ // Validate required fields+ if (!uri || !payload || !digest || !key || !keyId) {+ process.send({+ type: 'result',+ id,+ success: false,+ error: 'missing fields',+ });+ return;+ }++ pendingTaskIds.add(id);++ // Create AbortController for this task — allows immediate abort on shutdown+ const controller = new AbortController();+ activeTasks.set(id, controller);++ try {+ // SSRF check+ const { ok } = await check(uri);+ if (!ok) {+ process.send({+ type: 'result',+ id,+ success: false,+ error: 'SSRF check failed — reserved IP address',+ });+ pendingTaskIds.delete(id);+ activeTasks.delete(id);+ return;+ }++ // Sign+ const headers = await sign(key, keyId, uri, digest);++ // POST — redirect: 'manual' prevents SSRF via HTTP redirect+ // Combined signal: task-specific abort (shutdown) + 10s timeout+ const timeoutSignal = AbortSignal.timeout(10000);+ const combinedSignal = AbortSignal.any([controller.signal, timeoutSignal]);++ const response = await fetch(uri, {+ method: 'POST',+ headers: {+ ...headers,+ 'content-type': 'application/ld+json; profile="https://www.w3.org/ns/activitystreams"',+ },+ body: payload,+ signal: combinedSignal,+ redirect: 'manual',+ });++ // Validate Content-Length to prevent memory exhaustion+ const contentLength = response.headers.get('content-length');+ if (contentLength && parseInt(contentLength, 10) > MAX_BODY_SIZE) {+ process.send({+ type: 'result',+ id,+ success: false,+ error: 'Response body exceeds 10MB limit',+ });+ activeTasks.delete(id);+ pendingTaskIds.delete(id);+ return;+ }++ if (String(response.status).startsWith('2')) {+ process.send({+ type: 'result',+ id,+ success: true,+ });+ } else {+ let bodyText = '';+ try {+ bodyText = await response.text();+ } catch (e) { /* ignore */ }++ process.send({+ type: 'result',+ id,+ success: false,+ error: `HTTP ${response.status}: ${bodyText}`,+ });+ }+ } catch (e) {+ process.send({+ type: 'result',+ id,+ success: false,+ error: e.message || 'unknown error',+ });+ } finally {+ pendingTaskIds.delete(id);+ activeTasks.delete(id);+ }+ break;+ }++ default:+ winston.warn(`[activitypub/send] Unknown message type: ${message.type}`);+ break;+ }+});++// Uncaught exception handler — send error back if task ID is known, then exit+process.on('uncaughtException', (err) => {+ if (pendingTaskIds.size > 0) {+ // Send error for the first pending task (there should only be one)+ const taskId = pendingTaskIds.values().next().value;+ try {+ process.send({+ type: 'result',+ id: taskId,+ success: false,+ error: `uncaughtException: ${err.message}`,+ });+ } catch (e) { /* IPC may be broken */ }+ }+ winston.error(`[activitypub/send] Uncaught exception: ${err.stack}`);+ process.exit(1);+});++// Emit ready signal after startup+process.send({ type: 'ready' });diff --git a/src/start.js b/src/start.jsindex 89a1683703..44878daa80 100644--- a/src/start.js+++ b/src/start.js@@ -151,6 +151,8 @@ async function shutdown(code) {winston.info('[app] Web server closed to connections.');await require('./analytics').writeLocalData();winston.info('[app] Live analytics saved.');+ require('./activitypub').shutdown();+ winston.info('[app] ActivityPub send pool shut down.');const db = require('./database');await db.delete('locks');await db.close();