Self-hosted, Self-Improving, Federated Everything App — dev management, agent orchestration, persistent mind, digital twin, second brain, and creative pipeline from a single dashboard over Tailscale to port the user into the OS
JavaScript ai-agentsdashboarddeveloper-toolsdigital-twinquantified-selfsecond-brainself-hostedtailscale
fix: pin and revalidate every hop of the privacy broker scan fetch (#12426)
[._.]/ Adam Eivyd8363c353341
diff --git a/server/lib/README.md b/server/lib/README.mdindex 50b86e5821..13e2e20129 100644--- a/server/lib/README.md+++ b/server/lib/README.md@@ -487,7 +487,7 @@ The barrel `server/lib/index.js` is a machine-checkable enumeration of every pub| `federatedMediaRequest.js` | Builds and validates the versioned federated-media submission body for a visual kind, so every call site that persists a remote-job marker projects local params onto the wire identically. || `federatedMediaWire.js` | Versioned federated-media status schema/constants plus fail-closed consumer freshness checks (stale and clock-skewed snapshots are never assignable). `FEDERATED_MEDIA_FEATURES` + `federatedMediaSupports(status, feature, capability)` answer "does the peer BUILD speak this wire feature" from the status-root `features` list — the one place the "absent reads as the wire-v1 baseline" rule lives; a published list wins outright, with the `inputAssets` capability block retained as its only legacy overlap tell because it is genuinely per-model. `federatedMediaDeclaresFeatures(status)` and `federatedMediaDeniesFeature(status, feature, capability)` separate "positively denied" from "could not establish" for MESSAGE selection only — both gate identically, and which missing signal indicts the peer’s build is recorded per feature rather than at each call site. || `connectivity.js` | `isMachineOnline({timeoutMs?, hosts?})` — internet-reachability probe: bare TCP `connect` to public anycast resolvers on :443 (no DNS, no HTTP), resolves `true` on the first connect and `false` only after all fail; never rejects. Reuse when a caller needs a best-effort local reachability signal. Also `isPortReachable({host?, port, timeoutMs?})` — the same connect primitive aimed at ONE address, for "is anything listening there yet?"; use it rather than a bind probe whenever another process is expected to own the port, since a bind probe can win the race and kill that process with `EADDRINUSE`. |-| `safeUrlFetch.js` | SSRF-guarded public-URL fetch: `isPublicHttpUrlSafe`/`assertPublicHttpUrl` (scheme + blocked-host-literal via `catalogValidation.isBlockedIngestHost` + DNS-resolve), exported strict `isPrivateAddress`, plus `fetchPublicText`/`fetchPublicBinary` (timeout, redirect revalidation, size cap), and `resolvePublicUrl` (bounded, pinned short-link resolution with an optional hop allow-list) and `buildPinnedLookup`/`buildPinnedDispatcher` (the connect-time IP pin that closes the DNS-rebinding TOCTOU). Reuse instead of copying the SSRF guard for any "fetch this remote thing the user pointed us at" flow. Also `readBodyCapped(response, maxBytes)` bounds streamed bodies without choosing a network policy. |+| `safeUrlFetch.js` | SSRF-guarded public-URL fetch: `isPublicHttpUrlSafe`/`assertPublicHttpUrl` (scheme + blocked-host-literal via `catalogValidation.isBlockedIngestHost` + DNS-resolve), exported strict `isPrivateAddress`, plus `fetchPublicText`/`fetchPublicBinary`/`fetchPublicResponse` (status-preserving raw Response) (timeout, redirect revalidation, size cap), and `resolvePublicUrl` (bounded, pinned short-link resolution with an optional hop allow-list) and `buildPinnedLookup`/`buildPinnedDispatcher` (the connect-time IP pin that closes the DNS-rebinding TOCTOU). Reuse instead of copying the SSRF guard for any "fetch this remote thing the user pointed us at" flow. Also `readBodyCapped(response, maxBytes)` bounds streamed bodies without choosing a network policy. || `pinterestFeed.js` | Pure Pinterest board RSS helpers: `isPinterestHost(host)` (explicit domain allow-list) + `normalizePinterestFeedUrl(input)` (board URL or `.rss` → `{ feedUrl, boardUrl }`, host/path-gated) + `parsePinterestRss(xml)` (per-pin `pinUrl`/`imageUrl`/title/description, 736x size upgrade). Feeds the mood-board Pinterest importer. || `xPostMedia.js` | Pure x.com/twitter.com post helpers: `parseXPostUrl(input)` (post URL → `{ statusId, canonicalUrl }`, host-gated) + `buildSyndicationUrl`/`syndicationToken` (Twitter's public unauthenticated tweet-syndication endpoint) + `extractXPostMedia(json)` (→ `{ images, video }`). Feeds the mood-board X post importer. || `requestAbort.js` | `abortSignalFromResponse(res)` — AbortSignal that fires only when an Express client disconnects *before the response finishes* (keyed off `res` close + `writableEnded`). Plus `anyAbortSignal(signals)` — combine several signals into one (native `AbortSignal.any` with a Node-18 fallback). |diff --git a/server/lib/safeUrlFetch.js b/server/lib/safeUrlFetch.jsindex e372e9348c..aea87c67f9 100644--- a/server/lib/safeUrlFetch.js+++ b/server/lib/safeUrlFetch.js@@ -306,6 +306,16 @@ export async function readBodyCapped(res, maxBytes) {return Buffer.concat(chunks);}+/**+ * Fetch with every hop vetted and DNS-pinned, returning the raw Response of ANY+ * status (3xx without a followable Location excluded) so callers that classify+ * on status (privacy scan) keep it. Returns null on a network error or a+ * blocked/invalid redirect. The caller owns reading the body.+ */+export function fetchPublicResponse(url, { timeoutMs, headers, blockPrivate = false, throwOnUnsafe = true } = {}) {+ return fetchGuarded(url, { timeoutMs, headers, blockPrivate, throwOnUnsafe });+}+/*** Fetch a URL's body as text. Returns the string on a 2xx, or null on any* failure (network error, non-ok status, blocked redirect). `blockPrivate` optsdiff --git a/server/services/privacyScan.js b/server/services/privacyScan.jsindex a836c604fe..3284b8b84e 100644--- a/server/services/privacyScan.js+++ b/server/services/privacyScan.js@@ -27,6 +27,7 @@import { lookup } from 'dns/promises';import { isSafeIngestUrl, isBlockedIngestHost } from '../lib/catalogValidation.js';+import { fetchPublicResponse } from '../lib/safeUrlFetch.js';import { fetchUrlMainText } from './catalogIngestSources.js';import { listScanEligibleValues } from './privacyVault.js';import { listBrokers, listBrokerCases, recordScanVerdict } from './privacyBrokers.js';@@ -203,6 +204,13 @@ async function isScanUrlSafe(target) {return !(resolved?.address && isBlockedIngestHost(resolved.address));}+// Default HTTP lane transport: every hop (including redirects) is DNS-pinned and+// revalidated, and private/LAN addresses are refused, so a broker page cannot+// bounce the scan into loopback or LAN services. A refused hop yields null.+const pinnedScanFetch = (url, { headers }) => fetchPublicResponse(url, {+ headers, timeoutMs: HTTP_FETCH_TIMEOUT_MS, blockPrivate: true, throwOnUnsafe: false,+});+// Plain HTTP GET lane. Returns { status, html } or null on network failure.async function httpFetchLane(url, fetchImpl) {const controller = new AbortController();@@ -210,9 +218,9 @@ async function httpFetchLane(url, fetchImpl) {try {const res = await fetchImpl(url, {signal: controller.signal,- redirect: 'follow',headers: { 'User-Agent': 'Mozilla/5.0 (compatible; PortOS-PrivacyScan/1.0)' },});+ if (!res) return null;const html = await res.text();return { status: res.status, html };} catch {@@ -232,7 +240,7 @@ async function httpFetchLane(url, fetchImpl) {* `deps` injectable so tests/verification never hit the network.*/export async function probeBroker(broker, vectors, {- fetchImpl = fetch, browserFetch = fetchUrlMainText, urlSafe = isScanUrlSafe,+ fetchImpl = pinnedScanFetch, browserFetch = fetchUrlMainText, urlSafe = isScanUrlSafe,} = {}) {const template = broker?.urls?.search;const name = (vectors.names || [])[0];@@ -312,7 +320,7 @@ export async function probeBroker(broker, vectors, {* no usable search URL (blind scan not possible) or the URL is unsafe.*/export async function scanBroker(broker, vectors, {- fetchImpl = fetch, browserFetch = fetchUrlMainText, urlSafe = isScanUrlSafe, now = new Date(),+ fetchImpl = pinnedScanFetch, browserFetch = fetchUrlMainText, urlSafe = isScanUrlSafe, now = new Date(),subjectId,} = {}) {const resolvedSubjectId = resolveSubjectId(subjectId);@@ -332,7 +340,7 @@ export async function scanBroker(broker, vectors, {** Returns a summary { scanned, verdicts: {<verdict>: n}, skipped, brokers }.*/-export async function runScanPass({ concurrency = 3, fetchImpl = fetch, browserFetch = fetchUrlMainText, urlSafe = isScanUrlSafe, now = new Date(), subjectId } = {}) {+export async function runScanPass({ concurrency = 3, fetchImpl = pinnedScanFetch, browserFetch = fetchUrlMainText, urlSafe = isScanUrlSafe, now = new Date(), subjectId } = {}) {// CONSENT GATE — BEFORE any vault read or broker fetch. A subject without an// active `broker_scan` grant is refused outright (403); the UI hiding the Scan// button is not sufficient, since a cron/API/agent caller reaches this samediff --git a/server/services/privacyScan.redirect.test.js b/server/services/privacyScan.redirect.test.jsnew file mode 100644index 0000000000..71516677d7--- /dev/null+++ b/server/services/privacyScan.redirect.test.js@@ -0,0 +1,44 @@+/**+ * The default HTTP lane must not follow a broker redirect into loopback/LAN.+ * Exercises the real safeUrlFetch guard with only DNS and the socket mocked.+ */+import { describe, it, expect, vi } from 'vitest';++const lookupMock = vi.fn(async () => ({ address: '93.184.216.34', family: 4 }));+vi.mock('dns/promises', () => ({+ default: { lookup: (...a) => lookupMock(...a) },+ lookup: (...a) => lookupMock(...a),+}));+const fetchMock = vi.fn();+vi.mock('../lib/fetchWithTimeout.js', () => ({ fetchWithTimeout: (...a) => fetchMock(...a) }));+vi.mock('./catalogIngestSources.js', () => ({ fetchUrlMainText: vi.fn(async () => null) }));+vi.mock('./privacyVault.js', () => ({ listScanEligibleValues: vi.fn(async () => []) }));+vi.mock('./privacyBrokers.js', () => ({ listBrokers: vi.fn(), listBrokerCases: vi.fn(), recordScanVerdict: vi.fn() }));+vi.mock('./privacySubjects.js', () => ({ resolveSubjectId: (id) => id, assertSubjectConsent: vi.fn() }));++const { probeBroker } = await import('./privacyScan.js');++const broker = { id: 'b1', urls: { search: 'https://broker.example.com/{first}-{last}' } };+const vectors = { names: [{ full: 'Jane Doe', firstName: 'Jane', lastName: 'Doe' }], locations: [] };+const redirectTo = (location) => ({ status: 302, ok: false, headers: new Map([['location', location]]), text: async () => '' });++describe('probeBroker default HTTP lane', () => {+ it.each(['http://127.0.0.1:5555/api/x', 'http://169.254.169.254/x', 'http://192.168.1.10/x'])(+ 'does not follow a redirect to %s',+ async (target) => {+ fetchMock.mockReset();+ fetchMock.mockResolvedValueOnce(redirectTo(target));+ const res = await probeBroker(broker, vectors, { browserFetch: vi.fn(async () => null) });+ expect(fetchMock).toHaveBeenCalledTimes(1);+ expect(res).toEqual({ skipped: true, reason: 'fetch_failed' });+ },+ );++ it('keeps the status contract: a 404 stays inconclusive', async () => {+ fetchMock.mockReset();+ fetchMock.mockResolvedValueOnce({ status: 404, ok: false, headers: new Map(), text: async () => 'nope' });+ const res = await probeBroker(broker, vectors, { browserFetch: vi.fn(async () => null) });+ expect(res).toMatchObject({ skipped: true, verdict: null });+ expect(res.reason).not.toBe('fetch_failed');+ });+});