Skip to content

Self-hosted, Self-Improving, Federated Everything App — dev management, agent orchestration, persistent mind, digital twin, second brain, and creative pipeline from a single dashboard over Tailscale to port the user into the OS

fix: pin and revalidate every hop of the privacy broker scan fetch (#12426)

[._.]/ Adam Eivyd8363c353341
diff --git a/server/lib/README.md b/server/lib/README.md
index 50b86e5821..13e2e20129 100644
--- a/server/lib/README.md
+++ b/server/lib/README.md
@@ -487,7 +487,7 @@ The barrel `server/lib/index.js` is a machine-checkable enumeration of every pub
| `federatedMediaRequest.js` | Builds and validates the versioned federated-media submission body for a visual kind, so every call site that persists a remote-job marker projects local params onto the wire identically. |
| `federatedMediaWire.js` | Versioned federated-media status schema/constants plus fail-closed consumer freshness checks (stale and clock-skewed snapshots are never assignable). `FEDERATED_MEDIA_FEATURES` + `federatedMediaSupports(status, feature, capability)` answer "does the peer BUILD speak this wire feature" from the status-root `features` list — the one place the "absent reads as the wire-v1 baseline" rule lives; a published list wins outright, with the `inputAssets` capability block retained as its only legacy overlap tell because it is genuinely per-model. `federatedMediaDeclaresFeatures(status)` and `federatedMediaDeniesFeature(status, feature, capability)` separate "positively denied" from "could not establish" for MESSAGE selection only — both gate identically, and which missing signal indicts the peer’s build is recorded per feature rather than at each call site. |
| `connectivity.js` | `isMachineOnline({timeoutMs?, hosts?})` — internet-reachability probe: bare TCP `connect` to public anycast resolvers on :443 (no DNS, no HTTP), resolves `true` on the first connect and `false` only after all fail; never rejects. Reuse when a caller needs a best-effort local reachability signal. Also `isPortReachable({host?, port, timeoutMs?})` — the same connect primitive aimed at ONE address, for "is anything listening there yet?"; use it rather than a bind probe whenever another process is expected to own the port, since a bind probe can win the race and kill that process with `EADDRINUSE`. |
-| `safeUrlFetch.js` | SSRF-guarded public-URL fetch: `isPublicHttpUrlSafe`/`assertPublicHttpUrl` (scheme + blocked-host-literal via `catalogValidation.isBlockedIngestHost` + DNS-resolve), exported strict `isPrivateAddress`, plus `fetchPublicText`/`fetchPublicBinary` (timeout, redirect revalidation, size cap), and `resolvePublicUrl` (bounded, pinned short-link resolution with an optional hop allow-list) and `buildPinnedLookup`/`buildPinnedDispatcher` (the connect-time IP pin that closes the DNS-rebinding TOCTOU). Reuse instead of copying the SSRF guard for any "fetch this remote thing the user pointed us at" flow. Also `readBodyCapped(response, maxBytes)` bounds streamed bodies without choosing a network policy. |
+| `safeUrlFetch.js` | SSRF-guarded public-URL fetch: `isPublicHttpUrlSafe`/`assertPublicHttpUrl` (scheme + blocked-host-literal via `catalogValidation.isBlockedIngestHost` + DNS-resolve), exported strict `isPrivateAddress`, plus `fetchPublicText`/`fetchPublicBinary`/`fetchPublicResponse` (status-preserving raw Response) (timeout, redirect revalidation, size cap), and `resolvePublicUrl` (bounded, pinned short-link resolution with an optional hop allow-list) and `buildPinnedLookup`/`buildPinnedDispatcher` (the connect-time IP pin that closes the DNS-rebinding TOCTOU). Reuse instead of copying the SSRF guard for any "fetch this remote thing the user pointed us at" flow. Also `readBodyCapped(response, maxBytes)` bounds streamed bodies without choosing a network policy. |
| `pinterestFeed.js` | Pure Pinterest board RSS helpers: `isPinterestHost(host)` (explicit domain allow-list) + `normalizePinterestFeedUrl(input)` (board URL or `.rss` → `{ feedUrl, boardUrl }`, host/path-gated) + `parsePinterestRss(xml)` (per-pin `pinUrl`/`imageUrl`/title/description, 736x size upgrade). Feeds the mood-board Pinterest importer. |
| `xPostMedia.js` | Pure x.com/twitter.com post helpers: `parseXPostUrl(input)` (post URL → `{ statusId, canonicalUrl }`, host-gated) + `buildSyndicationUrl`/`syndicationToken` (Twitter's public unauthenticated tweet-syndication endpoint) + `extractXPostMedia(json)` (→ `{ images, video }`). Feeds the mood-board X post importer. |
| `requestAbort.js` | `abortSignalFromResponse(res)` — AbortSignal that fires only when an Express client disconnects *before the response finishes* (keyed off `res` close + `writableEnded`). Plus `anyAbortSignal(signals)` — combine several signals into one (native `AbortSignal.any` with a Node-18 fallback). |
diff --git a/server/lib/safeUrlFetch.js b/server/lib/safeUrlFetch.js
index e372e9348c..aea87c67f9 100644
--- a/server/lib/safeUrlFetch.js
+++ b/server/lib/safeUrlFetch.js
@@ -306,6 +306,16 @@ export async function readBodyCapped(res, maxBytes) {
return Buffer.concat(chunks);
}
+/**
+ * Fetch with every hop vetted and DNS-pinned, returning the raw Response of ANY
+ * status (3xx without a followable Location excluded) so callers that classify
+ * on status (privacy scan) keep it. Returns null on a network error or a
+ * blocked/invalid redirect. The caller owns reading the body.
+ */
+export function fetchPublicResponse(url, { timeoutMs, headers, blockPrivate = false, throwOnUnsafe = true } = {}) {
+ return fetchGuarded(url, { timeoutMs, headers, blockPrivate, throwOnUnsafe });
+}
+
/**
* Fetch a URL's body as text. Returns the string on a 2xx, or null on any
* failure (network error, non-ok status, blocked redirect). `blockPrivate` opts
diff --git a/server/services/privacyScan.js b/server/services/privacyScan.js
index a836c604fe..3284b8b84e 100644
--- a/server/services/privacyScan.js
+++ b/server/services/privacyScan.js
@@ -27,6 +27,7 @@
import { lookup } from 'dns/promises';
import { isSafeIngestUrl, isBlockedIngestHost } from '../lib/catalogValidation.js';
+import { fetchPublicResponse } from '../lib/safeUrlFetch.js';
import { fetchUrlMainText } from './catalogIngestSources.js';
import { listScanEligibleValues } from './privacyVault.js';
import { listBrokers, listBrokerCases, recordScanVerdict } from './privacyBrokers.js';
@@ -203,6 +204,13 @@ async function isScanUrlSafe(target) {
return !(resolved?.address && isBlockedIngestHost(resolved.address));
}
+// Default HTTP lane transport: every hop (including redirects) is DNS-pinned and
+// revalidated, and private/LAN addresses are refused, so a broker page cannot
+// bounce the scan into loopback or LAN services. A refused hop yields null.
+const pinnedScanFetch = (url, { headers }) => fetchPublicResponse(url, {
+ headers, timeoutMs: HTTP_FETCH_TIMEOUT_MS, blockPrivate: true, throwOnUnsafe: false,
+});
+
// Plain HTTP GET lane. Returns { status, html } or null on network failure.
async function httpFetchLane(url, fetchImpl) {
const controller = new AbortController();
@@ -210,9 +218,9 @@ async function httpFetchLane(url, fetchImpl) {
try {
const res = await fetchImpl(url, {
signal: controller.signal,
- redirect: 'follow',
headers: { 'User-Agent': 'Mozilla/5.0 (compatible; PortOS-PrivacyScan/1.0)' },
});
+ if (!res) return null;
const html = await res.text();
return { status: res.status, html };
} catch {
@@ -232,7 +240,7 @@ async function httpFetchLane(url, fetchImpl) {
* `deps` injectable so tests/verification never hit the network.
*/
export async function probeBroker(broker, vectors, {
- fetchImpl = fetch, browserFetch = fetchUrlMainText, urlSafe = isScanUrlSafe,
+ fetchImpl = pinnedScanFetch, browserFetch = fetchUrlMainText, urlSafe = isScanUrlSafe,
} = {}) {
const template = broker?.urls?.search;
const name = (vectors.names || [])[0];
@@ -312,7 +320,7 @@ export async function probeBroker(broker, vectors, {
* no usable search URL (blind scan not possible) or the URL is unsafe.
*/
export async function scanBroker(broker, vectors, {
- fetchImpl = fetch, browserFetch = fetchUrlMainText, urlSafe = isScanUrlSafe, now = new Date(),
+ fetchImpl = pinnedScanFetch, browserFetch = fetchUrlMainText, urlSafe = isScanUrlSafe, now = new Date(),
subjectId,
} = {}) {
const resolvedSubjectId = resolveSubjectId(subjectId);
@@ -332,7 +340,7 @@ export async function scanBroker(broker, vectors, {
*
* Returns a summary { scanned, verdicts: {<verdict>: n}, skipped, brokers }.
*/
-export async function runScanPass({ concurrency = 3, fetchImpl = fetch, browserFetch = fetchUrlMainText, urlSafe = isScanUrlSafe, now = new Date(), subjectId } = {}) {
+export async function runScanPass({ concurrency = 3, fetchImpl = pinnedScanFetch, browserFetch = fetchUrlMainText, urlSafe = isScanUrlSafe, now = new Date(), subjectId } = {}) {
// CONSENT GATE — BEFORE any vault read or broker fetch. A subject without an
// active `broker_scan` grant is refused outright (403); the UI hiding the Scan
// button is not sufficient, since a cron/API/agent caller reaches this same
diff --git a/server/services/privacyScan.redirect.test.js b/server/services/privacyScan.redirect.test.js
new file mode 100644
index 0000000000..71516677d7
--- /dev/null
+++ b/server/services/privacyScan.redirect.test.js
@@ -0,0 +1,44 @@
+/**
+ * The default HTTP lane must not follow a broker redirect into loopback/LAN.
+ * Exercises the real safeUrlFetch guard with only DNS and the socket mocked.
+ */
+import { describe, it, expect, vi } from 'vitest';
+
+const lookupMock = vi.fn(async () => ({ address: '93.184.216.34', family: 4 }));
+vi.mock('dns/promises', () => ({
+ default: { lookup: (...a) => lookupMock(...a) },
+ lookup: (...a) => lookupMock(...a),
+}));
+const fetchMock = vi.fn();
+vi.mock('../lib/fetchWithTimeout.js', () => ({ fetchWithTimeout: (...a) => fetchMock(...a) }));
+vi.mock('./catalogIngestSources.js', () => ({ fetchUrlMainText: vi.fn(async () => null) }));
+vi.mock('./privacyVault.js', () => ({ listScanEligibleValues: vi.fn(async () => []) }));
+vi.mock('./privacyBrokers.js', () => ({ listBrokers: vi.fn(), listBrokerCases: vi.fn(), recordScanVerdict: vi.fn() }));
+vi.mock('./privacySubjects.js', () => ({ resolveSubjectId: (id) => id, assertSubjectConsent: vi.fn() }));
+
+const { probeBroker } = await import('./privacyScan.js');
+
+const broker = { id: 'b1', urls: { search: 'https://broker.example.com/{first}-{last}' } };
+const vectors = { names: [{ full: 'Jane Doe', firstName: 'Jane', lastName: 'Doe' }], locations: [] };
+const redirectTo = (location) => ({ status: 302, ok: false, headers: new Map([['location', location]]), text: async () => '' });
+
+describe('probeBroker default HTTP lane', () => {
+ it.each(['http://127.0.0.1:5555/api/x', 'http://169.254.169.254/x', 'http://192.168.1.10/x'])(
+ 'does not follow a redirect to %s',
+ async (target) => {
+ fetchMock.mockReset();
+ fetchMock.mockResolvedValueOnce(redirectTo(target));
+ const res = await probeBroker(broker, vectors, { browserFetch: vi.fn(async () => null) });
+ expect(fetchMock).toHaveBeenCalledTimes(1);
+ expect(res).toEqual({ skipped: true, reason: 'fetch_failed' });
+ },
+ );
+
+ it('keeps the status contract: a 404 stays inconclusive', async () => {
+ fetchMock.mockReset();
+ fetchMock.mockResolvedValueOnce({ status: 404, ok: false, headers: new Map(), text: async () => 'nope' });
+ const res = await probeBroker(broker, vectors, { browserFetch: vi.fn(async () => null) });
+ expect(res).toMatchObject({ skipped: true, verdict: null });
+ expect(res.reason).not.toBe('fetch_failed');
+ });
+});