Self-hosted, Self-Improving, Federated Everything App — dev management, agent orchestration, persistent mind, digital twin, second brain, and creative pipeline from a single dashboard over Tailscale to port the user into the OS
JavaScript ai-agentsdashboarddeveloper-toolsdigital-twinquantified-selfsecond-brainself-hostedtailscale
fix(security): check past an ID3 tag longer than the manifest probe window
Adam Eivy42977cca227f
diff --git a/server/lib/README.md b/server/lib/README.mdindex f4c57a2bd5..f5279d955d 100644--- a/server/lib/README.md+++ b/server/lib/README.md@@ -388,7 +388,7 @@ The barrel `server/lib/index.js` is a machine-checkable enumeration of every pub| `homePath.js` | `scrubHomePath(value)` — replace the running user's home-directory prefix with `~` anywhere in a string, so `/Users/<name>/…` never embeds the OS username in anything a user pastes into a bug report. Non-strings pass through; a root-user container reporting `/` as home is left alone (substituting on it would rewrite every separator in every path). Zero-dependency by design: `agentRunEvents.js` re-exports it for the CoS ledger, and `scripts/doctor.js` imports it statically because it must load from a bare checkout with no `node_modules` (`scripts/pre-install-entrypoints.test.js` enforces that). || `jsonIo.js` | JSON/JSONL parsing and file IO, strict read sentinels, append/read/write helpers, and `createCachedStore` — a TTL-cached mutable JSON document with serialized `save`/`mutate`; its `load()` is strict (#4115) so a corrupt file rejects instead of being overwritten by the next `mutate`. || `settingsStore.js` | `createSettingsStore(filePath | () => filePath, defaults, { normalize? })` → `{ get, update, path }` — the one owner of a flat per-feature settings document with shipped defaults (`journal-settings.json`, `activity-digest-settings.json`, `goal-scorecard-settings.json`, …). `get()` reads strict (#4115: a corrupt file rejects rather than reading as the defaults, because `update` writes it back), merges the file over `defaults`, then runs `normalize`; `update(patch)` is a serialized `get → apply → atomicWrite` that skips `undefined` values and writes `null` as an explicit clear. Filter client-settable keys BEFORE calling `update`. Reach for `createCachedStore` for an in-place-edited document and `createCollectionStore` for per-record collections. |-| `mimeTypes.js` | MIME tables and extension allowlists (incl. `UPLOAD_AUDIO_EXTENSIONS`, the playable-audio subset of `/api/uploads`), image/video format detection (`detectImageFormat`, `detectVideoFormat`), `detectMediaManifest` (names bytes ffmpeg would open as an HLS/DASH/IMF/concat/SDP manifest that pulls in other files), and filename sanitization. |+| `mimeTypes.js` | MIME tables and extension allowlists (incl. `UPLOAD_AUDIO_EXTENSIONS`, the playable-audio subset of `/api/uploads`), image/video format detection (`detectImageFormat`, `detectVideoFormat`), `detectMediaManifest` (names bytes ffmpeg would open as an HLS/DASH/IMF/concat/SDP manifest that pulls in other files) with `id3v2TagLength`, and filename sanitization. || `paths.js` | Canonical source/install `PATHS`, `dataPath`, and home-directory expansion. || `pathContainment.js` | Dependency-free leaf holding `isPathAtOrInsideDir(dir, path)` (containment where the root itself counts as inside — what a guard policing a whole tree wants, as opposed to `pathSafety.js`'s strict `isPathInsideDir`; case-folded on Windows/macOS) and `canonicalizePath(path)` (realpath of the deepest EXISTING ancestor plus the not-yet-created tail, so a target that does not exist yet still resolves through symlinked ancestors). Split out so `fileCore.js`'s #6176 guard can reach them without dragging `errorHandler.js`/`paths.js` into the closure of every server suite — see the budget in `importScoping.test.js`. Keep it free of non-builtin imports. || `repoPublishPath.js` | `isSymlinkFreeRepoPath(realRepoRoot, absPath, { leaf: 'dir' \| 'file' })` — symlink-aware containment for publishing generated files into a managed app's repo. `lstat`-walks every existing component below the real root and refuses any symlink (dangling or contained), a wrong-kind leaf, or a realpath outside the root; a missing tail is allowed below a verified ancestor. The lexical `isPathInsideDir` cannot see a redirecting directory link (#10894, after #9668). Shared by the chiptune and sprite publishers. |diff --git a/server/lib/mimeTypes.js b/server/lib/mimeTypes.jsindex 5865e97afe..60c262064b 100644--- a/server/lib/mimeTypes.js+++ b/server/lib/mimeTypes.js@@ -187,7 +187,7 @@ export function detectVideoFormat(buf) {export const MEDIA_MANIFEST_PROBE_BYTES = 1 << 20;// Length of a leading ID3v2 tag, which ffmpeg strips before probing.-function id3v2TagLength(buf) {+export function id3v2TagLength(buf) {if (buf.length < 10 || buf.toString('latin1', 0, 3) !== 'ID3' || buf[3] === 0xff || buf[4] === 0xff) return 0;if ((buf[6] | buf[7] | buf[8] | buf[9]) & 0x80) return 0;const length = 10 + ((buf[6] << 21) | (buf[7] << 14) | (buf[8] << 7) | buf[9]);@@ -200,17 +200,19 @@ function id3v2TagLength(buf) {* whose bytes are an HLS playlist, DASH manifest, IMF composition, concat* script or SDP session makes a later probe or render read the local media* files (or network endpoints) it names; ffmpeg before 6.1 opens an HLS- * playlist under any extension. Mirrors the demuxer probes: the bytes after- * one optional ID3v2 tag, with substring probes stopping at the first NUL.+ * playlist under any extension. Mirrors the demuxer probes: a probe window+ * at the start and one after an optional ID3v2 tag, with substring probes+ * stopping at the first NUL. The post-tag window is checked even past+ * ffmpeg's own 1 MiB limit, so a larger probe size cannot reopen the gap.*- * @param {Buffer} buf - Leading bytes of the file+ * @param {Buffer} buf - The file's bytes, or at least its leading window* @returns {'hls'|'concat'|'dash'|'imf'|'sdp'|null}*/export function detectMediaManifest(buf) {if (!Buffer.isBuffer(buf)) return null;- const head = buf.subarray(0, MEDIA_MANIFEST_PROBE_BYTES);- const tagLength = id3v2TagLength(head);- const views = tagLength > 0 && tagLength < head.length ? [head, head.subarray(tagLength)] : [head];+ const tagLength = id3v2TagLength(buf);+ const views = [buf.subarray(0, MEDIA_MANIFEST_PROBE_BYTES)];+ if (tagLength > 0 && tagLength < buf.length) views.push(buf.subarray(tagLength, tagLength + MEDIA_MANIFEST_PROBE_BYTES));for (const view of views) {const nul = view.indexOf(0);const text = view.toString('latin1', 0, nul === -1 ? view.length : nul);diff --git a/server/lib/mimeTypes.test.js b/server/lib/mimeTypes.test.jsindex 8ebe3ecc09..73a8c31c3b 100644--- a/server/lib/mimeTypes.test.js+++ b/server/lib/mimeTypes.test.js@@ -3,7 +3,8 @@ import { detectMediaManifest } from './mimeTypes.js';// A minimal ID3v2.4 tag (syncsafe size in bytes 6-9) followed by `body`.const withId3 = (body, padding = 32) => Buffer.concat([- Buffer.from([0x49, 0x44, 0x33, 4, 0, 0, 0, 0, 0, padding]), Buffer.alloc(padding), Buffer.from(body),+ Buffer.from([0x49, 0x44, 0x33, 4, 0, 0, (padding >> 21) & 0x7f, (padding >> 14) & 0x7f, (padding >> 7) & 0x7f, padding & 0x7f]),+ Buffer.alloc(padding), Buffer.from(body),]);const PLAYLIST = '#EXTM3U\n#EXTINF:2.0,\n../private/clip.mp4\n#EXT-X-ENDLIST\n';@@ -13,6 +14,7 @@ describe('detectMediaManifest', () => {['an HLS playlist', Buffer.from(PLAYLIST), 'hls'],['an HLS playlist after a UTF-8 BOM and blank lines', Buffer.concat([Buffer.from([0xef, 0xbb, 0xbf]), Buffer.from(`\n\n${PLAYLIST}`)]), 'hls'],['an HLS playlist behind an ID3v2 tag (ffmpeg strips it before probing)', withId3(PLAYLIST), 'hls'],+ ['an HLS playlist behind an ID3v2 tag longer than the probe window', withId3(PLAYLIST, 2 * 1024 * 1024), 'hls'],['a concat script', Buffer.from('ffconcat version 1.0\nfile clip.mp4\n'), 'concat'],['a DASH manifest', Buffer.from('<?xml version="1.0"?><MPD profiles="urn:mpeg:dash:profile:isoff-on-demand:2011">'), 'dash'],['an IMF composition', Buffer.from('<?xml version="1.0"?><CompositionPlaylist><ContentTitle>x</ContentTitle>'), 'imf'],diff --git a/server/lib/uploads.js b/server/lib/uploads.jsindex 523ab43969..385fb9e95a 100644--- a/server/lib/uploads.js+++ b/server/lib/uploads.js@@ -6,8 +6,8 @@ import { MIRRORED_MIME_TYPES } from './beeperAttachmentPaths.js';import { ServerError } from './errorHandler.js';import { copyFileGuarded, ensureDir, pathExists, renameGuarded, unlinkGuarded, writeFileGuarded } from './fileCore.js';import {- detectImageFormat, detectMediaManifest, getFileExtension, getMimeType, MEDIA_MANIFEST_PROBE_BYTES,- RISKY_MIME_TYPES, sanitizeFilename,+ detectImageFormat, detectMediaManifest, getFileExtension, getMimeType, id3v2TagLength,+ MEDIA_MANIFEST_PROBE_BYTES, RISKY_MIME_TYPES, sanitizeFilename,} from './mimeTypes.js';import { PATHS } from './paths.js';import { assertSafeFilename, isPathInsideDir } from './pathSafety.js';@@ -27,12 +27,23 @@ function assertNotMediaManifest(buffer) {}}-/** `assertNotMediaManifest` over the leading bytes of a file on disk. */+/**+ * `assertNotMediaManifest` over a file on disk: its leading window, plus the+ * window after an ID3v2 tag too long to fit in it.+ */export async function assertFileNotMediaManifest(path) {const handle = await open(path, 'r');- const head = Buffer.alloc(MEDIA_MANIFEST_PROBE_BYTES);- const { bytesRead } = await handle.read(head, 0, head.length, 0).finally(() => handle.close());- assertNotMediaManifest(head.subarray(0, bytesRead));+ const readWindow = async (position) => {+ const window = Buffer.alloc(MEDIA_MANIFEST_PROBE_BYTES);+ const { bytesRead } = await handle.read(window, 0, window.length, position);+ return window.subarray(0, bytesRead);+ };+ const windows = await (async () => {+ const head = await readWindow(0);+ const tagLength = id3v2TagLength(head);+ return tagLength >= head.length && head.length === MEDIA_MANIFEST_PROBE_BYTES ? [head, await readWindow(tagLength)] : [head];+ })().finally(() => handle.close());+ for (const window of windows) assertNotMediaManifest(window);}/**diff --git a/server/services/pipeline/musicLibrary.test.js b/server/services/pipeline/musicLibrary.test.jsindex 20bb3e28e1..804db6cad7 100644--- a/server/services/pipeline/musicLibrary.test.js+++ b/server/services/pipeline/musicLibrary.test.js@@ -182,6 +182,19 @@ describe('importUploadedTrack', () => {expect(await readdir(FAKE_MUSIC_DIR)).toEqual([]);expect(existsSync(tmp)).toBe(false);});++ // The file check reads a second window when the tag outgrows the first.+ it('refuses a playlist hidden behind an ID3v2 tag longer than the read window', async () => {+ const tmp = join(TEST_HOME, 'upload-tmp.mp3');+ const tagSize = 2 * 1024 * 1024;+ const syncsafe = [21, 14, 7, 0].map((shift) => (tagSize >> shift) & 0x7f);+ await writeFile(tmp, Buffer.concat([+ Buffer.from([0x49, 0x44, 0x33, 4, 0, 0, ...syncsafe]), Buffer.alloc(tagSize),+ Buffer.from('#EXTM3U\n#EXTINF:2.0,\n../private/song.m4a\n'),+ ]));+ await expect(importUploadedTrack(tmp, 'song.mp3')).rejects.toMatchObject({ status: 400, code: 'INVALID_FILE_TYPE' });+ expect(await readdir(FAKE_MUSIC_DIR)).toEqual([]);+ });});describe('statMusicTrack', () => {