Skip to content

Self-hosted, Self-Improving, Federated Everything App — dev management, agent orchestration, persistent mind, digital twin, second brain, and creative pipeline from a single dashboard over Tailscale to port the user into the OS

fix(security): check past an ID3 tag longer than the manifest probe window

Adam Eivy42977cca227f
diff --git a/server/lib/README.md b/server/lib/README.md
index f4c57a2bd5..f5279d955d 100644
--- a/server/lib/README.md
+++ b/server/lib/README.md
@@ -388,7 +388,7 @@ The barrel `server/lib/index.js` is a machine-checkable enumeration of every pub
| `homePath.js` | `scrubHomePath(value)` — replace the running user's home-directory prefix with `~` anywhere in a string, so `/Users/<name>/…` never embeds the OS username in anything a user pastes into a bug report. Non-strings pass through; a root-user container reporting `/` as home is left alone (substituting on it would rewrite every separator in every path). Zero-dependency by design: `agentRunEvents.js` re-exports it for the CoS ledger, and `scripts/doctor.js` imports it statically because it must load from a bare checkout with no `node_modules` (`scripts/pre-install-entrypoints.test.js` enforces that). |
| `jsonIo.js` | JSON/JSONL parsing and file IO, strict read sentinels, append/read/write helpers, and `createCachedStore` — a TTL-cached mutable JSON document with serialized `save`/`mutate`; its `load()` is strict (#4115) so a corrupt file rejects instead of being overwritten by the next `mutate`. |
| `settingsStore.js` | `createSettingsStore(filePath | () => filePath, defaults, { normalize? })` → `{ get, update, path }` — the one owner of a flat per-feature settings document with shipped defaults (`journal-settings.json`, `activity-digest-settings.json`, `goal-scorecard-settings.json`, …). `get()` reads strict (#4115: a corrupt file rejects rather than reading as the defaults, because `update` writes it back), merges the file over `defaults`, then runs `normalize`; `update(patch)` is a serialized `get → apply → atomicWrite` that skips `undefined` values and writes `null` as an explicit clear. Filter client-settable keys BEFORE calling `update`. Reach for `createCachedStore` for an in-place-edited document and `createCollectionStore` for per-record collections. |
-| `mimeTypes.js` | MIME tables and extension allowlists (incl. `UPLOAD_AUDIO_EXTENSIONS`, the playable-audio subset of `/api/uploads`), image/video format detection (`detectImageFormat`, `detectVideoFormat`), `detectMediaManifest` (names bytes ffmpeg would open as an HLS/DASH/IMF/concat/SDP manifest that pulls in other files), and filename sanitization. |
+| `mimeTypes.js` | MIME tables and extension allowlists (incl. `UPLOAD_AUDIO_EXTENSIONS`, the playable-audio subset of `/api/uploads`), image/video format detection (`detectImageFormat`, `detectVideoFormat`), `detectMediaManifest` (names bytes ffmpeg would open as an HLS/DASH/IMF/concat/SDP manifest that pulls in other files) with `id3v2TagLength`, and filename sanitization. |
| `paths.js` | Canonical source/install `PATHS`, `dataPath`, and home-directory expansion. |
| `pathContainment.js` | Dependency-free leaf holding `isPathAtOrInsideDir(dir, path)` (containment where the root itself counts as inside — what a guard policing a whole tree wants, as opposed to `pathSafety.js`'s strict `isPathInsideDir`; case-folded on Windows/macOS) and `canonicalizePath(path)` (realpath of the deepest EXISTING ancestor plus the not-yet-created tail, so a target that does not exist yet still resolves through symlinked ancestors). Split out so `fileCore.js`'s #6176 guard can reach them without dragging `errorHandler.js`/`paths.js` into the closure of every server suite — see the budget in `importScoping.test.js`. Keep it free of non-builtin imports. |
| `repoPublishPath.js` | `isSymlinkFreeRepoPath(realRepoRoot, absPath, { leaf: 'dir' \| 'file' })` — symlink-aware containment for publishing generated files into a managed app's repo. `lstat`-walks every existing component below the real root and refuses any symlink (dangling or contained), a wrong-kind leaf, or a realpath outside the root; a missing tail is allowed below a verified ancestor. The lexical `isPathInsideDir` cannot see a redirecting directory link (#10894, after #9668). Shared by the chiptune and sprite publishers. |
diff --git a/server/lib/mimeTypes.js b/server/lib/mimeTypes.js
index 5865e97afe..60c262064b 100644
--- a/server/lib/mimeTypes.js
+++ b/server/lib/mimeTypes.js
@@ -187,7 +187,7 @@ export function detectVideoFormat(buf) {
export const MEDIA_MANIFEST_PROBE_BYTES = 1 << 20;
// Length of a leading ID3v2 tag, which ffmpeg strips before probing.
-function id3v2TagLength(buf) {
+export function id3v2TagLength(buf) {
if (buf.length < 10 || buf.toString('latin1', 0, 3) !== 'ID3' || buf[3] === 0xff || buf[4] === 0xff) return 0;
if ((buf[6] | buf[7] | buf[8] | buf[9]) & 0x80) return 0;
const length = 10 + ((buf[6] << 21) | (buf[7] << 14) | (buf[8] << 7) | buf[9]);
@@ -200,17 +200,19 @@ function id3v2TagLength(buf) {
* whose bytes are an HLS playlist, DASH manifest, IMF composition, concat
* script or SDP session makes a later probe or render read the local media
* files (or network endpoints) it names; ffmpeg before 6.1 opens an HLS
- * playlist under any extension. Mirrors the demuxer probes: the bytes after
- * one optional ID3v2 tag, with substring probes stopping at the first NUL.
+ * playlist under any extension. Mirrors the demuxer probes: a probe window
+ * at the start and one after an optional ID3v2 tag, with substring probes
+ * stopping at the first NUL. The post-tag window is checked even past
+ * ffmpeg's own 1 MiB limit, so a larger probe size cannot reopen the gap.
*
- * @param {Buffer} buf - Leading bytes of the file
+ * @param {Buffer} buf - The file's bytes, or at least its leading window
* @returns {'hls'|'concat'|'dash'|'imf'|'sdp'|null}
*/
export function detectMediaManifest(buf) {
if (!Buffer.isBuffer(buf)) return null;
- const head = buf.subarray(0, MEDIA_MANIFEST_PROBE_BYTES);
- const tagLength = id3v2TagLength(head);
- const views = tagLength > 0 && tagLength < head.length ? [head, head.subarray(tagLength)] : [head];
+ const tagLength = id3v2TagLength(buf);
+ const views = [buf.subarray(0, MEDIA_MANIFEST_PROBE_BYTES)];
+ if (tagLength > 0 && tagLength < buf.length) views.push(buf.subarray(tagLength, tagLength + MEDIA_MANIFEST_PROBE_BYTES));
for (const view of views) {
const nul = view.indexOf(0);
const text = view.toString('latin1', 0, nul === -1 ? view.length : nul);
diff --git a/server/lib/mimeTypes.test.js b/server/lib/mimeTypes.test.js
index 8ebe3ecc09..73a8c31c3b 100644
--- a/server/lib/mimeTypes.test.js
+++ b/server/lib/mimeTypes.test.js
@@ -3,7 +3,8 @@ import { detectMediaManifest } from './mimeTypes.js';
// A minimal ID3v2.4 tag (syncsafe size in bytes 6-9) followed by `body`.
const withId3 = (body, padding = 32) => Buffer.concat([
- Buffer.from([0x49, 0x44, 0x33, 4, 0, 0, 0, 0, 0, padding]), Buffer.alloc(padding), Buffer.from(body),
+ Buffer.from([0x49, 0x44, 0x33, 4, 0, 0, (padding >> 21) & 0x7f, (padding >> 14) & 0x7f, (padding >> 7) & 0x7f, padding & 0x7f]),
+ Buffer.alloc(padding), Buffer.from(body),
]);
const PLAYLIST = '#EXTM3U\n#EXTINF:2.0,\n../private/clip.mp4\n#EXT-X-ENDLIST\n';
@@ -13,6 +14,7 @@ describe('detectMediaManifest', () => {
['an HLS playlist', Buffer.from(PLAYLIST), 'hls'],
['an HLS playlist after a UTF-8 BOM and blank lines', Buffer.concat([Buffer.from([0xef, 0xbb, 0xbf]), Buffer.from(`\n\n${PLAYLIST}`)]), 'hls'],
['an HLS playlist behind an ID3v2 tag (ffmpeg strips it before probing)', withId3(PLAYLIST), 'hls'],
+ ['an HLS playlist behind an ID3v2 tag longer than the probe window', withId3(PLAYLIST, 2 * 1024 * 1024), 'hls'],
['a concat script', Buffer.from('ffconcat version 1.0\nfile clip.mp4\n'), 'concat'],
['a DASH manifest', Buffer.from('<?xml version="1.0"?><MPD profiles="urn:mpeg:dash:profile:isoff-on-demand:2011">'), 'dash'],
['an IMF composition', Buffer.from('<?xml version="1.0"?><CompositionPlaylist><ContentTitle>x</ContentTitle>'), 'imf'],
diff --git a/server/lib/uploads.js b/server/lib/uploads.js
index 523ab43969..385fb9e95a 100644
--- a/server/lib/uploads.js
+++ b/server/lib/uploads.js
@@ -6,8 +6,8 @@ import { MIRRORED_MIME_TYPES } from './beeperAttachmentPaths.js';
import { ServerError } from './errorHandler.js';
import { copyFileGuarded, ensureDir, pathExists, renameGuarded, unlinkGuarded, writeFileGuarded } from './fileCore.js';
import {
- detectImageFormat, detectMediaManifest, getFileExtension, getMimeType, MEDIA_MANIFEST_PROBE_BYTES,
- RISKY_MIME_TYPES, sanitizeFilename,
+ detectImageFormat, detectMediaManifest, getFileExtension, getMimeType, id3v2TagLength,
+ MEDIA_MANIFEST_PROBE_BYTES, RISKY_MIME_TYPES, sanitizeFilename,
} from './mimeTypes.js';
import { PATHS } from './paths.js';
import { assertSafeFilename, isPathInsideDir } from './pathSafety.js';
@@ -27,12 +27,23 @@ function assertNotMediaManifest(buffer) {
}
}
-/** `assertNotMediaManifest` over the leading bytes of a file on disk. */
+/**
+ * `assertNotMediaManifest` over a file on disk: its leading window, plus the
+ * window after an ID3v2 tag too long to fit in it.
+ */
export async function assertFileNotMediaManifest(path) {
const handle = await open(path, 'r');
- const head = Buffer.alloc(MEDIA_MANIFEST_PROBE_BYTES);
- const { bytesRead } = await handle.read(head, 0, head.length, 0).finally(() => handle.close());
- assertNotMediaManifest(head.subarray(0, bytesRead));
+ const readWindow = async (position) => {
+ const window = Buffer.alloc(MEDIA_MANIFEST_PROBE_BYTES);
+ const { bytesRead } = await handle.read(window, 0, window.length, position);
+ return window.subarray(0, bytesRead);
+ };
+ const windows = await (async () => {
+ const head = await readWindow(0);
+ const tagLength = id3v2TagLength(head);
+ return tagLength >= head.length && head.length === MEDIA_MANIFEST_PROBE_BYTES ? [head, await readWindow(tagLength)] : [head];
+ })().finally(() => handle.close());
+ for (const window of windows) assertNotMediaManifest(window);
}
/**
diff --git a/server/services/pipeline/musicLibrary.test.js b/server/services/pipeline/musicLibrary.test.js
index 20bb3e28e1..804db6cad7 100644
--- a/server/services/pipeline/musicLibrary.test.js
+++ b/server/services/pipeline/musicLibrary.test.js
@@ -182,6 +182,19 @@ describe('importUploadedTrack', () => {
expect(await readdir(FAKE_MUSIC_DIR)).toEqual([]);
expect(existsSync(tmp)).toBe(false);
});
+
+ // The file check reads a second window when the tag outgrows the first.
+ it('refuses a playlist hidden behind an ID3v2 tag longer than the read window', async () => {
+ const tmp = join(TEST_HOME, 'upload-tmp.mp3');
+ const tagSize = 2 * 1024 * 1024;
+ const syncsafe = [21, 14, 7, 0].map((shift) => (tagSize >> shift) & 0x7f);
+ await writeFile(tmp, Buffer.concat([
+ Buffer.from([0x49, 0x44, 0x33, 4, 0, 0, ...syncsafe]), Buffer.alloc(tagSize),
+ Buffer.from('#EXTM3U\n#EXTINF:2.0,\n../private/song.m4a\n'),
+ ]));
+ await expect(importUploadedTrack(tmp, 'song.mp3')).rejects.toMatchObject({ status: 400, code: 'INVALID_FILE_TYPE' });
+ expect(await readdir(FAKE_MUSIC_DIR)).toEqual([]);
+ });
});
describe('statMusicTrack', () => {