Skip to content

Open-source gym & body-weight tracker, now in the App Store (iPhone + Apple Watch) and as an Android APK. Plan routines, log workouts, see which muscles are trained, import from Strong/Hevy/FitNotes. Self-host it or use it on the phone alone.

API: PUT and GET /api/media/{hash}, /api/media/missing and /api/media/sweep, the caps in /api/config, and half an hour for a slow upload

Duarte Santos154b9d88778b
diff --git a/api/Dockerfile b/api/Dockerfile
index 69b8c5198..3db39e238 100644
--- a/api/Dockerfile
+++ b/api/Dockerfile
@@ -53,7 +53,7 @@ RUN npm ci --omit=dev --omit=optional && npm cache clean --force
# Every module server.js imports from the api root has to be named here: the image carries nothing
# else, and one that is missing only shows at boot, as ERR_MODULE_NOT_FOUND.
-COPY server.js push-messages.js verify-error.js password.js rate-limit.js passkeys-store.js device-link.js ./
+COPY server.js push-messages.js verify-error.js password.js rate-limit.js passkeys-store.js device-link.js media.js ./
COPY coach ./coach
ENV NODE_ENV=production
ENV PORT=3000
diff --git a/api/openapi.yaml b/api/openapi.yaml
index bd1a26567..3cffddb10 100644
--- a/api/openapi.yaml
+++ b/api/openapi.yaml
@@ -78,10 +78,19 @@ info:
`GET /api/config`. Off by default.
- `TRUST_PROXY=1` — the throttle reads the client address from proxy headers (set by
the bundled `docker-compose.yml`, where only the web container can reach the API).
+ - `MEDIA_UPLOADS` (default on; `0` removes the `media` routes and the `media` block of
+ `GET /api/config`), `MEDIA_QUOTA_MB` (200 per profile, `0` = no cap),
+ `MEDIA_IMAGE_MAX_MB` (2), `MEDIA_GIF_MAX_MB` (8), `MEDIA_VIDEO_MAX_MB` (40),
+ `MEDIA_VIDEO_MAX_SEC` (60), `MEDIA_GC_GRACE_DAYS` (14), `MEDIA_UPLOADS_PER_HOUR` (600
+ per profile), `MEDIA_MIN_FREE_MB` (512, `0` = no floor) — photos and videos of custom
+ exercises (tag `media`). Every MB here is 2^20 bytes.
### Conventions
- - Every response body is JSON with `Cache-Control: no-store`.
- - Errors are always `{"error": "<human-readable message>"}`.
+ - Every response body is JSON with `Cache-Control: no-store`. The one exception is
+ `GET /api/media/{hash}`, which answers with the stored file itself
+ (`Cache-Control: private, no-store`).
+ - Errors are always `{"error": "<human-readable message>"}`. The `password`, throttle and
+ `media` routes add a stable `code` for the client to word in its own language.
- Unknown method+path pairs return `404 {"error":"not found"}`; an unhandled
exception returns `500 {"error":"server error"}`.
- Request bodies are parsed as JSON regardless of Content-Type. A body that does
@@ -128,6 +137,11 @@ tags:
description: Live-workout presence heartbeat
- name: admin
description: Admin dashboard — requires an admin session (`ADMIN_UIDS` or `admin:true`)
+ - name: media
+ description: >-
+ The photo, GIF or video of a user's own custom exercise. Stored per profile and named
+ by the SHA-256 of its bytes; the state only carries a `MediaRef`. Every route here is a
+ 404 when the instance runs with `MEDIA_UPLOADS=0`.
- name: coach
description: >-
AI Coach — plans, reviews and debriefs. Every route here answers 503 unless the
@@ -169,6 +183,10 @@ paths:
environment variables on the server (`INVITE_ONLY`, `ALLOW_GUEST`), and so does
`password_login`, which is there only when `PASSWORD_LOGIN=1`.
+ `media` is there unless the instance runs with `MEDIA_UPLOADS=0`, for everybody: the
+ caps are not a secret, and its absence is how the app knows this server does not
+ store photos and videos at all.
+
A caller with a valid session also gets a `coach` key: the block when the
instance has the AI Coach switched on **and** a provider connected, and `null`
when it has not. A caller with no session gets no key at all — the block names
@@ -221,6 +239,8 @@ paths:
Present (and true) only when the instance runs with
PASSWORD_LOGIN=1 — the client then offers name-and-password
sign-in next to passkeys.
+ media:
+ $ref: '#/components/schemas/MediaConfig'
examples:
anonymous: { value: { invite_only: true, allow_guest: false } }
signedInNoCoach: { value: { invite_only: true, allow_guest: false, coach: null } }
@@ -230,6 +250,7 @@ paths:
allow_guest: false
coach: { enabled: true, provider: compatible, providerLabel: "OpenAI-compatible endpoint", authMode: instance, community: false }
passwordLogin: { value: { invite_only: true, allow_guest: false, password_login: true } }
+ media: { value: { invite_only: false, allow_guest: true, media: { imageMB: 2, gifMB: 8, videoMB: 40, videoSec: 60, quotaMB: 200 } } }
/api/me:
get:
@@ -1232,6 +1253,10 @@ paths:
from before revisions, and deliberate replaces such as a backup import). The
server stamps the new revision into the document as `_rev`; a `_rev` sent by
the client is ignored.
+
+ Every accepted write also starts the grace period of each stored file (tag `media`)
+ the new state no longer references, and stops it for each one it references again.
+ That bookkeeping never changes the answer.
requestBody:
required: true
content:
@@ -1310,6 +1335,248 @@ paths:
rev: { type: integer }
'401': { $ref: '#/components/responses/Unauthorized' }
+ /api/media/{hash}:
+ parameters:
+ - name: hash
+ in: path
+ required: true
+ description: The SHA-256 of the file's bytes, 64 lowercase hex characters.
+ schema: { type: string, pattern: '^[0-9a-f]{64}$' }
+ get:
+ tags: [media]
+ operationId: getMedia
+ summary: Download one of your photos or videos
+ description: |
+ The stored file, whole, from the caller's own folder only: another profile's file
+ answers exactly like one that was never uploaded (404). The body is the file, not
+ JSON, with headers that keep a browser from ever treating it as a page:
+ `Content-Type` from the sniffed type (never `text/*` or SVG),
+ `X-Content-Type-Options: nosniff`, `Content-Security-Policy: default-src 'none'; sandbox`,
+ `Cross-Origin-Resource-Policy: same-origin`, `Content-Disposition: inline`,
+ `X-Robots-Tag: noindex` and `Cache-Control: private, no-store` — the app keeps its own
+ verified copy, and an HTTP-cache copy would outlive signing out.
+
+ No `Range` and no `ETag`: the app downloads a file once, checks it against its hash and
+ keeps it. A paired phone fetches with its bearer token; `<img src>` cannot carry one.
+ Not throttled.
+ responses:
+ '200':
+ description: The file.
+ headers:
+ Content-Length: { schema: { type: integer } }
+ Content-Disposition: { schema: { type: string }, example: 'inline; filename="<hash>.webp"' }
+ content:
+ image/jpeg: { schema: { type: string, format: binary } }
+ image/png: { schema: { type: string, format: binary } }
+ image/webp: { schema: { type: string, format: binary } }
+ image/gif: { schema: { type: string, format: binary } }
+ video/mp4: { schema: { type: string, format: binary } }
+ video/quicktime: { schema: { type: string, format: binary } }
+ video/webm: { schema: { type: string, format: binary } }
+ '401': { $ref: '#/components/responses/Unauthorized' }
+ '404':
+ description: Not in the caller's folder — never uploaded, removed by the sweep, or someone else's.
+ content:
+ application/json:
+ schema: { $ref: '#/components/schemas/Error' }
+ example: { error: "no such file", code: "media-missing" }
+ put:
+ tags: [media]
+ operationId: putMedia
+ summary: Upload one photo, GIF or video
+ description: |
+ The body is the raw file; `Content-Type` must be one of the seven types below and
+ `Content-Length` is optional (chunked uploads are capped while they stream). The path
+ names the SHA-256 of the bytes, and the file is stored only if they hash to it.
+ Idempotent: uploading a file the server already has answers 200 with `existed: true`.
+
+ What is stored is decided by the file's first bytes, not by the declared type. The
+ declared type only has to be in the same category — a still (JPEG, PNG, WebP, GIF) or a
+ video (MP4, MOV, WebM): a JPEG declared as a GIF is stored as the JPEG it is, a video
+ declared as an image is refused. The cap of the kind the bytes turned out to be applies
+ (`MEDIA_IMAGE_MAX_MB`, `MEDIA_GIF_MAX_MB`, `MEDIA_VIDEO_MAX_MB`), and an MP4 or MOV must
+ be a well-formed file no longer than `MEDIA_VIDEO_MAX_SEC` (+1 s), read from its
+ headers. HEIC/AVIF, SVG, HTML and every other type are refused; the app converts HEIC
+ photos before uploading. The server never decodes or alters a file: the app has already
+ re-encoded photos and removed video metadata on the device.
+
+ Checked in this order: session (401), the hourly budget and at most two uploads at once
+ per profile (429), the declared type (415), a declared length over the cap (413 before
+ a byte is read), already stored (200), the quota (413 `media-quota`; files the profile
+ already dropped get one hour instead of the grace to make room), free disk (507), then
+ while and after streaming: size (413), hash (400), sniffed type (415), sniffed cap
+ (413), video checks (415/413). No bytes for 60 s ends the upload with 408. A refused
+ upload is read and discarded up to twice its cap, so the answer reaches the client.
+
+ A new file that the profile's state does not reference yet starts its grace period
+ at once, so a state push that never comes does not keep it forever.
+ requestBody:
+ required: true
+ content:
+ image/jpeg: { schema: { type: string, format: binary } }
+ image/png: { schema: { type: string, format: binary } }
+ image/webp: { schema: { type: string, format: binary } }
+ image/gif: { schema: { type: string, format: binary } }
+ video/mp4: { schema: { type: string, format: binary } }
+ video/quicktime: { schema: { type: string, format: binary } }
+ video/webm: { schema: { type: string, format: binary } }
+ responses:
+ '201':
+ description: Stored.
+ content:
+ application/json:
+ schema: { $ref: '#/components/schemas/MediaStored' }
+ example: { ok: true, hash: "9f86d081884c7d659a2feaa0c55ad015a3bf4f1b2b0b822cd15d6c15b0f00a08", mime: "image/webp", size: 183204, existed: false }
+ '200':
+ description: The server already had these bytes (from this or another of your devices).
+ content:
+ application/json:
+ schema: { $ref: '#/components/schemas/MediaStored' }
+ example: { ok: true, hash: "9f86d081884c7d659a2feaa0c55ad015a3bf4f1b2b0b822cd15d6c15b0f00a08", mime: "image/webp", size: 183204, existed: true }
+ '400':
+ description: The bytes do not hash to the name in the path (`hash-mismatch`).
+ content:
+ application/json:
+ schema: { $ref: '#/components/schemas/Error' }
+ example: { error: "the file does not match its name", code: "hash-mismatch" }
+ '401': { $ref: '#/components/responses/Unauthorized' }
+ '403':
+ description: A browser request from another origin (see *CSRF*).
+ content:
+ application/json:
+ schema: { $ref: '#/components/schemas/Error' }
+ example: { error: "cross-origin request refused" }
+ '408':
+ description: No bytes arrived for 60 seconds (`timeout`). The connection is closed.
+ content:
+ application/json:
+ schema: { $ref: '#/components/schemas/Error' }
+ example: { error: "the upload stalled", code: "timeout" }
+ '413':
+ description: |
+ - `media-too-large` — over the cap of its kind; `maxMB` says the cap.
+ - `media-quota` — the profile's space is full; `usedMB` and `quotaMB` say how full.
+ - `media-too-long` — a video longer than `maxSec` seconds.
+ content:
+ application/json:
+ schema: { $ref: '#/components/schemas/Error' }
+ examples:
+ tooLarge: { value: { error: "that file is too large", code: "media-too-large", maxMB: 2 } }
+ quota: { value: { error: "your space for photos and videos is full", code: "media-quota", usedMB: 199.4, quotaMB: 200 } }
+ tooLong: { value: { error: "that video is too long", code: "media-too-long", maxSec: 60 } }
+ '415':
+ description: >-
+ `media-type` — the declared type is not one of the seven, the bytes are none of
+ them, or they are a video declared as an image (or the reverse). `media-invalid` —
+ an MP4 or MOV whose boxes do not add up.
+ content:
+ application/json:
+ schema: { $ref: '#/components/schemas/Error' }
+ examples:
+ type: { value: { error: "that file type is not accepted", code: "media-type" } }
+ invalid: { value: { error: "that video could not be read", code: "media-invalid" } }
+ '429':
+ description: >-
+ `locked` — more than `MEDIA_UPLOADS_PER_HOUR` uploads this hour; `busy` — two
+ uploads of this profile are already in flight (`retryAfter` 5).
+ headers:
+ Retry-After: { schema: { type: integer } }
+ content:
+ application/json:
+ schema: { $ref: '#/components/schemas/Error' }
+ examples:
+ locked: { value: { error: "too many uploads — try again later", code: "locked", retryAfter: 1740 } }
+ busy: { value: { error: "too many uploads at once — try again in a moment", code: "busy", retryAfter: 5 } }
+ '507':
+ description: The server's disk has less than `MEDIA_MIN_FREE_MB` free (`storage-full`).
+ content:
+ application/json:
+ schema: { $ref: '#/components/schemas/Error' }
+ example: { error: "the server is running out of disk space", code: "storage-full" }
+
+ /api/media/missing:
+ post:
+ tags: [media]
+ operationId: mediaMissing
+ summary: Which of these files the server does not have
+ description: |
+ The app sends every hash its state references and uploads only what comes back, so the
+ same call moves a guest's files into a new account, a phone's into its pairing, a
+ backup's into the server, and re-sends a file the sweep removed while a device still
+ had it. Answered from the caller's own folder only.
+ requestBody:
+ required: true
+ content:
+ application/json:
+ schema:
+ type: object
+ required: [hashes]
+ properties:
+ hashes:
+ type: array
+ maxItems: 1000
+ items: { type: string, pattern: '^[0-9a-f]{64}$' }
+ responses:
+ '200':
+ description: The subset the server does not have, in the order sent, and the profile's usage.
+ content:
+ application/json:
+ schema:
+ type: object
+ properties:
+ missing:
+ type: array
+ items: { type: string }
+ usage: { $ref: '#/components/schemas/MediaUsage' }
+ example: { missing: ["b94d27b9934d3e08a52e52d7da7dabfac484efe37a5380ee9088f7ace2efcde9"], usage: { bytes: 52428800, count: 14, quotaBytes: 209715200 } }
+ '400':
+ description: '`hashes` is missing, not a list, longer than 1000, or holds something that is not a lowercase SHA-256 (`bad-request`).'
+ content:
+ application/json:
+ schema: { $ref: '#/components/schemas/Error' }
+ '401': { $ref: '#/components/responses/Unauthorized' }
+
+ /api/media/sweep:
+ post:
+ tags: [media]
+ operationId: mediaSweep
+ summary: Delete every file your current state does not reference
+ description: |
+ What "Reset everything" does after pushing the empty state: every stored file of the
+ caller that the stored state does not reference goes now, without the grace period.
+ A state that cannot be read removes nothing. At most 10 an hour per profile (429
+ `locked`). Recorded in the activity log as `media.sweep`.
+
+ Without this call nothing is lost either: the hourly sweep removes a file once the
+ state has not referenced it for `MEDIA_GC_GRACE_DAYS` days (14), and only for profiles
+ whose state reads — a missing or unreadable state, or a profile missing from `db.json`,
+ never deletes anything.
+ requestBody:
+ content:
+ application/json:
+ schema: { type: object }
+ example: {}
+ responses:
+ '200':
+ description: What was removed, and what is left.
+ content:
+ application/json:
+ schema:
+ type: object
+ properties:
+ removed: { type: integer }
+ freedBytes: { type: integer }
+ usage: { $ref: '#/components/schemas/MediaUsage' }
+ '401': { $ref: '#/components/responses/Unauthorized' }
+ '429':
+ description: More than 10 sweeps this hour (`locked`).
+ headers:
+ Retry-After: { schema: { type: integer } }
+ content:
+ application/json:
+ schema: { $ref: '#/components/schemas/Error' }
+ example: { error: "too many uploads — try again later", code: "locked", retryAfter: 1200 }
+
/api/push/public-key:
get:
tags: [push]
@@ -1675,8 +1942,9 @@ paths:
summary: 'Admin: delete a user and everything of theirs'
description: |
Removes the account for good: the user record, their passkeys, their push
- subscriptions, their live presence, their training history file and any Coach
- credential stored for them. Not reversible — `/api/admin/user/disable` is the
+ subscriptions, their live presence, their training history file, their uploaded
+ photos and videos (`DATA_DIR/uploads/<uid>/`) and any Coach credential stored for
+ them. Not reversible — `/api/admin/user/disable` is the
reversible one. An admin cannot delete their own account, and the last admin
cannot be deleted. Audited by name, since the id stops meaning anything.
requestBody:
@@ -2530,8 +2798,10 @@ components:
code:
type: string
description: >-
- A stable reason on the password and throttle routes, for a client to word in
- its own language (`bad-credentials`, `locked`, `too-short`, …).
+ A stable reason on the password, throttle and media routes, for a client to word
+ in its own language (`bad-credentials`, `locked`, `too-short`, `media-quota`, …).
+ Media refusals add the numbers to word it with (`maxMB`, `maxSec`, `usedMB`,
+ `quotaMB`).
retryAfter:
type: integer
description: 429 only — the same number of seconds as the Retry-After header.
@@ -2719,7 +2989,7 @@ components:
customEx:
type: array
description: User-created exercises (id, name, muscles…).
- items: { type: object, additionalProperties: true }
+ items: { $ref: '#/components/schemas/CustomExercise' }
exWeights:
type: object
description: 'Per-exercise last/best weight memory: exercise id → {w, d}.'
@@ -2748,6 +3018,84 @@ components:
additionalProperties: true
additionalProperties: true
+ CustomExercise:
+ type: object
+ description: >-
+ One exercise the user made (representative shape). The server stores it as sent and
+ reads only `media` from it, to know which uploaded files are still referenced.
+ properties:
+ id: { type: string }
+ n: { type: string, description: Name. }
+ bp: { type: string, description: Body part. }
+ custom: { type: boolean, const: true }
+ media:
+ $ref: '#/components/schemas/MediaRef'
+ url:
+ type: string
+ format: uri
+ maxLength: 2048
+ description: >-
+ One link — a video or a guide. http(s) only, no credentials. Neither the app nor
+ the server ever fetches it: the app opens it in a new browsing context on a tap.
+ _ts: { type: number, description: 'When this entry was last edited, ms — for merging two devices'' copies.' }
+ additionalProperties: true
+
+ MediaRef:
+ type: object
+ description: >-
+ What the state knows of an uploaded file. The bytes are behind
+ `GET /api/media/{hash}`; the server keeps a file as long as some `customEx[].media.hash`
+ or `.poster.hash` of the profile's state names it, and for `MEDIA_GC_GRACE_DAYS` after.
+ required: [kind, hash, mime, size, width, height, at]
+ properties:
+ kind: { type: string, enum: [image, gif, video], description: '`gif` means animated; a one-frame GIF is an `image`.' }
+ hash: { type: string, pattern: '^[0-9a-f]{64}$', description: SHA-256 of the stored file. }
+ mime: { type: string, enum: [image/webp, image/jpeg, image/png, image/gif, video/mp4, video/quicktime, video/webm] }
+ size: { type: integer, minimum: 1 }
+ width: { type: integer, minimum: 1, maximum: 16384 }
+ height: { type: integer, minimum: 1, maximum: 16384 }
+ dur: { type: number, minimum: 0, maximum: 3600, description: 'Seconds, one decimal; videos and animated GIFs, absent when unknown.' }
+ codec: { type: string, enum: [avc1, hvc1, av01, vp09, vp8, vp9, other], description: Videos only. }
+ poster:
+ type: object
+ description: A still of at most 480 px — always for images and GIFs, when possible for videos.
+ properties:
+ hash: { type: string, pattern: '^[0-9a-f]{64}$' }
+ mime: { type: string, enum: [image/webp, image/jpeg] }
+ size: { type: integer }
+ width: { type: integer }
+ height: { type: integer }
+ at: { type: number, description: When it was attached, ms. }
+
+ MediaConfig:
+ type: object
+ description: >-
+ The instance's media caps (`MEDIA_*`), in MB of 2^20 bytes. Absent when the instance
+ runs with `MEDIA_UPLOADS=0`.
+ properties:
+ imageMB: { type: number, description: 'A photo after the app re-encoded it, or a poster.' }
+ gifMB: { type: number }
+ videoMB: { type: number }
+ videoSec: { type: number }
+ quotaMB: { type: number, description: Per profile; 0 = no cap. }
+ example: { imageMB: 2, gifMB: 8, videoMB: 40, videoSec: 60, quotaMB: 200 }
+
+ MediaUsage:
+ type: object
+ properties:
+ bytes: { type: integer, description: What the profile's stored files take. }
+ count: { type: integer }
+ quotaBytes: { type: integer, description: The cap; 0 = no cap. }
+
+ MediaStored:
+ type: object
+ properties:
+ ok: { type: boolean, const: true }
+ hash: { type: string }
+ mime: { type: string, description: 'What the bytes are — not necessarily what was declared.' }
+ size: { type: integer }
+ existed: { type: boolean, description: The server already had this file. }
+
Workout:
type: object
description: One completed workout (representative shape).
diff --git a/api/server.js b/api/server.js
index 645ec29fa..4d3415bc9 100644
--- a/api/server.js
+++ b/api/server.js
@@ -7,6 +7,7 @@ import path from 'node:path';
import https from 'node:https';
import dns from 'node:dns';
import net from 'node:net';
+import { pipeline } from 'node:stream/promises';
import {
generateRegistrationOptions, verifyRegistrationResponse,
generateAuthenticationOptions, verifyAuthenticationResponse
@@ -28,6 +29,7 @@ import {
listPasskeys, addPasskeyRecord, renamePasskeyRecord, removePasskeyRecord, MAX_PASSKEYS
} from './passkeys-store.js';
import { createDeviceLink, findDeviceLink, burnDeviceLink, dropDeviceLinks } from './device-link.js';
+import { createMediaStore, mediaLimits, mediaConfig, MediaError, HASH_RE } from './media.js';
const PORT = +(process.env.PORT || 3000);
const DATA = process.env.DATA_DIR || '/data';
@@ -1407,6 +1409,126 @@ const passkeyRoutes = {
}
};
+/* ---------- photos & videos of custom exercises (api/media.js) ---------- */
+// One photo, GIF or short video per custom exercise, stored per profile and named by its sha256.
+// The state only ever carries a small ref; the bytes arrive and leave through the routes below.
+// MEDIA_UPLOADS=0 takes the routes and the /api/config block away, but the store is created
+// either way: an admin deleting a profile must still remove files uploaded while it was on.
+const MEDIA_LIMITS = mediaLimits(process.env);
+const MEDIA_ON = MEDIA_LIMITS.enabled;
+const MEDIA = createMediaStore({ dir: path.join(DATA, 'uploads'), limits: MEDIA_LIMITS, readState });
+// Leftovers of uploads the previous process was receiving when it stopped.
+try { MEDIA.cleanTmp(); } catch (e) { console.error('media: boot cleanup failed', e.message); }
+// Per profile, not per address: every upload is signed in, and behind a proxy one address can be
+// a whole household. The hourly budget is generous (an import of a backup re-uploads everything
+// at once); the two-in-flight cap in media.js is what keeps one phone from holding many sockets.
+const MEDIA_BURST = createWindow({ max: MEDIA_LIMITS.uploadsPerHour, windowMs: 3600000 });
+// Deleting everything unreferenced is what "Reset everything" does; ten an hour is plenty.
+const MEDIA_SWEEPS = createWindow({ max: 10, windowMs: 3600000 });
+// Uid -> until when its throttle has already been recorded, so a client hammering a closed
+// window writes one audit line per window, not one per request.
+const mediaThrottleNoted = new Map();
+setInterval(() => {
+ MEDIA_BURST.sweep(); MEDIA_SWEEPS.sweep();
+ for (const [k, until] of mediaThrottleNoted) if (until < Date.now()) mediaThrottleNoted.delete(k);
+}, 600000).unref();
+function mediaThrottle(req, user, win) {
+ const wait = win.take(user.id);
+ if (!wait) return;
+ if ((mediaThrottleNoted.get(user.id) || 0) < Date.now()) {
+ mediaThrottleNoted.set(user.id, Date.now() + wait * 1000);
+ audit(req, 'media.throttled', { ok: false, user, msg: win === MEDIA_SWEEPS ? 'sweep' : 'upload' });
+ }
+ throw new MediaError(429, 'locked', { retryAfter: wait }, { 'Retry-After': String(wait) });
+}
+// Removes files that no profile's readable state has referenced for MEDIA_GC_GRACE_DAYS. Only
+// profiles in db.json are swept, and only when their state parses (see media.js for why a
+// missing anything never means "delete"). First pass a few minutes after boot, so an instance
+// that is redeployed more often than hourly still gets one.
+function mediaSweepAll() {
+ try {
+ const r = MEDIA.sweepAll({ uids: db.users.map(u => u.id) });
+ if (r.removed || r.tmp) console.log(`media: swept ${r.removed} unreferenced file(s), ${(r.freedBytes / 1048576).toFixed(1)} MB, ${r.tmp} stale upload(s)`);
+ } catch (e) { console.error('media: sweep failed', e); }
+}
+if (MEDIA_ON) {
+ setInterval(mediaSweepAll, 3600000).unref();
+ setTimeout(mediaSweepAll, 5 * 60000).unref();
+}
+
+// A stored file, streamed. The headers are the point: the bytes were chosen by a user, so the
+// answer must never be something a browser would run. The Content-Type comes from the sniffed
+// extension (never text/* or SVG), nosniff stops a browser second-guessing it, the sandbox CSP
+// neuters it even if opened as a page, CORP keeps other sites from embedding it, and no-store
+// keeps a copy out of the HTTP cache — the app's own media store is the only client cache, and
+// an HTTP-cache copy would outlive the sign-out purge. No Range and no ETag: nothing asks.
+async function sendMediaFile(res, f, hash) {
+ let fd;
+ try { fd = fs.openSync(f.path, 'r'); }
+ catch { throw new MediaError(404, 'media-missing'); } // swept between the lookup and here
+ const size = fs.fstatSync(fd).size;
+ res.writeHead(200, {
+ 'Content-Type': f.mime,
+ 'Content-Length': String(size),
+ 'Cache-Control': 'private, no-store',
+ 'X-Content-Type-Options': 'nosniff',
+ 'Content-Security-Policy': "default-src 'none'; sandbox",
+ 'Cross-Origin-Resource-Policy': 'same-origin',
+ 'Content-Disposition': `inline; filename="${hash}.${f.ext}"`,
+ 'X-Robots-Tag': 'noindex',
+ // The paired phone reads the length before the body to refuse an oversized answer; it is
+ // CORS-safelisted already, this only spells it out for older WebViews.
+ 'Access-Control-Expose-Headers': 'Content-Length'
+ });
+ try { await pipeline(fs.createReadStream(null, { fd }), res); }
+ catch { res.destroy(); } // the client went away mid-download; nothing left to answer
+}
+
+const mediaRoutes = {
+ // The raw bytes of one file, named by the sha256 the client computed. media.js checks the
+ // hash, the magic bytes, the caps, the quota and the free disk; this route only adds the
+ // session and the hourly budget.
+ 'PUT /api/media/{hash}': async (req, res) => {
+ const user = readSession(req);
+ if (!user) return json(res, 401, { error: 'not signed in' });
+ mediaThrottle(req, user, MEDIA_BURST);
+ const r = await MEDIA.receive(user.id, req.mediaHash, req);
+ json(res, r.status, r.body);
+ },
+ 'GET /api/media/{hash}': async (req, res) => {
+ const user = readSession(req);
+ if (!user) return json(res, 401, { error: 'not signed in' });
+ // Only ever the caller's own folder: another profile's file is exactly as missing as one
+ // that was never uploaded.
+ const f = MEDIA.file(user.id, req.mediaHash);
+ if (!f) throw new MediaError(404, 'media-missing');
+ await sendMediaFile(res, f, req.mediaHash);
+ },
+ // Which of these the server does not have, so a device uploads only those. Answered from the
+ // in-memory list of the caller's own folder.
+ 'POST /api/media/missing': async (req, res) => {
+ const user = readSession(req);
+ if (!user) return json(res, 401, { error: 'not signed in' });
+ const body = await readBody(req);
+ const hashes = body.hashes;
+ if (!Array.isArray(hashes) || hashes.length > 1000 || !hashes.every(h => typeof h === 'string' && HASH_RE.test(h))) {
+ throw new MediaError(400, 'bad-request', { error: 'hashes must be a list of at most 1000 lowercase sha256 hex strings' });
+ }
+ json(res, 200, MEDIA.missing(user.id, hashes));
+ },
+ // "Reset everything": every file the caller's current state does not reference goes now,
+ // without the grace. A state that cannot be read removes nothing.
+ 'POST /api/media/sweep': async (req, res) => {
+ const user = readSession(req);
+ if (!user) return json(res, 401, { error: 'not signed in' });
+ await readBody(req);
+ mediaThrottle(req, user, MEDIA_SWEEPS);
+ const r = MEDIA.sweep(user.id, { graceMs: 0 });
+ audit(req, 'media.sweep', { user, msg: `${r.removed} file(s), ${(r.freedBytes / 1048576).toFixed(1)} MB${r.skipped ? ', state unreadable' : ''}` });
+ json(res, 200, { removed: r.removed, freedBytes: r.freedBytes, usage: MEDIA.usage(user.id) });
+ }
+};
+
/* ---------- routes ---------- */
const routes = {
'GET /api/health': async (req, res) => json(res, 200, { ok: true, users: db.users.length }),
@@ -1434,6 +1556,9 @@ const routes = {
invite_only: INVITE_ONLY, allow_guest: ALLOW_GUEST,
// Only when on, so an instance without passwords answers exactly as it did before (#118).
...(PASSWORD_LOGIN ? { password_login: true } : {}),
+ // Public like the two flags above: the caps are not a secret, and the absence of the
+ // block is how the app knows this server does not take photos and videos at all.
+ ...(MEDIA_ON ? { media: mediaConfig(MEDIA_LIMITS) } : {}),
...(readSession(req) ? { coach: coachConfig.publicConfig() } : {})
});
},
@@ -1725,6 +1850,12 @@ const routes = {
// the old revision until some later write happened to land on a different tick. This is the
// only writer of a state file in the tree, so evicting here is the whole fix.
stateCache.delete(user.id);
+ // Starts (or stops) the grace clock of every stored file this write stopped (or started)
+ // referencing. Bookkeeping only: the state is already saved, so nothing here may turn a
+ // successful write into an error.
+ if (MEDIA_ON) {
+ try { MEDIA.noteState(user.id, body.state); } catch (e) { console.error('media noteState', e); }
+ }
json(res, 200, { ok: true, ts: body.state._ts || null, rev: body.state._rev });
},
@@ -1907,6 +2038,8 @@ const routes = {
// The training history and any Coach credential of theirs, both outside db.json.
try { fs.unlinkSync(stateFile(u.id)); } catch { /* already gone */ }
try { coachConfig.clearProfileAuth(u.id); } catch { /* nothing stored */ }
+ // Their photos and videos — the one place a profile's folder under uploads/ is removed.
+ try { MEDIA.removeUser(u.id); } catch (e) { console.error('media: could not remove uploads of', u.id, e.message); }
saveDb();
// Logged with the name, because the id is about to mean nothing to anyone reading this back.
audit(req, 'admin.user.delete', { user: admin, msg: name });
@@ -1990,7 +2123,12 @@ const routes = {
// Routes live in coach/routes.js and are handed the helpers above rather than importing
// them: they are closures over db and SECRET, and passing them in keeps that module free of
// a cycle. Every one of them is inert while the feature is unconfigured.
- ...coachRoutes({ json, readBody, readSession, requireAdmin })
+ ...coachRoutes({ json, readBody, readSession, requireAdmin }),
+
+ /* ---------- photos & videos ---------- */
+ // Absent, not refusing, when MEDIA_UPLOADS=0: a 404 is what a server from before the feature
+ // answers, and the client already treats that as "this server does not store them".
+ ...(MEDIA_ON ? mediaRoutes : {})
};
/* ---------- Coach: boot recovery, notifications, scheduled reviews ---------- */
@@ -2031,7 +2169,11 @@ const server = http.createServer(async (req, res) => {
let url;
try { url = new URL(req.url, 'http://x'); }
catch { return json(res, 400, { error: 'bad request' }); }
- const key = req.method + ' ' + url.pathname;
+ let key = req.method + ' ' + url.pathname;
+ // The one route with a parameter in its path. Mapped onto its template key here so the table
+ // above stays a plain lookup, and so csrfOk and the catch-all see one name for every file.
+ const mm = /^\/api\/media\/([0-9a-f]{64})$/.exec(url.pathname);
+ if (mm) { key = req.method + ' /api/media/{hash}'; req.mediaHash = mm[1]; }
const handler = routes[key];
if (!handler) return json(res, 404, { error: 'not found' });
if (!csrfOk(req, key)) {
@@ -2056,6 +2198,11 @@ const server = http.createServer(async (req, res) => {
if (!res.headersSent) json(res, e.status, { error: e.message });
return;
}
+ // A refused upload or a missing file: the caller's to act on, never a logged 500.
+ if (e instanceof MediaError) {
+ if (!res.headersSent) json(res, e.status, { error: e.message, code: e.code, ...e.extra }, e.headers);
+ return;
+ }
// Every scrypt slot and the short queue behind them are taken (password.js).
if (e instanceof BusyError) {
if (!res.headersSent) json(res, 503, { error: 'the server is busy — try again in a moment', code: 'busy' }, { 'Retry-After': '2' });
@@ -2065,6 +2212,11 @@ const server = http.createServer(async (req, res) => {
if (!res.headersSent) json(res, 500, { error: 'server error' });
}
});
+// Node's default of 300 s for a whole request would answer 408 to a 40 MB video on a ~1 Mbit/s
+// uplink. Half an hour covers that; a stalled upload is cut much sooner by its own 60 s idle
+// timer in media.js, and a client that never finishes its headers still meets headersTimeout.
+server.requestTimeout = 30 * 60000;
+server.headersTimeout = 60000;
// The port is read back off the listener rather than echoed from PORT, so the line states the
// port that was actually bound: with PORT=0 the OS picks one, and a caller that did not choose it
// (the tests spawn the server that way, and so does anyone running two instances on one box) has
diff --git a/api/test/dockerfile-copy.test.js b/api/test/dockerfile-copy.test.js
new file mode 100644
index 000000000..a7c96c091
--- /dev/null
+++ b/api/test/dockerfile-copy.test.js
@@ -0,0 +1,35 @@
+/* The api image carries only the files named on api/Dockerfile's COPY line (plus coach/). A
+ root-level module that server.js imports and that line forgets is invisible to every test run
+ from the source tree and shows up only when the published image boots — as
+ ERR_MODULE_NOT_FOUND, on every instance that pulled it. So the line is checked here, for
+ server.js's own imports and for whatever those modules import from the api root in turn. */
+import { test } from 'node:test';
+import assert from 'node:assert/strict';
+import fs from 'node:fs';
+import path from 'node:path';
+import { fileURLToPath } from 'node:url';
+
+const API = path.join(path.dirname(fileURLToPath(import.meta.url)), '..');
+const read = f => fs.readFileSync(path.join(API, f), 'utf8');
+
+// Every `./name.js` a file imports, statically or dynamically — root-level only; coach/ is
+// copied as a whole directory.
+const rootImports = src => [...src.matchAll(/(?:from\s+|import\s*\(\s*)['"]\.\/([^'"/]+\.js)['"]/g)].map(m => m[1]);
+
+test('every root-level module server.js needs is on the Dockerfile COPY line', () => {
+ const line = read('Dockerfile').split('\n').find(l => /^COPY\s+server\.js\b/.test(l));
+ assert.ok(line, 'the COPY line that carries server.js');
+ const copied = new Set(line.trim().split(/\s+/).slice(1, -1));
+
+ const needed = new Set(['server.js']);
+ const queue = ['server.js'];
+ while (queue.length) {
+ for (const dep of rootImports(read(queue.shift()))) {
+ if (!needed.has(dep)) { needed.add(dep); queue.push(dep); }
+ }
+ }
+ assert.ok(needed.has('media.js'), 'the walk finds the media store');
+ const missing = [...needed].filter(f => !copied.has(f));
+ assert.deepEqual(missing, [], `add ${missing.join(', ')} to the COPY line in api/Dockerfile`);
+ for (const f of copied) assert.ok(fs.existsSync(path.join(API, f)), `${f} is copied but does not exist`);
+});
diff --git a/api/test/server-config-session.test.js b/api/test/server-config-session.test.js
index e4b2a7fc5..5f0a4da65 100644
--- a/api/test/server-config-session.test.js
+++ b/api/test/server-config-session.test.js
@@ -31,9 +31,11 @@ async function startServer(t, env = {}) {
// The fixture provider is "connected" by definition (coach/config.js isConnected), which is all
// publicConfig() needs to produce a block.
fs.writeFileSync(path.join(dataDir, 'coach.json'), JSON.stringify({ enabled: true, provider: 'fixture' }));
+ // MEDIA_UPLOADS=0: the answers below are compared whole, and they are about the login flags
+ // and the Coach block. The public media block has its own tests in server-media.test.js.
const child = spawn(process.execPath, ['server.js'], {
cwd: API, stdio: ['ignore', 'pipe', 'pipe'],
- env: { ...process.env, PORT: '0', DATA_DIR: dataDir, ORIGIN: 'http://localhost:8080', RP_ID: 'localhost', INVITE_ONLY: '1', ALLOW_GUEST: '0', COACH_DISABLED: '', ...env }
+ env: { ...process.env, PORT: '0', DATA_DIR: dataDir, ORIGIN: 'http://localhost:8080', RP_ID: 'localhost', INVITE_ONLY: '1', ALLOW_GUEST: '0', COACH_DISABLED: '', MEDIA_UPLOADS: '0', ...env }
});
t.after(() => { child.kill('SIGKILL'); fs.rmSync(dataDir, { recursive: true, force: true }); });
let log = '';
diff --git a/api/test/server-media.test.js b/api/test/server-media.test.js
new file mode 100644
index 000000000..674b6ee7a
--- /dev/null
+++ b/api/test/server-media.test.js
@@ -0,0 +1,437 @@
+/* The media routes on a real server.js in a child process: sessions, CSRF, CORS, the throttle,
+ the headers a stored file is served with, the config block, the PUT /api/data hook, admin
+ delete, and what is left on disk after each refusal. Caps are lowered through the MEDIA_*
+ variables where a test needs to cross one. media-receive.test.js and media-gc.test.js cover
+ the store itself in-process. */
+import { test } from 'node:test';
+import assert from 'node:assert/strict';
+import crypto from 'node:crypto';
+import fs from 'node:fs';
+import http from 'node:http';
+import os from 'node:os';
+import path from 'node:path';
+import { spawn } from 'node:child_process';
+import { fileURLToPath } from 'node:url';
+import { boundPort } from './helpers.mjs';
+import * as M from './media-samples.mjs';
+
+const API = path.join(path.dirname(fileURLToPath(import.meta.url)), '..');
+const ORIGIN = 'http://localhost:8080';
+const SECRET = crypto.randomBytes(32).toString('hex');
+const mint = uid => {
+ const payload = `${uid}:${Date.now() + 86400000}:0`;
+ return payload + '.' + crypto.createHmac('sha256', SECRET).update(payload).digest('base64url');
+};
+const U1 = 'u_media_1', U2 = 'u_media_2', ADMIN = 'u_media_adm';
+const cookie = uid => ({ Cookie: `gymsid=${mint(uid)}`, Origin: ORIGIN });
+const bearer = uid => ({ Authorization: `Bearer ${mint(uid)}` });
+const refState = (...hashes) => ({
+ unit: 'kg', routines: [], workouts: [],
+ customEx: hashes.map((h, i) => ({ id: 'cx' + i, n: 'Sandbag ' + i, bp: 'back', custom: true, media: { kind: 'image', hash: h, mime: 'image/jpeg', size: 1, width: 1, height: 1, at: 1 } }))
+});
+
+async function start(t, env = {}) {
+ const dataDir = fs.mkdtempSync(path.join(os.tmpdir(), 'gym-media-'));
+ fs.writeFileSync(path.join(dataDir, 'secret'), SECRET, { mode: 0o600 });
+ fs.writeFileSync(path.join(dataDir, 'db.json'), JSON.stringify({
+ users: [
+ { id: U1, name: 'Ana', created: new Date().toISOString() },
+ { id: U2, name: 'Bo', created: new Date().toISOString() },
+ { id: ADMIN, name: 'Admin', created: new Date().toISOString(), admin: true }
+ ],
+ creds: [], subs: [], invites: []
+ }));
+ const child = spawn(process.execPath, ['server.js'], {
+ cwd: API, stdio: ['ignore', 'pipe', 'pipe'],
+ env: { ...process.env, PORT: '0', DATA_DIR: dataDir, ORIGIN, RP_ID: 'localhost', MEDIA_MIN_FREE_MB: '1', ...env }
+ });
+ const h = { log: '', dataDir, uploads: path.join(dataDir, 'uploads') };
+ child.stdout.on('data', d => h.log += d);
+ child.stderr.on('data', d => h.log += d);
+ t.after(() => { child.kill('SIGKILL'); fs.rmSync(dataDir, { recursive: true, force: true }); });
+ h.port = await boundPort(child, () => h.log);
+ h.api = `http://127.0.0.1:${h.port}`;
+ h.put = (bytes, { uid = U1, declare = 'image/jpeg', hash = M.sha(bytes), headers } = {}) =>
+ fetch(`${h.api}/api/media/${hash}`, { method: 'PUT', headers: { ...(headers || cookie(uid)), 'Content-Type': declare }, body: bytes });
+ h.get = (hash, uid = U1, headers) => fetch(`${h.api}/api/media/${hash}`, { headers: headers || cookie(uid) });
+ h.post = (route, body, uid = U1) => fetch(`${h.api}${route}`, { method: 'POST', headers: { ...cookie(uid), 'Content-Type': 'application/json' }, body: JSON.stringify(body) });
+ h.pushState = (state, uid = U1) => fetch(`${h.api}/api/data`, { method: 'PUT', headers: { ...cookie(uid), 'Content-Type': 'application/json' }, body: JSON.stringify({ state }) });
+ h.files = uid => { try { return fs.readdirSync(path.join(h.uploads, uid)).filter(n => !n.startsWith('.')).sort(); } catch { return []; } };
+ h.tmp = uid => { try { return fs.readdirSync(path.join(h.uploads, uid, '.tmp')); } catch { return []; } };
+ h.stackFrames = () => h.log.split('\n').filter(l => /^\s+at /.test(l)).length;
+ return h;
+}
+
+/* A request written by hand, for what fetch cannot do: stop halfway, stream without a length,
+ or claim a length it never sends. Resolves `response` with {status, headers, body}. */
+function rawPut(h, hash, { headers = {}, declare = 'image/jpeg', length } = {}) {
+ const hd = { ...cookie(U1), 'Content-Type': declare, ...headers };
+ if (length != null) hd['Content-Length'] = String(length);
+ const req = http.request(`${h.api}/api/media/${hash}`, { method: 'PUT', headers: hd });
+ const response = new Promise((resolve, reject) => {
+ req.on('response', res => {
+ let body = '';
+ res.setEncoding('utf8');
+ res.on('data', d => body += d);
+ res.on('end', () => { let j = null; try { j = JSON.parse(body); } catch { /* not json */ } resolve({ status: res.statusCode, headers: res.headers, body: j }); });
+ });
+ req.on('error', reject);
+ });
+ return { req, response };
+}
+const sleep = ms => new Promise(r => setTimeout(r, ms));
+
+test('every media route wants a session', async t => {
+ const h = await start(t);
+ const hash = 'a'.repeat(64);
+ const noAuth = { Origin: ORIGIN };
+ assert.equal((await fetch(`${h.api}/api/media/${hash}`, { method: 'PUT', headers: { ...noAuth, 'Content-Type': 'image/jpeg' }, body: M.jpeg() })).status, 401);
+ assert.equal((await fetch(`${h.api}/api/media/${hash}`)).status, 401);
+ assert.equal((await fetch(`${h.api}/api/media/missing`, { method: 'POST', headers: noAuth, body: '{"hashes":[]}' })).status, 401);
+ assert.equal((await fetch(`${h.api}/api/media/sweep`, { method: 'POST', headers: noAuth, body: '{}' })).status, 401);
+ assert.equal(fs.existsSync(h.uploads), false);
+});
+
+test('an upload is served back whole, with the headers that keep it from ever being a page', async t => {
+ const h = await start(t);
+ const bytes = M.webp(5000);
+ const up = await h.put(bytes, { declare: 'image/webp' });
+ assert.equal(up.status, 201);
+ assert.deepEqual(await up.json(), { ok: true, hash: M.sha(bytes), mime: 'image/webp', size: 5000, existed: false });
+
+ const r = await h.get(M.sha(bytes));
+ assert.equal(r.status, 200);
+ assert.equal(r.headers.get('content-type'), 'image/webp');
+ assert.equal(r.headers.get('content-length'), '5000');
+ assert.equal(r.headers.get('cache-control'), 'private, no-store');
+ assert.equal(r.headers.get('x-content-type-options'), 'nosniff');
+ assert.equal(r.headers.get('content-security-policy'), "default-src 'none'; sandbox");
+ assert.equal(r.headers.get('cross-origin-resource-policy'), 'same-origin');
+ assert.equal(r.headers.get('content-disposition'), `inline; filename="${M.sha(bytes)}.webp"`);
+ assert.equal(r.headers.get('x-robots-tag'), 'noindex');
+ assert.equal(r.headers.get('etag'), null);
+ assert.equal(r.headers.get('accept-ranges'), null);
+ assert.ok(Buffer.from(await r.arrayBuffer()).equals(bytes));
+
+ // Again: already there, counted once.
+ const again = await h.put(bytes, { declare: 'image/webp' });
+ assert.equal(again.status, 200);
+ assert.equal((await again.json()).existed, true);
+ const miss = await (await h.post('/api/media/missing', { hashes: [M.sha(bytes)] })).json();
+ assert.deepEqual(miss, { missing: [], usage: { bytes: 5000, count: 1, quotaBytes: 200 * 1048576 } });
+ assert.equal(h.stackFrames(), 0, h.log);
+});
+
+test("another profile's file is exactly as missing as one that was never uploaded", async t => {
+ const h = await start(t);
+ const bytes = M.jpeg();
+ assert.equal((await h.put(bytes)).status, 201);
+ const r = await h.get(M.sha(bytes), U2);
+ assert.equal(r.status, 404);
+ assert.deepEqual(await r.json(), { error: 'no such file', code: 'media-missing' });
+ const never = await h.get('0'.repeat(64), U2);
+ assert.equal(never.status, 404);
+ assert.deepEqual((await (await h.post('/api/media/missing', { hashes: [M.sha(bytes)] }, U2)).json()).missing, [M.sha(bytes)]);
+ // A path that is not a hash is no media route at all.
+ assert.equal((await h.get('A'.repeat(64))).status, 404);
+ assert.equal((await fetch(`${h.api}/api/media/..%2Fstate-${U1}.json`, { headers: cookie(U1) })).status, 404);
+});
+
+test('refusals: a wrong hash, SVG, a video posing as an image — each leaves nothing on disk', async t => {
+ const h = await start(t);
+ const bytes = M.jpeg();
+ let r = await h.put(bytes, { hash: 'f'.repeat(64) });
+ assert.equal(r.status, 400);
+ assert.deepEqual(await r.json(), { error: 'the file does not match its name', code: 'hash-mismatch' });
+
+ r = await h.put(M.svg(), { declare: 'image/svg+xml' });
+ assert.deepEqual([r.status, (await r.json()).code], [415, 'media-type']);
+ r = await h.put(M.svg(), { declare: 'image/png' });
+ assert.deepEqual([r.status, (await r.json()).code], [415, 'media-type']);
+ r = await h.put(M.html(), { declare: 'image/jpeg' });
+ assert.deepEqual([r.status, (await r.json()).code], [415, 'media-type']);
+ r = await h.put(M.mp4(), { declare: 'image/jpeg' });
+ assert.deepEqual([r.status, (await r.json()).code], [415, 'media-type']);
+
+ assert.deepEqual(h.files(U1), []);
+ assert.deepEqual(h.tmp(U1), []);
+ assert.equal(h.stackFrames(), 0, `a refusal is not a server error:\n${h.log}`);
+});
+
+test('a declared length over the cap is answered before the body is sent', async t => {
+ const h = await start(t);
+ const hash = 'c'.repeat(64);
+ const { req, response } = rawPut(h, hash, { length: 100 * 1024 * 1024 });
+ req.write(M.jpeg(1024)); // and never the other 99.99 MB
+ const r = await response;
+ req.destroy();
+ assert.equal(r.status, 413);
+ assert.deepEqual(r.body, { error: 'that file is too large', code: 'media-too-large', maxMB: 2 });
+ assert.deepEqual(h.tmp(U1), []);
+});
+
+test('the cap of what the bytes are applies, and the quota says how full it is', async t => {
+ // 10 KB images, 1 MB GIFs, a 25 KB quota.
+ const h = await start(t, { MEDIA_IMAGE_MAX_MB: '0.009765625', MEDIA_GIF_MAX_MB: '1', MEDIA_QUOTA_MB: '0.0244140625' });
+ const jpegAsGif = M.jpeg(20000);
+ let r = await h.put(jpegAsGif, { declare: 'image/gif' });
+ assert.equal(r.status, 413);
+ assert.deepEqual(await r.json(), { error: 'that file is too large', code: 'media-too-large', maxMB: 0.009765625 });
+
+ assert.equal((await h.put(M.jpeg(9000), { uid: U2 })).status, 201);
+ assert.equal((await h.put(M.jpeg(9000), { uid: U2 })).status, 201);
+ r = await h.put(M.jpeg(9000), { uid: U2 });
+ assert.equal(r.status, 413);
+ assert.deepEqual(await r.json(), { error: 'your space for photos and videos is full', code: 'media-quota', usedMB: 0, quotaMB: 0.0244140625 });
+ assert.equal(h.files(U2).length, 2);
+});
+
+test('a video longer than MEDIA_VIDEO_MAX_SEC is refused, and a broken one too', async t => {
+ const h = await start(t);
+ let r = await h.put(M.mp4({ seconds: 90 }), { declare: 'video/mp4' });
+ assert.deepEqual([r.status, await r.json()], [413, { error: 'that video is too long', code: 'media-too-long', maxSec: 60 }]);
+ r = await h.put(Buffer.concat([M.ftyp(), M.box('mdat', Buffer.alloc(64))]), { declare: 'video/mp4' });
+ assert.deepEqual([r.status, (await r.json()).code], [415, 'media-invalid']);
+ r = await h.put(M.mp4({ brand: 'qt ', seconds: 30, moovAtEnd: true }), { declare: 'video/quicktime' });
+ assert.deepEqual([r.status, (await r.json()).mime], [201, 'video/quicktime']);
+});
+
+test('an upload without Content-Length works, and is cut off at the cap while it streams', async t => {
+ const h = await start(t, { MEDIA_IMAGE_MAX_MB: '0.1' });
+ const small = M.jpeg(50000);
+ let { req, response } = rawPut(h, M.sha(small));
+ req.write(small.subarray(0, 20000));
+ await sleep(20);
+ req.end(small.subarray(20000));
+ let r = await response;
+ assert.equal(r.status, 201, JSON.stringify(r.body));
+
+ const big = M.jpeg(1024 * 1024);
+ ({ req, response } = rawPut(h, M.sha(big)));
+ let answered = false;
+ response.then(() => { answered = true; }, () => { answered = true; });
+ let sent = 0;
+ while (!answered && sent < big.length) {
+ req.write(big.subarray(sent, sent + 16384));
+ sent += 16384;
+ await sleep(2);
+ }
+ r = await response;
+ req.destroy();
+ assert.equal(r.status, 413, 'the client gets the answer, not a reset');
+ assert.equal(r.body.code, 'media-too-large');
+ assert.ok(sent < big.length, `the answer came while the upload was still going (${sent} bytes sent)`);
+ assert.deepEqual(h.files(U1), [`${M.sha(small)}.jpg`]);
+ assert.deepEqual(h.tmp(U1), []);
+});
+
+test('the same file from two devices at once: one file, counted once', async t => {
+ const h = await start(t);
+ const bytes = M.jpeg(1024 * 1024);
+ const rs = await Promise.all([h.put(bytes), h.put(bytes, { headers: bearer(U1) })]);
+ assert.deepEqual(rs.map(r => r.status).sort(), [200, 201]);
+ assert.deepEqual(h.files(U1), [`${M.sha(bytes)}.jpg`]);
+ const miss = await (await h.post('/api/media/missing', { hashes: [M.sha(bytes)] })).json();
+ assert.deepEqual(miss.usage, { bytes: 1024 * 1024, count: 1, quotaBytes: 200 * 1048576 });
+});
+
+test('a cross-site browser upload is refused; a paired phone with its bearer token is not', async t => {
+ const h = await start(t);
+ const bytes = M.jpeg();
+ let r = await h.put(bytes, { headers: { ...cookie(U1), 'Sec-Fetch-Site': 'cross-site' } });
+ assert.equal(r.status, 403);
+ r = await h.put(bytes, { headers: { Cookie: `gymsid=${mint(U1)}`, Origin: 'https://evil.example' } });
+ assert.equal(r.status, 403);
+ assert.deepEqual(h.files(U1), []);
+
+ // The Android WebView's origin; the token is what authenticates, the origin is reflected.
+ const phone = { ...bearer(U1), Origin: 'https://localhost' };
+ r = await h.put(bytes, { headers: phone });
+ assert.equal(r.status, 201);
+ assert.equal(r.headers.get('access-control-allow-origin'), 'https://localhost');
+ assert.equal(r.headers.get('access-control-allow-credentials'), null);
+ const g = await h.get(M.sha(bytes), U1, phone);
+ assert.equal(g.status, 200);
+ assert.equal(g.headers.get('access-control-allow-origin'), 'https://localhost');
+ assert.ok(Buffer.from(await g.arrayBuffer()).equals(bytes));
+
+ // The preflight a WebView sends before that PUT.
+ const pre = await fetch(`${h.api}/api/media/${M.sha(bytes)}`, {
+ method: 'OPTIONS',
+ headers: { Origin: 'https://localhost', 'Access-Control-Request-Method': 'PUT', 'Access-Control-Request-Headers': 'content-type,authorization' }
+ });
+ assert.equal(pre.status, 204);
+ assert.match(pre.headers.get('access-control-allow-methods'), /\bPUT\b/);
+ assert.match(pre.headers.get('access-control-allow-headers'), /Content-Type/);
+ assert.match(pre.headers.get('access-control-allow-headers'), /Authorization/);
+ assert.equal(pre.headers.get('access-control-allow-origin'), 'https://localhost');
+});
+
+test('the hourly budget answers 429 with Retry-After, and is recorded once', async t => {
+ const h = await start(t, { MEDIA_UPLOADS_PER_HOUR: '2' });
+ assert.equal((await h.put(M.jpeg())).status, 201);
+ assert.equal((await h.put(M.jpeg())).status, 201);
+ for (let i = 0; i < 3; i++) {
+ const r = await h.put(M.jpeg());
+ assert.equal(r.status, 429);
+ const retry = +r.headers.get('retry-after');
+ assert.ok(retry > 3000 && retry <= 3600, `retry-after ${retry}`);
+ assert.deepEqual(await r.json(), { error: 'too many uploads — try again later', code: 'locked', retryAfter: retry });
+ }
+ assert.equal((await h.put(M.jpeg(), { uid: U2 })).status, 201, 'per profile, not per instance');
+ const audit = fs.readFileSync(path.join(h.dataDir, 'audit.log'), 'utf8').trim().split('\n').map(l => JSON.parse(l));
+ assert.equal(audit.filter(e => e.ev === 'media.throttled').length, 1);
+});
+
+test('a third upload while two are in flight is refused as busy', async t => {
+ const h = await start(t);
+ const a = rawPut(h, 'a'.repeat(64), { length: 1000000 });
+ const b = rawPut(h, 'b'.repeat(64), { length: 1000000 });
+ a.req.write(M.jpeg(100));
+ b.req.write(M.jpeg(100));
+ a.response.catch(() => {}); b.response.catch(() => {});
+ await sleep(150);
+ const r = await h.put(M.jpeg());
+ assert.equal(r.status, 429);
+ assert.equal(r.headers.get('retry-after'), '5');
+ assert.deepEqual(await r.json(), { error: 'too many uploads at once — try again in a moment', code: 'busy', retryAfter: 5 });
+ a.req.destroy(); b.req.destroy();
+ await sleep(150);
+ assert.equal((await h.put(M.jpeg())).status, 201, 'the slots come back when the two go away');
+ assert.deepEqual(h.tmp(U1), []);
+});
+
+test('507 when the disk has less free than MEDIA_MIN_FREE_MB', async t => {
+ const h = await start(t, { MEDIA_MIN_FREE_MB: String(1024 * 1024 * 1024) });
+ const r = await h.put(M.jpeg());
+ assert.equal(r.status, 507);
+ assert.deepEqual(await r.json(), { error: 'the server is running out of disk space', code: 'storage-full' });
+});
+
+test('/missing answers from the caller\'s own folder and refuses what is not a list of hashes', async t => {
+ const h = await start(t);
+ const a = M.jpeg(), b = M.png();
+ await h.put(a);
+ const r = await (await h.post('/api/media/missing', { hashes: [M.sha(b), M.sha(a), M.sha(b)] })).json();
+ assert.deepEqual(r, { missing: [M.sha(b)], usage: { bytes: 1000, count: 1, quotaBytes: 200 * 1048576 } });
+ for (const body of [{}, { hashes: 'x' }, { hashes: ['A'.repeat(64)] }, { hashes: [1] }, { hashes: Array(1001).fill('a'.repeat(64)) }]) {
+ const res = await h.post('/api/media/missing', body);
+ assert.equal(res.status, 400, JSON.stringify(body).slice(0, 60));
+ assert.equal((await res.json()).code, 'bad-request');
+ }
+});
+
+test('a state push starts the grace of a dropped file, and /sweep removes it at once', async t => {
+ const h = await start(t);
+ const a = M.jpeg(), b = M.jpeg(2000);
+ await h.put(a);
+ await h.put(b);
+ assert.equal((await h.pushState(refState(M.sha(a), M.sha(b)))).status, 200);
+ const gc = () => JSON.parse(fs.readFileSync(path.join(h.uploads, U1, '.gc.json'), 'utf8'));
+ assert.deepEqual(gc(), {}, 'both referenced: no marks');
+ assert.equal((await h.pushState(refState(M.sha(a)))).status, 200);
+ assert.deepEqual(Object.keys(gc()), [M.sha(b)]);
+
+ const r = await h.post('/api/media/sweep', {});
+ assert.equal(r.status, 200);
+ assert.deepEqual(await r.json(), { removed: 1, freedBytes: 2000, usage: { bytes: 1000, count: 1, quotaBytes: 200 * 1048576 } });
+ assert.equal((await h.get(M.sha(b))).status, 404);
+ assert.equal((await h.get(M.sha(a))).status, 200);
+ const audit = fs.readFileSync(path.join(h.dataDir, 'audit.log'), 'utf8');
+ assert.match(audit, /"ev":"media\.sweep"/);
+});
+
+test('/sweep with a state that does not parse removes nothing', async t => {
+ const h = await start(t);
+ const a = M.jpeg();
+ await h.put(a, { uid: U2 });
+ fs.writeFileSync(path.join(h.dataDir, `state-${U2}.json`), '{"customEx": [tor');
+ const r = await h.post('/api/media/sweep', {}, U2);
+ assert.equal(r.status, 200);
+ assert.equal((await r.json()).removed, 0);
+ assert.deepEqual(h.files(U2), [`${M.sha(a)}.jpg`]);
+ // And with no state file at all.
+ fs.unlinkSync(path.join(h.dataDir, `state-${U2}.json`));
+ assert.equal((await (await h.post('/api/media/sweep', {}, U2)).json()).removed, 0);
+ assert.deepEqual(h.files(U2), [`${M.sha(a)}.jpg`]);
+});
+
+test('the sweep route has its own budget of 10 an hour', async t => {
+ const h = await start(t);
+ for (let i = 0; i < 10; i++) assert.equal((await h.post('/api/media/sweep', {})).status, 200);
+ const r = await h.post('/api/media/sweep', {});
+ assert.equal(r.status, 429);
+ assert.equal((await r.json()).code, 'locked');
+});
+
+test('PUT /api/data still answers 200 when the media bookkeeping fails', async t => {
+ const h = await start(t);
+ const a = M.jpeg();
+ await h.put(a);
+ const gcPath = path.join(h.uploads, U1, '.gc.json');
+ fs.rmSync(gcPath, { force: true });
+ fs.mkdirSync(gcPath); // a write to it now throws
+ const r = await h.pushState(refState());
+ assert.equal(r.status, 200);
+ assert.equal((await r.json()).ok, true);
+ assert.match(h.log, /media noteState/);
+ const saved = JSON.parse(fs.readFileSync(path.join(h.dataDir, `state-${U1}.json`), 'utf8'));
+ assert.equal(saved._rev, 1, 'the state itself was saved');
+});
+
+test('folders are 0700 and files 0600, and an admin deleting the profile removes its folder', async t => {
+ const h = await start(t);
+ const a = M.jpeg();
+ await h.put(a);
+ await h.put(M.jpeg(), { uid: U2 });
+ const mode = p => fs.statSync(p).mode & 0o777;
+ assert.equal(mode(h.uploads), 0o700);
+ assert.equal(mode(path.join(h.uploads, U1)), 0o700);
+ assert.equal(mode(path.join(h.uploads, U1, '.tmp')), 0o700);
+ assert.equal(mode(path.join(h.uploads, U1, `${M.sha(a)}.jpg`)), 0o600);
+ assert.equal(mode(path.join(h.uploads, U1, '.gc.json')), 0o600);
+
+ const del = await fetch(`${h.api}/api/admin/user/delete`, { method: 'POST', headers: { ...cookie(ADMIN), 'Content-Type': 'application/json' }, body: JSON.stringify({ id: U1 }) });
+ assert.equal(del.status, 200);
+ assert.equal(fs.existsSync(path.join(h.uploads, U1)), false);
+ assert.equal(fs.existsSync(path.join(h.uploads, U2)), true, 'nobody else is touched');
+});
+
+test('/api/config carries the caps; MEDIA_UPLOADS=0 takes the block and every route away', async t => {
+ const on = await start(t, { MEDIA_QUOTA_MB: '100', MEDIA_VIDEO_MAX_SEC: '30' });
+ const cfg = await (await fetch(`${on.api}/api/config`)).json();
+ assert.deepEqual(cfg.media, { imageMB: 2, gifMB: 8, videoMB: 40, videoSec: 30, quotaMB: 100 });
+
+ const off = await start(t, { MEDIA_UPLOADS: '0' });
+ const cfgOff = await (await fetch(`${off.api}/api/config`)).json();
+ assert.equal('media' in cfgOff, false);
+ const bytes = M.jpeg();
+ assert.equal((await off.put(bytes)).status, 404);
+ assert.equal((await off.get(M.sha(bytes))).status, 404);
+ assert.equal((await off.post('/api/media/missing', { hashes: [] })).status, 404);
+ assert.equal((await off.post('/api/media/sweep', {})).status, 404);
+ assert.equal(fs.existsSync(off.uploads), false);
+});
+
+test('boot removes the temp files of uploads the previous process was receiving', async t => {
+ const dataDir = fs.mkdtempSync(path.join(os.tmpdir(), 'gym-media-boot-'));
+ t.after(() => fs.rmSync(dataDir, { recursive: true, force: true }));
+ const tmp = path.join(dataDir, 'uploads', U1, '.tmp');
+ fs.mkdirSync(tmp, { recursive: true });
+ fs.writeFileSync(path.join(tmp, 'half-an-upload'), 'x');
+ fs.writeFileSync(path.join(dataDir, 'secret'), SECRET);
+ const child = spawn(process.execPath, ['server.js'], { cwd: API, stdio: ['ignore', 'pipe', 'pipe'], env: { ...process.env, PORT: '0', DATA_DIR: dataDir } });
+ t.after(() => child.kill('SIGKILL'));
+ await boundPort(child);
+ assert.deepEqual(fs.readdirSync(tmp), []);
+});
+
+test('server.js gives a slow upload half an hour, not node\'s default five minutes', () => {
+ const src = fs.readFileSync(path.join(API, 'server.js'), 'utf8');
+ const m = /^server\.requestTimeout\s*=\s*([\d_\s*]+);/m.exec(src);
+ assert.ok(m, 'server.requestTimeout is set');
+ const ms = m[1].split('*').reduce((a, x) => a * Number(x.replace(/[_\s]/g, '')), 1);
+ assert.ok(ms >= 30 * 60000, `requestTimeout is ${ms} ms`);
+ const hm = /^server\.headersTimeout\s*=\s*([\d_\s*]+);/m.exec(src);
+ assert.ok(!hm || hm[1].split('*').reduce((a, x) => a * Number(x.replace(/[_\s]/g, '')), 1) <= 60000, 'headers still have to arrive within a minute');
+});
diff --git a/scripts/build-api-docs.mjs b/scripts/build-api-docs.mjs
index 6c11a2734..fdb5b6809 100644
--- a/scripts/build-api-docs.mjs
+++ b/scripts/build-api-docs.mjs
@@ -128,6 +128,7 @@ const TAGS = {
data: { title: 'Data', side: 'State sync' },
push: { title: 'Push', side: 'Notifications &amp; rest timer' },
activity: { title: 'Activity', side: 'Live presence' },
+ media: { title: 'Media', side: 'Photos &amp; videos of your own exercises' },
// A tag missing from this map renders nowhere at all, silently — so every tag in the
// spec needs a line here.
coach: { title: 'AI Coach', side: 'Plans, reviews, debriefs' },