Self-hosted, Self-Improving, Federated Everything App — dev management, agent orchestration, persistent mind, digital twin, second brain, and creative pipeline from a single dashboard over Tailscale to port the user into the OS
JavaScript ai-agentsdashboarddeveloper-toolsdigital-twinquantified-selfsecond-brainself-hostedtailscale
docs: clarify unreadable media queue admission refusal (#12054)
Adam Eivy966ae91c8c53
diff --git a/docs/STORAGE.md b/docs/STORAGE.mdindex 46fa95867c..054fb88964 100644--- a/docs/STORAGE.md+++ b/docs/STORAGE.md@@ -320,7 +320,7 @@ goal data together; interrupted intents remain retryable after restore.**Examples.**- Upload staging — `data/uploads/*`. Ephemeral; do not put in DB except as job references.-- Media job queue — runtime queue state can stay file-backed short term, but **job history and artifact lineage are `db-primary`** and should move to the DB. Admission is durable: `enqueueJob` resolves only once a snapshot containing the job is written, a failed write refuses the job (`503 MEDIA_QUEUE_PERSIST_FAILED`) before any dispatch, and graceful shutdown first stops dispatch (`quiesceMediaJobQueue()`, before its first await — waiting jobs stay queued for the next boot instead of starting during teardown and restoring as interrupted; Run Now answers `503 MEDIA_QUEUE_SHUTTING_DOWN`), then runs the bounded `flushMediaJobQueue()` before exit. The one exception is a boot that found the snapshot unreadable: persistence stays latched off to preserve that file, and the queue keeps accepting work in memory only. Waiting work is bounded: at 250 queued jobs across all lanes (`MAX_PENDING_MEDIA_JOBS`, separate from each lane's running concurrency) a new submission is refused with a retryable `429 MEDIA_QUEUE_FULL` before any state changes; a restored snapshot above the bound is kept whole and only blocks new admissions until it drains. Local video failure holds live in the existing `data/media-jobs.json` envelope as `videoHolds`, beside unchanged `jobs`. They are machine-local dispatch state, never federated and never sent in capability/status payloads to peers; the local queue API exposes them on retained jobs and through `GET /api/media-jobs/holds`, so status and resume remain available when no retained jobs remain. Three matching terminal causes hold one catalog model/runtime until explicit resume. If hold metadata is damaged, boot restores unrelated work but holds all local video, including new submissions, behind an explicit session-only recovery action; the original snapshot stays preserved until repaired and restarted. Migration 344 adds an empty hold list to legacy envelopes without changing jobs; fresh installs use the empty seed. The envelope retains its existing filesystem backup coverage, with no new store, search index, sync cursor, or tombstone. A completion that cannot enter backup publication admission retains an optional `durabilityPending` receipt in its existing job row, while the published status stays `running`. Boot restores that receipt for admitted completion and hook delivery instead of rerunning the provider or marking it interrupted. The receipt is machine-local recovery state and excluded from API/peer projections. Legacy rows without a receipt keep their existing recovery behavior; no existing data needs transformation.+- Media job queue — runtime queue state can stay file-backed short term, but **job history and artifact lineage are `db-primary`** and should move to the DB. Admission is durable: `enqueueJob` resolves only once a snapshot containing the job is written, a failed write refuses the job (`503 MEDIA_QUEUE_PERSIST_FAILED`) before any dispatch, and graceful shutdown first stops dispatch (`quiesceMediaJobQueue()`, before its first await — waiting jobs stay queued for the next boot instead of starting during teardown and restoring as interrupted; Run Now answers `503 MEDIA_QUEUE_SHUTTING_DOWN`), then runs the bounded `flushMediaJobQueue()` before exit. If boot finds the snapshot unreadable, persistence stays latched off to preserve that file, and new admissions are refused until the snapshot is repaired and the server restarts. Waiting work is bounded: at 250 queued jobs across all lanes (`MAX_PENDING_MEDIA_JOBS`, separate from each lane's running concurrency) a new submission is refused with a retryable `429 MEDIA_QUEUE_FULL` before any state changes; a restored snapshot above the bound is kept whole and only blocks new admissions until it drains. Local video failure holds live in the existing `data/media-jobs.json` envelope as `videoHolds`, beside unchanged `jobs`. They are machine-local dispatch state, never federated and never sent in capability/status payloads to peers; the local queue API exposes them on retained jobs and through `GET /api/media-jobs/holds`, so status and resume remain available when no retained jobs remain. Three matching terminal causes hold one catalog model/runtime until explicit resume. If hold metadata is damaged, boot restores unrelated work but holds all local video, including new submissions, behind an explicit session-only recovery action; the original snapshot stays preserved until repaired and restarted. Migration 344 adds an empty hold list to legacy envelopes without changing jobs; fresh installs use the empty seed. The envelope retains its existing filesystem backup coverage, with no new store, search index, sync cursor, or tombstone. A completion that cannot enter backup publication admission retains an optional `durabilityPending` receipt in its existing job row, while the published status stays `running`. Boot restores that receipt for admitted completion and hook delivery instead of rerunning the provider or marking it interrupted. The receipt is machine-local recovery state and excluded from API/peer projections. Legacy rows without a receipt keep their existing recovery behavior; no existing data needs transformation.- Browser CDP profile / downloads — `data/browser-profile/`, `data/browser-downloads/` — cache, non-overridable backup excludes.- Brain parity audit results — `data/brain_parity_reports.json` (`server/services/brainParity.js`, #4519). The last record-level brain-parity report per peer, keyed by peer `instanceId` (the same key `data/instances_sync_cursors.json` uses). `ephemeral-file` because it is a point-in-time *observation* about two installs, fully regenerable by re-running the audit, and stale the moment either side syncs — the durable state it describes lives in the brain stores. **Intentionally machine-local — never federated**: it is this install's view of a peer, and each peer computes its own. No sync cursor, no tombstone, no migration (an absent file reads as "nothing audited yet"). Per-type record lists are capped at 25 ids with a `truncated` flag so a badly diverged install can't grow the file without bound.- Persistent-mind rollups and decision journal — `data/cos/persistent-mind-rollups.json` (`server/services/persistentMindContext.js`) and `data/cos/persistent-mind-journal.json` (`server/services/persistentMindJournal.js`). The rollups are sealed prose summaries of ranges that have left raw retention; the journal is the typed, individually addressable record of what the mind decided, committed to, is still asking, and is watching, each entry citing the message sequences it was drawn from. `file-primary` rather than `ephemeral-file`: both outlive the raw events they were derived from, so neither is regenerable, and a corrupt store fails CLOSED rather than resetting to `[]`. **Intentionally machine-local — never federated**: both quote one human private conversation, so the [machine-local privacy ADR](./decisions/2026-08-08-privacy-records-machine-local.md) keeps them off the federation layer entirely — no `PORTOS_SCHEMA_VERSIONS` entry, no sync cursor, no tombstone. No migration and no `data.reference/` seed: an absent file is the correct empty state. Retirement is a STATUS TRANSITION, never a deletion — a superseded entry stays readable with a pointer to what replaced it. Bounded (100 rollups; 500 journal entries, shedding settled history before anything still active), and backed up with the rest of `data/cos/`. The `history` cleanup scope clears the journal along with the messages it cites; a context-only clear does not.