Skip to content

Self-hosted, Self-Improving, Federated Everything App — dev management, agent orchestration, persistent mind, digital twin, second brain, and creative pipeline from a single dashboard over Tailscale to port the user into the OS

15.7 KB
Raw Blame
/**
 * Code-rendered music: the Music Designer's "Code" engine.
 *
 *   writeMusicCode()       musical description → code the LLM wrote
 *                          (fence-stripped and size-checked).
 *   saveCodeTakeToTrack()  a WAV the browser recorded from that code → the
 *                          shared music library, appended to the track's render
 *                          history as an `engine: 'code'` take.
 *
 * Strudel/Tone.js playback runs in a sandboxed, network-blocked browser iframe
 * (client/src/components/music/strudelFrame.js); SuperCollider uses a contained
 * server renderer. These playback boundaries do not contain authoring: the
 * selected provider or fallback may launch a CLI/TUI agent with host tools.
 * The authoring HTTP entry point therefore requires operator authority.
 *
 * Nothing here runs on its own: every call is driven by an explicit button
 * press in the same request (AI Provider Usage Policy: no cold bootstrap).
 */

import { ServerError } from '../lib/errorHandler.js';
import { trimTo } from '../lib/textUtils.js';
import { wavDurationMs } from '../lib/wavAudioFile.js';
import { readFile } from 'fs/promises';
import { assertProvider, resolveProviderAndModel, runPromptThroughProvider } from './promptRunner.js';
import * as tracks from './tracks/index.js';
import { readSuperColliderPreview } from './superColliderRender.js';
import { publishMusicTake } from './musicTakePublication.js';

const CODE_ENGINE = 'code';
// Strudel was the first language; Tone.js the second; SuperCollider is rendered server-side (contained, offline; no browser frame) (client/src/components/music/strudelFrame.js
// mounts a frame document per language, same postMessage protocol for both).
export const MUSIC_CODE_LANGUAGES = Object.freeze(['strudel', 'tonejs', 'supercollider']);
// Longest code the designer accepts, from the LLM or back from the editor.
export const MUSIC_CODE_MAX = 20000;

const MAX_DESCRIPTION = 8000;
const MAX_LYRICS = 20000;
const MAX_GUIDANCE = 4000;

const MAX_SCENE = 20000;

const section = (label, body) => (body ? `\n\n${label}:\n${body}` : '');

// The piece's brief, shared by every language's contract. `scene` is the
// script of an animation the piece scores; `durationSec` is its fixed length.
function briefSections({ description, lyrics, scene, durationSec, guidance, current }) {
  return [
    section('MUSIC TO WRITE', trimTo(description, MAX_DESCRIPTION) || (scene ? '(score the animation below)' : '(none given)')),
    section('ANIMATION THIS SCORES (a soundtrack for the film, not a standalone song: unless the description says otherwise, write sound design synced to what happens on screen, such as whirrs, blips, sweeps, risers and impacts built from synths and noise at the moments they occur, over a musical bed that follows its mood and pacing; nothing is sung)', trimTo(scene, MAX_SCENE)),
    durationSec ? section('LENGTH', `The piece plays for exactly ${durationSec} seconds. Shape intro, build, peak and ending to fit that length, and land the ending at ${durationSec}s.`) : '',
    section('LYRICS / THEME (mood only; nothing is sung, so instruments carry the melody)', trimTo(lyrics, MAX_LYRICS)),
    section('ADDITIONAL GUIDANCE FROM THE USER', trimTo(guidance, MAX_GUIDANCE)),
    current ? section('CURRENT CODE (revise it per the request above; keep what works, change what is asked)', current) : '',
  ].join('');
}

/** The Strudel writing contract sent to the LLM. */
function buildStrudelPrompt(input) {
  return `You are a composer who writes music as Strudel code (the JavaScript port of TidalCycles, strudel.cc). The code runs in a browser with the @strudel/web bundle and is the whole piece: there is no audio model and no DAW.${briefSections(input)}

Return ONLY the Strudel code: no prose and no markdown fence. It is evaluated exactly like the strudel.cc REPL:
- Set the tempo first with setcps(BPM / 60 / 4), e.g. setcps(0.5) for 120 BPM in 4/4.
- End with ONE pattern expression, usually stack(...) with one layer per part (lead, bass, harmony, drums).
- Double-quoted strings are mini-notation: "c3 e3 g3" steps, "~" rests, "[a b]" subdivides a step, "<a b c>" picks one per cycle, "a*4" repeats, "a!2" duplicates, "a@3" lengthens.
- Pitch: note("c3 eb3 g3") or n("0 2 4 <5 7>").scale("C4:minor"). Chords: note("<[c3,eb3,g3] [ab2,c3,eb3]>").
- Sounds: ONLY these built-in synths exist: "sine", "triangle", "square", "sawtooth", "supersaw", "pulse", the noises "white", "pink", "brown", "crackle", and the synth kick "sbd". Pick one with .s("sawtooth"). NO samples are loaded, so s("bd"), s("hh"), s("piano"), and samples(...) fail. Never load or fetch anything.
- Drums from synths: kick s("sbd*4"); snare s("~ pink ~ pink").decay(.12).sustain(0); hats s("white*8").decay(.04).sustain(0).gain(.35).
- Shape sounds with .lpf(), .hpf(), .attack(), .decay(), .sustain(), .release(), .gain(), .pan(), .room(), .delay(), .vowel(); slow or vary with .slow(), .fast(), .every(4, x => x.rev()), .sometimes(...).
- Keep master levels sensible (layer gains around 0.3-0.8).

Compose real music: a memorable melody, a bassline that moves, harmony, and percussion that grooves unless the description asks otherwise. Stay in key, and use "<...>" alternation over several cycles so the piece develops instead of looping one bar.`;
}

/** The Tone.js writing contract sent to the LLM. */
function buildTonejsPrompt(input) {
  return `You are a composer who writes music as plain JavaScript against the Tone.js library (tonejs.github.io/), running against the global \`Tone\`. The code IS the whole piece: there is no audio model and no DAW, and it runs once per play.${briefSections(input)}

Return ONLY the JavaScript: no prose and no markdown fence. Rules for the code:
- The host has already unlocked audio and reset the transport before your code runs; do not call Tone.start().
- Build the graph, schedule the music (Tone.Transport.schedule, Tone.Transport.scheduleRepeat, or a Tone.Part/Tone.Sequence), then call Tone.Transport.start() as the LAST statement so playback actually begins.
- Only these built-in instruments exist, all playable with no external files: Tone.Synth, Tone.AMSynth, Tone.FMSynth, Tone.MonoSynth, Tone.DuoSynth, Tone.PluckSynth, Tone.MembraneSynth (kick/toms), Tone.MetalSynth (hats/cymbals), Tone.NoiseSynth (noise/snare), and Tone.PolySynth(Tone.Synth) (or another synth) for chords. NEVER use Tone.Player, Tone.GrainPlayer, Tone.Sampler, Tone.UserMedia, or any URL/buffer/sample loading — there is no network access and no sample files, so they always fail.
- Shape and route sound with the built-in effects: Tone.Filter, Tone.EQ3, Tone.Distortion, Tone.Chorus, Tone.FeedbackDelay, Tone.Reverb, Tone.Compressor, Tone.Gain, Tone.Panner. Chain with .connect() or .chain(), and end every audible chain with .toDestination().
- Note names are strings like "C3", "Eb4", "G#2"; durations are Tone.js notation like "8n", "4n.", "16t". Use Tone.Time/Tone.Frequency helpers as needed.
- Set the tempo with Tone.Transport.bpm.value = BPM before scheduling.
- Keep levels sensible: set instrument/gain .volume around -12 to -6 dB per layer so a full mix doesn't clip.
- Wrap the whole body in a try/catch only if you need to recover from an error yourself; otherwise let errors propagate so they can be reported.

Compose real music: a memorable melody, a bassline that moves, harmony, and percussion that grooves unless the description asks otherwise. Stay in key, and vary the pattern over multiple bars (with scheduleRepeat callbacks or a Part with several events) so the piece develops instead of looping one bar forever.`;
}

/** The SuperCollider writing contract sent to the LLM (rendered offline by a trusted, contained runner). */
function buildSupercolliderPrompt(input) {
  return `You are a composer who writes music as SuperCollider (sclang) code. The code is the whole piece: it is compiled and rendered OFFLINE by a trusted runner in a sandbox, never played live, and there is no audio model and no DAW.${briefSections(input)}

Return ONLY the sclang code: no prose and no markdown fence. Rules for the code:
- Define instruments with SynthDef(\\name, { |out=0, freq=440, amp=0.2, gate=1| ... }) using ONLY stock UGens (SinOsc, Saw, Pulse, LFTri, WhiteNoise, PinkNoise, RLPF, RHPF, LPF, HPF, EnvGen, Env, Pan2, FreeVerb, CombL, Splay, Mix, ...), finish each with Out.ar(out, signal) in stereo, and register it with .add (e.g. SynthDef(\\lead, { ... }).add;). Give every sustained instrument an envelope that frees it (doneAction: 2), gated by gate when it has one.
- Compose with Pbind/Pseq/Prand/Pwhite/Pn/Ppar patterns on those SynthDefs, using \\dur, \\degree or \\note, \\scale, \\octave and \\amp. Express timing in beats via \\dur; the runner fixes the tempo at 120 BPM. Use finite or long-running patterns; the runner cuts the piece at its length.
- The runner turns your patterns into a bounded offline score (it supplies the length, 48 kHz stereo, and the output file). The LAST expression of the code must be ONE pattern (usually Ppar([...]) layering the parts); the runner renders that value. Do NOT call s.boot, s.record, Server.default, Score.recordNRT, play, or write any file path.
- NEVER use unixCmd, unixCmdGetStdOut, systemCmd, String.runInTerminal, File, Pipe, Document, thisProcess.interpreter.executeFile, load, loadRelative, include, Quarks, SoundFile, Buffer.read or any sample/URL/path loading: there is no filesystem or network access and no sample files, so they always fail.
- Keep levels sensible (per-voice amp around 0.05-0.3) so a full mix does not clip.

Compose real music: a memorable melody, a bassline that moves, harmony, and percussion from synths that grooves unless the description asks otherwise. Stay in key and vary the patterns over several bars so the piece develops instead of looping one bar.`;
}

const PROMPT_BUILDERS = { strudel: buildStrudelPrompt, tonejs: buildTonejsPrompt, supercollider: buildSupercolliderPrompt };

const LANGUAGE_LABELS = { strudel: 'Strudel', tonejs: 'Tone.js', supercollider: 'SuperCollider' };

/** The exact prompt writeMusicCode sends for `input`. */
function buildMusicCodePrompt({ language = 'strudel', current, ...input } = {}) {
  const buildPrompt = PROMPT_BUILDERS[language] || buildStrudelPrompt;
  return buildPrompt({ ...input, current: trimTo(current, MUSIC_CODE_MAX) });
}

/**
 * What writing the code would send (#11317): the sent prompt, plus everything
 * in it beyond the description as labeled additions. Pure; no provider call.
 */
export function previewMusicCodePrompt(input = {}) {
  const language = PROMPT_BUILDERS[input.language] ? input.language : 'strudel';
  const sentPrompt = buildMusicCodePrompt({ ...input, language });
  const rules = buildMusicCodePrompt({ language }).replace(briefSections({}), '');
  const promptAdditions = [
    ['code-rules', `${LANGUAGE_LABELS[language]} writing rules`, rules],
    ['animation-script', 'Animation script', trimTo(input.scene, MAX_SCENE)],
    ['length', 'Length', input.durationSec ? `${input.durationSec} seconds` : ''],
    ['lyrics', 'Lyrics / theme', trimTo(input.lyrics, MAX_LYRICS)],
    ['guidance', 'Code guidance', trimTo(input.guidance, MAX_GUIDANCE)],
    ['current', 'Current code (revise)', trimTo(input.current, MUSIC_CODE_MAX)],
  ].filter(([, , text]) => text).map(([id, label, text]) => ({ id, label, text }));
  return { sentPrompt, promptAdditions };
}

// A reply usually arrives bare, but a fenced block (with or without prose
// around it) is common. Take the first fence's body when there is one.
const FENCE_RE = /```[^\n`]*\n([\s\S]*?)\n?```/;
function extractCode(text) {
  const raw = typeof text === 'string' ? text : '';
  const fenced = raw.match(FENCE_RE);
  return (fenced ? fenced[1] : raw).trim();
}

/**
 * Ask the chosen provider to write the piece as code. `current` (the code in
 * the editor) turns the call into a revision of it.
 */
export async function writeMusicCode({ language = 'strudel', providerId, model, effort, ...brief } = {}) {
  const { provider, selectedModel } = await resolveProviderAndModel({ providerId, model });
  assertProvider(provider, { message: 'No AI provider available to write the music code', code: 'NO_PROVIDER' });

  const run = await runPromptThroughProvider({
    provider,
    model: selectedModel ?? undefined,
    effort,
    prompt: buildMusicCodePrompt({ ...brief, language }),
    source: 'music-code',
  });

  const code = extractCode(run.text);
  if (!code) {
    throw new ServerError('The AI returned no code. Try rerunning or picking a stronger model.', { status: 502, code: 'MUSIC_CODE_EMPTY' });
  }
  if (code.length > MUSIC_CODE_MAX) {
    throw new ServerError(`The AI returned more than ${MUSIC_CODE_MAX} characters of code. Try rerunning or asking for a shorter piece.`, {
      status: 502, code: 'MUSIC_CODE_TOO_LONG',
    });
  }
  const ranModel = run.model ?? selectedModel ?? null;
  console.log(`🎹 Wrote ${language} music code via ${provider.id}/${ranModel || 'default'} (${code.length} chars)`);
  return { language, code, llm: { provider: run.provider?.id || provider.id, model: ranModel } };
}

/**
 * Land a browser-recorded WAV (a Buffer) in the shared music library and make
 * it the track's active take.
 */
export async function saveCodeTakeToTrack({ trackId, wav, prompt, title }) {
  const durationMs = wavDurationMs(wav);
  if (!durationMs) {
    throw new ServerError('The take must be a PCM WAV file', { status: 400, code: 'MUSIC_CODE_TAKE_NOT_WAV' });
  }
  const track = await tracks.getTrack(trackId);
  if (!track) throw new ServerError('Track not found', { status: 404, code: 'NOT_FOUND' });

  const durationSec = Math.max(1, Math.round(durationMs / 1000));
  const { track: updated, filename } = await publishMusicTake({
    trackId, wav, take: { prompt: prompt || track.prompt, engine: CODE_ENGINE, durationSec },
    patch: title ? { title } : {},
  });
  if (!updated) throw new ServerError('Track not found', { status: 404, code: 'NOT_FOUND' });
  console.log(`🎹 Saved code-rendered take (${durationSec}s)`);
  return { track: updated, filename, durationSec };
}

/**
 * Save a finished SuperCollider preview as the track's active take (#9414).
 * The audio is read back from the host-written preview store (never from a
 * container), re-checked as a PCM WAV, and stored with the provenance that
 * reproduces it: language, source, hash, seed, runtime version and settings.
 * Previewing alone never reaches here, so it cannot change a track.
 */
export async function saveSuperColliderTakeToTrack({ trackId, jobId, prompt, title }) {
  const found = await readSuperColliderPreview(jobId);
  if (!found) throw new ServerError('Preview not found or expired. Render it again.', { status: 404, code: 'SUPERCOLLIDER_PREVIEW_NOT_FOUND' });
  const track = await tracks.getTrack(trackId);
  if (!track) throw new ServerError('Track not found', { status: 404, code: 'NOT_FOUND' });
  const wav = await readFile(found.wavPath);
  const durationMs = wavDurationMs(wav);
  if (!durationMs) throw new ServerError('The preview is not a PCM WAV file', { status: 400, code: 'MUSIC_CODE_TAKE_NOT_WAV' });

  const { preview } = found;
  const durationSec = Math.max(1, Math.round(durationMs / 1000));
  const { track: updated, filename } = await publishMusicTake({
    trackId, wav,
    take: {
      prompt: prompt || track.prompt,
      engine: CODE_ENGINE,
      durationSec,
      codeProvenance: {
        language: preview.language,
        source: preview.source,
        sourceHash: preview.sourceHash,
        seed: preview.seed,
        runtimeVersion: preview.runtime?.version,
        policyVersion: preview.runtime?.policyVersion,
        settings: preview.settings,
      },
    },
    patch: title ? { title } : {},
  });
  if (!updated) throw new ServerError('Track not found', { status: 404, code: 'NOT_FOUND' });
  console.log(`🎛️ Saved SuperCollider take (${durationSec}s, seed ${preview.seed})`);
  return { track: updated, filename, durationSec };
}