Skip to content

Write HTML. Render video. Built for agents.

feat(cli): read a whole recording as voice, effect and music lines (#6056)

Xuanru Li2c1d9d50d239
diff --git a/docs/packages/cli.mdx b/docs/packages/cli.mdx
index 362b420d52..21681fca8c 100644
--- a/docs/packages/cli.mdx
+++ b/docs/packages/cli.mdx
@@ -583,6 +583,50 @@ first, and the result warns when the cut part held the file's loudest moment.
Word timing is not here: use [`transcribe`](#transcribe).
+#### Reading a whole recording
+
+`--read` turns one recording — an episode, a reference video — into
+time-stamped lines an agent can read or grep, instead of measurements of one sound:
+
+```bash
+npx hyperframes audio refs/episode.mp4 --read # the whole recording
+npx hyperframes audio refs/episode.mp4 --read --range 60,70 # only what overlaps 60–70 s
+```
+
+- **voice**: one line per sentence with its text, speaking rate against the
+ recording's mean, pitch when it is 3 semitones or more off the median, and a
+ rising or falling end; pauses of 0.4 s or more between sentences.
+- **effects**: each sound effect with a kind (`riser`, `boom`, `thump`, `whoosh`,
+ `click`, `tick`, `ding`, `hit`), the group of identical sounds it belongs to
+ (`S1` is the most used), band, attack, tail, how far it stands over the music,
+ and where it falls in a pause. Quick runs of one sound are a single line with a
+ count; hits on the music's beat are the music's own drums, and sounds more than
+ 20 dB under the recording are separation residue nobody hears; both are left out.
+- **music**: the bed split at its cut-outs, each part naming its piece, the
+ instruments that carry it and its tempo, whether it starts, resumes after a
+ break, returns or is new, and where it fades.
+
+Sentences come from a transcript: `--transcript <file>`, or
+`<recording>.transcript.json` beside the recording. Without one, voice lines
+carry no words.
+
+Effects and music are read from stems. At `--quality best` the
+recording is separated into vocals, drums, bass, guitar, piano and other by
+HeyGen's `/v3/audio/stems`, with your HeyGen credential (see
+[audio services](#audio-services)): it is uploaded as 192 kb/s MP3 through the
+same origin's direct-upload endpoints. The separation request waits for the
+stems; an in-progress response or dropped connection retries the same request
+with the same idempotency key (`HYPERFRAMES_AUDIO_STEMS_TIMEOUT_MS`, 30 minutes
+by default). Effects are read
+from the drums stem and the music from the other four. Without stems, effects are
+read from the mix outside speech and the music is not read; with neither stems nor
+a transcript, voice and effects are not read either, because the mix alone does
+not tell speech from music. Each case says so in a warning.
+
+The result is computed once per file content and kept in `.hyperframes/read-audio`
+under the project, with the stems; the text output ends with the path of the
+whole recording's text, and `--json` returns the analysis.
+
Every measurement comes from a provider ladder, tried in this order until one
answers:
@@ -593,13 +637,12 @@ answers:
| sections | `api`, `apple-mu`, `signal` |
| describe | `api`, `apple-sound` |
-- `api` is an audio analysis service, used when `HYPERFRAMES_AUDIO_API_URL` and
- `HYPERFRAMES_AUDIO_API_KEY` are both set. The audio is uploaded as 16 kHz mono
+- `api` is HeyGen's `/v3/audio/analysis`, used when a HeyGen credential is
+ found (see [audio services](#audio-services)). The audio is uploaded as 16 kHz mono
WAV through the same origin’s public asset direct-upload and completion endpoints.
Each decoded signal is uploaded once; analysis requests send its finalized asset ID as JSON.
- The API key needs asset upload and audio-analysis permissions. Channels are averaged unless cancellation removes over 90% of the loudest
- channel’s energy, in which case that channel is used. Requests carry the key in an `x-api-key` header and a new `Idempotency-Key` on each
- request; the answer may come wrapped in `{ "data": … }`. Every beat,
+ The credential needs asset upload and audio permissions. Channels are averaged unless cancellation removes over 90% of the loudest
+ channel’s energy, in which case that channel is used. Each request carries a new `Idempotency-Key`; the answer may come wrapped in `{ "data": … }`. Every beat,
downbeat or section boundary it answers moves onto the nearest beat signal
processing measured, when one is within a quarter beat; the rest are listed in
`snap.unsnapped`, and `snap.medianShiftMs` says how far the moved ones went.
@@ -621,13 +664,34 @@ answers:
helper compiled once with the Xcode command line tools.
A provider that cannot run, or fails, is passed over with a warning that says
-why. `--quality fast|local|best` limits which providers may run (`local` keeps
-the audio on this machine, `fast` allows only signal processing),
-`--provider beats=signal` pins one, and `--full` shows each ladder step.
+why. `--quality local|best|fast` limits which providers may run: `local`, the
+default, keeps the audio on this machine; `best` also sends it to the audio
+services below; `fast` allows only signal processing. When a capability goes
+unanswered because a service was left out, a warning says `--quality best` could
+answer it. `--provider beats=signal` pins a capability to one provider, and
+`--full` shows each ladder step.
Tune the analysis with `--onset-method`, `--threshold`, `--band lo,hi`,
`--hop-ms`, `--onset-ratio`, `--tail-db` and `--silence-db`. Decoding needs
FFmpeg.
+#### Audio services
+
+Only at `--quality best` is audio uploaded. Stems and the `api` provider then
+call HeyGen, which may bill your account, with the same credential as the other
+HeyGen commands: `HEYGEN_API_KEY`, `HYPERFRAMES_API_KEY`, `HEYGEN_ACCESS_TOKEN`,
+or a credential saved by `hyperframes auth login`, on the API at
+`HEYGEN_API_URL` when set. API keys are sent as `x-api-key`; OAuth credentials
+are sent as bearer tokens. If the service refuses a request, analysis falls
+back with a warning. Without a credential, stems are skipped and `api` is
+passed over. At `local`, the default, no credential is looked up and nothing
+leaves the machine.
+
+To use another service, set `HYPERFRAMES_AUDIO_STEMS_URL` or
+`HYPERFRAMES_AUDIO_API_URL` to its full URL together with
+`HYPERFRAMES_AUDIO_API_KEY`, which is sent as `x-api-key`. Your HeyGen
+credential is never sent to a URL set this way. Setting either URL to an empty
+string turns that service off, whatever credential is present.
+
## Look at it
### `normalize-audio`
diff --git a/packages/cli/src/audio/analysis/analyze.test.ts b/packages/cli/src/audio/analysis/analyze.test.ts
index 68e54b7873..3bb020c306 100644
--- a/packages/cli/src/audio/analysis/analyze.test.ts
+++ b/packages/cli/src/audio/analysis/analyze.test.ts
@@ -83,6 +83,34 @@ describe("analyzeSignal", () => {
});
});
+describe("analyzeSignal quality", () => {
+ const service: AudioAnalysisProvider<AudioSignal> = {
+ id: "service",
+ local: false,
+ capabilities: ["describe"],
+ analyze: async () => ({ tags: [{ label: "speech", score: 0.9 }] }),
+ };
+
+ it("keeps audio on this machine unless best is asked for, and says what best would add", async () => {
+ const local = await analyzeSignal(tone, { capabilities: ["describe"] }, [service]);
+ expect(local.values.describe).toBeNull();
+ expect(local.ladder.describe).toEqual([
+ {
+ provider: "service",
+ outcome: "excluded",
+ reason: "quality local keeps audio on this machine",
+ },
+ ]);
+ expect(local.warnings.at(-1)).toContain("--quality best");
+
+ const best = await analyzeSignal(tone, { capabilities: ["describe"], quality: "best" }, [
+ service,
+ ]);
+ expect(best.provenance.describe).toBe("service");
+ expect(best.warnings).toEqual([]);
+ });
+});
+
describe("envelopeHopMs", () => {
it("keeps short sounds at 5 ms and coarsens long ones to about 2,000 values", () => {
expect(envelopeHopMs(1)).toBe(5);
diff --git a/packages/cli/src/audio/analysis/analyze.ts b/packages/cli/src/audio/analysis/analyze.ts
index bc9850bd6c..60bc0adf67 100644
--- a/packages/cli/src/audio/analysis/analyze.ts
+++ b/packages/cli/src/audio/analysis/analyze.ts
@@ -36,6 +36,7 @@ function audioAnalysisProviders(): AudioAnalysisProvider<AudioSignal>[] {
export interface AnalyzeOptions {
capabilities: readonly AudioCapability[];
+ /** Defaults to local: audio leaves the machine only when best is asked for. */
quality?: AnalysisQuality;
/** Pin a capability to one provider, skipping the ladder. */
provider?: Partial<Record<AudioCapability, string>>;
@@ -103,6 +104,7 @@ export async function analyzeSignal(
request: AnalyzeOptions,
providers: readonly AudioAnalysisProvider<AudioSignal>[] = audioAnalysisProviders(),
): Promise<Analysis> {
+ const quality = request.quality ?? "local";
const { beats, bpm } = request.provider ?? {};
if (beats && bpm && beats !== bpm) throw new Error("beats and bpm must use the same provider");
const beatProvider = beats ?? bpm;
@@ -120,7 +122,7 @@ export async function analyzeSignal(
if (!result) {
const withContext = capability === "onsets" && framing.cut;
result = runLadder(capability, withContext ? signal : framing.windowed, providers, {
- quality: request.quality,
+ quality,
provider: capability === "beats" ? beatProvider : request.provider?.[capability],
options: optionsFor(capability, request, analysis.analyzed),
});
@@ -140,9 +142,23 @@ export async function analyzeSignal(
const said = new Set(analysis.warnings);
analysis.warnings.push(...result.warnings.filter((w) => !said.has(w)));
}
+ analysis.warnings.push(...serviceHint(quality, analysis, request.capabilities));
return analysis;
}
+function serviceHint(
+ quality: AnalysisQuality,
+ analysis: Analysis,
+ capabilities: readonly AudioCapability[],
+): string[] {
+ const leftToService = (capability: AudioCapability) =>
+ analysis.values[capability] == null &&
+ (analysis.ladder[capability] ?? []).some((s) => s.outcome === "excluded");
+ return quality === "local" && capabilities.some(leftToService)
+ ? ["a service could have answered what is missing: --quality best uploads the audio to HeyGen"]
+ : [];
+}
+
/** Tempo is read off the beats of whichever provider placed them, so the two never disagree. */
function tempoFromBeats(beats: LadderResult): LadderResult {
const times = (beats.value as Beats | null)?.times;
diff --git a/packages/cli/src/audio/analysis/decode.ts b/packages/cli/src/audio/analysis/decode.ts
index ef86377d24..9ba709330a 100644
--- a/packages/cli/src/audio/analysis/decode.ts
+++ b/packages/cli/src/audio/analysis/decode.ts
@@ -55,6 +55,11 @@ export interface DecodeWindow {
seconds?: number;
}
+export interface DecodeFormat {
+ sampleRate?: number;
+ mono?: boolean;
+}
+
export interface Decoded {
signal: AudioSignal;
stream: AudioStreamInfo;
@@ -65,9 +70,15 @@ export async function decodeAudio(
file: string,
window: DecodeWindow = {},
probed?: AudioStreamInfo,
+ format: DecodeFormat = {},
): Promise<Decoded | null> {
- const info = probed ?? (await probeAudio(file));
- if (!info) return null;
+ const probedInfo = probed ?? (await probeAudio(file));
+ if (!probedInfo) return null;
+ const info = {
+ ...probedInfo,
+ sampleRate: format.sampleRate ?? probedInfo.sampleRate,
+ channels: format.mono ? 1 : probedInfo.channels,
+ };
const seek = window.from ? ["-ss", String(window.from)] : [];
const length = window.seconds === undefined ? [] : ["-t", String(window.seconds)];
const { stdout } = await run(
@@ -81,6 +92,8 @@ export async function decodeAudio(
...length,
"-map",
"0:a:0",
+ ...(format.mono ? ["-ac", "1"] : []),
+ ...(format.sampleRate ? ["-ar", String(format.sampleRate)] : []),
"-f",
"f32le",
"-acodec",
diff --git a/packages/cli/src/audio/analysis/providers/api.test.ts b/packages/cli/src/audio/analysis/providers/api.test.ts
index 09f5067ce5..7dc5e0cdc3 100644
--- a/packages/cli/src/audio/analysis/providers/api.test.ts
+++ b/packages/cli/src/audio/analysis/providers/api.test.ts
@@ -1,6 +1,6 @@
import { createServer, type IncomingMessage, type Server } from "node:http";
import type { AddressInfo } from "node:net";
-import { afterAll, afterEach, beforeAll, describe, expect, it } from "vitest";
+import { afterAll, afterEach, beforeAll, beforeEach, describe, expect, it, vi } from "vitest";
import {
rhythm,
signalProvider,
@@ -28,6 +28,7 @@ function clicks(seconds: number): AudioSignal {
interface Received {
method: string;
+ url: string | undefined;
apiKey: string | undefined;
authorization: string | undefined;
idempotencyKey: string | undefined;
@@ -94,6 +95,7 @@ beforeAll(async () => {
};
received.push({
method: req.method!,
+ url: req.url,
apiKey: req.headers["x-api-key"] as string | undefined,
authorization: req.headers.authorization,
idempotencyKey: req.headers["idempotency-key"] as string | undefined,
@@ -108,32 +110,64 @@ beforeAll(async () => {
afterAll(() => new Promise<void>((done) => server.close(() => done())));
+const serverOrigin = () => `http://127.0.0.1:${(server.address() as AddressInfo).port}`;
+
+beforeEach(() => {
+ for (const name of ["HEYGEN_API_KEY", "HYPERFRAMES_API_KEY", "HEYGEN_ACCESS_TOKEN"])
+ vi.stubEnv(name, "");
+ vi.stubEnv("HEYGEN_API_URL", serverOrigin());
+});
+
afterEach(() => {
received = [];
uploads = 0;
enveloped = true;
+ vi.unstubAllEnvs();
delete process.env.HYPERFRAMES_AUDIO_API_URL;
delete process.env.HYPERFRAMES_AUDIO_API_KEY;
});
function configure() {
- process.env.HYPERFRAMES_AUDIO_API_URL = `http://127.0.0.1:${(server.address() as AddressInfo).port}/v1/audio`;
+ process.env.HYPERFRAMES_AUDIO_API_URL = `${serverOrigin()}/v1/audio`;
process.env.HYPERFRAMES_AUDIO_API_KEY = "test-key";
}
describe("apiProvider", () => {
- it("is unavailable until both the URL and the key are set", () => {
- expect(apiProvider.available!()).toEqual({
+ it("is unavailable without a HeyGen credential", async () => {
+ expect(await apiProvider.available!()).toEqual({
+ ok: false,
+ reason: expect.stringContaining("no HeyGen credential"),
+ });
+ vi.stubEnv("HEYGEN_API_KEY", "test-key");
+ expect(await apiProvider.available!()).toEqual({ ok: true });
+ });
+
+ it("calls HeyGen's audio analysis route with the CLI's HeyGen API key", async () => {
+ vi.stubEnv("HEYGEN_API_KEY", "test-key");
+ reply = () => ({ status: 200, body: { version: 1, beats: { times: [0.25, 0.75] } } });
+ await apiProvider.analyze("beats", clicks(2), {});
+ expect(received.map((r) => [r.url, r.apiKey])).toEqual([["/v3/audio/analysis", "test-key"]]);
+ });
+
+ it("stays off when the host blanks HYPERFRAMES_AUDIO_API_URL, whatever HeyGen credential it holds", async () => {
+ vi.stubEnv("HEYGEN_API_KEY", "gateway-token");
+ process.env.HYPERFRAMES_AUDIO_API_URL = "";
+ process.env.HYPERFRAMES_AUDIO_API_KEY = "";
+ expect(await apiProvider.available!()).toEqual({
ok: false,
- reason: "HYPERFRAMES_AUDIO_API_URL is not set",
+ reason: "HYPERFRAMES_AUDIO_API_URL is empty, which turns this service off",
});
+ });
+
+ it("never sends the HeyGen credential to a service named by HYPERFRAMES_AUDIO_API_URL", async () => {
+ vi.stubEnv("HEYGEN_API_KEY", "heygen-key");
process.env.HYPERFRAMES_AUDIO_API_URL = "http://127.0.0.1:1/";
- expect(apiProvider.available!()).toEqual({
+ expect(await apiProvider.available!()).toEqual({
ok: false,
- reason: "HYPERFRAMES_AUDIO_API_KEY is not set",
+ reason: "HYPERFRAMES_AUDIO_API_URL is set but HYPERFRAMES_AUDIO_API_KEY is not",
});
process.env.HYPERFRAMES_AUDIO_API_KEY = "k";
- expect(apiProvider.available!()).toEqual({ ok: true });
+ expect(await apiProvider.available!()).toEqual({ ok: true });
});
it("refuses redirects without sending the API key to another origin", async () => {
@@ -180,10 +214,11 @@ describe("apiProvider", () => {
downbeats: { times: measured.filter((_, k) => k % 4 === 0).map((t) => t - 0.02), meter: 4 },
} satisfies AudioApiResponse,
});
- const analysis = await analyzeSignal(signal, { capabilities: ["beats", "downbeats", "bpm"] }, [
- apiProvider,
- signalProvider,
- ]);
+ const analysis = await analyzeSignal(
+ signal,
+ { quality: "best", capabilities: ["beats", "downbeats", "bpm"] },
+ [apiProvider, signalProvider],
+ );
expect(received).toHaveLength(1);
const [call] = received;
@@ -225,6 +260,7 @@ describe("apiProvider", () => {
configure();
reply = () => ({ status: 200, body: { version: 1, beats: { times: [0.5, 1, 1.5] } } });
const analysis = await analyzeSignal(clicks(8), {
+ quality: "best",
capabilities,
provider: { bpm: "signal" },
});
@@ -247,6 +283,7 @@ describe("apiProvider", () => {
});
await expect(
analyzeSignal(clicks(8), {
+ quality: "best",
capabilities: ["describe", "beats", "bpm"],
provider,
}),
@@ -272,10 +309,11 @@ describe("apiProvider", () => {
describe: { text: "clicks", tags: [{ label: "click", score: 0.9 }] },
},
});
- const analysis = await analyzeSignal(signal, { capabilities: ["sections", "describe"] }, [
- apiProvider,
- signalProvider,
- ]);
+ const analysis = await analyzeSignal(
+ signal,
+ { quality: "best", capabilities: ["sections", "describe"] },
+ [apiProvider, signalProvider],
+ );
expect(received.map((r) => r.request.capabilities)).toEqual([["sections"], ["describe"]]);
expect(new Set(received.map((r) => r.idempotencyKey)).size).toBe(2);
expect(uploads).toBe(1);
@@ -296,7 +334,7 @@ describe("apiProvider", () => {
it("falls through to signal processing with a warning when the service fails or answers in another shape", async () => {
configure();
reply = () => ({ status: 503, body: { error: { message: "busy" } } });
- const failed = await analyzeSignal(clicks(8), { capabilities: ["beats"] }, [
+ const failed = await analyzeSignal(clicks(8), { quality: "best", capabilities: ["beats"] }, [
apiProvider,
signalProvider,
]);
@@ -306,7 +344,7 @@ describe("apiProvider", () => {
]);
reply = () => ({ status: 200, body: { version: 1, beats: { times: ["soon"] } } });
- const garbled = await analyzeSignal(clicks(8), { capabilities: ["beats"] }, [
+ const garbled = await analyzeSignal(clicks(8), { quality: "best", capabilities: ["beats"] }, [
apiProvider,
signalProvider,
]);
@@ -328,7 +366,7 @@ describe("apiProvider", () => {
reply = () => ({ status: 200, body: { version: 1, describe: { tags: [] } } });
await analyzeSignal(
{ sampleRate: rate, channels: [left, right] },
- { capabilities: ["describe"] },
+ { quality: "best", capabilities: ["describe"] },
[apiProvider],
);
@@ -378,7 +416,7 @@ describe("apiProvider", () => {
reply = () => ({ status: 200, body: { version: 1, describe: { tags: [] } } });
await analyzeSignal(
{ sampleRate: rate, channels: [left, right] },
- { capabilities: ["describe"] },
+ { quality: "best", capabilities: ["describe"] },
[apiProvider],
);
const wav = Buffer.from(received[0]!.audio.bytes);
@@ -393,14 +431,22 @@ describe("apiProvider", () => {
configure();
enveloped = false;
reply = () => ({ status: 200, body: { version: 1, describe: { text: "bare", tags: [] } } });
- const analysis = await analyzeSignal(clicks(2), { capabilities: ["describe"] }, [apiProvider]);
+ const analysis = await analyzeSignal(
+ clicks(2),
+ { quality: "best", capabilities: ["describe"] },
+ [apiProvider],
+ );
expect(analysis.values.describe).toEqual({ text: "bare", tags: [] });
});
it("never sends the key anywhere but the x-api-key header", async () => {
configure();
reply = () => ({ status: 401, body: { error: { message: "bad key" } } });
- const analysis = await analyzeSignal(clicks(8), { capabilities: ["describe"] }, [apiProvider]);
+ const analysis = await analyzeSignal(
+ clicks(8),
+ { quality: "best", capabilities: ["describe"] },
+ [apiProvider],
+ );
expect(JSON.stringify(analysis)).not.toContain("test-key");
const [call] = received;
expect(call).toMatchObject({ apiKey: "test-key", authorization: undefined });
diff --git a/packages/cli/src/audio/analysis/providers/api.ts b/packages/cli/src/audio/analysis/providers/api.ts
index 29451eccc8..5cc1ff7492 100644
--- a/packages/cli/src/audio/analysis/providers/api.ts
+++ b/packages/cli/src/audio/analysis/providers/api.ts
@@ -10,19 +10,22 @@ import {
type AudioSignal,
type Snapper,
} from "@hyperframes/core/audio-analysis";
-import { AVAILABLE, encodeWav, perSignal, rounded, unavailable } from "./shared.js";
+import {
+ AVAILABLE,
+ encodeWav,
+ perSignal,
+ rounded,
+ serviceAccess,
+ serviceFailure,
+ type ServiceAccess,
+ unavailable,
+} from "./shared.js";
/**
- * The audio analysis service contract.
- *
- * Request: `POST <HYPERFRAMES_AUDIO_API_URL>`, exactly as configured, with `x-api-key: <HYPERFRAMES_AUDIO_API_KEY>`,
- * a new `Idempotency-Key` per request, and an {@link AudioApiRequest} JSON body.
- * The analyzed span is uploaded once as 16-bit PCM WAV, mono at 16 kHz, using the same origin
- * public asset direct-upload and completion routes. The body references that finalized asset.
- *
- * Response: `200` with `application/json`, `{ "data": AudioApiResponse }`; a bare {@link AudioApiResponse} is read
- * too. Every time is in seconds from the start of the uploaded audio. A capability the service cannot answer is left
- * out of the response. Any other status is a failure; its body may be `{ "error": { "message": string } }`.
+ * POST an asset-backed request to the configured audio service with a fresh Idempotency-Key.
+ * Uploads are 16 kHz mono WAV through the same origin's direct-upload routes.
+ * Responses may be bare or wrapped in `data`; times start at the uploaded audio's beginning.
+ * Unavailable capabilities are omitted. Other HTTP statuses fail with the service's error.
*/
export interface AudioApiRequest {
version: 1;
@@ -115,11 +118,17 @@ function uploaded(signal: AudioSignal): AudioSignal {
const askedTogether = (capability: AudioApiCapability): AudioApiCapability[] =>
capability === "beats" || capability === "downbeats" ? ["beats", "downbeats"] : [capability];
-function config() {
- const url = process.env[URL_ENV]?.trim();
- const key = process.env[KEY_ENV]?.trim();
- const timeout = Number(process.env[TIMEOUT_ENV]);
- return { url, key, timeoutMs: timeout > 0 ? timeout : DEFAULT_TIMEOUT_MS };
+const access = () =>
+ serviceAccess(
+ { url: URL_ENV, key: KEY_ENV, timeout: TIMEOUT_ENV },
+ "/v3/audio/analysis",
+ DEFAULT_TIMEOUT_MS,
+ );
+
+async function granted(): Promise<ServiceAccess> {
+ const found = await access();
+ if (typeof found === "string") throw new Error(found);
+ return found;
}
const isNumberList = (value: unknown): value is number[] =>
@@ -146,20 +155,8 @@ function checked(capability: AudioApiCapability, body: AudioApiResponse): void {
throw new Error(`the service answered ${capability} in a shape this CLI cannot read`);
}
-async function failure(response: Response): Promise<Error> {
- const text = await response.text().catch(() => "");
- let detail = text.slice(0, 200);
- try {
- detail = (JSON.parse(text) as { error?: { message?: string } }).error?.message ?? detail;
- } catch {
- // not JSON: keep the text
- }
- return new Error(`the service answered ${response.status}${detail ? `: ${detail}` : ""}`);
-}
-
const uploadedAsset = perSignal(async (analyzed: AudioSignal) => {
- const { url, key, timeoutMs } = config();
- if (!url || !key) throw new Error(`${URL_ENV} and ${KEY_ENV} must both be set`);
+ const { url, headers, timeoutMs } = await granted();
const deadline = AbortSignal.timeout(timeoutMs);
const boundedFetch: typeof fetch = (input, init) =>
fetch(input, {
@@ -169,7 +166,7 @@ const uploadedAsset = perSignal(async (analyzed: AudioSignal) => {
});
const client = new HyperframesCloudClient({
baseUrl: new URL(url).origin,
- getAuthHeaders: () => ({ "x-api-key": key }),
+ getAuthHeaders: () => headers,
fetchImpl: boundedFetch,
});
const asset = await uploadAssetViaDirectUpload({
@@ -184,8 +181,7 @@ const uploadedAsset = perSignal(async (analyzed: AudioSignal) => {
});
async function ask(analyzed: AudioSignal, capabilities: AudioApiCapability[]) {
- const { url, key, timeoutMs } = config();
- if (!url || !key) throw new Error(`${URL_ENV} and ${KEY_ENV} must both be set`);
+ const { url, headers, timeoutMs } = await granted();
const request: AudioApiRequest = {
version: 1,
capabilities,
@@ -195,14 +191,14 @@ async function ask(analyzed: AudioSignal, capabilities: AudioApiCapability[]) {
method: "POST",
redirect: "error",
headers: {
- "x-api-key": key,
+ ...headers,
"idempotency-key": randomUUID(),
"content-type": "application/json",
},
body: JSON.stringify(request),
signal: AbortSignal.timeout(timeoutMs),
});
- if (!response.ok) throw await failure(response);
+ if (!response.ok) throw await serviceFailure(response, "the service");
const body = (await response.json()) as { data?: AudioApiResponse } | null;
return (body?.data ?? body ?? {}) as AudioApiResponse;
}
@@ -265,16 +261,14 @@ function measured(capability: AudioApiCapability, signal: AudioSignal, body: Aud
const API_CAPABILITIES: readonly AudioCapability[] = ["beats", "downbeats", "sections", "describe"];
-/** HeyGen's audio analysis service, or any service that speaks {@link AudioApiRequest}, configured by environment. */
+/** HeyGen's audio analysis service, or another that speaks {@link AudioApiRequest} when the environment names one. */
export const apiProvider: AudioAnalysisProvider<AudioSignal> = {
id: "api",
local: false,
capabilities: API_CAPABILITIES,
- available() {
- const { url, key } = config();
- if (!url) return unavailable(`${URL_ENV} is not set`);
- if (!key) return unavailable(`${KEY_ENV} is not set`);
- return AVAILABLE;
+ async available() {
+ const found = await access();
+ return typeof found === "string" ? unavailable(found) : AVAILABLE;
},
async analyze(capability, signal) {
const asked = capability as AudioApiCapability;
diff --git a/packages/cli/src/audio/analysis/providers/shared.ts b/packages/cli/src/audio/analysis/providers/shared.ts
index c6e88153d9..1d06a97b7f 100644
--- a/packages/cli/src/audio/analysis/providers/shared.ts
+++ b/packages/cli/src/audio/analysis/providers/shared.ts
@@ -5,6 +5,9 @@ import { join } from "node:path";
import { promisify } from "node:util";
import type { AudioSignal, Availability, Downbeats } from "@hyperframes/core/audio-analysis";
import { medianInterval } from "@hyperframes/core/audio-analysis";
+import { buildAuthHeaders } from "../../../auth/client.js";
+import { tryResolveCredential } from "../../../auth/resolver.js";
+import { refreshIfNeeded, resolveCloudBaseUrl } from "../../../cloud/auth.js";
export const runFile = promisify(execFile);
@@ -74,3 +77,58 @@ export function downbeatsWithMeter(beats: number[], downbeats: number[]): Downbe
const round = (t: number) => Math.round(t * 1000) / 1000;
export const rounded = (times: readonly number[]) => times.map(round);
+
+export interface ServiceAccess {
+ url: string;
+ headers: Record<string, string>;
+ timeoutMs: number;
+}
+
+const NO_HEYGEN_CREDENTIAL =
+ "no HeyGen credential: set HEYGEN_API_KEY or run `hyperframes auth login --api-key`";
+
+async function heygenHeaders(): Promise<Record<string, string> | string> {
+ try {
+ const credential = await tryResolveCredential();
+ return credential ? buildAuthHeaders(await refreshIfNeeded(credential)) : NO_HEYGEN_CREDENTIAL;
+ } catch (error) {
+ return error instanceof Error ? error.message : String(error);
+ }
+}
+
+/** An explicit service URL requires its own key; never forward the HeyGen credential to it. */
+export async function serviceAccess(
+ env: { url: string; key: string; timeout: string },
+ path: string,
+ defaultMs: number,
+): Promise<ServiceAccess | string> {
+ const timeout = Number(process.env[env.timeout]);
+ const timeoutMs = timeout > 0 ? timeout : defaultMs;
+ const url = process.env[env.url]?.trim();
+ const key = process.env[env.key]?.trim();
+ if (url === "") return `${env.url} is empty, which turns this service off`;
+ if (key)
+ return {
+ url: url || `${resolveCloudBaseUrl()}${path}`,
+ headers: { "x-api-key": key },
+ timeoutMs,
+ };
+ if (url) return `${env.url} is set but ${env.key} is not`;
+ const headers = await heygenHeaders();
+ if (typeof headers === "string") return headers;
+ return { url: `${resolveCloudBaseUrl()}${path}`, headers, timeoutMs };
+}
+
+function errorMessageIn(text: string): string | undefined {
+ try {
+ return (JSON.parse(text) as { error?: { message?: string } }).error?.message;
+ } catch {
+ return undefined;
+ }
+}
+
+export async function serviceFailure(response: Response, service: string): Promise<Error> {
+ const text = await response.text().catch(() => "");
+ const detail = errorMessageIn(text) ?? text.slice(0, 200);
+ return new Error(`${service} answered ${response.status}${detail ? `: ${detail}` : ""}`);
+}
diff --git a/packages/cli/src/audio/read/effects.test.ts b/packages/cli/src/audio/read/effects.test.ts
new file mode 100644
index 0000000000..ee8cbc9655
--- /dev/null
+++ b/packages/cli/src/audio/read/effects.test.ts
@@ -0,0 +1,120 @@
+import { describe, expect, it } from "vitest";
+import { findEffects } from "./effects.js";
+import { READ_RATE } from "./signal.js";
+
+let seed = 11;
+const noise = () =>
+ ((seed = (seed * 1_103_515_245 + 12_345) % 2_147_483_648) / 2_147_483_648) * 2 - 1;
+
+function track(seconds: number, ...sounds: ((t: number) => number)[]): Float32Array {
+ seed = 11;
+ const out = new Float32Array(Math.round(seconds * READ_RATE));
+ for (let i = 0; i < out.length; i++) {
+ const t = i / READ_RATE;
+ out[i] = 0.0005 * noise() + sounds.reduce((sum, sound) => sum + sound(t), 0);
+ }
+ return out;
+}
+
+const boom = (at: number) => (t: number) =>
+ t < at ? 0 : 0.8 * Math.exp(-(t - at) / 0.35) * Math.sin(2 * Math.PI * 55 * (t - at));
+const click = (at: number) => (t: number) =>
+ t < at || t > at + 0.03 ? 0 : 0.6 * Math.exp(-(t - at) / 0.006) * noise();
+const riser = (at: number, length: number) => (t: number) =>
+ t < at || t > at + length + 0.05
+ ? 0
+ : 0.5 *
+ Math.min(1, (t - at) / length) ** 2 *
+ noise() *
+ (t > at + length ? 1 - (t - at - length) / 0.05 : 1);
+
+describe("findEffects", () => {
+ it("names a low boom, a short bright tick and a long riser, each once", () => {
+ const source = track(8, boom(1), click(3.2), riser(5, 1.2));
+ const { events } = findEffects({
+ source,
+ sampleRate: READ_RATE,
+ music: null,
+ heard: null,
+ beats: [],
+ speech: [],
+ });
+ expect(events.map((e) => [Math.round(e.t * 10) / 10, e.kind])).toEqual([
+ [1, "boom"],
+ [3.2, "tick"],
+ [5.1, "riser"],
+ ]);
+ expect(events[0]).toMatchObject({ band: "low", count: 1, overMusicDb: null });
+ expect(events[0]!.tail).toBeGreaterThan(0.6);
+ expect(events[2]!.attackMs).toBeGreaterThan(600);
+ });
+
+ it("groups the same sound and reports a quick run of it as one train", () => {
+ const source = track(6, click(1), click(1.12), click(1.24), click(4));
+ const { events, groups } = findEffects({
+ source,
+ sampleRate: READ_RATE,
+ music: null,
+ heard: null,
+ beats: [],
+ speech: [],
+ });
+ expect(groups).toEqual([{ id: "S1", kind: "tick", count: 4 }]);
+ expect(events.map((e) => [e.t, e.count])).toEqual([
+ [1, 3],
+ [4, 1],
+ ]);
+ });
+
+ it("leaves out a sound that keeps landing on the music's beat, and skips onsets inside speech", () => {
+ const beats = Array.from({ length: 12 }, (_, i) => 0.5 + i * 0.5);
+ const source = track(7, ...beats.map((b) => click(b)), boom(6.2));
+ const result = findEffects({
+ source,
+ sampleRate: READ_RATE,
+ music: null,
+ heard: null,
+ beats,
+ speech: [],
+ });
+ expect(result.drumHits).toBe(12);
+ expect(result.events.map((e) => e.kind)).toEqual(["boom"]);
+ const muted = findEffects({
+ source: track(4, boom(1)),
+ sampleRate: READ_RATE,
+ music: null,
+ heard: null,
+ beats: [],
+ speech: [{ start: 0.8, end: 1.5 }],
+ });
+ expect(muted.events).toEqual([]);
+ });
+
+ it("measures an effect against the music playing under it", () => {
+ const source = track(4, boom(2));
+ const music = track(4, (t) => 0.05 * Math.sin(2 * Math.PI * 440 * t));
+ const [event] = findEffects({
+ source,
+ sampleRate: READ_RATE,
+ music,
+ heard: null,
+ beats: [],
+ speech: [],
+ }).events;
+ expect(event!.overMusicDb).toBeGreaterThan(10);
+ });
+
+ it("leaves out stem residue far under the recording, and gives no level over music that is out", () => {
+ const quiet = track(4, (t) => 0.01 * boom(1)(t), boom(3));
+ const loud = track(4, (t) => (t < 2 ? 0.5 * Math.sin(2 * Math.PI * 220 * t) : 0));
+ const { events } = findEffects({
+ source: quiet,
+ sampleRate: READ_RATE,
+ music: track(4, () => 0),
+ heard: quiet.map((v, i) => v + loud[i]!),
+ beats: [],
+ speech: [],
+ });
+ expect(events.map((e) => [Math.round(e.t), e.overMusicDb])).toEqual([[3, null]]);
+ });
+});
diff --git a/packages/cli/src/audio/read/effects.ts b/packages/cli/src/audio/read/effects.ts
new file mode 100644
index 0000000000..db2a1d85d5
--- /dev/null
+++ b/packages/cli/src/audio/read/effects.ts
@@ -0,0 +1,356 @@
+import { bandFramesOf, type Span } from "@hyperframes/core/audio-analysis";
+import { asSignal, levels, quantile, round } from "./signal.js";
+import type { Band, Effect, EffectGroup, EffectKind, Effects } from "./types.js";
+
+const HOP = 0.01;
+const FLOOR_FRAMES = 150;
+const FLOOR_EVERY = 10;
+const RISE_DB = 20;
+const STEP_DB = 6;
+const AUDIBLE_DB = -60;
+const MASKED_DB = 20;
+const MUSIC_OUT_DB = -60;
+const TAIL_DB = 30;
+const LONGEST = 4;
+const SAME_SOUND = 0.95;
+const TRAIN_GAP = 0.2;
+const BEAT_SLACK = 0.05;
+const ON_BEAT_SHARE = 0.7;
+const DRUM_GROUP_MIN = 8;
+const LOW_HZ = 150;
+const BAND_FPS = 100;
+
+interface Found {
+ t: number;
+ peak: number;
+ end: number;
+ peakDb: number;
+ riseDb: number;
+}
+
+/** A rolling low percentile of the level: what was playing before each frame. */
+function floors(db: Float32Array): Float32Array {
+ const out = new Float32Array(db.length);
+ let current = db[0] ?? -120;
+ for (let f = 0; f < db.length; f++) {
+ if (f % FLOOR_EVERY === 0 && f > 0)
+ current = quantile(db.subarray(Math.max(0, f - FLOOR_FRAMES), f), 0.2);
+ out[f] = current;
+ }
+ return out;
+}
+
+const BLOCK = 0.001;
+
+/** The first millisecond that rises halfway out of the floor: frame levels alone place an onset up to a hop early. */
+function refined(
+ samples: Float32Array,
+ sampleRate: number,
+ frame: number,
+ floorDb: number,
+): number {
+ const block = Math.max(1, Math.round(BLOCK * sampleRate));
+ const from = Math.max(0, Math.round((frame - 2) * HOP * sampleRate));
+ const to = Math.min(samples.length, Math.round((frame + 2) * HOP * sampleRate));
+ for (let a = from; a + block <= to; a += block) {
+ let power = 0;
+ for (let i = a; i < a + block; i++) power += samples[i]! ** 2;
+ if (10 * Math.log10(power / block + 1e-12) >= floorDb + RISE_DB / 2) return a / sampleRate;
+ }
+ return frame * HOP;
+}
+
+/** How far a sound found at frame `f` reaches: back to where it left the floor, on to its peak and through its tail. */
+function extent(db: Float32Array, f: number, floor: number) {
+ let onset = f;
+ while (onset > Math.max(0, f - 50) && db[onset - 1]! > floor + STEP_DB) onset--;
+ let peak = f;
+ let end = f;
+ const last = Math.min(db.length - 1, f + LONGEST / HOP);
+ for (let g = f; g <= last; g++) {
+ if (db[g]! > db[peak]!) peak = g;
+ end = g;
+ if (g > peak && (db[g]! < db[peak]! - TAIL_DB || db[g]! < floor + 3)) break;
+ }
+ return { onset, peak, end };
+}
+
+function startsHere(db: Float32Array, f: number, floor: number): boolean {
+ const level = db[f]!;
+ if (level < AUDIBLE_DB || level - floor < RISE_DB) return false;
+ return level - Math.min(db[f - 1]!, db[f - 2]!, db[f - 3]!, db[f - 4]!, db[f - 5]!) >= STEP_DB;
+}
+
+function detect(
+ db: Float32Array,
+ samples: Float32Array,
+ sampleRate: number,
+ skip: (t: number) => boolean,
+): Found[] {
+ const floor = floors(db);
+ const found: Found[] = [];
+ for (let f = 5; f < db.length; f++) {
+ if (!startsHere(db, f, floor[f]!)) continue;
+ const { onset, peak, end } = extent(db, f, floor[f]!);
+ const t = refined(samples, sampleRate, onset, floor[f]!);
+ if (!skip(t))
+ found.push({
+ t,
+ peak: peak * HOP,
+ end: end * HOP,
+ peakDb: db[peak]!,
+ riseDb: db[peak]! - floor[f]!,
+ });
+ f = Math.max(end, onset + 8);
+ }
+ return found;
+}
+
+const BIN_HZ = 22050 / 1024;
+const EDGES = (() => {
+ const edges: number[] = [];
+ for (let hz = 30; hz < 9000; hz *= 2 ** (1 / 5)) edges.push(Math.max(1, Math.round(hz / BIN_HZ)));
+ return edges;
+})();
+const bandHz = (k: number) => 30 * 2 ** ((k + 0.5) / 5);
+/** Bins in each band: core's band frames hold the mean power per bin, so a band's energy is that times its width. */
+const WIDTH = EDGES.slice(0, -1).map((edge, k) => Math.max(edge + 1, EDGES[k + 1]!) - edge);
+const energies = (power: Float64Array) => Array.from(power, (v, k) => Math.expm1(v) * WIDTH[k]!);
+const BANDS_PER_PRINT = 5;
+
+function shape(bands: Float64Array[], from: number, to: number): Float64Array {
+ const a = Math.max(0, Math.floor(from * BAND_FPS));
+ const b = Math.max(a + 1, Math.min(bands.length, Math.ceil(to * BAND_FPS)));
+ const size = bands[0]?.length ?? 0;
+ const mean = new Float64Array(size);
+ for (let f = a; f < b; f++) for (let k = 0; k < size; k++) mean[k]! += bands[f]![k]! / (b - a);
+ return mean;
+}
+
+function lowShare(power: Float64Array): number {
+ const e = energies(power);
+ const total = e.reduce((a, b) => a + b, 0);
+ const low = e.reduce((sum, p, k) => sum + (bandHz(k) < LOW_HZ ? p : 0), 0);
+ return total > 0 ? low / total : 0;
+}
+
+function centroidHz(power: Float64Array): number {
+ const e = energies(power);
+ const weight = e.reduce((a, b) => a + b, 0);
+ const sum = e.reduce((acc, p, k) => acc + p * Math.log2(bandHz(k)), 0);
+ return weight > 0 ? 2 ** (sum / weight) : 0;
+}
+
+const tonal = (power: Float64Array) => {
+ const sorted = Float64Array.from(power).sort();
+ return sorted[sorted.length - 1]! - sorted[Math.floor(sorted.length * 0.75)]! >= 2.5;
+};
+
+interface Shape {
+ length: number;
+ attack: number;
+ tail: number;
+ band: Band;
+ centroid: number;
+ tonal: boolean;
+}
+
+/** Tried in order; a sound none of them fits is a hit. */
+const KINDS: [(s: Shape) => boolean, (s: Shape) => EffectKind][] = [
+ [(s) => s.length >= 0.8 && s.attack >= 0.45 * s.length, () => "riser"],
+ [(s) => s.band === "low", (s) => (s.tail >= 0.6 ? "boom" : "thump")],
+ [(s) => s.length <= 0.15, (s) => (s.centroid >= 3000 ? "tick" : "click")],
+ [(s) => s.tonal && s.tail >= 0.4, () => "ding"],
+ [(s) => s.attack >= 0.06 && s.length <= 1.5, () => "whoosh"],
+];
+
+function bandOf(low: number, centroid: number): Band {
+ if (low >= 0.5) return "low";
+ return centroid >= 2500 ? "high" : "mid";
+}
+
+function kindOf(found: Found, power: Float64Array): { kind: EffectKind; band: Band } {
+ const centroid = centroidHz(power);
+ const shape: Shape = {
+ length: found.end - found.t,
+ attack: found.peak - found.t,
+ tail: found.end - found.peak,
+ band: bandOf(lowShare(power), centroid),
+ centroid,
+ tonal: tonal(power),
+ };
+ const kind = KINDS.find(([fits]) => fits(shape))?.[1](shape) ?? "hit";
+ return { kind, band: shape.band };
+}
+
+/** The sound's spectral shape: the square root of each octave's share of its energy, so loudness does not count. */
+function printOf(power: Float64Array): Float64Array {
+ const e = energies(power);
+ const octaves = new Float64Array(Math.ceil(e.length / BANDS_PER_PRINT));
+ e.forEach((p, k) => (octaves[Math.floor(k / BANDS_PER_PRINT)]! += p));
+ const total = octaves.reduce((a, b) => a + b, 0);
+ return octaves.map((p) => (total > 0 ? Math.sqrt(p / total) : 0));
+}
+
+function cosine(a: Float64Array, b: Float64Array): number {
+ let dot = 0;
+ let na = 0;
+ let nb = 0;
+ for (let k = 0; k < a.length; k++) {
+ dot += a[k]! * b[k]!;
+ na += a[k]! ** 2;
+ nb += b[k]! ** 2;
+ }
+ return na && nb ? dot / Math.sqrt(na * nb) : 0;
+}
+
+interface Measured extends Found {
+ kind: EffectKind;
+ band: Band;
+ print: Float64Array;
+ group: number;
+}
+
+function grouped(events: Measured[]): Float64Array[] {
+ const prints: { sum: Float64Array; n: number; length: number }[] = [];
+ for (const event of events) {
+ const length = event.end - event.t;
+ let best = -1;
+ let bestScore = SAME_SOUND;
+ prints.forEach((p, i) => {
+ const ratio = length / p.length;
+ if (ratio < 0.5 || ratio > 2) return;
+ const score = cosine(
+ event.print,
+ p.sum.map((v) => v / p.n),
+ );
+ if (score >= bestScore) {
+ best = i;
+ bestScore = score;
+ }
+ });
+ if (best < 0) {
+ prints.push({ sum: Float64Array.from(event.print), n: 1, length });
+ best = prints.length - 1;
+ } else {
+ const p = prints[best]!;
+ event.print.forEach((v, k) => (p.sum[k]! += v));
+ p.length = (p.length * p.n + length) / (p.n + 1);
+ p.n++;
+ }
+ event.group = best;
+ }
+ return prints.map((p) => p.sum);
+}
+
+function onBeat(t: number, beats: readonly number[]): boolean {
+ let lo = 0;
+ let hi = beats.length;
+ while (lo < hi) {
+ const mid = (lo + hi) >> 1;
+ if (beats[mid]! < t) lo = mid + 1;
+ else hi = mid;
+ }
+ const near = [beats[lo - 1], beats[lo]].filter((b): b is number => b !== undefined);
+ return near.some((b) => Math.abs(b - t) <= BEAT_SLACK);
+}
+
+function majority(kinds: EffectKind[]): EffectKind {
+ const counts = new Map<EffectKind, number>();
+ for (const k of kinds) counts.set(k, (counts.get(k) ?? 0) + 1);
+ return [...counts].sort((a, b) => b[1] - a[1])[0]![0];
+}
+
+export interface EffectInput {
+ /** The track effects are found in: the drum stem, or the whole mix without stems. */
+ source: Float32Array;
+ sampleRate: number;
+ /** The music without drums, to say how far an effect stands over it. */
+ music: Float32Array | null;
+ /** The whole recording, when the source is a stem: an effect far under it is separation residue nobody hears. */
+ heard: Float32Array | null;
+ beats: readonly number[];
+ /** Spans where an onset is speech, not an effect (the mix only). */
+ speech: readonly Span[];
+}
+
+function measured(input: EffectInput): Measured[] {
+ const db = levels(input.source, input.sampleRate, HOP);
+ const inSpeech = (t: number) => input.speech.some((s) => t >= s.start && t < s.end);
+ const heardDb = input.heard ? levels(input.heard, input.sampleRate, HOP, 0.5) : null;
+ const masked = (f: Found) => {
+ const level = heardDb?.[Math.round(f.peak / HOP)];
+ return level !== undefined && f.peakDb < level - MASKED_DB;
+ };
+ const found = detect(db, input.source, input.sampleRate, inSpeech).filter((f) => !masked(f));
+ const bands = found.length ? bandFramesOf(asSignal(input.source, input.sampleRate)).bands : [];
+ const out: Measured[] = found.map((f) => {
+ const power = shape(bands, f.t, Math.min(f.end, f.t + 0.4));
+ return { ...f, ...kindOf(f, power), print: printOf(power), group: -1 };
+ });
+ grouped(out);
+ return out;
+}
+
+/** Groups that keep landing on the music's beat are its own drums, not effects. */
+function drumGroups(members: Map<number, Measured[]>, beats: readonly number[]): Set<number> {
+ const drums = new Set<number>();
+ if (!beats.length) return drums;
+ for (const [id, list] of members) {
+ const onBeats = list.filter((m) => onBeat(m.t, beats)).length;
+ if (list.length >= DRUM_GROUP_MIN && onBeats >= ON_BEAT_SHARE * list.length) drums.add(id);
+ }
+ return drums;
+}
+
+function effectOf(
+ m: Measured,
+ group: string,
+ kind: EffectKind,
+ musicDb: Float32Array | null,
+): Effect {
+ const musicLevel = musicDb?.[Math.round(m.peak / HOP)];
+ const musicPlays = musicLevel !== undefined && musicLevel > MUSIC_OUT_DB;
+ return {
+ t: round(m.t),
+ end: round(m.end),
+ kind,
+ group,
+ band: m.band,
+ attackMs: Math.round((m.peak - m.t) * 1000),
+ tail: round(m.end - m.peak),
+ riseDb: Math.round(m.riseDb),
+ overMusicDb: musicPlays ? Math.round(m.peakDb - musicLevel) : null,
+ count: 1,
+ };
+}
+
+export function findEffects(input: EffectInput): Effects {
+ const sounds = measured(input);
+ const members = new Map<number, Measured[]>();
+ for (const m of sounds) members.set(m.group, [...(members.get(m.group) ?? []), m]);
+ const drums = drumGroups(members, input.beats);
+ const kept = [...members]
+ .filter(([id]) => !drums.has(id))
+ .sort((a, b) => b[1].length - a[1].length);
+ const names = new Map(kept.map(([id], i) => [id, `S${i + 1}`]));
+ const groups: EffectGroup[] = kept.map(([id, list]) => ({
+ id: names.get(id)!,
+ kind: majority(list.map((m) => m.kind)),
+ count: list.length,
+ }));
+ const kindOfGroup = new Map(groups.map((g) => [g.id, g.kind]));
+ const musicDb = input.music ? levels(input.music, input.sampleRate, HOP, 0.5) : null;
+ const events: Effect[] = [];
+ for (const m of sounds) {
+ if (drums.has(m.group)) continue;
+ const group = names.get(m.group)!;
+ const previous = events[events.length - 1];
+ if (previous?.group === group && m.t - previous.end <= TRAIN_GAP) {
+ previous.end = round(m.end);
+ previous.count++;
+ } else events.push(effectOf(m, group, kindOfGroup.get(group) ?? m.kind, musicDb));
+ }
+ const drumHits = [...drums].reduce((n, id) => n + members.get(id)!.length, 0);
+ return { events, groups, drumHits };
+}
diff --git a/packages/cli/src/audio/read/format.ts b/packages/cli/src/audio/read/format.ts
new file mode 100644
index 0000000000..bf54b06d0d
--- /dev/null
+++ b/packages/cli/src/audio/read/format.ts
@@ -0,0 +1,144 @@
+import type { Span } from "@hyperframes/core/audio-analysis";
+import { semitones } from "./pitch.js";
+import type { Effect, Entry, MusicSegment, ReadAudio, Sentence, Voice } from "./types.js";
+
+const PITCH_NOTE_SEMITONES = 3;
+const KIND_WIDTH = 7;
+
+interface Line {
+ start: number;
+ end: number | null;
+ kind: string;
+ text: string;
+}
+
+const s = (t: number) => t.toFixed(1);
+const signed = (db: number) => `${db >= 0 ? "+" : ""}${db} dB`;
+
+function when(line: Line): string {
+ return line.end === null ? s(line.start) : `${s(line.start)}–${s(line.end)}`;
+}
+
+function voiceLine(sentence: Sentence, voice: Voice): Line {
+ const { unit, meanRate, medianPitchHz } = voice;
+ const parts = [sentence.text === null ? "(no transcript)" : `"${sentence.text}"`];
+ if (sentence.rate !== null)
+ parts.push(`${sentence.rate} ${unit}/s${meanRate !== null ? ` (mean ${meanRate})` : ""}`);
+ if (
+ sentence.pitchHz !== null &&
+ medianPitchHz !== null &&
+ Math.abs(semitones(sentence.pitchHz, medianPitchHz)) >= PITCH_NOTE_SEMITONES
+ )
+ parts.push(`pitch ${sentence.pitchHz} Hz (median ${medianPitchHz})`);
+ if (sentence.ending && sentence.ending !== "level") parts.push(`${sentence.ending} end`);
+ return { start: sentence.start, end: sentence.end, kind: "voice", text: parts.join(" ") };
+}
+
+function placeOf(t: number, pauses: readonly Span[]): string | null {
+ const pause = pauses.find((p) => t >= p.start && t < p.end);
+ return pause ? `${(t - pause.start).toFixed(1)} s into the pause` : null;
+}
+
+function standing(effect: Effect, musicRead: boolean): string {
+ if (effect.overMusicDb !== null) return `${signed(effect.overMusicDb)} over music`;
+ return musicRead ? "while the music is out" : `${signed(effect.riseDb)} over what played before`;
+}
+
+function effectLine(effect: Effect, pauses: readonly Span[], musicRead: boolean): Line {
+ const parts = [
+ `${effect.kind}${effect.count > 1 ? ` ×${effect.count}` : ""} (${effect.group}): ${effect.band}`,
+ `attack ${effect.attackMs} ms`,
+ `tail ${effect.tail.toFixed(1)} s`,
+ standing(effect, musicRead),
+ ];
+ const place = placeOf(effect.t, pauses);
+ if (place) parts.push(place);
+ return { start: effect.t, end: null, kind: "effect", text: parts.join(", ") };
+}
+
+const ENTRY: Record<Entry, string> = {
+ start: "",
+ new: ", first appearance after a break",
+ resumes: ", resumes after a break",
+ returns: ", returns",
+};
+
+function musicLine(segment: MusicSegment, analysis: ReadAudio): Line {
+ const piece = analysis.music!.pieces.find((p) => p.id === segment.piece)!;
+ const about = [piece.instruments.join(", "), segment.bpm ? `${segment.bpm} BPM` : null]
+ .filter(Boolean)
+ .join(" · ");
+ const fade = segment.fadeFrom !== null ? `, fades from ${s(segment.fadeFrom)}` : "";
+ return {
+ start: segment.start,
+ end: segment.end,
+ kind: "music",
+ text: `piece ${piece.id}${about ? ` (${about})` : ""}${ENTRY[segment.entry]}${fade}`,
+ };
+}
+
+function overlaps(line: Line, range?: [number, number]): boolean {
+ if (!range) return true;
+ return line.start < range[1] && (line.end ?? line.start) >= range[0];
+}
+
+function header(analysis: ReadAudio): string[] {
+ const { voice, effects, music } = analysis;
+ const separated =
+ analysis.separation === "stems" ? `stems: ${analysis.stems.join(", ")}` : "not separated";
+ const lines = [`${analysis.file} · ${s(analysis.duration)} s · ${separated}`];
+ if (voice) {
+ const rate = voice.meanRate !== null ? `mean ${voice.meanRate} ${voice.unit}/s` : null;
+ const pitch = voice.medianPitchHz !== null ? `median pitch ${voice.medianPitchHz} Hz` : null;
+ lines.push(
+ `voice: ${voice.sentences.length} lines, ${voice.pauses.length} pauses${[rate, pitch]
+ .filter(Boolean)
+ .map((p) => `, ${p}`)
+ .join("")}`,
+ );
+ }
+ if (effects) {
+ const top = effects.groups
+ .slice(0, 6)
+ .map((g) => `${g.id} ${g.kind} ×${g.count}`)
+ .join(", ");
+ lines.push(
+ `effects: ${effects.events.length} in ${effects.groups.length} sounds${top ? ` (${top})` : ""}${
+ effects.drumHits ? `; ${effects.drumHits} drum hits on the beat left out` : ""
+ }`,
+ );
+ }
+ if (music) lines.push(`music: ${music.pieces.length} pieces, ${music.breaks.length} breaks`);
+ return lines;
+}
+
+/** The recording as time-stamped lines, all of it or the events overlapping `range`. */
+export function formatRead(analysis: ReadAudio, range?: [number, number]): string {
+ const { voice, effects } = analysis;
+ const pauses = voice?.pauses ?? [];
+ const lines: Line[] = [
+ ...(voice?.sentences.map((x) => voiceLine(x, voice)) ?? []),
+ ...pauses.map((p) => ({
+ start: p.start,
+ end: p.end,
+ kind: "pause",
+ text: `${(p.end - p.start).toFixed(1)} s`,
+ })),
+ ...(effects?.events.map((e) => effectLine(e, pauses, analysis.music !== null)) ?? []),
+ ...(analysis.music?.segments.map((m) => musicLine(m, analysis)) ?? []),
+ ...(analysis.music?.breaks.map((b) => ({
+ start: b.start,
+ end: b.end,
+ kind: "music",
+ text: `break ${(b.end - b.start).toFixed(1)} s`,
+ })) ?? []),
+ ]
+ .filter((line) => overlaps(line, range))
+ .sort((a, b) => a.start - b.start || (a.end ?? a.start) - (b.end ?? b.start));
+ const width = Math.max(0, ...lines.map((l) => when(l).length));
+ const body = lines.map((l) => `${when(l).padEnd(width)} ${l.kind.padEnd(KIND_WIDTH)} ${l.text}`);
+ const warnings = analysis.warnings.map((w) => `! ${w}`);
+ return [...header(analysis), "", ...body, ...(warnings.length ? ["", ...warnings] : [])].join(
+ "\n",
+ );
+}
diff --git a/packages/cli/src/audio/read/music.test.ts b/packages/cli/src/audio/read/music.test.ts
new file mode 100644
index 0000000000..2a8f83c467
--- /dev/null
+++ b/packages/cli/src/audio/read/music.test.ts
@@ -0,0 +1,53 @@
+import { describe, expect, it } from "vitest";
+import { readMusic } from "./music.js";
+import { READ_RATE } from "./signal.js";
+
+const chord = (hz: number[]) => (t: number) =>
+ hz.reduce((sum, f) => sum + 0.1 * Math.sin(2 * Math.PI * f * t), 0);
+const C_MAJOR = chord([261.63, 329.63, 392]);
+const F_SHARP_MINOR = chord([369.99, 440, 554.37]);
+
+function piano(seconds: number, parts: [number, number, (t: number) => number][]): Float32Array {
+ const out = new Float32Array(Math.round(seconds * READ_RATE));
+ for (const [from, to, sound] of parts)
+ for (let i = Math.round(from * READ_RATE); i < Math.round(to * READ_RATE); i++)
+ out[i] = sound(i / READ_RATE);
+ return out;
+}
+
+describe("readMusic", () => {
+ it("splits the bed at its cut-outs and tells a new piece from one that returns", () => {
+ const stem = piano(26, [
+ [0, 8, C_MAJOR],
+ [9, 16, F_SHARP_MINOR],
+ [17, 24, C_MAJOR],
+ ]);
+ const { music } = readMusic({ stems: { piano: stem }, drums: null, sampleRate: READ_RATE });
+ expect(
+ music.segments.map((s) => [Math.round(s.start), Math.round(s.end), s.piece, s.entry]),
+ ).toEqual([
+ [0, 8, 1, "start"],
+ [9, 16, 2, "new"],
+ [17, 24, 1, "returns"],
+ ]);
+ expect(music.breaks.map((b) => [Math.round(b.start), Math.round(b.end)])).toEqual([
+ [8, 9],
+ [16, 17],
+ ]);
+ expect(music.pieces[0]).toMatchObject({ id: 1, instruments: ["piano"], firstAt: 0 });
+ expect(music.energy.map((e) => e.start)).toEqual([0]);
+ });
+
+ it("names the stems that carry a piece, loudest first, and notes a fade at its end", () => {
+ const fadeOut = (t: number) => C_MAJOR(t) * Math.min(1, Math.max(0.001, (10 - t) / 3));
+ const loud = piano(10, [[0, 10, fadeOut]]);
+ const quiet = loud.map((v) => v * 0.6);
+ const { music } = readMusic({
+ stems: { piano: loud, guitar: quiet, bass: new Float32Array(loud.length) },
+ drums: null,
+ sampleRate: READ_RATE,
+ });
+ expect(music.pieces[0]!.instruments).toEqual(["piano", "guitar"]);
+ expect(music.segments[0]!.fadeFrom).toBeGreaterThan(6.5);
+ });
+});
diff --git a/packages/cli/src/audio/read/music.ts b/packages/cli/src/audio/read/music.ts
new file mode 100644
index 0000000000..c5fb5aebf2
--- /dev/null
+++ b/packages/cli/src/audio/read/music.ts
@@ -0,0 +1,273 @@
+import { bandFramesOf, medianInterval, rhythm } from "@hyperframes/core/audio-analysis";
+import { asSignal, levels, median, mixed, round } from "./signal.js";
+import type { Entry, Music, MusicSegment, Piece } from "./types.js";
+
+const HOP = 0.1;
+const ABSENT_DB = -55;
+const CUT_DB = 25;
+const RUNNING_FRAMES = 200;
+const LEVEL_WINDOW = 0.4;
+const CONTEXT = 2;
+const SHORTEST_BREAK = 0.5;
+const LONGEST_REST = 1;
+const SHORTEST_SEGMENT = 1;
+const SAME_PIECE = 0.92;
+const SAME_TEMPO = 0.06;
+const SAME_MIX = 0.25;
+const RESUMING_GAP = 5;
+const RESUMING_PIECE = 0.85;
+const RESUMING_MIX = 0.35;
+const FADE_DB = 10;
+const FADE_WINDOW = 3;
+const ENERGY_WINDOW = 30;
+const LEADING_SHARE = 0.2;
+
+export interface MusicInput {
+ /** Every pitched stem, by name: bass, guitar, piano, other. */
+ stems: Record<string, Float32Array>;
+ drums: Float32Array | null;
+ sampleRate: number;
+}
+
+interface Run {
+ start: number;
+ end: number;
+}
+
+type Span = { quiet: boolean; start: number; end: number };
+
+function joined(spans: Span[], tooShort: (span: Span) => boolean): Span[] {
+ const out: Span[] = [];
+ for (const span of spans) {
+ const last = out[out.length - 1];
+ if (last && (tooShort(span) || last.quiet === span.quiet)) last.end = span.end;
+ else out.push({ ...span });
+ }
+ return out;
+}
+
+function powerDb(db: Float32Array, start: number, end: number): number {
+ const a = Math.max(0, Math.round(start / HOP));
+ const b = Math.min(db.length, Math.round(end / HOP));
+ let sum = 0;
+ for (let f = a; f < b; f++) sum += 10 ** (db[f]! / 10);
+ return b > a ? 10 * Math.log10(sum / (b - a) || 1e-12) : Number.NEGATIVE_INFINITY;
+}
+
+/** A break holds the music far under what plays around it, or out altogether, across its whole length. */
+function holdsUp(db: Float32Array, span: Span): boolean {
+ const inside = powerDb(db, span.start, span.end);
+ const around = Math.max(
+ powerDb(db, span.start - CONTEXT, span.start),
+ powerDb(db, span.end, span.end + CONTEXT),
+ );
+ return inside < ABSENT_DB || inside < around - CUT_DB;
+}
+
+const tooShort = (span: Span) =>
+ span.end - span.start < (span.quiet ? SHORTEST_BREAK : SHORTEST_SEGMENT);
+
+function candidates(
+ quiet: Uint8Array,
+ from: number,
+ to: number,
+ absorbed: (span: Span) => boolean = tooShort,
+): Span[] {
+ const spans: Span[] = [];
+ for (let f = from; f < to; f++) {
+ const isQuiet = quiet[f] === 1;
+ const last = spans[spans.length - 1];
+ if (last && last.quiet === isQuiet) last.end = (f + 1) * HOP;
+ else spans.push({ quiet: isQuiet, start: f * HOP, end: (f + 1) * HOP });
+ }
+ return joined(spans, absorbed);
+}
+
+/** Inside a dip that does not hold up as a whole, the music is out wherever it is absent for longer than a rest between notes. */
+function silencesIn(db: Float32Array, span: Span): Span[] {
+ const from = Math.round(span.start / HOP);
+ const to = Math.min(db.length, Math.round(span.end / HOP));
+ const absent = new Uint8Array(db.length);
+ for (let f = from; f < to; f++) absent[f] = db[f]! < ABSENT_DB ? 1 : 0;
+ const shortSilence = (s: Span) => s.quiet && s.end - s.start < LONGEST_REST;
+ return candidates(absent, from, to, shortSilence).map((s) =>
+ s.quiet && !holdsUp(db, s) ? { ...s, quiet: false } : s,
+ );
+}
+
+/** Breaks are where the music drops far under its running level, or out altogether, for half a second or more. */
+function runs(db: Float32Array): { segments: Run[]; breaks: Run[] } {
+ const quiet = new Uint8Array(db.length);
+ const heard: number[] = [];
+ for (let f = 0; f < db.length; f++) {
+ const level = db[f]!;
+ const reference = heard.length
+ ? median(heard.slice(-RUNNING_FRAMES))
+ : Number.NEGATIVE_INFINITY;
+ quiet[f] = level < ABSENT_DB || level < reference - CUT_DB ? 1 : 0;
+ if (!quiet[f]) heard.push(level);
+ }
+ const merged = candidates(quiet, 0, db.length).flatMap((span) =>
+ span.quiet && !holdsUp(db, span) ? silencesIn(db, span) : [span],
+ );
+ const settled = joined(merged, (s) => !s.quiet && s.end - s.start < SHORTEST_SEGMENT);
+ return {
+ segments: settled.filter((s) => !s.quiet),
+ breaks: settled.filter((s) => s.quiet && s.start > 0 && s.end < db.length * HOP),
+ };
+}
+
+function chromaOf(chroma: Float64Array[], fps: number, run: Run): Float64Array {
+ const a = Math.floor(run.start * fps);
+ const b = Math.max(a + 1, Math.min(chroma.length, Math.ceil(run.end * fps)));
+ const sum = new Float64Array(12);
+ for (let f = a; f < b; f++) chroma[f]?.forEach((v, k) => (sum[k]! += v));
+ const norm = Math.hypot(...sum);
+ return norm ? sum.map((v) => v / norm) : sum;
+}
+
+const dot = (a: Float64Array, b: Float64Array) => a.reduce((n, v, k) => n + v * b[k]!, 0);
+
+function energyOf(samples: Float32Array, sampleRate: number, run: Run): number {
+ const a = Math.floor(run.start * sampleRate);
+ const b = Math.min(samples.length, Math.floor(run.end * sampleRate));
+ let sum = 0;
+ for (let i = a; i < b; i++) sum += samples[i]! ** 2;
+ return b > a ? sum / (b - a) : 0;
+}
+
+function shares(input: MusicInput, run: Run): Record<string, number> {
+ const energies = Object.entries(input.stems).map(
+ ([name, samples]) => [name, energyOf(samples, input.sampleRate, run)] as const,
+ );
+ const total = energies.reduce((n, [, e]) => n + e, 0);
+ return Object.fromEntries(energies.map(([name, e]) => [name, total ? e / total : 0]));
+}
+
+function mixDistance(a: Record<string, number>, b: Record<string, number>): number {
+ return Object.keys(a).reduce((n, k) => n + Math.abs(a[k]! - (b[k] ?? 0)), 0) / 2;
+}
+
+function bpmOf(beats: readonly number[], run: Run): number | null {
+ const inside = beats.filter((t) => t >= run.start && t < run.end);
+ const interval = inside.length >= 4 ? medianInterval(inside) : null;
+ return interval ? Math.round(60 / interval) : null;
+}
+
+const sameTempo = (a: number | null, b: number | null) =>
+ a === null || b === null || Math.abs(a - b) / Math.max(a, b) <= SAME_TEMPO;
+
+function fadeFrom(db: Float32Array, run: Run): number | null {
+ const a = Math.floor(run.start / HOP);
+ const b = Math.min(db.length, Math.floor(run.end / HOP));
+ const body = median(db.subarray(a, b));
+ const last = db[b - 1] ?? body;
+ if (body - last < FADE_DB) return null;
+ for (let f = Math.max(a, b - FADE_WINDOW / HOP); f < b; f++)
+ if (db[f]! < body - 3) return round(f * HOP, 1);
+ return null;
+}
+
+type Known = Piece & { print: Float64Array; mix: Record<string, number> };
+
+interface Heard {
+ run: Run;
+ print: Float64Array;
+ mix: Record<string, number>;
+ bpm: number | null;
+}
+
+const alike = (p: Known, heard: Heard, piece: number, mix: number) =>
+ dot(p.print, heard.print) >= piece &&
+ mixDistance(p.mix, heard.mix) <= mix &&
+ sameTempo(p.bpm, heard.bpm);
+
+/** The piece a part belongs to: the one before a short break, held loosely, else any heard before, held strictly. */
+function pieceOf(
+ pieces: Known[],
+ previous: MusicSegment | undefined,
+ heard: Heard,
+): Known | undefined {
+ const last = previous && pieces.find((p) => p.id === previous.piece);
+ if (
+ last &&
+ heard.run.start - previous.end <= RESUMING_GAP &&
+ alike(last, heard, RESUMING_PIECE, RESUMING_MIX)
+ )
+ return last;
+ return pieces.find((p) => alike(p, heard, SAME_PIECE, SAME_MIX));
+}
+
+function newPiece(id: number, heard: Heard): Known {
+ return {
+ id,
+ instruments: Object.entries(heard.mix)
+ .filter(([, share]) => share >= LEADING_SHARE)
+ .sort((a, b) => b[1] - a[1])
+ .map(([name]) => name),
+ bpm: heard.bpm,
+ firstAt: round(heard.run.start),
+ print: heard.print,
+ mix: heard.mix,
+ };
+}
+
+function energyCurve(melodic: Float32Array, sampleRate: number, duration: number) {
+ const energy = [];
+ for (let start = 0; start < duration; start += ENERGY_WINDOW) {
+ const end = Math.min(duration, start + ENERGY_WINDOW);
+ const power = energyOf(melodic, sampleRate, { start, end });
+ energy.push({
+ start,
+ end: round(end),
+ db: power > 0 ? Math.round(10 * Math.log10(power)) : -120,
+ });
+ }
+ return energy;
+}
+
+export function readMusic(input: MusicInput): { music: Music; beats: number[] } {
+ const melodic = mixed(Object.values(input.stems));
+ const db = levels(melodic, input.sampleRate, HOP, LEVEL_WINDOW);
+ const { segments: found, breaks } = runs(db);
+ const frames = bandFramesOf(asSignal(melodic, input.sampleRate));
+ const beats = rhythm(
+ asSignal(input.drums ? mixed([melodic, input.drums]) : melodic, input.sampleRate),
+ ).beats.times;
+ const pieces: Known[] = [];
+ const segments: MusicSegment[] = [];
+ for (const run of found) {
+ const heard: Heard = {
+ run,
+ print: chromaOf(frames.chroma, frames.chromaFps, run),
+ mix: shares(input, run),
+ bpm: bpmOf(beats, run),
+ };
+ const previous = segments[segments.length - 1];
+ let piece = pieceOf(pieces, previous, heard);
+ let entry: Entry;
+ if (piece) entry = previous?.piece === piece.id ? "resumes" : "returns";
+ else {
+ piece = newPiece(pieces.length + 1, heard);
+ pieces.push(piece);
+ entry = previous ? "new" : "start";
+ }
+ segments.push({
+ start: round(run.start),
+ end: round(run.end),
+ piece: piece.id,
+ entry,
+ bpm: heard.bpm,
+ fadeFrom: fadeFrom(db, run),
+ });
+ }
+ return {
+ music: {
+ segments,
+ pieces: pieces.map(({ print: _print, mix: _mix, ...piece }) => piece),
+ breaks: breaks.map((b) => ({ start: round(b.start), end: round(b.end) })),
+ energy: energyCurve(melodic, input.sampleRate, db.length * HOP),
+ },
+ beats,
+ };
+}
diff --git a/packages/cli/src/audio/read/pitch.ts b/packages/cli/src/audio/read/pitch.ts
new file mode 100644
index 0000000000..ba4facec45
--- /dev/null
+++ b/packages/cli/src/audio/read/pitch.ts
@@ -0,0 +1,74 @@
+import { PITCH_RATE } from "./signal.js";
+
+const FRAME = 320;
+const HOP = 320;
+const LOW_HZ = 70;
+const HIGH_HZ = 500;
+const VOICED_DB = -35;
+const PERIODIC = 0.15;
+
+export interface PitchPoint {
+ t: number;
+ hz: number;
+}
+
+const MIN_LAG = Math.floor(PITCH_RATE / HIGH_HZ);
+const MAX_LAG = Math.ceil(PITCH_RATE / LOW_HZ);
+
+function voiced(pcm: Float32Array, start: number): boolean {
+ let energy = 0;
+ for (let i = 0; i < FRAME; i++) energy += pcm[start + i]! ** 2;
+ return 10 * Math.log10(energy / FRAME + 1e-12) >= VOICED_DB;
+}
+
+function difference(pcm: Float32Array, start: number, tau: number): number {
+ let sum = 0;
+ for (let i = 0; i < FRAME; i++) {
+ const delta = pcm[start + i]! - pcm[start + i + tau]!;
+ sum += delta * delta;
+ }
+ return sum;
+}
+
+/** The first lag whose normalized difference dips under the threshold, followed down to its minimum; -1 if none. */
+function periodOf(pcm: Float32Array, start: number, d: Float64Array): number {
+ let running = 0;
+ let lag = -1;
+ for (let tau = 1; tau <= MAX_LAG; tau++) {
+ const sum = difference(pcm, start, tau);
+ running += sum;
+ d[tau] = running > 0 ? (sum * tau) / running : 1;
+ if (lag === -1) {
+ if (tau >= MIN_LAG && d[tau]! < PERIODIC) lag = tau;
+ } else if (d[tau]! > d[tau - 1]!) break;
+ else if (d[tau]! < d[lag]!) lag = tau;
+ }
+ return lag;
+}
+
+/** The fundamental of each voiced 40 ms frame of 8 kHz mono, by YIN's cumulative mean normalized difference. */
+export function pitchTrack(pcm: Float32Array): PitchPoint[] {
+ const out: PitchPoint[] = [];
+ const d = new Float64Array(MAX_LAG + 1);
+ for (let start = 0; start + FRAME + MAX_LAG <= pcm.length; start += HOP) {
+ if (!voiced(pcm, start)) continue;
+ const lag = periodOf(pcm, start, d);
+ if (lag > 0) out.push({ t: (start + FRAME / 2) / PITCH_RATE, hz: PITCH_RATE / lag });
+ }
+ return out;
+}
+
+export function within(track: readonly PitchPoint[], start: number, end: number): number[] {
+ let lo = 0;
+ let hi = track.length;
+ while (lo < hi) {
+ const mid = (lo + hi) >> 1;
+ if (track[mid]!.t < start) lo = mid + 1;
+ else hi = mid;
+ }
+ const out: number[] = [];
+ for (let i = lo; i < track.length && track[i]!.t < end; i++) out.push(track[i]!.hz);
+ return out;
+}
+
+export const semitones = (a: number, b: number) => 12 * Math.log2(a / b);
diff --git a/packages/cli/src/audio/read/read.test.ts b/packages/cli/src/audio/read/read.test.ts
new file mode 100644
index 0000000000..c6145f1803
--- /dev/null
+++ b/packages/cli/src/audio/read/read.test.ts
@@ -0,0 +1,257 @@
+import { createServer, type IncomingMessage, type Server } from "node:http";
+import type { AddressInfo } from "node:net";
+import { existsSync, mkdtempSync, readFileSync, rmSync, writeFileSync } from "node:fs";
+import { tmpdir } from "node:os";
+import { join } from "node:path";
+import { afterAll, afterEach, beforeAll, beforeEach, describe, expect, it, vi } from "vitest";
+import { findFfBinary } from "@hyperframes/parsers/ff-binaries";
+import { readAudio } from "./read.js";
+import { cachedStems, fetchStems } from "./stems.js";
+
+const RATE = 22_050;
+const dir = mkdtempSync(join(tmpdir(), "hf-audio-read-"));
+afterAll(() => rmSync(dir, { recursive: true, force: true }));
+
+function wav(name: string, seconds: number, fill: (t: number) => number): string {
+ const frames = Math.round(seconds * RATE);
+ const data = Buffer.alloc(frames * 2);
+ for (let i = 0; i < frames; i++)
+ data.writeInt16LE(Math.round(Math.max(-1, Math.min(1, fill(i / RATE))) * 32767), i * 2);
+ const header = Buffer.alloc(44);
+ header.write("RIFF", 0);
+ header.writeUInt32LE(36 + data.length, 4);
+ header.write("WAVEfmt ", 8);
+ header.writeUInt32LE(16, 16);
+ header.writeUInt16LE(1, 20);
+ header.writeUInt16LE(1, 22);
+ header.writeUInt32LE(RATE, 24);
+ header.writeUInt32LE(RATE * 2, 28);
+ header.writeUInt16LE(2, 32);
+ header.writeUInt16LE(16, 34);
+ header.write("data", 36);
+ header.writeUInt32LE(data.length, 40);
+ const path = join(dir, name);
+ writeFileSync(path, Buffer.concat([header, data]));
+ return path;
+}
+
+const voiced = (t: number) =>
+ (t >= 0.5 && t < 2) || (t >= 3 && t < 4.5) ? 0.3 * Math.sin(2 * Math.PI * 180 * t) : 0;
+const boom = (t: number) =>
+ t < 2.4 ? 0 : 0.7 * Math.exp(-(t - 2.4) / 0.3) * Math.sin(2 * Math.PI * 55 * (t - 2.4));
+const bed = (t: number) =>
+ t > 5 && t < 6
+ ? 0
+ : 0.08 * Math.sin(2 * Math.PI * 261.63 * t) + 0.08 * Math.sin(2 * Math.PI * 392 * t);
+
+let server: Server;
+let stalledPath: string | null = null;
+let onStalled = () => {};
+let stemCalls = 0;
+let stemRequest: unknown = null;
+const stemKeys: unknown[] = [];
+const stemFiles: Record<string, string> = {};
+
+async function body(req: IncomingMessage) {
+ const chunks: Uint8Array[] = [];
+ for await (const chunk of req) chunks.push(chunk);
+ return Buffer.concat(chunks);
+}
+
+const origin = () => `http://127.0.0.1:${(server.address() as AddressInfo).port}`;
+
+type Reply = (status: number, value: unknown) => void;
+
+const routes = new Map<string, (reply: Reply, req: IncomingMessage, bytes: Buffer) => void>(
+ Object.entries({
+ "POST /v3/assets/direct-uploads": (reply) =>
+ reply(200, {
+ data: { asset_id: "episode", upload_url: `${origin()}/put`, upload_headers: {} },
+ }),
+ "PUT /put": (reply) => reply(200, {}),
+ "POST /v3/assets/episode/complete": (reply) => reply(200, { data: { asset_id: "episode" } }),
+ "POST /v3/audio/stems": (reply, req, bytes) => {
+ expect(req.headers["x-api-key"]).toBe("test-key");
+ stemCalls++;
+ stemRequest = JSON.parse(bytes.toString());
+ stemKeys.push(req.headers["idempotency-key"]);
+ if (stemCalls === 1)
+ return reply(409, { error: { code: "request_in_progress", message: "Retry shortly." } });
+ reply(200, {
+ data: {
+ id: "job-1",
+ status: "completed",
+ stems: Object.keys(stemFiles).map((name) => ({
+ name,
+ url: `${origin()}/stem/${name}.wav`,
+ })),
+ },
+ });
+ },
+ }),
+);
+
+beforeAll(async () => {
+ server = createServer(async (req, res) => {
+ if (req.url === stalledPath) {
+ onStalled();
+ return;
+ }
+ const bytes = await body(req);
+ const reply: Reply = (status, value) =>
+ res.writeHead(status, { "content-type": "application/json" }).end(JSON.stringify(value));
+ const route = routes.get(`${req.method} ${req.url}`);
+ if (typeof route === "function") return route(reply, req, bytes);
+ const stem = req.url?.match(/^\/stem\/(\w+)\.wav$/)?.[1];
+ if (stem && stemFiles[stem]) return res.writeHead(200).end(readFileSync(stemFiles[stem]));
+ res.writeHead(404).end();
+ });
+ await new Promise<void>((done) => server.listen(0, "127.0.0.1", done));
+});
+
+afterAll(() => new Promise<void>((done) => server.close(() => done())));
+
+beforeEach(() => {
+ for (const name of ["HEYGEN_API_KEY", "HYPERFRAMES_API_KEY", "HEYGEN_ACCESS_TOKEN"])
+ vi.stubEnv(name, "");
+ vi.stubEnv("HEYGEN_API_URL", origin());
+});
+
+afterEach(() => {
+ stalledPath = null;
+ vi.restoreAllMocks();
+ vi.unstubAllEnvs();
+ delete process.env.HYPERFRAMES_AUDIO_STEMS_URL;
+ delete process.env.HYPERFRAMES_AUDIO_API_KEY;
+});
+
+const ffmpeg = findFfBinary("ffmpeg") && findFfBinary("ffprobe");
+
+describe.skipIf(!ffmpeg)("readAudio", () => {
+ it("reads the mix with its sidecar transcript when there are no stems, and reuses the cache", async () => {
+ const path = wav("talk.wav", 6, (t) => voiced(t));
+ writeFileSync(
+ `${path}.transcript.json`,
+ JSON.stringify({
+ audio: "talk.wav",
+ words: [
+ { text: "一口气", start: 0.5, end: 1.4 },
+ { text: "了解。", start: 1.4, end: 2 },
+ { text: "伊朗", start: 3, end: 4.5 },
+ ],
+ }),
+ );
+ const cacheDir = join(dir, "cache-mix");
+ const first = await readAudio(path, "talk.wav", { cacheDir, quality: "best" });
+ expect(first.cached).toBe(false);
+ expect(first.analysis).toMatchObject({ separation: "mix", stems: [], music: null });
+ expect(first.analysis.voice!.sentences.map((s) => s.text)).toEqual(["一口气了解。", "伊朗"]);
+ expect(first.analysis.voice!.medianPitchHz).toBeGreaterThan(170);
+ expect(first.analysis.warnings.join("\n")).toContain("no HeyGen credential");
+ const text = readFileSync(first.textPath, "utf8");
+ expect(text).toContain('voice "一口气了解。"');
+ expect(text).toContain("pause 1.0 s");
+ const again = await readAudio(path, "talk.wav", { cacheDir, quality: "best" });
+ expect(again.cached).toBe(true);
+ writeFileSync(
+ `${path}.transcript.json`,
+ JSON.stringify({ words: [{ text: "换了", start: 0.5, end: 2 }] }),
+ );
+ const edited = await readAudio(path, "talk.wav", { cacheDir, quality: "best" });
+ expect(edited.cached).toBe(false);
+ expect(edited.analysis.voice!.sentences[0]!.text).toBe("换了");
+ });
+
+ it("separates through HeyGen's stems route with the CLI's HeyGen API key, then reads effects from the drums and the bed from the rest", async () => {
+ stemFiles.vocals = wav("vocals.wav", 8, voiced);
+ stemFiles.drums = wav("drums.wav", 8, boom);
+ stemFiles.piano = wav("piano.wav", 8, bed);
+ stemFiles.bass = wav("bass.wav", 8, () => 0);
+ stemFiles.guitar = stemFiles.bass;
+ stemFiles.other = stemFiles.bass;
+ stemFiles["../../escape"] = stemFiles.bass;
+ const path = wav("episode.wav", 8, (t) => voiced(t) + boom(t) + bed(t));
+ vi.stubEnv("HEYGEN_API_KEY", "test-key");
+ const cacheDir = join(dir, "cache-stems");
+ const { analysis, textPath } = await readAudio(path, "episode.wav", {
+ cacheDir,
+ quality: "best",
+ });
+ expect(stemRequest).toEqual({ version: 1, audio: { type: "asset_id", asset_id: "episode" } });
+ expect(stemKeys).toHaveLength(2);
+ expect(stemKeys[1]).toBe(stemKeys[0]);
+ expect(analysis).toMatchObject({
+ separation: "stems",
+ stems: ["bass", "drums", "guitar", "other", "piano", "vocals"],
+ warnings: [expect.stringContaining("no transcript")],
+ });
+ expect(analysis.voice!.source).toBe("activity");
+ expect(analysis.effects!.events.map((e) => [Math.round(e.t * 10) / 10, e.kind])).toEqual([
+ [2.4, "boom"],
+ ]);
+ expect(analysis.effects!.events[0]!.overMusicDb).toBeGreaterThan(5);
+ expect(analysis.music!.breaks.map((b) => Math.round(b.start))).toEqual([5]);
+ expect(analysis.music!.segments.map((s) => s.entry)).toEqual(["start", "resumes"]);
+ expect(readFileSync(textPath, "utf8")).toContain("effect boom (S1): low");
+ expect(existsSync(join(cacheDir, "stems"))).toBe(true);
+ await readAudio(path, "episode.wav", { cacheDir, quality: "best" });
+ expect(stemCalls).toBe(2);
+ });
+
+ it("falls back on partial stems and retries instead of caching unmeasured layers as silence", async () => {
+ for (const name of Object.keys(stemFiles)) delete stemFiles[name];
+ stemFiles.vocals = wav("partial-vocals.wav", 2, voiced);
+ const path = wav("partial.wav", 2, voiced);
+ vi.stubEnv("HEYGEN_API_KEY", "test-key");
+ const cacheDir = join(dir, "cache-partial");
+ const partial = await readAudio(path, "partial.wav", { cacheDir, quality: "best" });
+ expect(partial.analysis).toMatchObject({ separation: "mix", effects: null, music: null });
+ expect(partial.analysis.warnings.join("\n")).toContain("incomplete separation: missing drums");
+ const callsBefore = stemCalls;
+ const silence = wav("partial-silence.wav", 2, () => 0);
+ for (const name of ["drums", "bass", "guitar", "piano", "other"]) stemFiles[name] = silence;
+ const complete = await readAudio(path, "partial.wav", { cacheDir, quality: "best" });
+ expect(complete.cached).toBe(false);
+ expect(complete.analysis.separation).toBe("stems");
+ expect(stemCalls).toBe(callsBefore + 1);
+
+ const stemDir = join(cacheDir, "stems", complete.analysis.sha256.slice(0, 12));
+ writeFileSync(join(stemDir, "stems.json"), JSON.stringify({ vocals: "vocals.audio" }));
+ expect(cachedStems(stemDir)).toBeNull();
+ const repaired = await readAudio(path, "partial.wav", { cacheDir, quality: "best" });
+ expect(repaired.cached).toBe(false);
+ expect(stemCalls).toBe(callsBefore + 2);
+ expect(Object.keys(cachedStems(stemDir)!)).toHaveLength(6);
+ });
+
+ it.each(["/put", "/v3/audio/stems"])(
+ "aborts a stalled %s request at the separation deadline",
+ async (endpoint) => {
+ const controller = new AbortController();
+ const timeout = vi.spyOn(AbortSignal, "timeout").mockReturnValue(controller.signal);
+ stalledPath = endpoint;
+ onStalled = () => controller.abort();
+ const path = wav("timeout.wav", 1, voiced);
+ await expect(
+ fetchStems(path, join(dir, `timeout-${endpoint.replaceAll("/", "-")}`), {
+ url: `${origin()}/v3/audio/stems`,
+ headers: { "x-api-key": "test-key" },
+ timeoutMs: 1000,
+ }),
+ ).rejects.toMatchObject({ name: "AbortError" });
+ expect(timeout).toHaveBeenCalledWith(1000);
+ },
+ );
+
+ it("keeps audio on the machine by default, and reads neither voice nor effects from a bare mix", async () => {
+ const path = wav("local.wav", 3, voiced);
+ vi.stubEnv("HEYGEN_API_KEY", "test-key");
+ const callsBefore = stemCalls;
+ const { analysis } = await readAudio(path, "local.wav", { cacheDir: join(dir, "cache-local") });
+ expect(analysis).toMatchObject({ separation: "mix", voice: null, effects: null });
+ expect(analysis.warnings[0]).toContain("quality local keeps audio on this machine");
+ expect(analysis.warnings[0]).toContain("--quality best");
+ expect(stemCalls).toBe(callsBefore);
+ expect(analysis.warnings[1]).toContain("no transcript and no stems");
+ });
+});
diff --git a/packages/cli/src/audio/read/read.ts b/packages/cli/src/audio/read/read.ts
new file mode 100644
index 0000000000..e8f9c0c03a
--- /dev/null
+++ b/packages/cli/src/audio/read/read.ts
@@ -0,0 +1,208 @@
+import { createHash } from "node:crypto";
+import {
+ createReadStream,
+ existsSync,
+ mkdirSync,
+ readFileSync,
+ renameSync,
+ writeFileSync,
+} from "node:fs";
+import { basename, extname, join } from "node:path";
+import type { AnalysisQuality } from "@hyperframes/core/audio-analysis";
+import { loadTranscript } from "../../whisper/normalize.js";
+import { findEffects } from "./effects.js";
+import { formatRead } from "./format.js";
+import { readMusic } from "./music.js";
+import { pitchTrack } from "./pitch.js";
+import { PITCH_RATE, READ_RATE, decodeMono, mixed, round } from "./signal.js";
+import { DRUM_STEM, VOICE_STEM, cachedStems, fetchStems, stemsAccess } from "./stems.js";
+import { READ_VERSION, type ReadAudio } from "./types.js";
+import { voiceFromActivity, voiceFromWords, type TimedWord } from "./voice.js";
+
+export interface ReadOptions {
+ /** Where analyses and stems are kept, normally the project's `.hyperframes/read-audio`. */
+ cacheDir: string;
+ /** A transcript of the recording; `<file>.transcript.json` next to it is used when absent. */
+ transcript?: string;
+ quality?: AnalysisQuality;
+}
+
+export interface ReadResult {
+ analysis: ReadAudio;
+ /** The whole recording as text, for grep. */
+ textPath: string;
+ jsonPath: string;
+ cached: boolean;
+}
+
+async function sha256(path: string): Promise<string> {
+ const hash = createHash("sha256");
+ for await (const chunk of createReadStream(path)) hash.update(chunk as Buffer);
+ return hash.digest("hex");
+}
+
+function transcriptFor(path: string, given?: string): string | null {
+ if (given) return given;
+ const sidecar = `${path}.transcript.json`;
+ return existsSync(sidecar) ? sidecar : null;
+}
+
+function wordsOf(transcript: string): TimedWord[] {
+ return loadTranscript(transcript).words.map(({ text, start, end }) => ({ text, start, end }));
+}
+
+const fingerprint = (path: string | null) =>
+ path ? createHash("sha256").update(readFileSync(path)).digest("hex").slice(0, 16) : "none";
+
+function writeAtomic(path: string, content: string) {
+ const temporary = `${path}.${process.pid}.tmp`;
+ writeFileSync(temporary, content);
+ renameSync(temporary, path);
+}
+
+interface CacheEntry {
+ analysis: ReadAudio;
+ transcriptKey: string;
+}
+
+async function stemsFor(
+ path: string,
+ dir: string,
+ quality: AnalysisQuality,
+ warnings: string[],
+): Promise<Record<string, string> | null> {
+ const cached = cachedStems(dir);
+ if (cached) return cached;
+ if (quality !== "best") {
+ warnings.push(
+ `not separated into stems: quality ${quality} keeps audio on this machine; --quality best separates it through HeyGen`,
+ );
+ return null;
+ }
+ const access = await stemsAccess();
+ if (typeof access === "string") {
+ warnings.push(`not separated into stems: ${access}`);
+ return null;
+ }
+ try {
+ return await fetchStems(path, dir, access);
+ } catch (error) {
+ warnings.push(`not separated into stems: ${error instanceof Error ? error.message : error}`);
+ return null;
+ }
+}
+
+/** Effects from the drums and the music from the other non-voice stems, measured against the whole recording. */
+async function fromStems(path: string, stemPaths: Record<string, string>) {
+ const pitched: Record<string, Float32Array> = {};
+ for (const [name, stemPath] of Object.entries(stemPaths))
+ if (name !== VOICE_STEM && name !== DRUM_STEM) pitched[name] = await decodeMono(stemPath);
+ const drums = stemPaths[DRUM_STEM] ? await decodeMono(stemPaths[DRUM_STEM]) : null;
+ const bed = Object.keys(pitched).length ? mixed(Object.values(pitched)) : null;
+ const { music, beats } = bed
+ ? readMusic({ stems: pitched, drums, sampleRate: READ_RATE })
+ : { music: null, beats: [] };
+ const effects = findEffects({
+ source: drums ?? bed ?? new Float32Array(0),
+ sampleRate: READ_RATE,
+ music: bed,
+ heard: await decodeMono(path),
+ beats,
+ speech: [],
+ });
+ return { effects, music };
+}
+
+const transcribeHint = (file: string) => `run hyperframes transcribe ${file} and pass --transcript`;
+
+async function analyze(
+ path: string,
+ file: string,
+ sha: string,
+ transcript: string | null,
+ stemDir: string,
+ quality: AnalysisQuality,
+): Promise<ReadAudio> {
+ const warnings: string[] = [];
+ const stemPaths = await stemsFor(path, stemDir, quality, warnings);
+ const words = transcript ? wordsOf(transcript) : [];
+ const voicePath = stemPaths?.[VOICE_STEM] ?? path;
+ const voiceSamples = await decodeMono(voicePath);
+ const about = base(file, sha, round(voiceSamples.length / READ_RATE, 3), stemPaths, transcript);
+ if (!stemPaths && !words.length) {
+ warnings.push(
+ `no transcript and no stems: voice and effects are not read, because the mix alone does not tell speech from music; ${transcribeHint(file)}`,
+ );
+ return { ...about, voice: null, effects: null, music: null, warnings };
+ }
+ if (!words.length)
+ warnings.push(`no transcript: voice lines carry no words; ${transcribeHint(file)}`);
+ const pitch = pitchTrack(await decodeMono(voicePath, PITCH_RATE));
+ const voice = words.length
+ ? voiceFromWords(words, pitch)
+ : voiceFromActivity(voiceSamples, READ_RATE, pitch);
+ if (stemPaths) return { ...about, voice, ...(await fromStems(path, stemPaths)), warnings };
+ warnings.push("without stems, effects under speech are not found and music is not read");
+ const effects = findEffects({
+ source: voiceSamples,
+ sampleRate: READ_RATE,
+ music: null,
+ heard: null,
+ beats: [],
+ speech: voice.sentences,
+ });
+ return { ...about, voice, effects, music: null, warnings };
+}
+
+function base(
+ file: string,
+ sha: string,
+ duration: number,
+ stemPaths: Record<string, string> | null,
+ transcript: string | null,
+): Omit<ReadAudio, "voice" | "effects" | "music" | "warnings"> {
+ return {
+ version: READ_VERSION,
+ file,
+ sha256: sha,
+ duration,
+ separation: stemPaths ? "stems" : "mix",
+ stems: stemPaths ? Object.keys(stemPaths).sort() : [],
+ transcript: transcript ? basename(transcript) : null,
+ };
+}
+
+/** Read a recording as time-stamped voice, effect and music events, computed once per file content. */
+export async function readAudio(
+ path: string,
+ file: string,
+ options: ReadOptions,
+): Promise<ReadResult> {
+ const quality = options.quality ?? "local";
+ const sha = await sha256(path);
+ const transcript = transcriptFor(path, options.transcript);
+ const transcriptKey = fingerprint(transcript);
+ const stem = `${basename(file, extname(file)).replace(/[^\w.-]/g, "_")}-${sha.slice(0, 12)}`;
+ mkdirSync(options.cacheDir, { recursive: true });
+ const jsonPath = join(options.cacheDir, `${stem}.json`);
+ const textPath = join(options.cacheDir, `${stem}.txt`);
+ const stemDir = join(options.cacheDir, "stems", sha.slice(0, 12));
+ if (existsSync(jsonPath) && existsSync(textPath)) {
+ const entry = JSON.parse(readFileSync(jsonPath, "utf8")) as CacheEntry;
+ const stemsNowPossible =
+ entry.analysis.separation === "mix" &&
+ quality === "best" &&
+ (cachedStems(stemDir) !== null || typeof (await stemsAccess()) !== "string");
+ if (
+ (entry.analysis.separation !== "stems" || cachedStems(stemDir) !== null) &&
+ entry.analysis.version === READ_VERSION &&
+ entry.transcriptKey === transcriptKey &&
+ !stemsNowPossible
+ )
+ return { analysis: entry.analysis, textPath, jsonPath, cached: true };
+ }
+ const analysis = await analyze(path, file, sha, transcript, stemDir, quality);
+ writeAtomic(jsonPath, JSON.stringify({ analysis, transcriptKey } satisfies CacheEntry));
+ writeAtomic(textPath, formatRead(analysis));
+ return { analysis, textPath, jsonPath, cached: false };
+}
diff --git a/packages/cli/src/audio/read/signal.test.ts b/packages/cli/src/audio/read/signal.test.ts
new file mode 100644
index 0000000000..cc88231670
--- /dev/null
+++ b/packages/cli/src/audio/read/signal.test.ts
@@ -0,0 +1,13 @@
+import { describe, expect, it } from "vitest";
+import { levels, READ_RATE } from "./signal.js";
+
+describe("levels", () => {
+ it("keeps fractional sample hops aligned late in a recording", () => {
+ const samples = new Float32Array(601 * READ_RATE);
+ samples.fill(0.5, 600 * READ_RATE, Math.round(600.1 * READ_RATE));
+ const db = levels(samples, READ_RATE, 0.01);
+ expect(db).toHaveLength(60100);
+ expect(db[60005]).toBeCloseTo(20 * Math.log10(0.5), 3);
+ expect(db[59990]).toBe(-120);
+ });
+});
diff --git a/packages/cli/src/audio/read/signal.ts b/packages/cli/src/audio/read/signal.ts
new file mode 100644
index 0000000000..cb776fed98
--- /dev/null
+++ b/packages/cli/src/audio/read/signal.ts
@@ -0,0 +1,61 @@
+import type { AudioSignal } from "@hyperframes/core/audio-analysis";
+import { decodeAudio } from "../analysis/decode.js";
+
+/** Every layer is read from mono at this rate: enough for onsets and band shapes, small enough for a long episode. */
+export const READ_RATE = 22050;
+export const PITCH_RATE = 8000;
+const SILENT_DB = -120;
+
+export async function decodeMono(path: string, sampleRate = READ_RATE): Promise<Float32Array> {
+ const decoded = await decodeAudio(path, {}, undefined, { sampleRate, mono: true });
+ if (!decoded) throw new Error(`${path} has no audio stream`);
+ return decoded.signal.channels[0]!;
+}
+
+export const asSignal = (samples: Float32Array, sampleRate = READ_RATE): AudioSignal => ({
+ sampleRate,
+ channels: [samples],
+});
+
+export function mixed(tracks: readonly Float32Array[]): Float32Array {
+ const length = Math.max(0, ...tracks.map((t) => t.length));
+ const sum = new Float32Array(length);
+ for (const track of tracks) for (let i = 0; i < track.length; i++) sum[i]! += track[i]!;
+ return sum;
+}
+
+/** RMS level in dBFS of each `hop`-second frame, measured over a `window` centred on it. */
+export function levels(
+ samples: Float32Array,
+ sampleRate: number,
+ hop: number,
+ window = hop * 2,
+): Float32Array {
+ const hopSamples = Math.max(1, hop * sampleRate);
+ const half = Math.max(1, Math.round((window * sampleRate) / 2));
+ const squares = new Float64Array(samples.length + 1);
+ for (let i = 0; i < samples.length; i++) squares[i + 1] = squares[i]! + samples[i]! ** 2;
+ const frames = Math.ceil(samples.length / hopSamples);
+ const out = new Float32Array(frames);
+ for (let f = 0; f < frames; f++) {
+ const center = Math.round(f * hopSamples);
+ const a = Math.max(0, center - half);
+ const b = Math.min(samples.length, center + half);
+ const power = b > a ? (squares[b]! - squares[a]!) / (b - a) : 0;
+ out[f] = power > 0 ? Math.max(SILENT_DB, 10 * Math.log10(power)) : SILENT_DB;
+ }
+ return out;
+}
+
+export function quantile(values: ArrayLike<number>, q: number): number {
+ if (!values.length) return NaN;
+ const sorted = Float64Array.from(values).sort();
+ return sorted[Math.min(sorted.length - 1, Math.max(0, Math.floor(q * (sorted.length - 1))))]!;
+}
+
+export const median = (values: ArrayLike<number>) => quantile(values, 0.5);
+
+export const round = (n: number, places = 2) => {
+ const k = 10 ** places;
+ return Math.round(n * k) / k;
+};
diff --git a/packages/cli/src/audio/read/stems.ts b/packages/cli/src/audio/read/stems.ts
new file mode 100644
index 0000000000..1a33ac4d81
--- /dev/null
+++ b/packages/cli/src/audio/read/stems.ts
@@ -0,0 +1,176 @@
+import { randomUUID } from "node:crypto";
+import { existsSync, mkdirSync, readFileSync, statSync, writeFileSync } from "node:fs";
+import { mkdtemp, rm } from "node:fs/promises";
+import { tmpdir } from "node:os";
+import { join } from "node:path";
+import { findFfBinary } from "@hyperframes/parsers/ff-binaries";
+import { setTimeout as sleep } from "node:timers/promises";
+import { downloadToFile } from "../../cloud/download.js";
+import { HyperframesCloudClient } from "../../cloud/_gen/client.js";
+import { uploadAssetViaDirectUpload } from "../../cloud/upload.js";
+import {
+ runFile,
+ serviceAccess,
+ serviceFailure,
+ type ServiceAccess,
+} from "../analysis/providers/shared.js";
+
+/**
+ * Upload 192 kb/s stereo MP3 through the service origin's direct-upload routes, then POST
+ * `{ version: 1, audio: { type: "asset_id", asset_id } }` with one Idempotency-Key.
+ * A completed response wraps StemsJob in `data`. Retry 409 `request_in_progress` or a dropped
+ * connection with the same key and body; other HTTP errors carry `{ error: { code, message } }`.
+ * A complete separation contains vocals, drums, bass, guitar, piano and other.
+ */
+interface StemsJob {
+ id: string;
+ status: "pending" | "completed" | "failed";
+ stems?: { name: string; url: string }[];
+ failure_message?: string;
+}
+
+const URL_ENV = "HYPERFRAMES_AUDIO_STEMS_URL";
+const KEY_ENV = "HYPERFRAMES_AUDIO_API_KEY";
+const TIMEOUT_ENV = "HYPERFRAMES_AUDIO_STEMS_TIMEOUT_MS";
+const DEFAULT_TIMEOUT_MS = 30 * 60_000;
+const RETRY_MS = 5000;
+const DONE = "stems.json";
+const STEM_NAMES = ["vocals", "drums", "bass", "guitar", "piano", "other"] as const;
+
+export const VOICE_STEM = "vocals";
+export const DRUM_STEM = "drums";
+
+/** How to reach the stems service, or why it cannot be reached. */
+export const stemsAccess = () =>
+ serviceAccess(
+ { url: URL_ENV, key: KEY_ENV, timeout: TIMEOUT_ENV },
+ "/v3/audio/stems",
+ DEFAULT_TIMEOUT_MS,
+ );
+
+/** Stems already fetched into `dir`, by name; null when there are none yet. */
+export function cachedStems(dir: string): Record<string, string> | null {
+ const done = join(dir, DONE);
+ if (!existsSync(done)) return null;
+ const names = JSON.parse(readFileSync(done, "utf8")) as Record<string, string>;
+ if (!STEM_NAMES.every((name) => typeof names[name] === "string")) return null;
+ const paths = Object.fromEntries(STEM_NAMES.map((name) => [name, join(dir, names[name]!)]));
+ return Object.values(paths).every((p) => existsSync(p)) ? paths : null;
+}
+
+class StillRunning extends Error {}
+
+const DROPPED = new Set(["UND_ERR_HEADERS_TIMEOUT", "UND_ERR_SOCKET"]);
+
+const dropped = (error: unknown) =>
+ error instanceof TypeError &&
+ DROPPED.has((error.cause as { code?: string } | undefined)?.code ?? "");
+
+async function stillRunning(response: Response): Promise<boolean> {
+ if (response.status !== 409) return false;
+ const body = (await response
+ .clone()
+ .json()
+ .catch(() => null)) as { error?: { code?: string } } | null;
+ return body?.error?.code === "request_in_progress";
+}
+
+async function job(response: Response): Promise<StemsJob> {
+ if (await stillRunning(response)) throw new StillRunning();
+ if (!response.ok) throw await serviceFailure(response, "the stems service");
+ const body = (await response.json()) as { data?: StemsJob } | StemsJob | null;
+ const value = (body && "data" in body ? body.data : body) as StemsJob | undefined;
+ if (!value?.id || !value.status) throw new Error("the stems service answered without a job");
+ return value;
+}
+
+async function encoded(path: string, dir: string, signal: AbortSignal): Promise<string> {
+ const ffmpeg = findFfBinary("ffmpeg", { configuredMustExist: true });
+ if (!ffmpeg) throw new Error("ffmpeg is required to separate stems; run: npx hyperframes doctor");
+ const out = join(dir, "upload.mp3");
+ await runFile(
+ ffmpeg,
+ ["-v", "error", "-y", "-i", path, "-vn", "-ac", "2", "-ar", "44100", "-b:a", "192k", out],
+ { windowsHide: true, maxBuffer: 8 * 1024 * 1024, signal },
+ );
+ return out;
+}
+
+async function separated(
+ url: string,
+ request: RequestInit,
+ signal: AbortSignal,
+): Promise<StemsJob> {
+ for (;;) {
+ try {
+ return await job(await fetch(url, { ...request, signal }));
+ } catch (error) {
+ if (!(error instanceof StillRunning || dropped(error))) throw error;
+ await sleep(RETRY_MS, undefined, { signal });
+ }
+ }
+}
+
+/** Separate the recording through the configured service and save each stem into `dir`. */
+export async function fetchStems(
+ path: string,
+ dir: string,
+ { url, headers, timeoutMs }: ServiceAccess,
+): Promise<Record<string, string>> {
+ const signal = AbortSignal.timeout(timeoutMs);
+ const scratch = await mkdtemp(join(tmpdir(), "hf-stems-"));
+ try {
+ const mp3 = await encoded(path, scratch, signal);
+ const client = new HyperframesCloudClient({
+ baseUrl: new URL(url).origin,
+ getAuthHeaders: () => headers,
+ fetchImpl: (input, init) => fetch(input, { ...init, redirect: "error", signal }),
+ });
+ const asset = await uploadAssetViaDirectUpload({
+ client,
+ path: mp3,
+ size: statSync(mp3).size,
+ filename: "audio.mp3",
+ contentType: "audio/mpeg",
+ idempotencyKey: randomUUID(),
+ signal,
+ });
+ const current = await separated(
+ url,
+ {
+ method: "POST",
+ redirect: "error",
+ headers: {
+ ...headers,
+ "idempotency-key": randomUUID(),
+ "content-type": "application/json",
+ },
+ body: JSON.stringify({ version: 1, audio: { type: "asset_id", asset_id: asset.asset_id } }),
+ },
+ signal,
+ );
+ if (current.status !== "completed")
+ throw new Error(
+ `stem separation failed${current.failure_message ? `: ${current.failure_message}` : ""}`,
+ );
+ const stems = current.stems ?? [];
+ const missing = STEM_NAMES.filter((name) => !stems.some((stem) => stem.name === name));
+ if (missing.length)
+ throw new Error(
+ `the stems service returned an incomplete separation: missing ${missing.join(", ")}`,
+ );
+ mkdirSync(dir, { recursive: true });
+ const names: Record<string, string> = {};
+ for (const name of STEM_NAMES) {
+ const stem = stems.find((candidate) => candidate.name === name);
+ if (!stem) continue;
+ const file = `${name}.audio`;
+ await downloadToFile(stem.url, join(dir, file), { signal });
+ names[name] = file;
+ }
+ writeFileSync(join(dir, DONE), JSON.stringify(names));
+ return cachedStems(dir)!;
+ } finally {
+ await rm(scratch, { recursive: true, force: true });
+ }
+}
diff --git a/packages/cli/src/audio/read/types.ts b/packages/cli/src/audio/read/types.ts
new file mode 100644
index 0000000000..ad8c877019
--- /dev/null
+++ b/packages/cli/src/audio/read/types.ts
@@ -0,0 +1,104 @@
+import type { Span } from "@hyperframes/core/audio-analysis";
+
+export const READ_VERSION = 2;
+
+export type Separation = "stems" | "mix";
+
+export type RateUnit = "chars" | "words";
+
+export type Ending = "rising" | "falling" | "level";
+
+export interface Sentence {
+ start: number;
+ end: number;
+ /** Null when there is no transcript and the span comes from where the voice is active. */
+ text: string | null;
+ rate: number | null;
+ pitchHz: number | null;
+ ending: Ending | null;
+}
+
+export interface Voice {
+ source: "transcript" | "activity";
+ unit: RateUnit;
+ sentences: Sentence[];
+ pauses: Span[];
+ meanRate: number | null;
+ medianPitchHz: number | null;
+}
+
+export type EffectKind = "riser" | "boom" | "thump" | "whoosh" | "click" | "tick" | "ding" | "hit";
+
+export type Band = "low" | "mid" | "high";
+
+export interface Effect {
+ t: number;
+ end: number;
+ kind: EffectKind;
+ group: string;
+ band: Band;
+ attackMs: number;
+ tail: number;
+ /** How far the effect rises out of what was playing just before it. */
+ riseDb: number;
+ /** The effect's peak against the music at that moment; null without a music stem or while the music is out. */
+ overMusicDb: number | null;
+ /** More than one when a quick run of the same sound is reported as one train. */
+ count: number;
+}
+
+export interface EffectGroup {
+ id: string;
+ kind: EffectKind;
+ count: number;
+}
+
+export interface Effects {
+ events: Effect[];
+ groups: EffectGroup[];
+ /** Hits that repeat on the music's beat, which are the music's own drums and not listed as effects. */
+ drumHits: number;
+}
+
+export type Entry = "start" | "resumes" | "returns" | "new";
+
+export interface MusicSegment {
+ start: number;
+ end: number;
+ piece: number;
+ entry: Entry;
+ bpm: number | null;
+ /** Where a fade out begins, when the segment ends on one. */
+ fadeFrom: number | null;
+}
+
+export interface Piece {
+ id: number;
+ /** The stems that carry it, loudest first. */
+ instruments: string[];
+ bpm: number | null;
+ firstAt: number;
+}
+
+export interface Music {
+ segments: MusicSegment[];
+ pieces: Piece[];
+ breaks: Span[];
+ /** Level of the music every 30 s, in dBFS. */
+ energy: { start: number; end: number; db: number }[];
+}
+
+export interface ReadAudio {
+ version: typeof READ_VERSION;
+ file: string;
+ sha256: string;
+ duration: number;
+ separation: Separation;
+ stems: string[];
+ transcript: string | null;
+ /** Null when the mix alone cannot tell speech from music: no stems and no transcript. */
+ voice: Voice | null;
+ effects: Effects | null;
+ music: Music | null;
+ warnings: string[];
+}
diff --git a/packages/cli/src/audio/read/voice.test.ts b/packages/cli/src/audio/read/voice.test.ts
new file mode 100644
index 0000000000..901bca7637
--- /dev/null
+++ b/packages/cli/src/audio/read/voice.test.ts
@@ -0,0 +1,71 @@
+import { describe, expect, it } from "vitest";
+import { voiceFromActivity, voiceFromWords } from "./voice.js";
+import { READ_RATE } from "./signal.js";
+
+const glide = (start: number, end: number, from: number, to: number, step = 0.04) => {
+ const points = [];
+ for (let t = start; t < end; t += step)
+ points.push({ t, hz: from + ((to - from) * (t - start)) / (end - start) });
+ return points;
+};
+
+describe("voiceFromWords", () => {
+ const words = [
+ { text: "伊", start: 1.0, end: 1.2 },
+ { text: "朗", start: 1.2, end: 1.4 },
+ { text: "经", start: 1.4, end: 1.6 },
+ { text: "济。", start: 1.6, end: 2.0 },
+ { text: "涨", start: 3.0, end: 3.25 },
+ { text: "了", start: 3.25, end: 3.5 },
+ ];
+
+ it("splits sentences at end punctuation, counts Han characters per second and lists pauses", () => {
+ const voice = voiceFromWords(words, []);
+ expect(voice.unit).toBe("chars");
+ expect(voice.sentences.map((s) => [s.text, s.start, s.end, s.rate])).toEqual([
+ ["伊朗经济。", 1, 2, 4],
+ ["涨了", 3, 3.5, 4],
+ ]);
+ expect(voice.pauses).toEqual([{ start: 2, end: 3 }]);
+ expect(voice.meanRate).toBe(4);
+ });
+
+ it("says a sentence ends rising when its last third lifts two semitones or more", () => {
+ const track = [...glide(1, 1.7, 180, 180), ...glide(1.7, 2, 180, 240)];
+ const voice = voiceFromWords(words, track);
+ expect(voice.sentences[0]).toMatchObject({ ending: "rising", pitchHz: 180 });
+ expect(voice.sentences[1]!.ending).toBeNull();
+ });
+
+ it("counts words, not letters, in a Latin transcript", () => {
+ const voice = voiceFromWords(
+ [
+ { text: "Prices", start: 0, end: 0.5 },
+ { text: "rose", start: 0.5, end: 0.8 },
+ { text: "fast.", start: 0.8, end: 1 },
+ ],
+ [],
+ );
+ expect(voice.unit).toBe("words");
+ expect(voice.sentences[0]).toMatchObject({ text: "Prices rose fast.", rate: 3 });
+ });
+});
+
+describe("voiceFromActivity", () => {
+ it("finds the voiced spans and the pause between them without a transcript", () => {
+ const samples = new Float32Array(4 * READ_RATE);
+ for (let i = 0; i < samples.length; i++) {
+ const t = i / READ_RATE;
+ if ((t >= 0.5 && t < 1.5) || (t >= 2.5 && t < 3.5))
+ samples[i] = 0.3 * Math.sin(2 * Math.PI * 200 * t);
+ }
+ const voice = voiceFromActivity(samples, READ_RATE, []);
+ expect(voice.source).toBe("activity");
+ expect(voice.sentences.map((s) => [s.text, Math.round(s.start * 10) / 10])).toEqual([
+ [null, 0.5],
+ [null, 2.5],
+ ]);
+ expect(voice.pauses).toHaveLength(1);
+ expect(voice.pauses[0]!.end - voice.pauses[0]!.start).toBeCloseTo(1, 0);
+ });
+});
diff --git a/packages/cli/src/audio/read/voice.ts b/packages/cli/src/audio/read/voice.ts
new file mode 100644
index 0000000000..ec99528925
--- /dev/null
+++ b/packages/cli/src/audio/read/voice.ts
@@ -0,0 +1,162 @@
+import type { Span } from "@hyperframes/core/audio-analysis";
+import { median, round, levels, quantile } from "./signal.js";
+import { semitones, within, type PitchPoint } from "./pitch.js";
+import type { Ending, RateUnit, Sentence, Voice } from "./types.js";
+
+export interface TimedWord {
+ text: string;
+ start: number;
+ end: number;
+}
+
+const SENTENCE_END = /[。!?!?…;;.]["'”’」』)]*$/;
+const SENTENCE_GAP = 0.6;
+const LONGEST_SENTENCE = 60;
+const PAUSE_SECONDS = 0.4;
+const CJK = /[\p{Script=Han}\p{Script=Hiragana}\p{Script=Katakana}\p{Script=Hangul}]/gu;
+const LETTERS = /\p{L}/u;
+const ENDING_SEMITONES = 2;
+const MIN_PITCH_FRAMES = 5;
+
+function rateUnit(words: readonly TimedWord[]): RateUnit {
+ const text = words.map((w) => w.text).join("");
+ return (text.match(CJK)?.length ?? 0) * 2 >= text.replace(/\s/g, "").length ? "chars" : "words";
+}
+
+function spoken(words: readonly TimedWord[], unit: RateUnit): number {
+ if (unit === "chars") return words.reduce((n, w) => n + (w.text.match(CJK)?.length ?? 0), 0);
+ return words.filter((w) => LETTERS.test(w.text)).length;
+}
+
+function joined(words: readonly TimedWord[], unit: RateUnit): string {
+ return unit === "chars"
+ ? words.map((w) => w.text.trim()).join("")
+ : words
+ .map((w) => w.text.trim())
+ .join(" ")
+ .replace(/\s+([,.!?;:])/g, "$1");
+}
+
+function sentenceGroups(words: readonly TimedWord[]): TimedWord[][] {
+ const groups: TimedWord[][] = [];
+ let current: TimedWord[] = [];
+ let chars = 0;
+ words.forEach((word, i) => {
+ current.push(word);
+ chars += word.text.length;
+ const next = words[i + 1];
+ if (
+ !next ||
+ SENTENCE_END.test(word.text.trim()) ||
+ next.start - word.end >= SENTENCE_GAP ||
+ chars >= LONGEST_SENTENCE
+ ) {
+ groups.push(current);
+ current = [];
+ chars = 0;
+ }
+ });
+ return groups;
+}
+
+function ending(track: readonly PitchPoint[], start: number, end: number): Ending | null {
+ const split = start + (end - start) * 0.7;
+ const body = within(track, start, split);
+ const tail = within(track, split, end);
+ if (body.length < MIN_PITCH_FRAMES || tail.length < 2) return null;
+ const shift = semitones(median(tail), median(body));
+ if (shift >= ENDING_SEMITONES) return "rising";
+ if (shift <= -ENDING_SEMITONES) return "falling";
+ return "level";
+}
+
+function pitchOf(track: readonly PitchPoint[], start: number, end: number): number | null {
+ const hz = within(track, start, end);
+ return hz.length >= MIN_PITCH_FRAMES ? Math.round(median(hz)) : null;
+}
+
+function summary(sentences: Sentence[], pauses: Span[], track: readonly PitchPoint[]) {
+ const timed = sentences.filter((s) => s.rate !== null);
+ const seconds = timed.reduce((n, s) => n + (s.end - s.start), 0);
+ const meanRate = seconds
+ ? round(timed.reduce((n, s) => n + s.rate! * (s.end - s.start), 0) / seconds, 1)
+ : null;
+ return {
+ sentences,
+ pauses,
+ meanRate,
+ medianPitchHz: track.length ? Math.round(median(track.map((p) => p.hz))) : null,
+ };
+}
+
+export function voiceFromWords(words: readonly TimedWord[], track: readonly PitchPoint[]): Voice {
+ const ordered = words.filter((w) => w.end > w.start).sort((a, b) => a.start - b.start);
+ const unit = rateUnit(ordered);
+ const sentences = sentenceGroups(ordered).map((group): Sentence => {
+ const start = group[0]!.start;
+ const end = group[group.length - 1]!.end;
+ const count = spoken(group, unit);
+ return {
+ start: round(start),
+ end: round(end),
+ text: joined(group, unit),
+ rate: count && end > start ? round(count / (end - start), 1) : null,
+ pitchHz: pitchOf(track, start, end),
+ ending: ending(track, start, end),
+ };
+ });
+ const pauses: Span[] = [];
+ for (let i = 1; i < ordered.length; i++) {
+ const gap = ordered[i]!.start - ordered[i - 1]!.end;
+ if (gap >= PAUSE_SECONDS)
+ pauses.push({ start: round(ordered[i - 1]!.end), end: round(ordered[i]!.start) });
+ }
+ return { source: "transcript", unit, ...summary(sentences, pauses, track) };
+}
+
+const ACTIVITY_HOP = 0.05;
+const SHORTEST_SPEECH = 0.3;
+
+/** Where the voice is active, for a recording without a transcript: spans between pauses, without words. */
+export function voiceFromActivity(
+ vocals: Float32Array,
+ sampleRate: number,
+ track: readonly PitchPoint[],
+): Voice {
+ const db = levels(vocals, sampleRate, ACTIVITY_HOP);
+ const threshold = Math.max(-45, quantile(db, 0.9) - 30);
+ const spans: Span[] = [];
+ let start: number | null = null;
+ let quietSince: number | null = null;
+ db.forEach((level, f) => {
+ const t = f * ACTIVITY_HOP;
+ if (level >= threshold) {
+ if (start === null) start = t;
+ quietSince = null;
+ } else if (start !== null) {
+ quietSince ??= t;
+ if (t - quietSince >= PAUSE_SECONDS) {
+ if (quietSince - start >= SHORTEST_SPEECH) spans.push({ start, end: quietSince });
+ start = null;
+ quietSince = null;
+ }
+ }
+ });
+ const end = db.length * ACTIVITY_HOP;
+ if (start !== null && (quietSince ?? end) - start >= SHORTEST_SPEECH)
+ spans.push({ start, end: quietSince ?? end });
+ const sentences = spans.map(
+ (s): Sentence => ({
+ start: round(s.start),
+ end: round(s.end),
+ text: null,
+ rate: null,
+ pitchHz: pitchOf(track, s.start, s.end),
+ ending: ending(track, s.start, s.end),
+ }),
+ );
+ const pauses = spans
+ .slice(1)
+ .map((s, i) => ({ start: round(spans[i]!.end), end: round(s.start) }));
+ return { source: "activity", unit: "chars", ...summary(sentences, pauses, track) };
+}
diff --git a/packages/cli/src/cloud/upload.ts b/packages/cli/src/cloud/upload.ts
index 2598ab5305..419a911299 100644
--- a/packages/cli/src/cloud/upload.ts
+++ b/packages/cli/src/cloud/upload.ts
@@ -56,6 +56,7 @@ export interface UploadZipViaDirectOptions {
idempotencyKey?: string;
fetchImpl?: typeof fetch;
onProgress?: (event: UploadProgressEvent) => void;
+ signal?: AbortSignal;
}
// Per api-docs: completeAssetUpload can return 409 "Uploaded object not found
@@ -103,17 +104,19 @@ async function putBytesToPresignedUrl(
uploadHeaders: Record<string, unknown>,
body: Uint8Array | UploadFileSource,
contentType: string,
+ signal?: AbortSignal,
): Promise<void> {
const headers: Record<string, string> = {
"content-type": contentType,
...normalizeUploadHeaders(uploadHeaders),
};
- if (!(body instanceof Uint8Array)) return putFileToPresignedUrl(uploadUrl, headers, body);
+ if (!(body instanceof Uint8Array)) return putFileToPresignedUrl(uploadUrl, headers, body, signal);
// `Uint8Array<ArrayBufferLike>` is a valid `BodyInit` at runtime but
// not strictly assignable per lib.dom.d.ts — cast rather than copy,
// since a 200MB buffer copy would be wasteful.
const res = await fetchImpl(uploadUrl, {
method: "PUT",
+ signal,
headers,
body: body as unknown as BodyInit,
});
@@ -132,10 +135,12 @@ async function putFileToPresignedUrl(
uploadUrl: string,
headers: Record<string, string>,
file: UploadFileSource,
+ signal?: AbortSignal,
): Promise<void> {
const send = new URL(uploadUrl).protocol === "https:" ? httpsRequest : httpRequest;
const req = send(uploadUrl, {
method: "PUT",
+ signal,
headers: { ...headers, "content-length": String(file.size) },
});
const response = new Promise<IncomingMessage>((resolve, reject) => {
@@ -221,6 +226,7 @@ export async function uploadAssetViaDirectUpload(
initialize.upload_headers,
body,
opts.contentType,
+ opts.signal,
);
opts.onProgress?.({ phase: "upload", percent: 100 });
diff --git a/packages/cli/src/commands/audio.ts b/packages/cli/src/commands/audio.ts
index 85cc7900a5..609e72305b 100644
--- a/packages/cli/src/commands/audio.ts
+++ b/packages/cli/src/commands/audio.ts
@@ -1,4 +1,4 @@
-import { resolve } from "node:path";
+import { join, resolve } from "node:path";
import { defineCommand } from "citty";
import type {
AnalysisQuality,
@@ -10,6 +10,8 @@ import { DEFAULT_CAPABILITIES, type AnalyzeOptions } from "../audio/analysis/ana
import { formatReport } from "../audio/analysis/format.js";
import { analyzeTargets, previewTarget } from "../audio/analysis/run.js";
import { filmTargets, type AudioTarget, type FilmQuery } from "../audio/analysis/targets.js";
+import { formatRead } from "../audio/read/format.js";
+import { readAudio } from "../audio/read/read.js";
import { describeProject } from "../timeline/describeProject.js";
import { CliUsageError } from "../utils/commandResult.js";
import { ensureDOMParser } from "../utils/dom.js";
@@ -29,6 +31,10 @@ export const examples: Example[] = [
"The beat grid, tempo and sections of a music bed",
"hyperframes audio music.mp3 --only beats,downbeats,bpm,sections",
],
+ [
+ "A whole recording as voice, effect and music lines, 60–70 s shown",
+ "hyperframes audio refs/episode.mp4 --read --range 60,70",
+ ],
[
"Only onsets, from 60–200 Hz, finer",
"hyperframes audio drums.wav --only onsets --band 60,200 --hop-ms 2",
@@ -114,11 +120,19 @@ interface AudioFlags {
"onset-ratio"?: string;
"tail-db"?: string;
"silence-db"?: string;
+ read?: boolean;
+ transcript?: string;
+ json?: boolean;
+}
+
+function parseQuality(flags: AudioFlags): AnalysisQuality {
+ const quality = (flags.quality ?? "local") as AnalysisQuality;
+ if (!QUALITIES.has(quality)) throw new CliUsageError("--quality is local, best or fast");
+ return quality;
}
function analyzeRequest(flags: AudioFlags): AnalyzeOptions {
- const quality = (flags.quality ?? "best") as AnalysisQuality;
- if (!QUALITIES.has(quality)) throw new CliUsageError("--quality is best, local or fast");
+ const quality = parseQuality(flags);
const method = flags["onset-method"] as OnsetMethod | undefined;
if (method && method !== "spectral-flux" && method !== "energy")
throw new CliUsageError("--onset-method is spectral-flux or energy");
@@ -160,6 +174,28 @@ async function clipTargets(flags: AudioFlags, query: FilmQuery | null): Promise<
return filmTargets(project.dir, timeline, query ?? { by: "all" });
}
+function readQuality(files: string[], flags: AudioFlags): AnalysisQuality {
+ if (files.length !== 1) throw new CliUsageError("--read reads exactly one file");
+ if (flags.element || flags.at || flags["place-at"])
+ throw new CliUsageError("--read takes a file and --range, not a film query");
+ return parseQuality(flags);
+}
+
+async function read(files: string[], flags: AudioFlags) {
+ const quality = readQuality(files, flags);
+ const range = pair("range", flags.range);
+ const result = await readAudio(resolve(process.cwd(), files[0]!), files[0]!, {
+ cacheDir: join(resolve(flags.dir ?? process.cwd()), ".hyperframes", "read-audio"),
+ transcript: flags.transcript ? resolve(process.cwd(), flags.transcript) : undefined,
+ quality,
+ });
+ if (flags.json) {
+ console.log(JSON.stringify(withMeta(result)));
+ return;
+ }
+ console.log(`${formatRead(result.analysis, range)}\n\nwhole recording: ${result.textPath}`);
+}
+
export default defineCommand({
meta: {
name: "audio",
@@ -188,7 +224,8 @@ export default defineCommand({
},
quality: {
type: "string",
- description: "best (any provider), local (this machine), fast (signal only)",
+ description:
+ "local (this machine, the default), best (also uploads to HeyGen's audio services, which may bill your account), fast (signal only)",
},
provider: { type: "string", description: "Pin capabilities to providers: onsets=signal,..." },
"onset-method": { type: "string", description: "spectral-flux (default) or energy" },
@@ -212,6 +249,16 @@ export default defineCommand({
description: "Include each capability's provider ladder",
default: false,
},
+ read: {
+ type: "boolean",
+ description:
+ "Read one whole recording as voice, effect and music lines (--range picks what is shown)",
+ default: false,
+ },
+ transcript: {
+ type: "string",
+ description: "With --read: the recording's transcript (default <file>.transcript.json)",
+ },
json: { type: "boolean", description: "Output as JSON", default: false },
},
setup({ rawArgs, cmd }) {
@@ -220,6 +267,8 @@ export default defineCommand({
async run({ args }) {
const flags = args as unknown as AudioFlags;
const files = args._.map(String);
+ if (flags.read) return read(files, flags);
+ if (flags.transcript) throw new CliUsageError("--transcript only works with --read");
const query = filmQuery(flags);
if (files.length && query) throw new CliUsageError("give files or a film query, not both");
const targets = files.length