Skip to content

Node-based terminal manager for AI coding agents — tmux-backed terminals and parallel agent sessions as draggable nodes on an infinite pan/zoom canvas. macOS, Linux, and a browser Server Edition.

Merge origin/main into fix/smart-whisper-context-params

review7d15e243397a
diff --cc CLAUDE.md
index 4f7ad1e5e,4b2c9ba85..028b1f1ba
--- a/CLAUDE.md
+++ b/CLAUDE.md
@@@ -9893,17 -10079,15 +10079,26 @@@ Voice-to-text input captured via microp
SIMD flags in ggml), a GPU backend (this smart-whisper has no Vulkan build option), cancelling a
running transcription, and the Server Edition (installed with `--ignore-scripts`, so it keeps the
baseline; the startup log line `[speech] whisper binary: …` says which binary loaded).
- - **smart-whisper's model params are patched to start from whisper.cpp's defaults** (second patch in
+ - **macOS: smart-whisper's Metal shader path is patched to the unpacked copy** (second patch in
+ `scripts/patch-smart-whisper.mjs`, marker `NODETERM-PATCH(smart-whisper-metal-asar)`). smart-whisper
+ sets `GGML_METAL_PATH_RESOURCES` from its own `__dirname`, which in the packaged app is inside
+ `app.asar`; native code cannot read inside an asar ("Not a directory"), so Metal init failed and
+ whisper.cpp ran on the CPU with no error shown. The shader file is already unpacked (asarUnpack), so
+ the patch rewrites `app.asar` to `app.asar.unpacked` in that path. MEASURED with the app's own binary
+ on an M5 Pro, large-v3-turbo, 2.7 s clip: 5.97 s on the CPU, 0.64 s on Metal; in the app, dictation
+ went from several seconds to under one. A dev install and an explicit `GGML_METAL_PATH_RESOURCES`
+ are unchanged.
++- **smart-whisper's model params are patched to start from whisper.cpp's defaults** (third patch in
+ `scripts/patch-smart-whisper.mjs`, marker `NODETERM-PATCH(smart-whisper-ctx-params)`, every
+ platform). Its `LoadModel` declared `whisper_context_params params;` and set only `use_gpu`, so DTW,
+ `gpu_device`, `flash_attn` and the rest were uninitialized stack memory and the result depended on
+ what was there. MEASURED on an M5 Pro, Electron 42, large-v3-turbo: the shipped binary crashed in 4
+ of 4 runs outside the app (`aheads_masks_init` then a crash in `whisper_free_state`, because garbage
+ switched DTW on with large-v3's alignment heads); with the defaults it ran without a crash, 0.56 s
+ for a 2.7 s clip and 0.93 s for 21 s on Metal. The slow dictation inside the packaged app is a
+ separate bug: Metal cannot read its shaders inside app.asar. A newer whisper.cpp is
+ faster again (0.22 s / 0.43 s, measured the same way) but needs a build change on every platform;
+ that is a separate follow-up.
## Packaging & auto-update
diff --cc scripts/patch-smart-whisper.d.mts
index c6c0e1c51,6b33e2f9c..1f81e6c7c
--- a/scripts/patch-smart-whisper.d.mts
+++ b/scripts/patch-smart-whisper.d.mts
@@@ -3,5 -3,5 +3,6 @@@
export declare const PATCH_MARKER: string
export declare const VARIANT_TARGET: string
export declare const SIMD_FLAGS: readonly string[]
+export declare const CTX_PARAMS_MARKER: string
export declare function patchSmartWhisper(dir: string): 'patched' | 'already-patched'
+ export declare const METAL_MARKER: string
diff --cc scripts/patch-smart-whisper.mjs
index a3f841e98,4984a5bb0..737d5c303
--- a/scripts/patch-smart-whisper.mjs
+++ b/scripts/patch-smart-whisper.mjs
@@@ -28,19 -28,16 +28,28 @@@
* no-op when nothing sets the global.
* - Runs before electron-rebuild in `postinstall` / `rebuild`, so the native build sees it.
*
- * SECOND PATCH: initialized context params (every platform)
+ * SECOND PATCH: Metal shaders in a packaged app (macOS)
+ * smart-whisper tells whisper.cpp where its Metal shader source is through
+ * GGML_METAL_PATH_RESOURCES, built from its own __dirname. In the packaged app dist/ is inside
+ * app.asar, and native code cannot read inside an asar ("Not a directory"), so Metal init fails
+ * and whisper.cpp silently runs on the CPU. The shader file is already unpacked beside it
+ * (asarUnpack), so the patch points at app.asar.unpacked. Measured on an M5 Pro, Electron 42,
+ * large-v3-turbo, 2.7 s clip, the app's own binary: 5.97 s on the CPU vs 0.64 s on Metal. An
+ * explicit GGML_METAL_PATH_RESOURCES and a dev install are left unchanged.
+ *
- * GUARD: src/core/speech/smart-whisper-patch.test.ts asserts the marker is present in the
++ * THIRD PATCH: initialized context params (every platform)
+ * smart-whisper's `LoadModel` (src/binding/model.cc) declares `whisper_context_params params;`
+ * and sets only `use_gpu`, so every other field (dtw_token_timestamps, dtw_aheads_preset,
+ * gpu_device, flash_attn, ...) is uninitialized stack memory. What it holds decides the outcome:
+ * with DTW switched on by garbage, whisper_init_state() fails for large-v3-turbo ("tried to set
+ * alignment head on text layer 14, but model only has 4 text layers") and the app aborts in
+ * whisper_free_state. Measured on an M5 Pro, Electron 42, large-v3-turbo, 2.7 s and 21 s clips:
+ * the shipped binary crashed in 4 of 4 runs outside the app; with the params initialized it ran
+ * without a crash (0.56 s and 0.93 s on Metal). The slow dictation seen inside the packaged app is
- * a separate bug (Metal cannot read its shaders inside app.asar). The fix starts from
++ * a separate bug, the second patch above. The fix starts from
+ * whisper_context_default_params(), exactly what whisper.cpp's own examples do.
+ *
- * GUARD: src/core/speech/smart-whisper-patch.test.ts asserts both markers are present in the
++ * GUARD: src/core/speech/smart-whisper-patch.test.ts asserts every marker is present in the
* installed sources. If it is red, your node_modules is unpatched — run `npm run rebuild`.
*/
@@@ -77,37 -74,32 +86,50 @@@ var module2 = (() =>
const GYP_FIND = ' }\n ],\n}'
+ export const METAL_MARKER = 'NODETERM-PATCH(smart-whisper-metal-asar)'
+
+ const METAL_FIND =
+ 'process.env.GGML_METAL_PATH_RESOURCES = process.env.GGML_METAL_PATH_RESOURCES || import_node_path.default.join(__dirname, "../whisper.cpp/ggml/src");'
+
+ const METAL_REPLACE = `// ${METAL_MARKER}: native code cannot read inside app.asar; use the unpacked copy.
+ process.env.GGML_METAL_PATH_RESOURCES = process.env.GGML_METAL_PATH_RESOURCES || import_node_path.default.join(__dirname, "../whisper.cpp/ggml/src").replace(/([\\\\/])app\\.asar(?=[\\\\/])/, "$1app.asar.unpacked");`
+
+export const CTX_PARAMS_MARKER = 'NODETERM-PATCH(smart-whisper-ctx-params)'
+
+const PARAMS_FIND = ' whisper_context_params params;\n params.use_gpu'
+
+const PARAMS_REPLACE =
+ ` // ${CTX_PARAMS_MARKER}: start from the defaults; the bare declaration left every\n` +
+ ` // other field (DTW, gpu_device, flash_attn, ...) as uninitialized stack memory.\n` +
+ ' whisper_context_params params = whisper_context_default_params();\n' +
+ ' params.use_gpu'
+
/** Patches the smart-whisper package in `dir`. Returns 'patched' | 'already-patched'. */
export function patchSmartWhisper(dir) {
const gypPath = path.join(dir, 'binding.gyp')
const jsPath = path.join(dir, 'dist', 'index.js')
const gyp = fs.readFileSync(gypPath, 'utf8')
const js = fs.readFileSync(jsPath, 'utf8')
+ const modelPath = path.join(dir, 'src', 'binding', 'model.cc')
+ const model = readOrFail(modelPath, 'src/binding/model.cc')
const gypDone = gyp.includes(PATCH_MARKER)
const jsDone = js.includes(PATCH_MARKER)
+ const metalDone = js.includes(METAL_MARKER)
- if (gypDone && jsDone && metalDone) return 'already-patched'
+ const modelDone = model.includes(CTX_PARAMS_MARKER)
- if (gypDone && jsDone && modelDone) return 'already-patched'
++ if (gypDone && jsDone && metalDone && modelDone) return 'already-patched'
+
+ if (!modelDone) {
+ expectOnce(model, PARAMS_FIND, 'model.cc context params')
+ fs.writeFileSync(modelPath, model.replace(PARAMS_FIND, PARAMS_REPLACE))
+ }
- if (!jsDone) {
- expectOnce(js, LOADER_FIND, 'dist/index.js binary require')
- fs.writeFileSync(jsPath, js.replace(LOADER_FIND, LOADER_REPLACE))
+ if (!jsDone || !metalDone) {
+ if (!jsDone) expectOnce(js, LOADER_FIND, 'dist/index.js binary require')
+ if (!metalDone) expectOnce(js, METAL_FIND, 'dist/index.js Metal path')
+ let out = js
+ if (!jsDone) out = out.replace(LOADER_FIND, LOADER_REPLACE)
+ if (!metalDone) out = out.replace(METAL_FIND, METAL_REPLACE)
+ fs.writeFileSync(jsPath, out)
}
if (!gypDone) {
diff --cc src/core/speech/smart-whisper-patch.test.ts
index c72e01624,5b0af9045..57c614f2e
--- a/src/core/speech/smart-whisper-patch.test.ts
+++ b/src/core/speech/smart-whisper-patch.test.ts
@@@ -2,13 -2,7 +2,14 @@@ import { describe, expect, it } from 'v
import fs from 'node:fs'
import path from 'node:path'
import { createRequire } from 'node:module'
-import { METAL_MARKER, PATCH_MARKER, SIMD_FLAGS, VARIANT_TARGET, patchSmartWhisper } from '../../../scripts/patch-smart-whisper.mjs'
+import {
+ CTX_PARAMS_MARKER,
++ METAL_MARKER,
+ PATCH_MARKER,
+ SIMD_FLAGS,
+ VARIANT_TARGET,
+ patchSmartWhisper
+} from '../../../scripts/patch-smart-whisper.mjs'
import { WHISPER_SIMD_FLAGS } from './speech-service'
import { testTmpDir } from '../test-tmp'
@@@ -20,11 -14,10 +21,10 @@@
*/
const PKG = path.resolve(__dirname, '../../../node_modules/smart-whisper')
const installed = fs.existsSync(path.join(PKG, 'binding.gyp'))
-/** Pristine smart-whisper 0.8.1 binding.gyp + the loader lines of its dist/index.js (dist-index.js). */
+/** Pristine smart-whisper 0.8.1 binding.gyp, src/binding/model.cc, and its dist/index.js (dist-index.js). */
const FIXTURE = path.resolve(__dirname, '__fixtures__/smart-whisper-0.8.1')
- function copyPackage(): string {
- const dir = testTmpDir('sw-patch-')
+ function copyPackage(dir = testTmpDir('sw-patch-')): string {
fs.mkdirSync(path.join(dir, 'dist'), { recursive: true })
fs.copyFileSync(path.join(FIXTURE, 'binding.gyp'), path.join(dir, 'binding.gyp'))
// Stored flat: the repo's .gitignore drops every dist/ directory.
@@@ -39,27 -30,7 +39,28 @@@ describe('patch-smart-whisper', () =>
const howto = 'node_modules is unpatched — run `npm run rebuild`.'
expect(fs.readFileSync(path.join(PKG, 'binding.gyp'), 'utf8'), howto).toContain(PATCH_MARKER)
expect(fs.readFileSync(path.join(PKG, 'dist', 'index.js'), 'utf8'), howto).toContain(PATCH_MARKER)
+ expect(fs.readFileSync(path.join(PKG, 'dist', 'index.js'), 'utf8'), howto).toContain(METAL_MARKER)
+ expect(fs.readFileSync(path.join(PKG, 'src', 'binding', 'model.cc'), 'utf8'), howto).toContain(CTX_PARAMS_MARKER)
+ })
+
+ it('starts the model context params from whisper.cpp defaults and is idempotent', () => {
+ // The pristine declaration leaves every field but use_gpu as stack garbage (DTW among them).
+ const dir = copyPackage()
+ const modelPath = path.join(dir, 'src', 'binding', 'model.cc')
+ expect(fs.readFileSync(modelPath, 'utf8')).toMatch(/^\s*whisper_context_params params;$/m)
+ expect(patchSmartWhisper(dir)).toBe('patched')
+ const model = fs.readFileSync(modelPath, 'utf8')
+ expect(model).toContain('whisper_context_params params = whisper_context_default_params();')
+ expect(model).not.toMatch(/^\s*whisper_context_params params;$/m)
+ expect(model).toContain('params.use_gpu = info.Length() == 2')
+ expect(patchSmartWhisper(dir)).toBe('already-patched')
+ expect(fs.readFileSync(modelPath, 'utf8')).toBe(model)
+ })
+
+ it('refuses a model.cc whose params declaration moved', () => {
+ const dir = copyPackage()
+ fs.writeFileSync(path.join(dir, 'src', 'binding', 'model.cc'), '// rewritten upstream\n')
+ expect(() => patchSmartWhisper(dir)).toThrow(/anchor "model.cc context params"/)
})
it('builds for exactly the CPU flags the runtime checks', () => {