Skip to content

armgit / ruflo

Public Imported from GitHub
0

🌊 The original agent harness. Deploy intelligent multi-player swarms, coordinate autonomous workflows, and build conversational AI systems. Features adaptive memory, self-learning intelligence, federation, vector RAG integration, and native Claude Code / Codex / Hermes and many more Integrated

fix(embeddings): writable model cache dir + retryable degraded state (#3692) (#3831)

Nick Ruest6afffb777cf2
diff --git a/v3/@claude-flow/cli/__tests__/adr-454-memory-initializer-retry.test.ts b/v3/@claude-flow/cli/__tests__/adr-454-memory-initializer-retry.test.ts
new file mode 100644
index 0000000000..a03cf5155f
--- /dev/null
+++ b/v3/@claude-flow/cli/__tests__/adr-454-memory-initializer-retry.test.ts
@@ -0,0 +1,235 @@
+/**
+ * ADR-454 β€” writable embedding model cache + retryable degraded-embedder state.
+ *
+ * #3692: a system-prefix global install's package-relative model cache dir
+ * is root-owned, so an ordinary user hits EACCES the first time
+ * `loadLocalEmbeddingChain()` tries to initialize transformers.js. Before
+ * this fix, ANY failure on that path (including a transient/fixable one)
+ * memoized `embeddingModelState.loaded = true` with the hash fallback
+ * permanently for the rest of the process's life β€” the "already-running
+ * workers required a restart" symptom. This suite proves:
+ *
+ * 1. A real backend that THROWS during init (EACCES-shaped) is marked
+ * `degradedReason: 'retryable'`, not permanent.
+ * 2. A retryable failure does NOT re-attempt before `MODEL_INIT_RETRY_INTERVAL_MS`
+ * elapses (no retry-storm regression).
+ * 3. A retryable failure DOES re-attempt β€” and recovers to `backend: 'onnx'`
+ * β€” once the interval has elapsed, without a process restart.
+ * 4. No real backend importable at all is marked `degradedReason: 'permanent'`
+ * and is NEVER retried, even long after the interval elapses.
+ * 5. `env.cacheDir` is resolved and set before `pipelineFn(...)` is invoked.
+ * 6. Hot path: once healthy (backend 'onnx'), repeated embed calls do not
+ * re-run cache-dir resolution / re-invoke the pipeline factory.
+ * 7. ADR-390's `backend: 'onnx' | 'mock'` contract keeps reporting accurately
+ * in both the recovered and the permanently-degraded case.
+ *
+ * London-school: the transformers packages, agentic-flow, and ruvector are
+ * all mocked via hoisted mutable state (mirrors the established pattern in
+ * `issue-3375-rescue-local-embedder.test.ts`) so each test can drive the
+ * exact failure shape without touching the network or a real ONNX model.
+ * `generateLocalEmbedding()` is used directly β€” per its own contract it never
+ * consults the AgentDB bridge, so no bridge setup is needed here.
+ *
+ * `embeddingModelState` is private module state, so every test gets an
+ * isolated instance via `vi.resetModules()` + a fresh dynamic import.
+ */
+import { afterEach, beforeEach, describe, expect, it, vi } from 'vitest';
+import { existsSync, mkdtempSync, rmSync } from 'node:fs';
+import { tmpdir } from 'node:os';
+import { join } from 'node:path';
+
+type PipelineMode = 'absent' | 'throw' | 'succeed';
+
+const local = vi.hoisted(() => ({
+ hfMode: 'absent' as PipelineMode,
+ xenMode: 'absent' as PipelineMode,
+ hfEnv: {} as Record<string, unknown>,
+ xenEnv: {} as Record<string, unknown>,
+ hfCacheDirAtCallTime: undefined as string | undefined,
+ xenCacheDirAtCallTime: undefined as string | undefined,
+ pipelineCallCount: 0,
+ vec(text: string): number[] {
+ let h = 0;
+ for (let i = 0; i < text.length; i++) h = (h * 31 + text.charCodeAt(i)) | 0;
+ return Array.from({ length: 384 }, (_, i) => Math.sin(h + i * 7) + 1.5);
+ },
+}));
+
+function makePipelineGetter(which: 'hf' | 'xen') {
+ return function pipeline() {
+ const mode = which === 'hf' ? local.hfMode : local.xenMode;
+ if (mode === 'absent') return undefined;
+ return async (_task: string, _model?: string) => {
+ local.pipelineCallCount++;
+ const env = which === 'hf' ? local.hfEnv : local.xenEnv;
+ if (which === 'hf') local.hfCacheDirAtCallTime = env.cacheDir as string | undefined;
+ else local.xenCacheDirAtCallTime = env.cacheDir as string | undefined;
+ if (mode === 'throw') {
+ const err: NodeJS.ErrnoException = new Error('EACCES: permission denied, mkdir');
+ err.code = 'EACCES';
+ throw err;
+ }
+ // The resolved "extractor" β€” the callable returned by a real
+ // pipeline('feature-extraction', ...) call.
+ return async (text: string) => ({ data: Float32Array.from(local.vec(text)) });
+ };
+ };
+}
+
+vi.mock('@huggingface/transformers', () => ({
+ get pipeline() {
+ return makePipelineGetter('hf')();
+ },
+ get env() {
+ return local.hfEnv;
+ },
+}));
+vi.mock('@xenova/transformers', () => ({
+ get pipeline() {
+ return makePipelineGetter('xen')();
+ },
+ get env() {
+ return local.xenEnv;
+ },
+}));
+// Every fallback below the transformers branches is disabled so the hash
+// fallback (the branch under test) is reliably reached.
+vi.mock('agentic-flow/reasoningbank', () => ({ computeEmbedding: undefined }));
+vi.mock('agentic-flow', () => ({ embeddings: undefined }));
+vi.mock('ruvector', () => ({ initOnnxEmbedder: undefined, getOptimizedOnnxEmbedder: undefined }));
+
+async function freshInit() {
+ vi.resetModules();
+ return await import('../src/memory/memory-initializer.js');
+}
+
+let tmpRoot: string;
+
+beforeEach(() => {
+ vi.useFakeTimers();
+ local.hfMode = 'absent';
+ local.xenMode = 'absent';
+ local.hfEnv = {};
+ local.xenEnv = {};
+ local.hfCacheDirAtCallTime = undefined;
+ local.xenCacheDirAtCallTime = undefined;
+ local.pipelineCallCount = 0;
+ delete process.env.TRANSFORMERS_CACHE;
+ delete process.env.HF_HOME;
+ delete process.env.RUFLO_REQUIRE_REAL_EMBEDDINGS;
+ // generateLocalEmbedding() never touches the bridge per its own contract,
+ // but generateEmbedding() (exercised once below) is bridge-first β€” force
+ // the bridge off so that call stays on the local chain under test too,
+ // instead of importing the real memory-bridge.js / AgentDB machinery.
+ process.env.CLAUDE_FLOW_DISABLE_BRIDGE = '1';
+ tmpRoot = mkdtempSync(join(tmpdir(), 'ruflo-mem-init-adr454-'));
+});
+
+afterEach(() => {
+ vi.useRealTimers();
+ delete process.env.CLAUDE_FLOW_DISABLE_BRIDGE;
+ rmSync(tmpRoot, { recursive: true, force: true });
+});
+
+describe('ADR-454 permanent degraded state (no transformers package importable)', () => {
+ it('marks the hash fallback permanent and never retries, even long after the retry interval elapses', async () => {
+ const init = await freshInit();
+
+ const out1 = await init.generateLocalEmbedding('hello');
+ expect(out1.backend).toBe('mock');
+ expect(out1.dimensions).toBe(128);
+ expect(local.pipelineCallCount).toBe(0);
+
+ // Advance well past MODEL_INIT_RETRY_INTERVAL_MS (60s) and even flip the
+ // backend to "would now succeed" β€” a permanent state must not care.
+ vi.advanceTimersByTime(10 * 60_000);
+ local.hfMode = 'succeed';
+
+ const out2 = await init.generateLocalEmbedding('hello again, much later');
+ expect(out2.backend).toBe('mock');
+ expect(local.pipelineCallCount).toBe(0);
+ });
+});
+
+describe('ADR-454 retryable degraded state (a real backend existed but init threw)', () => {
+ it('marks retryable on an EACCES-shaped pipeline failure, does not retry before the interval elapses, and recovers to onnx after it does', async () => {
+ local.hfMode = 'throw';
+ const init = await freshInit();
+
+ const out1 = await init.generateLocalEmbedding('alpha task');
+ expect(out1.backend).toBe('mock');
+ expect(out1.dimensions).toBe(128);
+ expect(local.pipelineCallCount).toBe(1);
+
+ // Too soon (30s < 60s interval): must return the cached degraded state,
+ // not re-attempt β€” even though the backend has since "recovered".
+ vi.advanceTimersByTime(30_000);
+ local.hfMode = 'succeed'; // e.g. an admin fixed the cache-dir permissions
+ const out2 = await init.generateLocalEmbedding('beta task, still too soon');
+ expect(out2.backend).toBe('mock');
+ expect(local.pipelineCallCount).toBe(1); // not re-attempted
+
+ // Interval elapsed (30_000 + 30_001 = 60_001ms since the failed attempt):
+ // must re-attempt and recover without a process restart.
+ vi.advanceTimersByTime(30_001);
+ const out3 = await init.generateLocalEmbedding('gamma task, now it works');
+ expect(out3.backend).toBe('onnx');
+ expect(out3.dimensions).toBe(384);
+ expect(local.pipelineCallCount).toBe(2);
+ });
+});
+
+describe('ADR-454 env.cacheDir set before pipelineFn(...) is called (memory-initializer.ts copy)', () => {
+ it('resolves TRANSFORMERS_CACHE, creates the directory, and sets env.cacheDir before the hf pipeline is invoked', async () => {
+ const cacheDir = join(tmpRoot, 'models');
+ process.env.TRANSFORMERS_CACHE = cacheDir;
+ local.hfMode = 'succeed';
+ const init = await freshInit();
+
+ const out = await init.generateLocalEmbedding('cache dir probe');
+ expect(out.backend).toBe('onnx');
+ expect(local.hfCacheDirAtCallTime).toBe(cacheDir);
+ expect(existsSync(cacheDir)).toBe(true);
+ });
+});
+
+describe('ADR-454 hot path: healthy onnx state does no redundant work per embed call', () => {
+ it('does not re-resolve the cache dir or re-invoke the pipeline factory on subsequent healthy calls', async () => {
+ const cacheDir = join(tmpRoot, 'models');
+ process.env.TRANSFORMERS_CACHE = cacheDir;
+ local.hfMode = 'succeed';
+ const init = await freshInit();
+
+ const out1 = await init.generateLocalEmbedding('task one');
+ expect(out1.backend).toBe('onnx');
+ expect(local.pipelineCallCount).toBe(1);
+
+ // Many more calls on an already-healthy state must not touch the
+ // pipeline factory (which is where cache-dir resolution + mkdir live)
+ // again β€” only the retry-interval `Date.now()` comparison should run.
+ await init.generateLocalEmbedding('task two');
+ await init.generateLocalEmbedding('task three');
+ await init.generateLocalEmbedding('task four');
+ expect(local.pipelineCallCount).toBe(1);
+ });
+});
+
+describe('ADR-454 / ADR-390 backend field regression guard', () => {
+ it('reports backend:"onnx" in the recovered case and backend:"mock" in the permanently-degraded case', async () => {
+ // Permanently degraded: no backend importable at all.
+ const degradedInit = await freshInit();
+ const degraded = await degradedInit.generateLocalEmbedding('no backend available');
+ expect(degraded.backend).toBe('mock');
+
+ // Recovered: a real backend loads successfully from a cold state.
+ local.hfMode = 'succeed';
+ const recoveredInit = await freshInit();
+ const recovered = await recoveredInit.generateLocalEmbedding('real backend available');
+ expect(recovered.backend).toBe('onnx');
+
+ // generateEmbedding() (bridge-first, but no bridge configured here so it
+ // falls through to the local chain) must report the same backend truthfully.
+ const viaGenerateEmbedding = await recoveredInit.generateEmbedding('real backend available via generateEmbedding');
+ expect(viaGenerateEmbedding.backend).toBe('onnx');
+ });
+});
diff --git a/v3/@claude-flow/cli/__tests__/adr-454-task-embedder-retry.test.ts b/v3/@claude-flow/cli/__tests__/adr-454-task-embedder-retry.test.ts
new file mode 100644
index 0000000000..108c15cae6
--- /dev/null
+++ b/v3/@claude-flow/cli/__tests__/adr-454-task-embedder-retry.test.ts
@@ -0,0 +1,151 @@
+/**
+ * ADR-454 β€” task-embedder.ts's own permanent-vs-retryable bug.
+ *
+ * Separate from `memory-initializer.ts`'s bug: `loadExtractor()` cached
+ * `_extractorPromise` permanently even when it resolved to `null` on a
+ * transient/fixable failure (e.g. EACCES on the model cache dir), with no
+ * way to distinguish that from a genuinely missing dependency. This suite
+ * proves the same retryable/permanent distinction holds here, plus the
+ * cache-dir fix and the hot-path no-redundant-work requirement.
+ *
+ * `embedTaskWithCache()` never throws and never consults any bridge, so no
+ * bridge setup is needed. Module state is reset via the file's own test
+ * seam, `__resetTaskEmbedderForTests()`, rather than `vi.resetModules()`.
+ */
+import { afterEach, beforeEach, describe, expect, it, vi } from 'vitest';
+import { existsSync, mkdtempSync, rmSync } from 'node:fs';
+import { tmpdir } from 'node:os';
+import { join } from 'node:path';
+import {
+ __resetTaskEmbedderForTests,
+ embedTaskWithCache,
+ embedTaskWithCacheBatch,
+} from '../src/ruvector/task-embedder.js';
+
+type PipelineMode = 'absent' | 'throw' | 'succeed';
+
+const local = vi.hoisted(() => ({
+ mode: 'absent' as PipelineMode,
+ env: {} as Record<string, unknown>,
+ cacheDirAtCallTime: undefined as string | undefined,
+ pipelineCallCount: 0,
+}));
+
+vi.mock('@xenova/transformers', () => ({
+ get pipeline() {
+ if (local.mode === 'absent') return undefined;
+ return async (_task: string, _model?: string, _opts?: unknown) => {
+ local.pipelineCallCount++;
+ local.cacheDirAtCallTime = local.env.cacheDir as string | undefined;
+ if (local.mode === 'throw') {
+ const err: NodeJS.ErrnoException = new Error('EACCES: permission denied, mkdir');
+ err.code = 'EACCES';
+ throw err;
+ }
+ // The "extractor" callable returned by a real pipeline(...) call,
+ // supporting both single-text and array-input modes.
+ return async (input: string | string[], _opts: unknown) => {
+ const texts = Array.isArray(input) ? input : [input];
+ const dim = 384;
+ const data = new Float32Array(texts.length * dim).fill(0.5);
+ return { data, dims: [texts.length, dim] };
+ };
+ };
+ },
+ get env() {
+ return local.env;
+ },
+}));
+
+let tmpRoot: string;
+
+beforeEach(() => {
+ vi.useFakeTimers();
+ __resetTaskEmbedderForTests();
+ local.mode = 'absent';
+ local.env = {};
+ local.cacheDirAtCallTime = undefined;
+ local.pipelineCallCount = 0;
+ delete process.env.TRANSFORMERS_CACHE;
+ delete process.env.HF_HOME;
+ tmpRoot = mkdtempSync(join(tmpdir(), 'ruflo-task-embedder-adr454-'));
+});
+
+afterEach(() => {
+ vi.useRealTimers();
+ rmSync(tmpRoot, { recursive: true, force: true });
+});
+
+describe('ADR-454 permanent degraded state (no @xenova/transformers pipeline export)', () => {
+ it('returns undefined and never retries, even long after the retry interval elapses', async () => {
+ const out1 = await embedTaskWithCache('alpha task');
+ expect(out1).toBeUndefined();
+ expect(local.pipelineCallCount).toBe(0);
+
+ vi.advanceTimersByTime(10 * 60_000);
+ local.mode = 'succeed'; // "the package got installed" β€” must not matter
+
+ const out2 = await embedTaskWithCache('beta task, much later');
+ expect(out2).toBeUndefined();
+ expect(local.pipelineCallCount).toBe(0);
+ });
+});
+
+describe('ADR-454 retryable degraded state (pipeline present but init threw)', () => {
+ it('returns undefined on an EACCES-shaped failure, does not retry before the interval elapses, and recovers after it does', async () => {
+ local.mode = 'throw';
+
+ const out1 = await embedTaskWithCache('alpha task');
+ expect(out1).toBeUndefined();
+ expect(local.pipelineCallCount).toBe(1);
+
+ // Too soon: must not re-attempt.
+ vi.advanceTimersByTime(30_000);
+ local.mode = 'succeed';
+ const out2 = await embedTaskWithCache('beta task, still too soon');
+ expect(out2).toBeUndefined();
+ expect(local.pipelineCallCount).toBe(1);
+
+ // Interval elapsed: must re-attempt and recover.
+ vi.advanceTimersByTime(30_001);
+ const out3 = await embedTaskWithCache('gamma task, now it works');
+ expect(out3).toBeDefined();
+ expect(out3).toHaveLength(384);
+ expect(local.pipelineCallCount).toBe(2);
+ });
+});
+
+describe('ADR-454 env.cacheDir set before pipeline(...) is called (task-embedder.ts copy)', () => {
+ it('resolves TRANSFORMERS_CACHE, creates the directory, and sets env.cacheDir before the pipeline is invoked', async () => {
+ const cacheDir = join(tmpRoot, 'models');
+ process.env.TRANSFORMERS_CACHE = cacheDir;
+ local.mode = 'succeed';
+
+ const out = await embedTaskWithCache('cache dir probe task');
+ expect(out).toBeDefined();
+ expect(local.cacheDirAtCallTime).toBe(cacheDir);
+ expect(existsSync(cacheDir)).toBe(true);
+ });
+});
+
+describe('ADR-454 hot path: healthy extractor does no redundant work per embed call', () => {
+ it('loads the pipeline once and does not re-invoke it on subsequent cache-miss calls', async () => {
+ const cacheDir = join(tmpRoot, 'models');
+ process.env.TRANSFORMERS_CACHE = cacheDir;
+ local.mode = 'succeed';
+
+ const out1 = await embedTaskWithCache('task one');
+ expect(out1).toBeDefined();
+ expect(local.pipelineCallCount).toBe(1);
+
+ // Distinct task text each time so these are cache MISSES, not hits β€”
+ // the embedder itself runs again, but the pipeline factory (where
+ // cache-dir resolution + mkdir live) must not re-run.
+ await embedTaskWithCache('task two');
+ await embedTaskWithCache('task three');
+ const batch = await embedTaskWithCacheBatch(['task four', 'task five']);
+ expect(batch.every((v) => v !== undefined)).toBe(true);
+
+ expect(local.pipelineCallCount).toBe(1);
+ });
+});
diff --git a/v3/@claude-flow/cli/__tests__/model-cache-dir-adr454.test.ts b/v3/@claude-flow/cli/__tests__/model-cache-dir-adr454.test.ts
new file mode 100644
index 0000000000..93e88a28a4
--- /dev/null
+++ b/v3/@claude-flow/cli/__tests__/model-cache-dir-adr454.test.ts
@@ -0,0 +1,47 @@
+/**
+ * ADR-454 β€” `@claude-flow/cli`'s own copy of `resolveModelCacheDir()`.
+ *
+ * `v3/@claude-flow/cli/src/memory/model-cache-dir.ts` is an intentional,
+ * dependency-free duplicate of `@claude-flow/embeddings/src/transformers-loader.ts`'s
+ * helper of the same name (the CLI package has no dependency on
+ * `@claude-flow/embeddings`, to avoid reintroducing the ADR-094 circular
+ * optional-dep issue). Verified independently per the ADR: testing one copy
+ * does not cover the other.
+ */
+import { afterEach, beforeEach, describe, expect, it } from 'vitest';
+import { homedir } from 'node:os';
+import { join } from 'node:path';
+import { resolveModelCacheDir } from '../src/memory/model-cache-dir.js';
+
+const savedEnv: { TRANSFORMERS_CACHE?: string; HF_HOME?: string } = {};
+
+beforeEach(() => {
+ savedEnv.TRANSFORMERS_CACHE = process.env.TRANSFORMERS_CACHE;
+ savedEnv.HF_HOME = process.env.HF_HOME;
+ delete process.env.TRANSFORMERS_CACHE;
+ delete process.env.HF_HOME;
+});
+
+afterEach(() => {
+ if (savedEnv.TRANSFORMERS_CACHE === undefined) delete process.env.TRANSFORMERS_CACHE;
+ else process.env.TRANSFORMERS_CACHE = savedEnv.TRANSFORMERS_CACHE;
+ if (savedEnv.HF_HOME === undefined) delete process.env.HF_HOME;
+ else process.env.HF_HOME = savedEnv.HF_HOME;
+});
+
+describe('ADR-454 resolveModelCacheDir() precedence (@claude-flow/cli copy)', () => {
+ it('prefers TRANSFORMERS_CACHE when set', () => {
+ process.env.TRANSFORMERS_CACHE = '/custom/transformers-cache';
+ process.env.HF_HOME = '/custom/hf-home';
+ expect(resolveModelCacheDir()).toBe('/custom/transformers-cache');
+ });
+
+ it('falls back to HF_HOME when TRANSFORMERS_CACHE is unset', () => {
+ process.env.HF_HOME = '/custom/hf-home';
+ expect(resolveModelCacheDir()).toBe('/custom/hf-home');
+ });
+
+ it('falls back to ~/.cache/ruflo/models when neither env var is set', () => {
+ expect(resolveModelCacheDir()).toBe(join(homedir(), '.cache', 'ruflo', 'models'));
+ });
+});
diff --git a/v3/@claude-flow/cli/src/memory/memory-initializer.ts b/v3/@claude-flow/cli/src/memory/memory-initializer.ts
index 38bb512473..85b2fffb2d 100644
--- a/v3/@claude-flow/cli/src/memory/memory-initializer.ts
+++ b/v3/@claude-flow/cli/src/memory/memory-initializer.ts
@@ -22,6 +22,7 @@ import { createRequire } from 'node:module';
import { readFileMaybeEncrypted, writeDbImagePreservingFormat, writeFileAtomic, writeFileRestricted } from '../fs-secure.js';
import { restoreMemoryDbFromBackup } from '../services/memory-backup.js';
import { validateIdentifier } from '../mcp-tools/validate-input.js';
+import { resolveModelCacheDir } from './model-cache-dir.js';
/**
* ADR-323 β€” typed memory provenance. Distinguishes WHO/WHAT wrote a memory
@@ -2260,8 +2261,19 @@ interface EmbeddingModel {
model: unknown;
tokenizer: unknown;
dimensions: number;
+ /**
+ * ADR-454: distinguishes a hash-fallback state that's legitimately
+ * permanent (no transformers package importable at all) from one caused
+ * by an environmental/fixable failure (EACCES, ENOSPC, a transient
+ * download failure) that should be retried rather than memoized forever.
+ */
+ degradedReason?: 'permanent' | 'retryable';
+ lastAttemptAt?: number;
}
+/** ADR-454: minimum interval before a 'retryable' degraded state re-attempts real init. */
+const MODEL_INIT_RETRY_INTERVAL_MS = 60_000;
+
/**
* State of the LOCAL embedding chain only (transformers.js / agentic-flow /
* ruvector ONNX / hash). #3375: the AgentDB bridge's result is cached
@@ -2331,9 +2343,18 @@ async function loadLocalEmbeddingChain(verbose = false, startTime = Date.now()):
error?: string;
}> {
if (embeddingModelState?.loaded) {
- return { success: true, dimensions: embeddingModelState.dimensions, modelName: 'cached', loadTime: 0 };
+ const isRetryableDue =
+ embeddingModelState.degradedReason === 'retryable' &&
+ Date.now() - (embeddingModelState.lastAttemptAt ?? 0) >= MODEL_INIT_RETRY_INTERVAL_MS;
+ if (!isRetryableDue) {
+ return { success: true, dimensions: embeddingModelState.dimensions, modelName: 'cached', loadTime: 0 };
+ }
+ // Fall through to re-attempt real initialization β€” the retryable
+ // failure (e.g. EACCES on the model cache dir) may now be fixed.
}
+ let sawRetryableFailure = false;
+
try {
// ADR-094: prefer @huggingface/transformers (clears protobufjs <7.5.5
// critical RCE chain), fall back to legacy @xenova/transformers.
@@ -2342,6 +2363,7 @@ async function loadLocalEmbeddingChain(verbose = false, startTime = Date.now()):
// @claude-flow/embeddings/src/transformers-loader.ts.
let transformersSource: '@huggingface/transformers' | '@xenova/transformers' | null = null;
let pipelineFn: ((task: string, model?: string) => Promise<unknown>) | null = null;
+ let transformersModule: Record<string, unknown> | null = null;
{
const tryLoad = async (specifier: string): Promise<Record<string, unknown> | null> => {
@@ -2352,11 +2374,13 @@ async function loadLocalEmbeddingChain(verbose = false, startTime = Date.now()):
if (hf && typeof hf.pipeline === 'function') {
pipelineFn = hf.pipeline as (t: string, m?: string) => Promise<unknown>;
transformersSource = '@huggingface/transformers';
+ transformersModule = hf;
} else {
const xen = await tryLoad('@xenova/transformers');
if (xen && typeof xen.pipeline === 'function') {
pipelineFn = xen.pipeline as (t: string, m?: string) => Promise<unknown>;
transformersSource = '@xenova/transformers';
+ transformersModule = xen;
}
}
}
@@ -2374,6 +2398,15 @@ async function loadLocalEmbeddingChain(verbose = false, startTime = Date.now()):
if (verbose) {
console.log(`Loading ONNX embedding model via ${transformersSource} (all-MiniLM-L6-v2)...`);
}
+
+ // ADR-454: route model downloads to a user-writable cache dir β€”
+ // the package-relative default is root-owned under a system-prefix
+ // global install, which throws EACCES for the runtime user.
+ const cacheDir = resolveModelCacheDir();
+ fs.mkdirSync(cacheDir, { recursive: true });
+ const env = transformersModule?.env as { cacheDir: string } | undefined;
+ if (env) env.cacheDir = cacheDir;
+
const embedder = await pipelineFn('feature-extraction', 'Xenova/all-MiniLM-L6-v2');
embeddingModelState = {
@@ -2390,6 +2423,10 @@ async function loadLocalEmbeddingChain(verbose = false, startTime = Date.now()):
loadTime: Date.now() - startTime
};
} catch (err) {
+ // ADR-454: a real backend was importable and callable β€” this failure
+ // (EACCES, ENOSPC, transient network error, etc.) is environmental,
+ // not a missing dependency, so it must not be memoized permanently.
+ sawRetryableFailure = true;
if (verbose) {
console.warn(
`${transformersSource} pipeline init failed (${err instanceof Error ? err.message : String(err)}); ` +
@@ -2468,12 +2505,18 @@ async function loadLocalEmbeddingChain(verbose = false, startTime = Date.now()):
// No agentic-flow root-entry fallback: no published root exports
// `embeddings`, and importing it runs its CLI main() in 3.0.0-alpha.x.
- // No ONNX model available - use fallback
+ // No ONNX model available - use fallback.
+ // ADR-454: mark whether this is a permanent (no backend importable at
+ // all) or retryable (a backend existed but init failed for an
+ // environmental reason) degraded state, so a later call can re-attempt
+ // real initialization once the condition is fixed.
embeddingModelState = {
loaded: true,
model: null, // Will use simple hash-based fallback
tokenizer: null,
- dimensions: 128 // Smaller fallback dimensions
+ dimensions: 128, // Smaller fallback dimensions
+ degradedReason: sawRetryableFailure ? 'retryable' : 'permanent',
+ lastAttemptAt: Date.now()
};
return {
@@ -2553,7 +2596,16 @@ export async function generateLocalEmbedding(text: string): Promise<{
// loadEmbeddingModel(), which is bridge-first β€” when the bridge answered
// there, no local model was ever loaded and this function always returned
// the hash fallback.
- if (!embeddingModelState?.loaded) {
+ //
+ // ADR-454: also re-attempt when the memoized state is a 'retryable'
+ // degraded fallback whose retry interval has elapsed β€” otherwise this
+ // guard short-circuits past loadLocalEmbeddingChain()'s own retry check
+ // and the fix (e.g. a now-writable cache dir) never gets exercised.
+ const shouldRetryLoad =
+ !embeddingModelState?.loaded ||
+ (embeddingModelState.degradedReason === 'retryable' &&
+ Date.now() - (embeddingModelState.lastAttemptAt ?? 0) >= MODEL_INIT_RETRY_INTERVAL_MS);
+ if (shouldRetryLoad) {
await loadLocalEmbeddingChain();
}
diff --git a/v3/@claude-flow/cli/src/memory/model-cache-dir.ts b/v3/@claude-flow/cli/src/memory/model-cache-dir.ts
new file mode 100644
index 0000000000..f86082e28c
--- /dev/null
+++ b/v3/@claude-flow/cli/src/memory/model-cache-dir.ts
@@ -0,0 +1,19 @@
+import * as os from 'node:os';
+import * as path from 'node:path';
+
+/**
+ * Resolve a writable cache directory for ONNX model downloads.
+ *
+ * ADR-454: duplicated intentionally from
+ * `@claude-flow/embeddings/src/transformers-loader.ts`'s
+ * `resolveModelCacheDir()` rather than imported, since `@claude-flow/cli`
+ * has no dependency on `@claude-flow/embeddings` (see the circular
+ * optional-dep note in `memory-initializer.ts`).
+ */
+export function resolveModelCacheDir(): string {
+ return (
+ process.env.TRANSFORMERS_CACHE ||
+ process.env.HF_HOME ||
+ path.join(os.homedir(), '.cache', 'ruflo', 'models')
+ );
+}
diff --git a/v3/@claude-flow/cli/src/ruvector/task-embedder.ts b/v3/@claude-flow/cli/src/ruvector/task-embedder.ts
index a8657c620d..376f65f449 100644
--- a/v3/@claude-flow/cli/src/ruvector/task-embedder.ts
+++ b/v3/@claude-flow/cli/src/ruvector/task-embedder.ts
@@ -20,6 +20,9 @@
* @module task-embedder
*/
+import * as fs from 'node:fs';
+import { resolveModelCacheDir } from '../memory/model-cache-dir.js';
+
// ============================================================================
// FNV-1a-32 hash (matches scripts/gen-seed-corpus.mjs + router-trajectory.ts)
// ============================================================================
@@ -52,15 +55,49 @@ type ExtractorFn = (input: string | string[], opts: { pooling: 'mean'; normalize
// so we don't pay the ONNX model-load cost twice.
let _extractorPromise: Promise<ExtractorFn | null> | null = null;
+/**
+ * ADR-454: distinguishes a null-extractor result caused by a genuinely
+ * missing dependency (permanent β€” nothing changes without a reinstall) from
+ * one caused by an environmental/fixable failure during `pipeline(...)`
+ * init, such as EACCES on the model cache dir (retryable).
+ */
+let _extractorDegradedReason: 'permanent' | 'retryable' | null = null;
+let _extractorLastAttemptAt = 0;
+const EXTRACTOR_RETRY_INTERVAL_MS = 60_000;
+
function loadExtractor(): Promise<ExtractorFn | null> {
- if (_extractorPromise !== null) return _extractorPromise;
+ if (_extractorPromise !== null) {
+ const isRetryableDue =
+ _extractorDegradedReason === 'retryable' &&
+ Date.now() - _extractorLastAttemptAt >= EXTRACTOR_RETRY_INTERVAL_MS;
+ if (!isRetryableDue) return _extractorPromise;
+ // Fall through to re-attempt β€” the retryable failure may now be fixed.
+ }
+ _extractorLastAttemptAt = Date.now();
_extractorPromise = (async () => {
+ let packageImportable = false;
try {
const specifier = '@xenova/transformers';
const mod = await import(/* @vite-ignore */ specifier).catch(() => null);
- if (!mod || typeof mod.pipeline !== 'function') return null;
- return await mod.pipeline('feature-extraction', 'Xenova/all-MiniLM-L6-v2', { quantized: true });
+ if (!mod || typeof mod.pipeline !== 'function') {
+ _extractorDegradedReason = 'permanent';
+ return null;
+ }
+ packageImportable = true;
+
+ // ADR-454: route model downloads to a user-writable cache dir β€” the
+ // package-relative default is root-owned under a system-prefix
+ // global install, which throws EACCES for the runtime user.
+ const cacheDir = resolveModelCacheDir();
+ fs.mkdirSync(cacheDir, { recursive: true });
+ const env = (mod as Record<string, unknown>).env as { cacheDir: string } | undefined;
+ if (env) env.cacheDir = cacheDir;
+
+ const extractor = await mod.pipeline('feature-extraction', 'Xenova/all-MiniLM-L6-v2', { quantized: true });
+ _extractorDegradedReason = null;
+ return extractor;
} catch {
+ _extractorDegradedReason = packageImportable ? 'retryable' : 'permanent';
return null;
}
})();
@@ -242,4 +279,6 @@ export function __resetTaskEmbedderForTests(): void {
_hits = 0;
_misses = 0;
_extractorPromise = null;
+ _extractorDegradedReason = null;
+ _extractorLastAttemptAt = 0;
}
diff --git a/v3/@claude-flow/embeddings/__tests__/transformers-loader-adr454.test.ts b/v3/@claude-flow/embeddings/__tests__/transformers-loader-adr454.test.ts
new file mode 100644
index 0000000000..7eed48f45b
--- /dev/null
+++ b/v3/@claude-flow/embeddings/__tests__/transformers-loader-adr454.test.ts
@@ -0,0 +1,154 @@
+/**
+ * ADR-454 β€” writable embedding model cache dir.
+ *
+ * Covers `transformers-loader.ts`'s `resolveModelCacheDir()` precedence
+ * (TRANSFORMERS_CACHE > HF_HOME > ~/.cache/ruflo/models) and the requirement
+ * that `env.cacheDir` is set on the resolved transformers module BEFORE the
+ * returned `pipeline` function is ever invoked by a caller, for both the
+ * `@huggingface/transformers` and `@xenova/transformers` branches.
+ *
+ * London-school: both packages are mocked via hoisted mutable state so each
+ * test can flip availability/behavior without re-declaring `vi.mock`. Module
+ * state (`cached`/`cacheChecked`) is private to transformers-loader.ts, so
+ * every test gets an isolated instance via `vi.resetModules()` + a fresh
+ * dynamic import.
+ */
+import { afterEach, beforeEach, describe, expect, it, vi } from 'vitest';
+import { existsSync, mkdtempSync, rmSync } from 'node:fs';
+import { homedir, tmpdir } from 'node:os';
+import { join } from 'node:path';
+
+const state = vi.hoisted(() => ({
+ hfHasPipeline: false,
+ xenHasPipeline: false,
+ hfEnv: {} as Record<string, unknown>,
+ xenEnv: {} as Record<string, unknown>,
+ hfCacheDirAtCallTime: undefined as string | undefined,
+ xenCacheDirAtCallTime: undefined as string | undefined,
+}));
+
+vi.mock('@huggingface/transformers', () => ({
+ get pipeline() {
+ if (!state.hfHasPipeline) return undefined;
+ return async (_task: string, _model?: string) => {
+ state.hfCacheDirAtCallTime = state.hfEnv.cacheDir as string | undefined;
+ return async () => ({ data: new Float32Array(384) });
+ };
+ },
+ get env() {
+ return state.hfEnv;
+ },
+ version: '4.0.0-test',
+}));
+
+vi.mock('@xenova/transformers', () => ({
+ get pipeline() {
+ if (!state.xenHasPipeline) return undefined;
+ return async (_task: string, _model?: string) => {
+ state.xenCacheDirAtCallTime = state.xenEnv.cacheDir as string | undefined;
+ return async () => ({ data: new Float32Array(384) });
+ };
+ },
+ get env() {
+ return state.xenEnv;
+ },
+ version: '2.17.0-test',
+}));
+
+let tmpRoot: string;
+
+beforeEach(() => {
+ vi.resetModules();
+ state.hfHasPipeline = false;
+ state.xenHasPipeline = false;
+ state.hfEnv = {};
+ state.xenEnv = {};
+ state.hfCacheDirAtCallTime = undefined;
+ state.xenCacheDirAtCallTime = undefined;
+ delete process.env.TRANSFORMERS_CACHE;
+ delete process.env.HF_HOME;
+ tmpRoot = mkdtempSync(join(tmpdir(), 'ruflo-embeddings-adr454-'));
+});
+
+afterEach(() => {
+ rmSync(tmpRoot, { recursive: true, force: true });
+});
+
+describe('ADR-454 resolveModelCacheDir() precedence', () => {
+ it('prefers TRANSFORMERS_CACHE when set', async () => {
+ process.env.TRANSFORMERS_CACHE = '/custom/transformers-cache';
+ process.env.HF_HOME = '/custom/hf-home';
+ const { resolveModelCacheDir } = await import('../src/transformers-loader.js');
+ expect(resolveModelCacheDir()).toBe('/custom/transformers-cache');
+ });
+
+ it('falls back to HF_HOME when TRANSFORMERS_CACHE is unset', async () => {
+ process.env.HF_HOME = '/custom/hf-home';
+ const { resolveModelCacheDir } = await import('../src/transformers-loader.js');
+ expect(resolveModelCacheDir()).toBe('/custom/hf-home');
+ });
+
+ it('falls back to ~/.cache/ruflo/models when neither env var is set', async () => {
+ const { resolveModelCacheDir } = await import('../src/transformers-loader.js');
+ expect(resolveModelCacheDir()).toBe(join(homedir(), '.cache', 'ruflo', 'models'));
+ });
+});
+
+describe('ADR-454 env.cacheDir set before pipeline(...) is called', () => {
+ it('@huggingface/transformers branch: cache dir is created and env.cacheDir is set before the pipeline function is invoked', async () => {
+ const cacheDir = join(tmpRoot, 'models');
+ process.env.TRANSFORMERS_CACHE = cacheDir;
+ state.hfHasPipeline = true;
+
+ const { loadTransformersPipeline } = await import('../src/transformers-loader.js');
+ const handle = await loadTransformersPipeline();
+
+ expect(handle?.source).toBe('@huggingface/transformers');
+ // env.cacheDir must already be the resolved path immediately after
+ // loadTransformersPipeline() resolves, before any downstream pipeline() call.
+ expect(state.hfEnv.cacheDir).toBe(cacheDir);
+ expect(existsSync(cacheDir)).toBe(true);
+
+ // Invoking the returned pipeline function (as a real caller would) must
+ // see the cache dir already configured β€” proves ordering, not just
+ // end-state.
+ await handle!.pipeline('feature-extraction', 'Xenova/all-MiniLM-L6-v2');
+ expect(state.hfCacheDirAtCallTime).toBe(cacheDir);
+ });
+
+ it('@xenova/transformers branch: cache dir is created and env.cacheDir is set before the pipeline function is invoked', async () => {
+ const cacheDir = join(tmpRoot, 'models');
+ process.env.HF_HOME = cacheDir;
+ state.hfHasPipeline = false; // force fallback to xenova
+ state.xenHasPipeline = true;
+
+ const { loadTransformersPipeline } = await import('../src/transformers-loader.js');
+ const handle = await loadTransformersPipeline();
+
+ expect(handle?.source).toBe('@xenova/transformers');
+ expect(state.xenEnv.cacheDir).toBe(cacheDir);
+ expect(existsSync(cacheDir)).toBe(true);
+
+ await handle!.pipeline('feature-extraction', 'Xenova/all-MiniLM-L6-v2');
+ expect(state.xenCacheDirAtCallTime).toBe(cacheDir);
+ });
+
+ it('prefers @huggingface/transformers over @xenova/transformers when both are available', async () => {
+ const cacheDir = join(tmpRoot, 'models');
+ process.env.TRANSFORMERS_CACHE = cacheDir;
+ state.hfHasPipeline = true;
+ state.xenHasPipeline = true;
+
+ const { loadTransformersPipeline } = await import('../src/transformers-loader.js');
+ const handle = await loadTransformersPipeline();
+
+ expect(handle?.source).toBe('@huggingface/transformers');
+ });
+
+ it('returns null and caches the miss when neither package exposes a pipeline function', async () => {
+ const { loadTransformersPipeline, getCachedTransformersSource } = await import('../src/transformers-loader.js');
+ const handle = await loadTransformersPipeline();
+ expect(handle).toBeNull();
+ expect(getCachedTransformersSource()).toBeNull();
+ });
+});
diff --git a/v3/@claude-flow/embeddings/src/transformers-loader.ts b/v3/@claude-flow/embeddings/src/transformers-loader.ts
index 17e59d8b92..16aa4c0eed 100644
--- a/v3/@claude-flow/embeddings/src/transformers-loader.ts
+++ b/v3/@claude-flow/embeddings/src/transformers-loader.ts
@@ -14,6 +14,10 @@
* users can see which package satisfied the runtime.
*/
+import * as fs from 'node:fs';
+import * as os from 'node:os';
+import * as path from 'node:path';
+
export type PipelineFn = (
task: string,
model?: string,
@@ -29,6 +33,24 @@ export interface TransformersHandle {
let cached: TransformersHandle | null = null;
let cacheChecked = false;
+/**
+ * Resolve a writable cache directory for ONNX model downloads.
+ *
+ * ADR-454: a system-prefix global install's package directory is
+ * root-owned, so defaulting the loader's cache into
+ * `node_modules/@huggingface/transformers/.cache` fails with EACCES
+ * for the runtime user. Honor an explicit env override first, then
+ * fall back to the repo's existing `~/.cache/ruflo/<subpath>`
+ * convention (see `gaia-loader.ts`, `gaia-judge.ts`).
+ */
+export function resolveModelCacheDir(): string {
+ return (
+ process.env.TRANSFORMERS_CACHE ||
+ process.env.HF_HOME ||
+ path.join(os.homedir(), '.cache', 'ruflo', 'models')
+ );
+}
+
/**
* Load a working transformers pipeline. Returns null if neither
* package is installed. Caches the first successful resolution so
@@ -57,6 +79,9 @@ export async function loadTransformersPipeline(): Promise<TransformersHandle | n
// Prefer the maintained successor.
const hf = await tryLoad('@huggingface/transformers');
if (hf && typeof hf.pipeline === 'function') {
+ const cacheDir = resolveModelCacheDir();
+ fs.mkdirSync(cacheDir, { recursive: true });
+ if (hf.env) (hf.env as { cacheDir: string }).cacheDir = cacheDir;
cached = {
pipeline: hf.pipeline as PipelineFn,
source: '@huggingface/transformers',
@@ -68,6 +93,9 @@ export async function loadTransformersPipeline(): Promise<TransformersHandle | n
// Fall back to the legacy package so existing installs keep working.
const xen = await tryLoad('@xenova/transformers');
if (xen && typeof xen.pipeline === 'function') {
+ const cacheDir = resolveModelCacheDir();
+ fs.mkdirSync(cacheDir, { recursive: true });
+ if (xen.env) (xen.env as { cacheDir: string }).cacheDir = cacheDir;
cached = {
pipeline: xen.pipeline as PipelineFn,
source: '@xenova/transformers',
diff --git a/v3/docs/adr/ADR-454-writable-embedding-model-cache-and-retryable-degraded-state.md b/v3/docs/adr/ADR-454-writable-embedding-model-cache-and-retryable-degraded-state.md
new file mode 100644
index 0000000000..b743800c40
--- /dev/null
+++ b/v3/docs/adr/ADR-454-writable-embedding-model-cache-and-retryable-degraded-state.md
@@ -0,0 +1,185 @@
+# ADR-454 β€” Writable Embedding Model Cache and Retryable Degraded-Embedder State
+
+**Status**: Proposed
+**Date**: 2026-10-05
+**Related**: ADR-094 (xenova β†’ @huggingface/transformers migration), ADR-390 (semantic router uses the real embedder), #3692, `v3/@claude-flow/cli/src/benchmarks/gaia-loader.ts` / `gaia-judge.ts` (prior art for `~/.cache/ruflo/*`)
+**Surfaces**: `v3/@claude-flow/embeddings/src/transformers-loader.ts`, `v3/@claude-flow/embeddings/src/embedding-service.ts`, `v3/@claude-flow/cli/src/memory/memory-initializer.ts`, `v3/@claude-flow/cli/src/ruvector/task-embedder.ts`
+
+## Context
+
+Issue #3692: on a system-prefix global install (`npm install -g` as root), an
+ordinary user running `@claude-flow/cli` hits `EACCES` the first time any of
+three independent call sites load the ONNX embedding model
+(`Xenova/all-MiniLM-L6-v2`) via `@huggingface/transformers` /
+`@xenova/transformers`'s `pipeline('feature-extraction', ...)`:
+
+1. `v3/@claude-flow/embeddings/src/transformers-loader.ts`, resolved by
+ `TransformersEmbeddingService.initialize()` in
+ `v3/@claude-flow/embeddings/src/embedding-service.ts:398`.
+2. `v3/@claude-flow/cli/src/memory/memory-initializer.ts:2337`,
+ `loadLocalEmbeddingChain()`'s `pipelineFn('feature-extraction',
+ 'Xenova/all-MiniLM-L6-v2')`.
+3. `v3/@claude-flow/cli/src/ruvector/task-embedder.ts:62`, same pipeline call
+ with `{ quantized: true }`.
+
+None of the three passes a cache-dir override. Without one, the HF/Xenova
+loader defaults to a path under its own package's
+`node_modules/@huggingface/transformers/.cache`. Under a system-prefix global
+install that directory tree is root-owned, so the runtime user's download
+fails with `EACCES`. Reproduced by the reporter across five existing servers;
+creating only the exact model-cache directory as the runtime user (`0700`),
+without touching package ownership, let the normal loader download the model
+and produce real embeddings β€” confirming the cache path, not the package
+install, is the fixable boundary.
+
+A second, compounding bug lives only in `memory-initializer.ts`. The `EACCES`
+throw is caught by `loadLocalEmbeddingChain()`'s inner try/catch (around
+line 2352, which carries a comment referencing #2461) and falls through to
+the hash fallback at line ~2448:
+
+```ts
+embeddingModelState = {
+ loaded: true,
+ model: null, // Will use simple hash-based fallback
+ tokenizer: null,
+ dimensions: 128
+};
+```
+
+Every caller short-circuits on `embeddingModelState?.loaded` β€” inside
+`loadLocalEmbeddingChain` itself (line 2293) and inside
+`generateLocalEmbedding` (line 2532). Because `loaded: true` is set
+permanently on a transient, environmental failure, the degraded hash state is
+memoized for the rest of the process's life β€” this is exactly the "already
+running workers required a restart/refresh" symptom from the issue. Fixing
+the cache-directory permissions mid-run does not help a worker that already
+tripped this path; only a process restart does.
+
+ADR-390 already established that a degraded embedder must be reported
+truthfully rather than silently returning meaningless vectors, via an
+explicit `backend: 'onnx' | 'mock'` field on `generateLocalEmbedding` /
+`generateEmbedding`'s return value (`memory-initializer.ts`, line 2526 type
+definition, lines 2562 and 2578 for assignments) and an `embedder: 'minilm' |
+'hash'` field on router results. That reporting already exists and already
+satisfies part of the issue's "report mock/degraded status truthfully"
+expectation β€” the gap this ADR closes is specifically (a) where the model is
+allowed to try to write, and (b) how long a fixable failure is allowed to
+stay remembered.
+
+The repo already has a convention for a user-writable, XDG-adjacent cache
+outside any package directory: `path.join(os.homedir(), '.cache', 'ruflo',
+...)`, used today by `gaia-loader.ts` (`~/.cache/ruflo/gaia`) and
+`gaia-judge.ts` (`~/.cache/ruflo/gaia/judgments`).
+
+## Decision
+
+1. **One writable cache-dir default, applied at all three call sites.**
+ Resolve a cache directory for ONNX model downloads with this precedence:
+ - `TRANSFORMERS_CACHE` or `HF_HOME`, if the environment already sets one
+ (don't override a choice the user or host environment already made);
+ - otherwise `~/.cache/ruflo/models`, matching the existing
+ `~/.cache/ruflo/<subpath>` convention.
+
+ Centralize this resolution in one shared helper (in
+ `transformers-loader.ts`, since it is the only one of the three call
+ sites that is already a shared module) rather than tripling the
+ precedence logic across `embedding-service.ts`, `memory-initializer.ts`,
+ and `task-embedder.ts`. Each call site passes the resolved directory to
+ the loaded package's cache-dir mechanism (`env.cacheDir` on the
+ `@huggingface/transformers` / `@xenova/transformers` `env` singleton)
+ before calling `pipeline('feature-extraction', ...)`.
+
+2. **Never assume a system package directory is writable.** State this as
+ the root cause, not an edge case: a global, system-prefix install is
+ root-owned by construction, and a runtime user downloading a model into
+ it is the fragile path, not a rare misconfiguration. The fix is to stop
+ defaulting into package-relative storage at all, not to special-case
+ root-owned installs.
+
+3. **Make the degraded state retryable, not permanent.** Distinguish two
+ failure classes in `loadLocalEmbeddingChain()`:
+ - **No better backend exists at all** β€” e.g. neither
+ `@huggingface/transformers` nor `@xenova/transformers` is installed, or
+ `pipeline` isn't exported. This is legitimately permanent for the
+ process's dependency set; memoizing `loaded: true` with the hash
+ fallback is correct here, since nothing will change without a
+ reinstall.
+ - **A real backend exists but initialization failed for an
+ environmental/fixable reason** (`EACCES`, `ENOSPC`, a transient
+ network failure on first download, etc.). This must **not** be
+ memoized as permanent degraded state. Either:
+ - don't set `embeddingModelState.loaded = true` on this path β€” leave it
+ `null`/unset so the next call to `loadLocalEmbeddingChain()`
+ re-attempts initialization, or
+ - keep the memoized hash state but record the failure class alongside
+ it (e.g. `embeddingModelState.degradedReason: 'permanent' |
+ 'retryable'`) and have `generateLocalEmbedding` re-attempt the real
+ loader when the reason is `'retryable'`, bounded by a minimum retry
+ interval so a persistently broken environment doesn't retry on every
+ call.
+
+ Either shape removes the "worker requires a restart after fixing cache
+ permissions" symptom β€” the next real request should recover without a
+ process restart once the writable cache directory exists.
+
+4. **Keep ADR-390's truthful `backend` / `embedder` reporting unchanged.**
+ This ADR extends it (the retryable/permanent distinction above should be
+ visible through the existing status surface, e.g. alongside
+ `embeddings_status.ruvectorStatus`) but does not replace or regress it.
+
+## Consequences
+
+**Positive:**
+- Closes the actual failure mode in #3692 without requiring provider
+ credentials or manual model downloads, matching the reporter's expected
+ behavior.
+- A single resolution point for the cache directory means a future change
+ (e.g. supporting an additional env var) lands once instead of three times.
+- Recovery after an admin fixes cache-dir permissions no longer requires
+ restarting already-running workers.
+
+**Negative:**
+- `task-embedder.ts` and `memory-initializer.ts` each inline their own
+ dynamic `import('@xenova/transformers')` rather than routing through
+ `transformers-loader.ts`'s ADR-094 provider-agnostic loader (the
+ `memory-initializer.ts` site does this deliberately, per ADR-094's
+ "Implementation status" note, to avoid a circular optional-dependency at
+ install time). The shared cache-dir helper this ADR proposes must be
+ importable from all three without reintroducing that circularity β€” it
+ should be a small, dependency-free utility (just path resolution plus
+ setting `env.cacheDir`), not bundled with the loader's provider-selection
+ logic.
+- Honoring `TRANSFORMERS_CACHE` / `HF_HOME` means a host that already has
+ one of those set to a non-writable location keeps failing β€” deliberately
+ not overriding an explicit environment choice is weighed above
+ auto-recovering from a user's own misconfiguration.
+
+**Risk:**
+- The retryable/permanent distinction adds a small amount of state-machine
+ complexity to `loadLocalEmbeddingChain()`, which this ADR explicitly keeps
+ inside `memory-initializer.ts`'s existing inline state rather than
+ introducing a new shared abstraction, to avoid growing the file past the
+ 500-line-per-file guidance beyond what's already there.
+
+## Verification
+
+- Unit tests (London-school, with the transformers package mocked):
+ - cache-dir resolution prefers `TRANSFORMERS_CACHE`/`HF_HOME` when set, and
+ falls back to `~/.cache/ruflo/models` otherwise.
+ - `env.cacheDir` (or equivalent) is set before `pipeline(...)` is called,
+ at all three call sites.
+ - an `EACCES` (or similarly environmental) failure during pipeline init
+ does **not** leave `embeddingModelState.loaded === true` pinned to the
+ hash fallback in a way that blocks a subsequent real attempt β€” a
+ follow-up call after the condition is fixed succeeds with
+ `backend: 'onnx'`.
+ - a genuinely missing dependency (no transformers package importable)
+ still memoizes the hash fallback permanently β€” this ADR must not
+ regress that case into unbounded retry loops.
+ - `generateLocalEmbedding` / `generateEmbedding`'s `backend`/`embedder`
+ fields keep reporting accurately in both the recovered and the
+ permanently-degraded case (ADR-390 regression guard).
+- Manual repro: on a system-prefix global install as root with no
+ pre-existing writable cache, run as an unprivileged user; confirm the
+ model downloads into `~/.cache/ruflo/models` (or `TRANSFORMERS_CACHE` if
+ set) rather than attempting a write under the package directory.