π The original agent harness. Deploy intelligent multi-player swarms, coordinate autonomous workflows, and build conversational AI systems. Features adaptive memory, self-learning intelligence, federation, vector RAG integration, and native Claude Code / Codex / Hermes and many more Integrated
TypeScript agentic-aiagentic-frameworkagentic-workflowagentsai-agentsai-assistantai-skillsautonomous-agentsclaudecodexdsh-pluginharnessmcp-servermulti-agentmulti-agent-systemsnpmskillsswarmswarm-intelligencetypescript
docs(adr-094): migration plan for @xenova β @huggingface/transformers
Reuvenc32ddead28f8
diff --git a/v3/docs/adr/ADR-094-xenova-to-huggingface-transformers-migration.md b/v3/docs/adr/ADR-094-xenova-to-huggingface-transformers-migration.mdnew file mode 100644index 0000000000..9c09e0c6f1--- /dev/null+++ b/v3/docs/adr/ADR-094-xenova-to-huggingface-transformers-migration.md@@ -0,0 +1,119 @@+# ADR-094: Migrate `@xenova/transformers` β `@huggingface/transformers`++**Status**: Proposed+**Date**: 2026-05-03+**Version**: targets v3.7.x+**Supersedes**: nothing+**Related**: ADR-093, npm audit CVE chain through `protobufjs`++## Context++`@xenova/[email protected]` is the legacy ONNX inference package we use for client-side embeddings (`pipeline('feature-extraction', ...)`). The package was deprecated in favor of `@huggingface/transformers` (currently 4.2.0) in late 2024. The legacy package has not received updates in over a year and pins old transitive deps that have since picked up critical CVEs:++- `protobufjs <7.5.5` β **CRITICAL β Arbitrary code execution** β comes through `onnxruntime-web` β `onnx-proto`+- The chain is: `@xenova/transformers` β `onnxruntime-web@<1.17` β `onnx-proto` β `protobufjs@<7.5.5`++`@huggingface/[email protected]+` uses modern `onnxruntime-web` versions that pull in `protobufjs >=7.5.5`, eliminating this CVE class entirely.++We had previously documented this as a deferred ADR-093 follow-up. The npm overrides we shipped in 3.6.18 cannot resolve this chain because (a) overrides only apply at the install root and (b) the version range required by `@xenova/transformers`'s manifests forbids the safer protobufjs.++### Current usage surface (verified by grep)++Only two call sites import `@xenova/transformers` in the v3 monorepo:++| File | Import | Usage |+|---|---|---|+| `v3/@claude-flow/embeddings/src/embedding-service.ts:387` | `const { pipeline } = await import('@xenova/transformers')` | Used as the ONNX backend for `feature-extraction` |+| `v3/@claude-flow/cli/src/memory/memory-initializer.ts:1539` | `const transformers = await import('@xenova/transformers').catch(() => null)` | Optional ONNX provider for embedding generation |++Both are dynamic imports wrapped in try/catch β the migration risk is bounded.++### API parity++A direct exports comparison (probed against `@xenova/[email protected]` and `@huggingface/[email protected]`):++- `@xenova/transformers`: 501 exports+- `@huggingface/transformers`: 935 exports (superset)+- **472 shared** exports, including all four we use: `pipeline`, `env`, `AutoTokenizer`, `AutoModel`+- 29 xenova-only exports (none used by us per grep)++Pipeline calls in both packages:+- xenova: `pipeline('feature-extraction', modelId)` β 384/768-dim Float32Array via `output.data`+- huggingface: same signature, same return shape (verified in HF migration guide)++## Decision++Migrate both call sites to a **provider-agnostic loader** that prefers `@huggingface/transformers`, falls back to `@xenova/transformers` for backwards compat with consumers who haven't installed the new package, and reports honest status via `embeddings_status.ruvectorStatus` (already structured per ADR-093 F5).++```ts+// New helper in @claude-flow/embeddings/src/transformers-loader.ts+export async function loadTransformersPipeline(): Promise<{+ pipeline: PipelineFn;+ source: '@huggingface/transformers' | '@xenova/transformers';+ version?: string;+} | null> {+ // Prefer the maintained successor+ try {+ const mod = await import('@huggingface/transformers');+ if (typeof mod.pipeline === 'function') {+ return { pipeline: mod.pipeline, source: '@huggingface/transformers', version: mod.version };+ }+ } catch { /* fall through */ }+ // Fall back to legacy for backwards compat+ try {+ const mod = await import('@xenova/transformers');+ if (typeof mod.pipeline === 'function') {+ return { pipeline: mod.pipeline, source: '@xenova/transformers', version: mod.version };+ }+ } catch { /* both unavailable */ }+ return null;+}+```++Update both call sites to use the loader:++```ts+// embedding-service.ts:387 (was: const { pipeline } = await import('@xenova/transformers'))+const t = await loadTransformersPipeline();+if (!t) throw new Error('No ONNX transformers package available');+this.pipeline = await t.pipeline('feature-extraction', this.modelName);+this.transformersSource = t.source;+```++### Dependency changes++`@claude-flow/embeddings/package.json`:+- Move `@xenova/transformers` from `dependencies` to `optionalDependencies` (keeps install size small for users who don't need ONNX)+- Add `@huggingface/transformers: "^4.2.0"` as a `peerDependency` (optional) and `optionalDependencies` (auto-install)+- Document in README that consumers can install either; the loader will use whichever is present++`@claude-flow/cli/package.json`: no direct change (transformers is a transitive of @claude-flow/embeddings).++### Validation plan++1. **Determinism check**: same input β identical 384-dim vectors across both packages (HF uses the same ONNX model weights for `Xenova/all-MiniLM-L6-v2`). Snapshot the first 8 dims for `"hello world"` and verify byte-identical output between packages.+2. **HNSW round-trip**: store a pattern with HF-generated embedding, search with same query β should hit with cosine similarity > 0.99.+3. **Mixed install**: install only `@huggingface/transformers` and verify embeddings work; install only `@xenova/transformers` and verify the fallback works; install neither and verify the graceful "no provider" path.+4. **Re-run the 6-agent verification swarm** against the build with the migration to confirm no regressions in `embeddings_*` MCP tools.+5. **`npm audit`** post-migration: `protobufjs <7.5.5` should drop from the prod tree.++## Consequences++**Positive:**+- Removes the critical RCE CVE chain through `protobufjs`. The 14-critical / 4-high audit count from 3.6.17 should drop substantially.+- Aligns with HuggingFace's official guidance (xenova was deprecated upstream).+- HF package is actively maintained (releases monthly vs xenova's last release in early 2025).+- API-compatible β bounded migration risk.++**Negative:**+- Bigger dependency tree (HF has more bundled features). Mitigated by making `@xenova/transformers` optional rather than removing entirely (consumers can opt for the smaller package).+- One more peer/optional dependency for `@claude-flow/embeddings` consumers to be aware of.++**Risk:**+- HF's `pipeline('feature-extraction', model)` might produce subtly different outputs vs xenova for the same model (e.g. different default normalization). Validation step #1 (byte-identical output check) catches this before merge.++## Notes++- This ADR is the migration plan. Implementation is queued for the next /loop iteration cycle.+- The `@huggingface/transformers` package itself currently audits clean against npm advisory database as of 2026-05-03.+- The provider-agnostic loader pattern (try-then-fallback) matches the existing `getQueryEmbedding` pattern in `hooks-tools.ts:3050`, so callers familiar with that codebase will recognize the shape.