Skip to content

armgit / ruflo

Public Imported from GitHub
0

🌊 The original agent harness. Deploy intelligent multi-player swarms, coordinate autonomous workflows, and build conversational AI systems. Features adaptive memory, self-learning intelligence, federation, vector RAG integration, and native Claude Code / Codex / Hermes and many more Integrated

fix+perf+docs(intelligence): self-learning audit, hardening fixes, honest perf numbers (3.10.7) (#2228)

rUv08b3623759da
diff --git a/CLAUDE.md b/CLAUDE.md
index 4805697dbd..6c219809b2 100644
--- a/CLAUDE.md
+++ b/CLAUDE.md
@@ -340,7 +340,7 @@ This project is configured with Claude Flow V3 (Anti-Drift Defaults):
- **Strategy**: specialized (clear roles, no overlap)
- **Consensus**: raft (leader maintains authoritative state)
- **Memory Backend**: hybrid (SQLite + AgentDB)
-- **HNSW Indexing**: Enabled (150x-12,500x faster)
+- **HNSW Indexing**: Enabled (measured ~1.9x at N=20k, ~3.2x–4.7x at N=5k vs brute force; ANN wins above the crossover)
- **Neural Learning**: Enabled (SONA)
## V3 CLI Commands (26 Commands, 140+ Subcommands)
@@ -352,7 +352,7 @@ This project is configured with Claude Flow V3 (Anti-Drift Defaults):
| `init` | 4 | Project initialization with wizard, presets, skills, hooks |
| `agent` | 8 | Agent lifecycle (spawn, list, status, stop, metrics, pool, health, logs) |
| `swarm` | 6 | Multi-agent swarm coordination and orchestration |
-| `memory` | 11 | AgentDB memory with vector search (150x-12,500x faster) |
+| `memory` | 11 | AgentDB memory with HNSW vector search (measured ~1.9x–4.7x vs brute force above crossover) |
| `mcp` | 9 | MCP server management and tool execution |
| `task` | 6 | Task creation, assignment, and lifecycle |
| `session` | 7 | Session state management and persistence |
@@ -374,7 +374,7 @@ This project is configured with Claude Flow V3 (Anti-Drift Defaults):
| `providers` | 5 | AI providers (list, add, remove, test, configure) |
| `plugins` | 5 | Plugin management (list, install, uninstall, enable, disable) |
| `deployment` | 5 | Deployment management (deploy, rollback, status, environments, release) |
-| `embeddings` | 4 | Vector embeddings (embed, batch, search, init) - 75x faster with agentic-flow |
+| `embeddings` | 4 | Vector embeddings (embed, batch, search, init) β€” agentic-flow ONNX backend (speedup unverified, no benchmark) |
| `claims` | 4 | Claims-based authorization (check, grant, revoke, list) |
| `migrate` | 5 | V2 to V3 migration with rollback support |
| `process` | 4 | Background process management |
@@ -759,12 +759,12 @@ npx claude-flow@v3alpha hooks worker status
## Intelligence System (RuVector)
-V3 includes the RuVector Intelligence System:
-- **SONA**: Self-Optimizing Neural Architecture (<0.05ms adaptation)
-- **MoE**: Mixture of Experts for specialized routing
-- **HNSW**: 150x-12,500x faster pattern search
+V3 includes the RuVector Intelligence System (measured numbers: see [audit](docs/reviews/intelligence-system-audit-2026-05-29.md) + [`scripts/benchmark-intelligence.mjs`](scripts/benchmark-intelligence.mjs)):
+- **SONA**: Self-Optimizing Neural Architecture (measured 0.0043ms/adapt, target <0.05ms met)
+- **MoE**: Mixture of Experts for specialized routing (gate converges β€” confidence 0.13β†’0.88 after rewards)
+- **HNSW**: measured ~1.9x at N=20k, ~3.2x–4.7x at N=5k vs brute force (recall@10 ~0.99); ANN wins above the crossover, ruvector NAPI backend (WASM not active on test host)
- **EWC++**: Elastic Weight Consolidation (prevents forgetting)
-- **Flash Attention**: 2.49x-7.47x speedup
+- **Flash Attention**: unverified β€” no benchmark exists for this claim
The 4-step intelligence pipeline:
1. **RETRIEVE** β€” Fetch relevant patterns via HNSW
@@ -779,7 +779,7 @@ Features:
- **Document chunking**: Configurable overlap and size
- **Normalization**: L2, L1, min-max, z-score
- **Hyperbolic embeddings**: Poincare ball model for hierarchical data
-- **75x faster**: With agentic-flow ONNX integration
+- **agentic-flow ONNX integration**: speedup unverified (no benchmark; backend reported `onnx`, model all-MiniLM-L6-v2, 384-dim)
- **Neural substrate**: Integration with RuVector
## Hive-Mind Consensus
@@ -799,15 +799,18 @@ Features:
## V3 Performance Targets
-| Metric | Target | Status |
-|--------|--------|--------|
-| HNSW Search | 150x-12,500x faster | **Implemented** (persistent) |
-| Memory Reduction | 50-75% with quantization | **Implemented** (3.92x Int8) |
-| SONA Integration | Pattern learning | **Implemented** (ReasoningBank) |
-| Flash Attention | 2.49x-7.47x speedup | In progress |
-| MCP Response | <100ms | Achieved |
-| CLI Startup | <500ms | Achieved |
-| SONA Adaptation | <0.05ms | In progress |
+> Source of truth: [`docs/reviews/intelligence-system-audit-2026-05-29.md`](docs/reviews/intelligence-system-audit-2026-05-29.md) + [`scripts/benchmark-intelligence.mjs`](scripts/benchmark-intelligence.mjs). Numbers below are measured unless marked "target/unverified".
+
+| Metric | Measured / Target | Status |
+|--------|-------------------|--------|
+| HNSW Search | ~1.9x at N=20k, ~3.2x–4.7x at N=5k vs brute force (recall@10 ~0.99); ties/loses below crossover | **Measured** (ruvector NAPI; 150x-12,500x NOT reproduced β€” was brute-force fallback) |
+| Int8 Quantization | 3.84x compression, reconstruction cosine 0.99999 | **Measured** |
+| RaBitQ Quantization | 32x compression, 0.60ms/query (14,760-vec index) | **Measured** |
+| SONA Adaptation | 0.0043ms/adapt (target <0.05ms met) | **Measured** |
+| MoE Gate | converges β€” confidence 0.13β†’0.88, Q 0β†’99.8 after rewards | **Measured** |
+| Flash Attention | 2.49x-7.47x | **Unverified** (no benchmark exists) |
+| MCP Response | <100ms | target |
+| CLI Startup | <500ms | target |
## Environment Variables
diff --git a/README.md b/README.md
index d1e6974743..f77136cc3e 100644
--- a/README.md
+++ b/README.md
@@ -202,7 +202,7 @@ claude mcp add ruflo -- npx ruflo@latest mcp start
| πŸ“‘ **Comms Layer** | Zero-trust federation β€” agents across machines/orgs discover, authenticate, and exchange work securely |
| 🐝 **Swarm Coordination** | Hierarchical, mesh, and adaptive topologies with consensus |
| 🧠 **Self-Learning** | SONA neural patterns, ReasoningBank, trajectory learning |
-| πŸ’Ύ **Vector Memory** | HNSW-indexed AgentDB with 150x-12,500x faster search |
+| πŸ’Ύ **Vector Memory** | HNSW-indexed AgentDB β€” measured ~1.9x faster at N=20k, ~3.2x–4.7x at N=5k vs brute force (recall@10 ~0.99); ANN wins above the crossover, ties/loses at small N. See [audit](docs/reviews/intelligence-system-audit-2026-05-29.md) + [`scripts/benchmark-intelligence.mjs`](scripts/benchmark-intelligence.mjs) |
| ⚑ **Background Workers** | 12 auto-triggered workers (audit, optimize, testgaps, etc.) |
| 🧩 **Plugin Marketplace** | 32 native Claude Code plugins + 21 npm plugins |
| πŸ”Œ **Multi-Provider** | Claude, GPT, Gemini, Cohere, Ollama with smart routing |
@@ -228,7 +228,7 @@ claude mcp add ruflo -- npx ruflo@latest mcp start
| πŸ› οΈ | **~210 tools, ready to call** | 5 server groups (Core, Intelligence, Agents, Memory, DevTools) plus an 18-tool gallery that runs entirely in your browser β€” works offline. |
| πŸ”Œ | **Bring your own MCP servers** | Click the **MCP (n)** pill in the chat input β†’ *Add Server* and paste any MCP endpoint (HTTP, SSE, or stdio). Your tools join RuFlo's native ones in the same parallel-execution flow. Run a local MCP server on `localhost:3000` and it just works. |
| ⚑ | **Tools run in parallel** | One model response can fire 4–6+ tools at the same time. The UI shows them as cards with a *Step 1 β€” 2 tools completed* badge so you can see exactly what ran. |
-| πŸ’Ύ | **Memory that sticks** | Say *"remember my favorite color is indigo"* and ask weeks later β€” RuFlo recalls it. Backed by AgentDB + HNSW vector search (β‰₯150Γ— faster than brute force). |
+| πŸ’Ύ | **Memory that sticks** | Say *"remember my favorite color is indigo"* and ask weeks later β€” RuFlo recalls it. Backed by AgentDB + HNSW vector search (measured ~1.9x–4.7x faster than brute force above the crossover, recall@10 ~0.99). |
| πŸ“˜ | **Built-in capabilities tour** | Click the question-mark icon in the sidebar β€” a "RuFlo Capabilities" modal opens with the full tool list, model strengths, architecture, and keyboard shortcuts. |
| 🏠 | **Self-hostable** | Web UI is shipped as Docker (`ruflo/src/ruvocal/Dockerfile`) with embedded Mongo. Deploy to your own Cloud Run / Fly / Kubernetes / docker-compose. The hosted [flo.ruv.io](https://flo.ruv.io/) demo is one option; running your own is fully supported. |
| πŸš€ | **Zero install to try** | Open the hosted URL, pick a model, type a question. That's the whole onboarding. |
diff --git a/docs/reviews/intelligence-system-audit-2026-05-29.md b/docs/reviews/intelligence-system-audit-2026-05-29.md
new file mode 100644
index 0000000000..08d7ded9a8
--- /dev/null
+++ b/docs/reviews/intelligence-system-audit-2026-05-29.md
@@ -0,0 +1,89 @@
+# Intelligence / Self-Learning System β€” Empirical Capability Audit
+
+**Date:** 2026-05-29 Β· **Version audited:** `@claude-flow/[email protected]` (built `dist/`) Β· **Host:** darwin-arm64, Node 22
+**Method:** 6 parallel auditors ran **real measurements** against the built `dist` exports, CLI, and MCP tool handlers β€” not documentation. Every claim is graded by evidence. Throwaway scripts were deleted; no source was modified during the audit.
+
+> **Honesty mandate.** This audit was commissioned to *measure, benchmark, and confirm* β€” including confirming where claims do **not** hold. Headline performance multipliers in `CLAUDE.md` are largely hardcoded doc strings with no benchmark behind them; several are unsubstantiated and one is fabricated at runtime. At the same time, the **core self-learning loop is genuinely real and was measured end-to-end.** Both halves of that are reported plainly below.
+
+---
+
+## TL;DR
+
+- **The learning loop is real.** A success/failure verdict on a recorded trajectory measurably and directionally changes stored pattern confidence, persisted to disk, surviving across separate processes. Q-learning routing feedback genuinely steers the next routing decision cross-process. This is not theater.
+- **The big "Γ—" numbers are not.** "HNSW 150x–12,500x" measured **1.48Γ— peak**; "Flash Attention 2.49–7.47Γ—" is **fabricated with `Math.random()` at runtime**; "75Γ— embeddings" and "RaBitQ 2.70Γ— retrieval" have **no benchmark** anywhere.
+- **Confirmed-real with honest numbers:** Int8 **3.92Γ—** memory, RaBitQ **32Γ—** memory, MoE 8-expert gating (measured convergence), SONA WASM adapt **0.0042 ms** (beats the <0.05 ms claim).
+- **One critical correctness bug:** the CLI **silently inverts negative reward** (`route feedback -r -1.0` records **+1.00**) β€” a user training against a bad agent the documented way *reinforces* it.
+
+---
+
+## Capability matrix
+
+### βœ… CONFIRMED-REAL (measured)
+
+| Capability | Evidence (measured) |
+|---|---|
+| **4-step learning loop** RETRIEVE→JUDGE→DISTILL | Success verdict pushed pattern confidence **0.906→1.0**; subsequent failure pulled **1.0→0.952**; counters persist across separate processes. Steps share one `LocalReasoningBank`+`SonaCoordinator` and feed each other. |
+| **ReasoningBank file persistence** | Cross-process: stored in proc 1 β†’ reloaded from disk in proc 2. `.claude-flow/neural/patterns.json` (+`stats.json`). |
+| **Pattern store→search roundtrip** (#2226 fix) | Holds on both direct and MCP paths (`controller:"bridge-store", impl:"real-hnsw-indexed"`). 3.10.6 fix verified end-to-end. |
+| **Memory bridge** (import_claude / bridge_status / search_unified) | Real import (14 entries, 3 projects), results carry `source:"claude-code"` attribution. Persists to `.swarm/memory.db` ns `claude-memories`. |
+| **MoE β€” 8 experts + gating** | Genuine 384β†’128β†’8 softmax MLP + REINFORCE backprop. Measured: coder expert probability **0.081β†’0.994** after 200 rewards. |
+| **SONA <0.05 ms adaptation** (WASM) | `SonaInstantWasm.instantAdapt` measured **0.00417 ms/call** (200k warmed) β€” 12Γ— under the claim. EMA + adaptive-rank schedule. |
+| **Q-learning self-improvement** (mechanism) | Q-table is read at inference and argmax'd (no static fallback). Cross-process: penalize `architect` β†’ router switches greedy pick to `researcher`, persisted to `.swarm/q-learning-model.json`. |
+| **Int8 quantization 3.92Γ—** | Measured **3.918Γ—** (1536β†’392 bytes), reconstruction cosine **0.99999**. |
+| **RaBitQ 32Γ— memory** | Real WASM (`@ruvector/[email protected]`), builds a real 1-bit index, compressionRatio **32**. Not a stub. |
+| **3-tier model routing** (hybrid) | "typo"β†’sonnet/11%, "architect distributed consensus"β†’opus/60%. Static keyword complexity + a **real persisted Beta (Thompson) bandit** that measurably shifts model choice after `model-outcome` feedback. |
+| **MCP trajectory start/end** | Real lifecycle; `end` triggers SONA learning (pattern @ 55% confidence), persists to `.swarm/memory.db` + `.swarm/sona-patterns.json`, cross-process verified. |
+
+### 🟑 PARTIAL (real core, overstated or with gaps)
+
+| Capability | What's real / what's not |
+|---|---|
+| **EWC++** | Penalty math `Β½Β·Ξ£ Fα΅’(ΞΈα΅’βˆ’ΞΈ*α΅’)Β²` is correctly implemented and runs. **But "Fisher information" is a heuristic proxy** β€” `Fα΅’ = \|wα΅’\|Β·Ξ»` (ruvllm) / `embeddingα΅’Β²` (TS), not gradient curvature E[gΒ²]. `forgettingRate = 1βˆ’e^(βˆ’tasksΒ·0.1)` is a label, not a measurement. |
+| **MicroLoRA** | JS `LoraAdapter` forward/backward is real low-rank math. **The WASM adapter the MCP tools actually call is inert** β€” output L1 delta **0.000000** after 5000 adapts (B stays zero; gradients accumulate but never flush). |
+| **CONSOLIDATE (MCP path)** | Real `EWCConsolidator`, but MCP `trajectory-end` feeds it a **synthetic `Math.sin()` gradient** (`hooks-tools.js:2481`), not the embedding-derived one the library path uses. Fisher file didn't persist in a clean run. |
+| **ONNX embeddings** | 384-dim shape correct, but on this host `sharp` native build fails β†’ **silent fallback to mock embeddings still labeled `Xenova/all-MiniLM-L6-v2`**. Synonyms scored **βˆ’0.988**, unrelated text **+0.775**. Operator cannot tell mock from real β€” an observability defect regardless of environment. |
+| **Trajectory tracking** | MCP tools real & persist; **CLI `hooks intelligence trajectory-*` subcommands are no-op stubs** (positional arg ignored β†’ always render the status dashboard). MCP `step` is **in-memory only** between start/end (lost if server restarts mid-trajectory). |
+| **neural train** | Real epoch loop with contrastive loss + LoRA backend (JS fallback does real rank-2 algebra). **But emits zero output in non-TTY** β€” work happens invisibly. |
+
+### ❌ UNSUBSTANTIATED / FABRICATED
+
+| Claim | Reality |
+|---|---|
+| **HNSW "150x–12,500x faster"** | Measured **peak 1.48Γ— at N=20k**; *slower* than brute force below Nβ‰ˆ5k. The multipliers are hardcoded doc strings; the benchmark command's "recall" is a hardcoded `0.99` constant. Baseline undefined. (A real ANN index exists β€” only the *magnitude* and missing baseline are the problem; 150Γ— would need millions of vectors.) |
+| **Flash Attention "2.49x–7.47x"** | **Fabricated at runtime:** `attention-coordinator.ts:972` β†’ `flashSpeedup = 2.49 + Math.random()*4.98`. A correct tiled kernel exists (`blockAttention`, RMSE 2.6e-8) but measures only **~1.1Γ—**; the *default* path is a lossy top-K sparse approximation (12% of keys, RMSE 0.17) measuring **0.77×–3.62Γ—**, never 7.47Γ—. |
+| **RaBitQ "2.70x retrieval"** | Not measured anywhere in source. (The 32Γ— *memory* claim is confirmed; the retrieval-speed number is not.) |
+| **Embeddings "75x faster"** | Doc literal, no baseline, never measured. |
+
+---
+
+## Bugs & defects found (prioritized)
+
+| # | Sev | Defect | Location |
+|---|-----|--------|----------|
+| 1 | πŸ”΄ **Critical** | **CLI inverts negative reward.** `route feedback -r -1.0` (and `--reward -1.0`) parses as **+1.00** β€” negative feedback reinforces the bad agent. Only `--reward=-1.0` (equals form) preserves the sign; the command's own help example is broken. | `src/commands/route.ts` flag parser |
+| 2 | 🟠 High | **Fabricated speedup metric** reported as real telemetry via `Math.random()`. | `v3/@claude-flow/swarm/src/attention-coordinator.ts:972` |
+| 3 | 🟠 High | **Silent mock-embedding fallback mislabeled as the real ONNX model** β€” no way to distinguish mock from real output. | `memory-initializer.ts` embedding path; agentdb `EmbeddingService` |
+| 4 | 🟑 Med | **`hooks_intelligence_learn` is cosmetic** β€” reads/echoes stats; does not run a learning cycle despite its name. | `hooks-tools.js` (~:2920) |
+| 5 | 🟑 Med | **MCP `trajectory-end` consolidation uses a synthetic `Math.sin()` gradient**, not the trajectory's real embeddings. | `hooks-tools.js:2481` |
+| 6 | 🟑 Med | **CLI `hooks intelligence trajectory-*` subcommands are no-op stubs** (render the status dashboard). | `src/commands/hooks.js:1758` |
+| 7 | 🟑 Med | **WASM MicroLoRA `apply()` is inert** (B never flushed β†’ output unchanged). | `@ruvector/ruvllm-wasm`; `ruvllm-wasm.ts` |
+| 8 | 🟒 Low | **Benchmark "recall" is a hardcoded `0.99`** constant, not measured. | `commands/ruvector/benchmark.js:377` |
+| 9 | 🟒 Low | **`neural train` / `hooks intelligence` emit nothing in non-TTY**; documented `node dist/src/index.js` entry prints nothing (use `bin/cli.js`). | `commands/neural.js`, `commands/hooks.js` |
+| 10 | 🟒 Low | Q-router: stale route cache hides learning until 50 updates; `--explore false` is ignored; Beta bandit priors are global-per-model, not per-task. | `q-learning-router.ts`, `route.ts`, `model-router.ts` |
+
+---
+
+## Honest bottom line
+
+The self-learning system is **substantially real where it counts**: there is a genuine closed loop in which task outcomes update persisted pattern confidence and routing Q-values, and those persisted values demonstrably change subsequent behavior across processes. MoE gating, SONA WASM latency, Int8/RaBitQ compression, and the memory bridge all hold up under direct measurement. This is a real reinforcement-style memory system, not a faΓ§ade.
+
+What does **not** hold up is the **performance-multiplier marketing**: the HNSW and Flash-Attention speedups are unsubstantiated (and one is literally randomized at runtime), and the headline embedding/quantization "Γ—N faster" figures have no benchmark behind them. Several capabilities are real algorithms that are **implemented but inert or disconnected** (WASM MicroLoRA, MCP consolidation gradient, the `learn` tool, CLI trajectory subcommands). And one genuine **correctness bug** (negative-reward inversion) means the documented self-improvement path can train the system *backwards*.
+
+**Recommended next steps** (not yet applied β€” this branch is the audit only):
+1. **Fix #1 (negative-reward inversion)** β€” highest priority; it actively corrupts learning. (Follow-up to #2222.)
+2. **Remove the `Math.random()` speedup (#2)** β€” replace with a real measurement or drop the metric; it is a credibility liability.
+3. **Make the mock-embedding fallback observable (#3)** β€” surface `embeddingBackend: "mock"|"onnx"` everywhere the model name is reported.
+4. **Correct the `CLAUDE.md` / perf-table claims** to measured values (Int8 3.92Γ—, RaBitQ 32Γ— memory, SONA 0.004 ms confirmed; HNSW/Flash/embeddings marked "unverified / approximate" with the real numbers).
+5. Wire or remove the inert pieces (#4–#7) so named capabilities are either real or not advertised.
+
+*Per-subsystem raw evidence is preserved in the audit run; load-bearing file:line references are inline above.*
diff --git a/package-lock.json b/package-lock.json
index 0758cb1592..77b586a168 100644
--- a/package-lock.json
+++ b/package-lock.json
@@ -1,12 +1,12 @@
{
"name": "claude-flow",
- "version": "3.10.6",
+ "version": "3.10.7",
"lockfileVersion": 3,
"requires": true,
"packages": {
"": {
"name": "claude-flow",
- "version": "3.10.6",
+ "version": "3.10.7",
"license": "MIT",
"dependencies": {
"@claude-flow/cli-core": "^3.7.0-alpha.5",
diff --git a/package.json b/package.json
index f3e248daa6..fa2f5f7af2 100644
--- a/package.json
+++ b/package.json
@@ -1,6 +1,6 @@
{
"name": "claude-flow",
- "version": "3.10.6",
+ "version": "3.10.7",
"description": "Ruflo - Enterprise AI agent orchestration for Claude Code. Deploy 60+ specialized agents in coordinated swarms with self-learning, fault-tolerant consensus, vector memory, and MCP integration",
"main": "dist/index.js",
"type": "module",
diff --git a/ruflo/README.md b/ruflo/README.md
index 0b7eee30a0..d1e6974743 100644
--- a/ruflo/README.md
+++ b/ruflo/README.md
@@ -1,6 +1,6 @@
<div align="center">
-[![Ruflo Banner](ruflo/assets/ruflo-small.jpeg)](https://flo.ruv.io/)
+[![Ruflo Banner](ruflo/assets/ruflo-small.jpeg)](https://cognitum.one/agentic-engineering)
[![Try the UI Beta β€” flo.ruv.io](https://img.shields.io/badge/_Try_the_UI_Beta-flo.ruv.io-6366f1?style=for-the-badge&logoColor=white&logo=svelte)](https://flo.ruv.io/)
[![Goal Planner β€” goal.ruv.io](https://img.shields.io/badge/_Goal_Planner-goal.ruv.io-8b5cf6?style=for-the-badge&logoColor=white&logo=react)](https://goal.ruv.io/)
@@ -374,13 +374,17 @@ User --> Claude Code / CLI
## Documentation
-Three docs for three audiences:
+Four docs for four audiences:
| Doc | When to read it |
|-----|-----------------|
| **[Status](docs/STATUS.md)** | See what currently works β€” capability counts, test baselines, recent fixes, what's next. The *is-it-ready* doc. |
| **[User Guide](docs/USERGUIDE.md)** | Daily reference β€” every command, every config flag, every plugin. The *how-do-I* doc. |
+| **[Benchmarks](https://gist.github.com/ruvnet/298f8c668c8859b369f91734a0e9cbbe)** | v3.8.0 SOTA matrix vs LangGraph / AutoGen / CrewAI on darwin-arm64 + linux-x64. ruflo wins cold start, single turn, RSS by 1.3×–1953Γ—. The *is-it-fast* doc. |
| **[Verification](verification.md)** | Cryptographically prove your installed bytes match the signed witness β€” `ruflo verify`. The *trust-but-verify* doc. |
+| **[Team Gateway Checklist](docs/TEAM-GATEWAY-CHECKLIST.md)** | Before-merge gates, dual-mode handoff, memory namespace sharing, and witness manifest entry per merge. The *safer-team-workflows* doc. |
+
+Benchmark internals (for reproduction): [`sota-workload-spec.md`](https://github.com/ruvnet/ruflo/blob/perf/sota-comparator-benchmarks/docs/benchmarks/sota-workload-spec.md) Β· [`SOTA-PROGRESS.md`](https://github.com/ruvnet/ruflo/blob/perf/sota-comparator-benchmarks/docs/benchmarks/SOTA-PROGRESS.md) Β· [raw matrix JSON: darwin](https://github.com/ruvnet/ruflo/blob/perf/sota-comparator-benchmarks/docs/benchmarks/sota-matrix.json) Β· [linux](https://github.com/ruvnet/ruflo/blob/perf/sota-comparator-benchmarks/docs/benchmarks/sota-matrix-linux.json)
User Guide section index:
diff --git a/ruflo/package.json b/ruflo/package.json
index cc8e86c847..8b581195f7 100644
--- a/ruflo/package.json
+++ b/ruflo/package.json
@@ -1,6 +1,6 @@
{
"name": "ruflo",
- "version": "3.10.6",
+ "version": "3.10.7",
"description": "Ruflo - Enterprise AI agent orchestration platform. Deploy 60+ specialized agents in coordinated swarms with self-learning, fault-tolerant consensus, vector memory, and MCP integration",
"main": "bin/ruflo.js",
"type": "module",
diff --git a/scripts/benchmark-intelligence.mjs b/scripts/benchmark-intelligence.mjs
new file mode 100755
index 0000000000..18ba56e567
--- /dev/null
+++ b/scripts/benchmark-intelligence.mjs
@@ -0,0 +1,635 @@
+#!/usr/bin/env node
+/**
+ * benchmark-intelligence.mjs β€” Real, reusable benchmark harness for the
+ * RuVector / AgentDB intelligence stack.
+ *
+ * Measures, on the machine it runs on, against the BUILT exports under
+ * v3/@claude-flow/cli/dist/src/...
+ * (never against source, never hardcoded):
+ *
+ * 1. HNSW search vs in-process brute-force cosine baseline
+ * at N = 1000, 5000, 20000, 50000:
+ * - per-query ms (HNSW + brute force)
+ * - speedup ratio (brute / hnsw)
+ * - recall@10 (HNSW results vs exact top-10)
+ * 2. Int8 quantization: measured compression ratio + reconstruction cosine.
+ * 3. RaBitQ: memory compression ratio (+ retrieval speed if a populated
+ * index is feasible, else null with a reason).
+ * 4. SONA WASM adapt latency (ms/call, warmed).
+ * 5. MoE gate: confirm the gate LEARNS (probability/Q shift after rewards).
+ * 6. Embedding backend actually in use (onnx vs mock) β€” honest.
+ *
+ * DESIGN NOTES
+ * ------------
+ * - All vectors are generated with a seeded deterministic RNG so the run is
+ * reproducible and safe to re-run. We deliberately do NOT route HNSW/quant
+ * benchmarks through the embedding backend: that backend can be mock on a
+ * given machine, which would make the structural benchmarks non-deterministic
+ * and conflate two separate measurements. The embedding backend is instead
+ * reported HONESTLY as its own item (#6).
+ * - Every number printed/emitted comes from a measurement in THIS process.
+ * Unmeasurable items are emitted as `null` with a `reason` string.
+ * - Exit code is 0 on success (a benchmark being unmeasurable is not a failure;
+ * only an unexpected crash is). The script prints a markdown table to stdout
+ * and writes a machine-readable JSON object after a `===BENCH_JSON===` marker.
+ *
+ * USAGE
+ * node scripts/benchmark-intelligence.mjs # default sizes
+ * node scripts/benchmark-intelligence.mjs --sizes 1000,5000
+ * node scripts/benchmark-intelligence.mjs --queries 50 --dims 384
+ * node scripts/benchmark-intelligence.mjs --json-only
+ *
+ * Created for the ruflo intelligence stack. Co-Authored-By: RuFlo <[email protected]>
+ */
+
+import { fileURLToPath } from 'node:url';
+import path from 'node:path';
+
+const __dirname = path.dirname(fileURLToPath(import.meta.url));
+const REPO_ROOT = path.resolve(__dirname, '..');
+const DIST = path.join(REPO_ROOT, 'v3', '@claude-flow', 'cli', 'dist', 'src');
+
+// ----------------------------------------------------------------------------
+// CLI args
+// ----------------------------------------------------------------------------
+function parseArgs(argv) {
+ const args = { sizes: [1000, 5000, 20000, 50000], queries: 30, dims: 384, jsonOnly: false };
+ for (let i = 2; i < argv.length; i++) {
+ const a = argv[i];
+ if (a === '--sizes') args.sizes = argv[++i].split(',').map((s) => parseInt(s.trim(), 10)).filter(Boolean);
+ else if (a === '--queries') args.queries = parseInt(argv[++i], 10);
+ else if (a === '--dims') args.dims = parseInt(argv[++i], 10);
+ else if (a === '--json-only') args.jsonOnly = true;
+ else if (a === '--help' || a === '-h') {
+ console.log('Usage: node scripts/benchmark-intelligence.mjs [--sizes N,N] [--queries K] [--dims D] [--json-only]');
+ process.exit(0);
+ }
+ }
+ return args;
+}
+const ARGS = parseArgs(process.argv);
+
+const log = (...m) => { if (!ARGS.jsonOnly) console.log(...m); };
+
+// ----------------------------------------------------------------------------
+// Deterministic RNG (mulberry32) + unit-vector generator
+// ----------------------------------------------------------------------------
+function mulberry32(seed) {
+ let a = seed >>> 0;
+ return function () {
+ a |= 0; a = (a + 0x6d2b79f5) | 0;
+ let t = Math.imul(a ^ (a >>> 15), 1 | a);
+ t = (t + Math.imul(t ^ (t >>> 7), 61 | t)) ^ t;
+ return ((t ^ (t >>> 14)) >>> 0) / 4294967296;
+ };
+}
+
+/** Generate a clustered set of unit-length Float32 vectors (deterministic). */
+function makeDataset(n, dims, seed) {
+ const rng = mulberry32(seed);
+ // A handful of cluster centroids so nearest-neighbour structure is non-trivial.
+ const numClusters = Math.max(8, Math.round(Math.sqrt(n) / 4));
+ const centroids = [];
+ for (let c = 0; c < numClusters; c++) {
+ const v = new Float32Array(dims);
+ for (let d = 0; d < dims; d++) v[d] = rng() * 2 - 1;
+ normalize(v);
+ centroids.push(v);
+ }
+ const vectors = new Array(n);
+ for (let i = 0; i < n; i++) {
+ const base = centroids[i % numClusters];
+ const v = new Float32Array(dims);
+ for (let d = 0; d < dims; d++) v[d] = base[d] + (rng() * 2 - 1) * 0.35; // jitter around centroid
+ normalize(v);
+ vectors[i] = v;
+ }
+ return vectors;
+}
+
+function normalize(v) {
+ let s = 0;
+ for (let i = 0; i < v.length; i++) s += v[i] * v[i];
+ const inv = s > 0 ? 1 / Math.sqrt(s) : 0;
+ for (let i = 0; i < v.length; i++) v[i] *= inv;
+ return v;
+}
+
+function cosine(a, b) {
+ let dot = 0, na = 0, nb = 0;
+ for (let i = 0; i < a.length; i++) { dot += a[i] * b[i]; na += a[i] * a[i]; nb += b[i] * b[i]; }
+ const denom = Math.sqrt(na) * Math.sqrt(nb);
+ return denom > 0 ? dot / denom : 0;
+}
+
+/** Exact top-k by cosine (brute force) β€” returns array of ids. */
+function bruteTopK(vectors, query, k) {
+ const scored = new Array(vectors.length);
+ for (let i = 0; i < vectors.length; i++) scored[i] = [i, cosine(query, vectors[i])];
+ scored.sort((x, y) => y[1] - x[1]);
+ return scored.slice(0, k).map((s) => s[0]);
+}
+
+const now = () => performance.now();
+const round = (x, d = 4) => (x == null || Number.isNaN(x) ? null : Number(x.toFixed(d)));
+
+// ----------------------------------------------------------------------------
+// 1. HNSW vs brute-force
+// ----------------------------------------------------------------------------
+async function benchHnsw() {
+ const out = { unit: 'ms/query', backend: null, byN: {}, note: '' };
+ let createVectorDB, getStatus, loadRuVector;
+ try {
+ ({ createVectorDB, getStatus, loadRuVector } = await import(path.join(DIST, 'ruvector', 'vector-db.js')));
+ await loadRuVector();
+ out.backend = getStatus();
+ } catch (e) {
+ out.error = `failed to load ruvector vector-db: ${e.message}`;
+ return out;
+ }
+
+ // ruvector createVectorDB enforces 384 dims on this build; honour that.
+ const dims = ARGS.dims;
+ const K = 10;
+
+ for (const N of ARGS.sizes) {
+ const entry = { n: N };
+ let dataset, queries;
+ try {
+ dataset = makeDataset(N, dims, 1234 + N);
+ // Queries: reuse a deterministic subset perturbed slightly so they are
+ // near (but not identical to) indexed points.
+ const qRng = mulberry32(99 + N);
+ queries = [];
+ for (let q = 0; q < ARGS.queries; q++) {
+ const src = dataset[Math.floor(qRng() * N)];
+ const v = new Float32Array(dims);
+ for (let d = 0; d < dims; d++) v[d] = src[d] + (qRng() * 2 - 1) * 0.05;
+ queries.push(normalize(v));
+ }
+ } catch (e) {
+ entry.error = `dataset build failed: ${e.message}`;
+ out.byN[N] = entry;
+ continue;
+ }
+
+ // --- Build HNSW index ---
+ let db, buildMs;
+ try {
+ const t0 = now();
+ db = await createVectorDB(dims);
+ for (let i = 0; i < N; i++) await db.insert(dataset[i], String(i));
+ buildMs = now() - t0;
+ } catch (e) {
+ entry.error = `hnsw build failed: ${e.message}`;
+ out.byN[N] = entry;
+ continue;
+ }
+ entry.buildMs = round(buildMs, 2);
+ entry.indexSize = await db.size();
+
+ // --- HNSW query timing + recall ---
+ // Warm the index first: the native NAPI search path has per-call marshalling
+ // + JIT + page-fault overhead that dominates the FIRST touch of each query.
+ // The brute-force baseline below runs in an already-hot JS loop, so timing
+ // HNSW cold would compare a cold path against a warm one. We discard one full
+ // pass so both sides are measured at steady state β€” this measures search
+ // performance, not first-call overhead.
+ for (const q of queries) await db.search(q, K);
+
+ let hnswTotal = 0, recallHits = 0, recallTotal = 0;
+ const exactByQuery = queries.map((q) => bruteTopK(dataset, q, K)); // ground truth
+ for (let qi = 0; qi < queries.length; qi++) {
+ const q = queries[qi];
+ const t0 = now();
+ const res = await db.search(q, K);
+ hnswTotal += now() - t0;
+ const ids = (res || []).map((r) => parseInt(r.id, 10));
+ const truth = new Set(exactByQuery[qi]);
+ for (const id of ids) if (truth.has(id)) recallHits++;
+ recallTotal += K;
+ }
+ const hnswPerQuery = hnswTotal / queries.length;
+
+ // --- Brute-force baseline timing (independent of ground-truth precompute) ---
+ let bruteTotal = 0;
+ for (const q of queries) {
+ const t0 = now();
+ bruteTopK(dataset, q, K);
+ bruteTotal += now() - t0;
+ }
+ const brutePerQuery = bruteTotal / queries.length;
+
+ entry.hnswMsPerQuery = round(hnswPerQuery, 5);
+ entry.bruteMsPerQuery = round(brutePerQuery, 5);
+ entry.speedup = round(hnswPerQuery > 0 ? brutePerQuery / hnswPerQuery : null, 2);
+ entry.recallAt10 = round(recallTotal > 0 ? recallHits / recallTotal : null, 4);
+ out.byN[N] = entry;
+
+ if (db.clear) await db.clear();
+ log(` HNSW N=${N}: build=${entry.buildMs}ms hnsw=${entry.hnswMsPerQuery}ms brute=${entry.bruteMsPerQuery}ms speedup=${entry.speedup}x recall@10=${entry.recallAt10}`);
+ }
+ return out;
+}
+
+// ----------------------------------------------------------------------------
+// 2. Int8 quantization
+// ----------------------------------------------------------------------------
+async function benchInt8() {
+ const out = {};
+ let encodeEmbedding, decodeEmbedding, encodedByteSize;
+ try {
+ ({ encodeEmbedding, decodeEmbedding, encodedByteSize } = await import(path.join(DIST, 'memory', 'embedding-quantization.js')));
+ } catch (e) {
+ return { error: `failed to load embedding-quantization: ${e.message}` };
+ }
+ const dims = ARGS.dims;
+ const SAMPLES = 200;
+ const rng = mulberry32(4242);
+ let cosSum = 0;
+ let rawBytes = 0, encBase64Bytes = 0, quantRawBytes = 0;
+ for (let s = 0; s < SAMPLES; s++) {
+ const v = new Float32Array(dims);
+ for (let d = 0; d < dims; d++) v[d] = rng() * 2 - 1;
+ normalize(v);
+ const encoded = encodeEmbedding(v); // "inline:<base64>"
+ const decoded = decodeEmbedding(encoded); // Float32Array | null
+ if (!decoded) continue;
+ cosSum += cosine(v, decoded);
+ rawBytes += dims * 4; // float32 source bytes
+ // measured base64 transport payload (the blob, sans "inline:" prefix)
+ const b64 = encoded.startsWith('inline:') ? encoded.slice(7) : encoded;
+ encBase64Bytes += b64.length; // chars == bytes for base64 ASCII
+ // measured raw quantized byte count: decode the base64 to get its true
+ // pre-encoding size (header + 1 byte/dim). We do NOT assume the format.
+ quantRawBytes += Buffer.from(b64, 'base64').length;
+ }
+ out.dims = dims;
+ out.samples = SAMPLES;
+ out.reconstructionCosine = round(cosSum / SAMPLES, 6);
+ out.rawBytesPerVec = round(rawBytes / SAMPLES, 1); // 1536 for 384-d f32
+ out.quantizedRawBytesPerVec = round(quantRawBytes / SAMPLES, 1); // ~400 (header+int8)
+ out.base64BytesPerVec = round(encBase64Bytes / SAMPLES, 1); // ~536 transport
+ out.encodedByteSizeReported = encodedByteSize ? encodedByteSize(dims) : null;
+ // Honest int8 compression = float32 source bytes / int8 quantized bytes.
+ out.compressionRatioInt8 = round(rawBytes / quantRawBytes, 3);
+ // Transport (base64) ratio β€” what actually lands in the embedding_ref column.
+ out.compressionRatioBase64 = round(rawBytes / encBase64Bytes, 3);
+ log(` Int8: reconstructionCosine=${out.reconstructionCosine} compression(int8)=${out.compressionRatioInt8}x compression(base64)=${out.compressionRatioBase64}x`);
+ return out;
+}
+
+// ----------------------------------------------------------------------------
+// 3. RaBitQ
+// ----------------------------------------------------------------------------
+async function benchRabitq() {
+ const out = {};
+ let buildRabitqIndex, getRabitqStatus, searchRabitq;
+ try {
+ ({ buildRabitqIndex, getRabitqStatus, searchRabitq } = await import(path.join(DIST, 'memory', 'rabitq-index.js')));
+ } catch (e) {
+ return { error: `failed to load rabitq-index: ${e.message}` };
+ }
+ // Memory compression ratio is a structural property: 1-bit packing of a
+ // float32 vector = 32x. The WASM module reports the measured ratio at build
+ // time. buildRabitqIndex pulls from a SQLite memory DB β€” if no populated DB
+ // exists we cannot measure retrieval speed; report that honestly.
+ const status = getRabitqStatus();
+ out.available = !!status.available;
+ out.statusBefore = status;
+
+ let build = null;
+ try {
+ build = await buildRabitqIndex({ dimensions: ARGS.dims });
+ } catch (e) {
+ out.buildError = e.message;
+ }
+ out.build = build;
+
+ if (build && build.success && build.vectorCount > 0) {
+ out.compressionRatio = round(build.compressionRatio, 3);
+ out.buildTimeMs = round(build.buildTimeMs, 3);
+ // Retrieval timing on the populated index.
+ try {
+ const rng = mulberry32(7);
+ const q = Array.from({ length: ARGS.dims }, () => rng() * 2 - 1);
+ const reps = 20;
+ const t0 = now();
+ for (let i = 0; i < reps; i++) await searchRabitq(q, { k: 10 });
+ out.searchMsPerQuery = round((now() - t0) / reps, 5);
+ } catch (e) {
+ out.searchMsPerQuery = null;
+ out.searchNote = `retrieval not measured: ${e.message}`;
+ }
+ } else {
+ // No populated SQLite vector store on this machine β€” compression ratio is
+ // reported from status/build if the WASM module surfaced it, else from the
+ // documented 1-bit packing invariant is NOT assumed; we mark it null.
+ out.compressionRatio = build && typeof build.compressionRatio === 'number' && build.compressionRatio > 0
+ ? round(build.compressionRatio, 3)
+ : (typeof status.compressionRatio === 'number' && status.compressionRatio > 0 ? round(status.compressionRatio, 3) : null);
+ out.searchMsPerQuery = null;
+ out.searchNote = 'not measured: no populated RaBitQ/SQLite index available on this machine (build returned vectorCount=0)';
+ }
+ log(` RaBitQ: available=${out.available} compressionRatio=${out.compressionRatio} searchMsPerQuery=${out.searchMsPerQuery ?? 'null'}`);
+ return out;
+}
+
+// ----------------------------------------------------------------------------
+// 4. SONA WASM adapt latency (warmed)
+// ----------------------------------------------------------------------------
+async function benchSona() {
+ const out = {};
+ let isRuvllmWasmAvailable, initRuvllmWasm, createSonaInstant;
+ try {
+ ({ isRuvllmWasmAvailable, initRuvllmWasm, createSonaInstant } = await import(path.join(DIST, 'ruvector', 'ruvllm-wasm.js')));
+ } catch (e) {
+ return { error: `failed to load ruvllm-wasm: ${e.message}` };
+ }
+ const available = await isRuvllmWasmAvailable();
+ out.wasmAvailable = available;
+ if (!available) {
+ out.adaptMsPerCall = null;
+ out.note = 'not measured: @ruvector/ruvllm-wasm not available on this machine';
+ log(' SONA: WASM not available');
+ return out;
+ }
+ await initRuvllmWasm();
+ const sona = await createSonaInstant({ hiddenDim: 64 });
+ // Warm-up (JIT + WASM page faults).
+ for (let i = 0; i < 1000; i++) sona.adapt(0.7 + (i % 3) * 0.1);
+ // Measured loop.
+ const ITER = 20000;
+ const rng = mulberry32(11);
+ const t0 = now();
+ for (let i = 0; i < ITER; i++) sona.adapt(rng());
+ const totalMs = now() - t0;
+ out.iterations = ITER;
+ out.totalMs = round(totalMs, 3);
+ out.adaptMsPerCall = round(totalMs / ITER, 6);
+ out.targetMet_0_05ms = out.adaptMsPerCall != null ? out.adaptMsPerCall < 0.05 : null;
+ if (sona.reset) sona.reset();
+ log(` SONA: ${out.adaptMsPerCall} ms/adapt-call (warmed, ${ITER} iters)`);
+ return out;
+}
+
+// ----------------------------------------------------------------------------
+// 5. MoE gate learns (Q-value / probability shift after rewards)
+// ----------------------------------------------------------------------------
+async function benchMoeGate() {
+ const out = {};
+ let createQLearningRouter;
+ try {
+ ({ createQLearningRouter } = await import(path.join(DIST, 'ruvector', 'q-learning-router.js')));
+ } catch (e) {
+ return { error: `failed to load q-learning-router: ${e.message}` };
+ }
+ let router;
+ try {
+ router = createQLearningRouter({ saveInterval: 1e9 }); // never auto-persist during bench
+ } catch (e) {
+ return { error: `failed to construct router: ${e.message}` };
+ }
+
+ const task = 'optimize the database query performance bottleneck';
+ // Discover available actions from an initial (greedy) decision.
+ const before = router.route(task, false);
+ const actions = (before.alternatives && before.alternatives.length)
+ ? before.alternatives.map((a) => a.route)
+ : [before.route];
+ if (actions.length < 2) {
+ out.note = 'gate exposed fewer than 2 actions; cannot demonstrate competitive shift';
+ }
+ // Pick a "good" action and reward it repeatedly; pick a "bad" action and
+ // penalise it. We then verify the gate's Q-value / probability for the good
+ // action rose relative to before.
+ const good = before.route; // initial greedy pick (argmax of qValues)
+ const bad = actions.find((a) => a !== good) || good;
+
+ // The greedy decision's chosen route is the argmax, so its Q is the max of
+ // the qValues vector. We read the Q of whatever route is greedily chosen
+ // before and after training. After rewarding `good`, a learning gate should
+ // (a) keep choosing `good` and (b) have raised its Q for that context.
+ const maxQ = (decision) => (decision.qValues && decision.qValues.length ? Math.max(...decision.qValues) : null);
+ // Q assigned specifically to the `good` route: if `good` is the greedy pick
+ // it equals maxQ; otherwise it's its score in alternatives.
+ const qForRoute = (decision, route) => {
+ if (decision.route === route) return maxQ(decision);
+ const alt = (decision.alternatives || []).find((a) => a.route === route);
+ return alt ? alt.score : null;
+ };
+
+ const beforeGoodQ = qForRoute(before, good);
+ const beforeConf = before.confidence;
+
+ // Train: reward `good`, penalise `bad`.
+ const REWARDS = 200;
+ for (let i = 0; i < REWARDS; i++) {
+ router.update(task, good, 1.0, task);
+ if (bad !== good) router.update(task, bad, -1.0, task);
+ }
+
+ const after = router.route(task, false); // greedy after training
+ const afterGoodQ = qForRoute(after, good);
+ const afterConf = after.confidence;
+ out.afterGreedyRoute = after.route;
+ out.goodStillChosen = after.route === good;
+
+ out.actionsObserved = actions.length;
+ out.goodAction = good;
+ out.badAction = bad;
+ out.rewardsApplied = REWARDS;
+ out.beforeGoodQ = round(beforeGoodQ, 5);
+ out.afterGoodQ = round(afterGoodQ, 5);
+ out.qShift = (beforeGoodQ != null && afterGoodQ != null) ? round(afterGoodQ - beforeGoodQ, 5) : null;
+ out.beforeConfidence = round(beforeConf, 5);
+ out.afterConfidence = round(afterConf, 5);
+ out.confidenceShift = (beforeConf != null && afterConf != null) ? round(afterConf - beforeConf, 5) : null;
+ // The gate "learns" if reward changed its internal valuation in the rewarded
+ // direction (Q rose) OR its confidence in the greedy pick rose.
+ out.gateLearned = (out.qShift != null && out.qShift > 0) || (out.confidenceShift != null && out.confidenceShift > 0);
+ const stats = router.getStats ? router.getStats() : null;
+ out.routerStats = stats;
+ log(` MoE gate: qShift=${out.qShift} confShift=${out.confidenceShift} learned=${out.gateLearned}`);
+ return out;
+}
+
+// ----------------------------------------------------------------------------
+// 6. Embedding backend honesty
+// ----------------------------------------------------------------------------
+async function benchEmbeddingBackend() {
+ const out = {};
+ let generateEmbedding;
+ try {
+ ({ generateEmbedding } = await import(path.join(DIST, 'memory', 'memory-initializer.js')));
+ } catch (e) {
+ return { error: `failed to load memory-initializer: ${e.message}` };
+ }
+ try {
+ const r = await generateEmbedding('benchmark probe: authentication and database optimization patterns');
+ out.backend = r.backend; // 'onnx' | 'mock' β€” the authoritative signal
+ out.model = r.model;
+ out.dimensions = r.dimensions;
+ out.honest = `backend=${r.backend} (model string '${r.model}' is reported regardless of backend; backend field is authoritative)`;
+ } catch (e) {
+ out.backend = null;
+ out.note = `not measured: ${e.message}`;
+ }
+ log(` Embedding backend: ${out.backend} (model=${out.model}, dims=${out.dimensions})`);
+ return out;
+}
+
+// ----------------------------------------------------------------------------
+// Markdown report
+// ----------------------------------------------------------------------------
+function printMarkdown(results) {
+ const lines = [];
+ lines.push('');
+ lines.push('## Intelligence Benchmark β€” Measured Results');
+ lines.push('');
+ lines.push(`- Host: ${process.platform}/${process.arch}, Node ${process.version}`);
+ lines.push(`- dist: ${path.relative(REPO_ROOT, DIST)}`);
+ lines.push(`- dims=${ARGS.dims}, queries/size=${ARGS.queries}`);
+ lines.push('');
+
+ // HNSW table
+ lines.push('### 1. HNSW vs brute-force cosine');
+ const h = results.hnsw;
+ if (h && h.byN && !h.error) {
+ lines.push(`backend: \`${JSON.stringify(h.backend)}\``);
+ lines.push('');
+ lines.push('| N | build ms | HNSW ms/q | brute ms/q | speedup | recall@10 |');
+ lines.push('|--:|--:|--:|--:|--:|--:|');
+ for (const N of ARGS.sizes) {
+ const e = h.byN[N];
+ if (!e) continue;
+ if (e.error) { lines.push(`| ${N} | error: ${e.error} | | | | |`); continue; }
+ lines.push(`| ${N} | ${e.buildMs} | ${e.hnswMsPerQuery} | ${e.bruteMsPerQuery} | ${e.speedup}x | ${e.recallAt10} |`);
+ }
+ } else {
+ lines.push(`error: ${h?.error ?? 'no data'}`);
+ }
+ lines.push('');
+
+ // Int8
+ lines.push('### 2. Int8 quantization');
+ const q = results.int8;
+ if (q && !q.error) {
+ lines.push('| metric | value |');
+ lines.push('|--|--:|');
+ lines.push(`| reconstruction cosine | ${q.reconstructionCosine} |`);
+ lines.push(`| compression (int8 quantized) | ${q.compressionRatioInt8}x |`);
+ lines.push(`| compression (base64 transport) | ${q.compressionRatioBase64}x |`);
+ lines.push(`| f32 source bytes/vec | ${q.rawBytesPerVec} |`);
+ lines.push(`| int8 quantized bytes/vec | ${q.quantizedRawBytesPerVec} |`);
+ lines.push(`| base64 transport bytes/vec | ${q.base64BytesPerVec} |`);
+ } else { lines.push(`error: ${q?.error ?? 'no data'}`); }
+ lines.push('');
+
+ // RaBitQ
+ lines.push('### 3. RaBitQ');
+ const rb = results.rabitq;
+ if (rb && !rb.error) {
+ lines.push('| metric | value |');
+ lines.push('|--|--:|');
+ lines.push(`| available | ${rb.available} |`);
+ lines.push(`| compression ratio | ${rb.compressionRatio ?? 'null'} |`);
+ lines.push(`| search ms/query | ${rb.searchMsPerQuery ?? 'null'} |`);
+ if (rb.searchNote) lines.push(`| note | ${rb.searchNote} |`);
+ } else { lines.push(`error: ${rb?.error ?? 'no data'}`); }
+ lines.push('');
+
+ // SONA
+ lines.push('### 4. SONA WASM adapt latency (warmed)');
+ const s = results.sona;
+ if (s && !s.error) {
+ lines.push('| metric | value |');
+ lines.push('|--|--:|');
+ lines.push(`| wasm available | ${s.wasmAvailable} |`);
+ lines.push(`| adapt ms/call | ${s.adaptMsPerCall ?? 'null'} |`);
+ if (s.iterations) lines.push(`| iterations | ${s.iterations} |`);
+ if (s.targetMet_0_05ms != null) lines.push(`| < 0.05ms target met | ${s.targetMet_0_05ms} |`);
+ if (s.note) lines.push(`| note | ${s.note} |`);
+ } else { lines.push(`error: ${s?.error ?? 'no data'}`); }
+ lines.push('');
+
+ // MoE
+ lines.push('### 5. MoE gate learning');
+ const m = results.moeGate;
+ if (m && !m.error) {
+ lines.push('| metric | value |');
+ lines.push('|--|--:|');
+ lines.push(`| actions observed | ${m.actionsObserved} |`);
+ lines.push(`| rewards applied | ${m.rewardsApplied} |`);
+ lines.push(`| good-action Q before β†’ after | ${m.beforeGoodQ} β†’ ${m.afterGoodQ} (Ξ” ${m.qShift}) |`);
+ lines.push(`| confidence before β†’ after | ${m.beforeConfidence} β†’ ${m.afterConfidence} (Ξ” ${m.confidenceShift}) |`);
+ lines.push(`| **gate learned** | **${m.gateLearned}** |`);
+ } else { lines.push(`error: ${m?.error ?? 'no data'}`); }
+ lines.push('');
+
+ // Embedding backend
+ lines.push('### 6. Embedding backend (honest)');
+ const eb = results.embeddingBackend;
+ if (eb && !eb.error) {
+ lines.push('| metric | value |');
+ lines.push('|--|--|');
+ lines.push(`| **backend in use** | **${eb.backend}** |`);
+ lines.push(`| model string | ${eb.model} |`);
+ lines.push(`| dimensions | ${eb.dimensions} |`);
+ if (eb.note) lines.push(`| note | ${eb.note} |`);
+ } else { lines.push(`error: ${eb?.error ?? 'no data'}`); }
+ lines.push('');
+
+ console.log(lines.join('\n'));
+}
+
+// ----------------------------------------------------------------------------
+// Main
+// ----------------------------------------------------------------------------
+async function main() {
+ log('Intelligence benchmark β€” measuring against built dist exports...');
+ log(`dist: ${DIST}`);
+
+ const results = {
+ meta: {
+ timestamp: new Date().toISOString(),
+ platform: `${process.platform}/${process.arch}`,
+ node: process.version,
+ dist: DIST,
+ dims: ARGS.dims,
+ queriesPerSize: ARGS.queries,
+ sizes: ARGS.sizes,
+ },
+ };
+
+ // Each benchmark is isolated: a failure in one does not abort the rest.
+ log('\n[1/6] HNSW vs brute-force...');
+ try { results.hnsw = await benchHnsw(); } catch (e) { results.hnsw = { error: e.stack || e.message }; }
+
+ log('\n[2/6] Int8 quantization...');
+ try { results.int8 = await benchInt8(); } catch (e) { results.int8 = { error: e.stack || e.message }; }
+
+ log('\n[3/6] RaBitQ...');
+ try { results.rabitq = await benchRabitq(); } catch (e) { results.rabitq = { error: e.stack || e.message }; }
+
+ log('\n[4/6] SONA WASM adapt...');
+ try { results.sona = await benchSona(); } catch (e) { results.sona = { error: e.stack || e.message }; }
+
+ log('\n[5/6] MoE gate learning...');
+ try { results.moeGate = await benchMoeGate(); } catch (e) { results.moeGate = { error: e.stack || e.message }; }
+
+ log('\n[6/6] Embedding backend...');
+ try { results.embeddingBackend = await benchEmbeddingBackend(); } catch (e) { results.embeddingBackend = { error: e.stack || e.message }; }
+
+ if (!ARGS.jsonOnly) printMarkdown(results);
+
+ // Machine-readable block (always emitted, after a stable marker).
+ console.log('\n===BENCH_JSON===');
+ console.log(JSON.stringify(results));
+
+ return results;
+}
+
+main()
+ .then(() => process.exit(0))
+ .catch((e) => { console.error('FATAL:', e.stack || e.message); process.exit(1); });
diff --git a/v3/@claude-flow/cli/CLAUDE.md b/v3/@claude-flow/cli/CLAUDE.md
index 498ece87ac..f617cc31a4 100644
--- a/v3/@claude-flow/cli/CLAUDE.md
+++ b/v3/@claude-flow/cli/CLAUDE.md
@@ -287,7 +287,7 @@ Bash("npx @claude-flow/cli@latest hooks worker dispatch --trigger optimize")
| `init` | 4 | Project initialization with wizard, presets, skills, hooks |
| `agent` | 8 | Agent lifecycle (spawn, list, status, stop, metrics, pool, health, logs) |
| `swarm` | 6 | Multi-agent swarm coordination and orchestration |
-| `memory` | 11 | AgentDB memory with vector search (150x-12,500x faster) |
+| `memory` | 11 | AgentDB memory with HNSW vector search (measured ~1.9x–4.7x vs brute force above crossover) |
| `mcp` | 9 | MCP server management and tool execution |
| `task` | 6 | Task creation, assignment, and lifecycle |
| `session` | 7 | Session state management and persistence |
@@ -308,7 +308,7 @@ Bash("npx @claude-flow/cli@latest hooks worker dispatch --trigger optimize")
| `providers` | 5 | AI providers (list, add, remove, test, configure) |
| `plugins` | 5 | Plugin management (list, install, uninstall, enable, disable) |
| `deployment` | 5 | Deployment management (deploy, rollback, status, environments, release) |
-| `embeddings` | 4 | Vector embeddings (embed, batch, search, init) - 75x faster with agentic-flow |
+| `embeddings` | 4 | Vector embeddings (embed, batch, search, init) β€” agentic-flow ONNX backend (speedup unverified, no benchmark) |
| `claims` | 4 | Claims-based authorization (check, grant, revoke, list) |
| `migrate` | 5 | V2 to V3 migration with rollback support |
| `doctor` | 1 | System diagnostics with health checks |
@@ -480,12 +480,12 @@ npx @claude-flow/cli@latest migrate validate
## 🧠 Intelligence System (RuVector)
-V3 includes the RuVector Intelligence System:
-- **SONA**: Self-Optimizing Neural Architecture (<0.05ms adaptation)
-- **MoE**: Mixture of Experts for specialized routing
-- **HNSW**: 150x-12,500x faster pattern search
+V3 includes the RuVector Intelligence System (measured numbers: see [audit](../../../docs/reviews/intelligence-system-audit-2026-05-29.md) + [`scripts/benchmark-intelligence.mjs`](../../../scripts/benchmark-intelligence.mjs)):
+- **SONA**: Self-Optimizing Neural Architecture (measured 0.0043ms/adapt, target <0.05ms met)
+- **MoE**: Mixture of Experts for specialized routing (gate converges β€” confidence 0.13β†’0.88 after rewards)
+- **HNSW**: measured ~1.9x at N=20k, ~3.2x–4.7x at N=5k vs brute force (recall@10 ~0.99); ANN wins above the crossover, ruvector NAPI backend (WASM not active on test host)
- **EWC++**: Elastic Weight Consolidation (prevents forgetting)
-- **Flash Attention**: 2.49x-7.47x speedup
+- **Flash Attention**: unverified β€” no benchmark exists for this claim
The 4-step intelligence pipeline:
1. **RETRIEVE** - Fetch relevant patterns via HNSW
@@ -500,7 +500,7 @@ Features:
- **Document chunking**: Configurable overlap and size
- **Normalization**: L2, L1, min-max, z-score
- **Hyperbolic embeddings**: PoincarΓ© ball model for hierarchical data
-- **75x faster**: With agentic-flow ONNX integration
+- **agentic-flow ONNX integration**: speedup unverified (no benchmark; backend reported `onnx`, model all-MiniLM-L6-v2, 384-dim)
- **Neural substrate**: Integration with RuVector
## 🐝 Hive-Mind Consensus
@@ -520,14 +520,18 @@ Features:
## V3 Performance Targets
-| Metric | Target |
-|--------|--------|
-| Flash Attention | 2.49x-7.47x speedup |
-| HNSW Search | 150x-12,500x faster |
-| Memory Reduction | 50-75% with quantization |
-| MCP Response | <100ms |
-| CLI Startup | <500ms |
-| SONA Adaptation | <0.05ms |
+> Source of truth: [`docs/reviews/intelligence-system-audit-2026-05-29.md`](../../../docs/reviews/intelligence-system-audit-2026-05-29.md) + [`scripts/benchmark-intelligence.mjs`](../../../scripts/benchmark-intelligence.mjs). Numbers below are measured unless marked "target/unverified".
+
+| Metric | Measured / Target | Status |
+|--------|-------------------|--------|
+| HNSW Search | ~1.9x at N=20k, ~3.2x–4.7x at N=5k vs brute force (recall@10 ~0.99) | **Measured** (ruvector NAPI; 150x-12,500x NOT reproduced) |
+| Int8 Quantization | 3.84x compression, reconstruction cosine 0.99999 | **Measured** |
+| RaBitQ Quantization | 32x compression, 0.60ms/query | **Measured** |
+| SONA Adaptation | 0.0043ms/adapt (target <0.05ms met) | **Measured** |
+| MoE Gate | converges (confidence 0.13β†’0.88) | **Measured** |
+| Flash Attention | 2.49x-7.47x | **Unverified** (no benchmark) |
+| MCP Response | <100ms | target |
+| CLI Startup | <500ms | target |
## πŸ“Š Performance Optimization Protocol
diff --git a/v3/@claude-flow/cli/README.md b/v3/@claude-flow/cli/README.md
index a4556f5964..d1e6974743 100644
--- a/v3/@claude-flow/cli/README.md
+++ b/v3/@claude-flow/cli/README.md
@@ -1,15 +1,14 @@
<div align="center">
-[![Ruflo Banner](ruflo/assets/ruflo-small.jpeg)](https://flo.ruv.io/)
+[![Ruflo Banner](ruflo/assets/ruflo-small.jpeg)](https://cognitum.one/agentic-engineering)
[![Try the UI Beta β€” flo.ruv.io](https://img.shields.io/badge/_Try_the_UI_Beta-flo.ruv.io-6366f1?style=for-the-badge&logoColor=white&logo=svelte)](https://flo.ruv.io/)
[![Goal Planner β€” goal.ruv.io](https://img.shields.io/badge/_Goal_Planner-goal.ruv.io-8b5cf6?style=for-the-badge&logoColor=white&logo=react)](https://goal.ruv.io/)
[![Live Agents β€” goal.ruv.io/agents](https://img.shields.io/badge/_Live_Agents-goal.ruv.io%2Fagents-10b981?style=for-the-badge&logoColor=white&logo=react)](https://goal.ruv.io/agents)
[![npm version (ruflo)](https://img.shields.io/npm/v/ruflo?label=ruflo&style=for-the-badge&logo=npm&color=cb3837)](https://www.npmjs.com/package/ruflo)
-[![npm downloads (ruflo)](https://img.shields.io/npm/dm/ruflo?label=ruflo%20downloads&style=for-the-badge&logo=npm&color=cb3837)](https://www.npmjs.com/package/ruflo)
-[![npm version (claude-flow)](https://img.shields.io/npm/v/claude-flow?label=claude-flow&style=for-the-badge&logo=npm&color=blue)](https://www.npmjs.com/package/claude-flow)
-[![npm downloads (claude-flow)](https://img.shields.io/npm/dm/claude-flow?label=claude-flow%20downloads&style=for-the-badge&logo=npm&color=blue)](https://www.npmjs.com/package/claude-flow)
+[![Ecosystem downloads](https://img.shields.io/badge/ecosystem%20downloads-22.2M%2B-blue?style=for-the-badge&logo=npm)](https://github.com/ruvnet/ruflo/blob/main/data/clone-data.proof.json)
+[![Git clones (14d)](https://img.shields.io/badge/git%20clones%2014d-115k-blueviolet?style=for-the-badge&logo=github)](https://github.com/ruvnet/ruflo/blob/main/data/clone-data.ledger.json)
[![Star on GitHub](https://img.shields.io/github/stars/ruvnet/claude-flow?style=for-the-badge&logo=github&color=gold)](https://github.com/ruvnet/claude-flow)
[![MIT License](https://img.shields.io/badge/License-MIT-yellow?style=for-the-badge)](https://opensource.org/licenses/MIT)
@@ -73,14 +72,14 @@ There are **two different install paths** with very different surface areas. Pic
# Install core + any plugins you need
/plugin install ruflo-core@ruflo
/plugin install ruflo-swarm@ruflo
-/plugin install ruflo-autopilot@ruflo
-/plugin install ruflo-federation@ruflo
+/plugin install ruflo-rag-memory@ruflo
+/plugin install ruflo-neural-trader@ruflo
```
This adds slash commands and agent definitions only. The Ruflo MCP server is NOT registered, so `memory_store`, `swarm_init`, `agent_spawn`, etc. won't be callable from Claude. For the full loop, use Path B below.
<details>
-<summary><strong>πŸ”Œ All 32 plugins</strong></summary>
+<summary><strong>πŸ”Œ All 33 plugins</strong></summary>
#### Core & Orchestration
@@ -108,6 +107,7 @@ This adds slash commands and agent definitions only. The Ruflo MCP server is NOT
| Plugin | What it does |
|--------|-------------|
| [**ruflo-intelligence**](plugins/ruflo-intelligence/README.md) | Agents learn from past successes and get smarter |
+| [**ruflo-graph-intelligence**](plugins/ruflo-graph-intelligence/) | Sublinear graph reasoning β€” PageRank, delta updates, complexity-aware execution (ADR-123) |
| [**ruflo-daa**](plugins/ruflo-daa/README.md) | Dynamic agent behavior and cognitive patterns |
| [**ruflo-ruvllm**](plugins/ruflo-ruvllm/README.md) | Run local LLMs (Ollama, etc.) with smart routing |
| [**ruflo-goals**](plugins/ruflo-goals/README.md) | Break big goals into plans and track progress |
@@ -374,13 +374,17 @@ User --> Claude Code / CLI
## Documentation
-Three docs for three audiences:
+Four docs for four audiences:
| Doc | When to read it |
|-----|-----------------|
| **[Status](docs/STATUS.md)** | See what currently works β€” capability counts, test baselines, recent fixes, what's next. The *is-it-ready* doc. |
| **[User Guide](docs/USERGUIDE.md)** | Daily reference β€” every command, every config flag, every plugin. The *how-do-I* doc. |
+| **[Benchmarks](https://gist.github.com/ruvnet/298f8c668c8859b369f91734a0e9cbbe)** | v3.8.0 SOTA matrix vs LangGraph / AutoGen / CrewAI on darwin-arm64 + linux-x64. ruflo wins cold start, single turn, RSS by 1.3×–1953Γ—. The *is-it-fast* doc. |
| **[Verification](verification.md)** | Cryptographically prove your installed bytes match the signed witness β€” `ruflo verify`. The *trust-but-verify* doc. |
+| **[Team Gateway Checklist](docs/TEAM-GATEWAY-CHECKLIST.md)** | Before-merge gates, dual-mode handoff, memory namespace sharing, and witness manifest entry per merge. The *safer-team-workflows* doc. |
+
+Benchmark internals (for reproduction): [`sota-workload-spec.md`](https://github.com/ruvnet/ruflo/blob/perf/sota-comparator-benchmarks/docs/benchmarks/sota-workload-spec.md) Β· [`SOTA-PROGRESS.md`](https://github.com/ruvnet/ruflo/blob/perf/sota-comparator-benchmarks/docs/benchmarks/SOTA-PROGRESS.md) Β· [raw matrix JSON: darwin](https://github.com/ruvnet/ruflo/blob/perf/sota-comparator-benchmarks/docs/benchmarks/sota-matrix.json) Β· [linux](https://github.com/ruvnet/ruflo/blob/perf/sota-comparator-benchmarks/docs/benchmarks/sota-matrix-linux.json)
User Guide section index:
diff --git a/v3/@claude-flow/cli/__tests__/bug-cluster-2219-2226.test.ts b/v3/@claude-flow/cli/__tests__/bug-cluster-2219-2226.test.ts
index d7b2fcb3c3..ad11b751b5 100644
--- a/v3/@claude-flow/cli/__tests__/bug-cluster-2219-2226.test.ts
+++ b/v3/@claude-flow/cli/__tests__/bug-cluster-2219-2226.test.ts
@@ -24,6 +24,8 @@ import { systemTools } from '../src/mcp-tools/system-tools.js';
import { hooksTools } from '../src/mcp-tools/hooks-tools.js';
import { agentdbPatternStore, agentdbPatternSearch } from '../src/mcp-tools/agentdb-tools.js';
import { createQLearningRouter } from '../src/ruvector/q-learning-router.js';
+import { CommandParser } from '../src/parser.js';
+import { routeCommand } from '../src/commands/route.js';
function findTool(tools: any[], name: string) {
const t = tools.find((x) => x.name === name);
@@ -85,6 +87,58 @@ describe('#2222 β€” route feedback persists the Q-table to disk', () => {
});
});
+describe('audit #1 β€” negative reward keeps its sign across all flag syntaxes', () => {
+ // Repro: `route feedback -r -1.0` and `--reward -1.0` used to parse as reward=true
+ // (β†’ +1.0 after numeric coercion), so NEGATIVE feedback REINFORCED the bad agent.
+ // Only the `--reward=-1.0` equals-form preserved the sign. Fix lives in the shared
+ // parser (isFlagValue), so all three forms must now yield reward = -1.0.
+ function parseFeedback(args: string[]) {
+ const parser = new CommandParser({ allowUnknownFlags: true });
+ parser.registerCommand(routeCommand);
+ return parser.parse(['route', 'feedback', ...args]);
+ }
+
+ const baseArgs = ['-t', 'write tests', '-a', 'tester'];
+
+ it('-r -1.0 (short flag, space form) β†’ reward = -1.0', () => {
+ const { flags } = parseFeedback([...baseArgs, '-r', '-1.0']);
+ expect(flags.reward).toBe(-1.0);
+ });
+
+ it('--reward -1.0 (long flag, space form) β†’ reward = -1.0', () => {
+ const { flags } = parseFeedback([...baseArgs, '--reward', '-1.0']);
+ expect(flags.reward).toBe(-1.0);
+ });
+
+ it('--reward=-1.0 (equals form) β†’ reward = -1.0', () => {
+ const { flags } = parseFeedback([...baseArgs, '--reward=-1.0']);
+ expect(flags.reward).toBe(-1.0);
+ });
+
+ it('all three syntaxes agree (no sign-flip regression)', () => {
+ const a = parseFeedback([...baseArgs, '-r', '-1.0']).flags.reward;
+ const b = parseFeedback([...baseArgs, '--reward', '-1.0']).flags.reward;
+ const c = parseFeedback([...baseArgs, '--reward=-1.0']).flags.reward;
+ expect(a).toBe(b);
+ expect(b).toBe(c);
+ expect(c).toBe(-1.0);
+ });
+
+ it('positive and fractional negative values are preserved', () => {
+ expect(parseFeedback([...baseArgs, '-r', '0.9']).flags.reward).toBe(0.9);
+ expect(parseFeedback([...baseArgs, '-r', '-0.5']).flags.reward).toBe(-0.5);
+ expect(parseFeedback([...baseArgs, '--reward', '-3.14']).flags.reward).toBe(-3.14);
+ });
+
+ it('real flags after a value-flag are still parsed as flags, not values', () => {
+ // `-a tester` then `-t ...`: the -t must NOT be swallowed as the value of -a.
+ const { flags } = parseFeedback(['-a', 'tester', '-t', 'write tests', '-r', '-1.0']);
+ expect(flags.agent).toBe('tester');
+ expect(flags.task).toBe('write tests');
+ expect(flags.reward).toBe(-1.0);
+ });
+});
+
describe('#2226 β€” pattern store and search share a backend', () => {
it('stores a pattern and finds it via search', async () => {
const marker = `oauth-refresh-token-rotation-${process.pid}-${process.hrtime.bigint()}`;
diff --git a/v3/@claude-flow/cli/__tests__/hooks-intelligence-learning.test.ts b/v3/@claude-flow/cli/__tests__/hooks-intelligence-learning.test.ts
new file mode 100644
index 0000000000..03bbe9b551
--- /dev/null
+++ b/v3/@claude-flow/cli/__tests__/hooks-intelligence-learning.test.ts
@@ -0,0 +1,152 @@
+/**
+ * Hooks Intelligence Learning β€” Audit Fix Smoke Tests
+ *
+ * AUDIT FINDINGS #4 & #5:
+ * (A) hooks_intelligence_trajectory-end must feed the EWC consolidator a
+ * gradient derived from the trajectory's REAL embedding (via
+ * generateEmbedding) β€” NOT a synthetic sine wave. If no real embedding is
+ * available, it must SKIP the EWC update rather than inject noise.
+ * (B) hooks_intelligence_learn must actually TRIGGER a real learning/
+ * consolidation cycle (distillLearning), not merely echo stats.
+ *
+ * These tests isolate the handlers with mocks so no real DB/ONNX is required.
+ */
+
+import { describe, it, expect, vi, beforeEach } from 'vitest';
+
+// --- Mocks (must be declared before importing the module under test) ---
+
+// Capture the gradient passed to EWC so we can assert it is the real embedding.
+const recordGradient = vi.fn();
+const getConsolidationStats = vi.fn(() => ({
+ avgPenalty: 0.42,
+ consolidationCount: 1,
+ highImportancePatterns: 0,
+ totalPatterns: 1,
+}));
+
+vi.mock('../src/memory/ewc-consolidation.js', () => ({
+ getEWCConsolidator: vi.fn(async () => ({
+ recordGradient,
+ getConsolidationStats,
+ })),
+}));
+
+// SONA optimizer present and "learns" so the success branch runs.
+vi.mock('../src/memory/sona-optimizer.js', () => ({
+ getSONAOptimizer: vi.fn(async () => ({
+ processTrajectoryOutcome: vi.fn(() => ({
+ learned: true,
+ patternKey: 'pattern-x',
+ confidence: 0.9,
+ })),
+ getStats: vi.fn(() => ({
+ totalPatterns: 3,
+ successfulRoutings: 2,
+ failedRoutings: 1,
+ trajectoriesProcessed: 3,
+ avgConfidence: 0.7,
+ })),
+ })),
+}));
+
+// REAL embedding signal β€” a distinctive vector that is clearly NOT a sine wave.
+const REAL_EMBEDDING = new Array(384).fill(0).map((_, i) => (i % 2 === 0 ? 0.5 : 0.25));
+vi.mock('../src/memory/memory-initializer.js', () => ({
+ generateEmbedding: vi.fn(async () => ({
+ embedding: REAL_EMBEDDING,
+ dimensions: REAL_EMBEDDING.length,
+ model: 'mock-onnx',
+ })),
+ // storeEntry must succeed so the success/learning branch (which contains the
+ // EWC update) is reached in trajectory-end.
+ storeEntry: vi.fn(async () => ({ success: true, id: 'mock-trajectory-id' })),
+}));
+
+// Intelligence layer β€” distillLearning is the real DISTILL+CONSOLIDATE path.
+const distillLearning = vi.fn(async () => ({ patternsDistilled: 5, ewcPenalty: 0.13 }));
+const runBackgroundLearning = vi.fn(async () => {});
+vi.mock('../src/memory/intelligence.js', () => ({
+ distillLearning,
+ runBackgroundLearning,
+}));
+
+// graph-edge-writer is fire-and-forget; stub it.
+vi.mock('../src/memory/graph-edge-writer.js', () => ({
+ insertGraphEdge: vi.fn(async () => {}),
+}));
+
+import {
+ hooksTrajectoryStart,
+ hooksTrajectoryStep,
+ hooksTrajectoryEnd,
+ hooksIntelligenceLearn,
+} from '../src/mcp-tools/hooks-tools.js';
+import { generateEmbedding } from '../src/memory/memory-initializer.js';
+
+describe('AUDIT FIX #4 β€” trajectory-end feeds REAL embedding gradient to EWC', () => {
+ beforeEach(() => {
+ recordGradient.mockClear();
+ (generateEmbedding as ReturnType<typeof vi.fn>).mockClear();
+ });
+
+ it('records a gradient equal to the real trajectory embedding, not a sine wave', async () => {
+ const started = (await hooksTrajectoryStart.handler({ task: 'do work', agent: 'coder' })) as any;
+ const trajectoryId = started.trajectoryId as string;
+
+ await hooksTrajectoryStep.handler({ trajectoryId, action: 'edit-file', result: 'ok', quality: 0.9 });
+
+ const ended = (await hooksTrajectoryEnd.handler({ trajectoryId, success: true })) as any;
+
+ // generateEmbedding was used to derive the gradient (mirrors DISTILL path)
+ expect(generateEmbedding).toHaveBeenCalledTimes(1);
+
+ // EWC received the REAL embedding, not synthetic noise
+ expect(recordGradient).toHaveBeenCalledTimes(1);
+ const passedGradient = recordGradient.mock.calls[0][1] as number[];
+ expect(passedGradient).toEqual(REAL_EMBEDDING);
+
+ // Guard: the OLD synthetic gradient was sin(i*0.01)*(steps/10). With 1 step
+ // that produced values like sin(0)=0, sin(0.01)*0.1 ~= 0.001 β€” confirm the
+ // gradient is NOT that degenerate sine sequence.
+ expect(passedGradient[0]).not.toBeCloseTo(Math.sin(0) * (1 / 10), 6);
+ expect(passedGradient[1]).not.toBeCloseTo(Math.sin(0.01) * (1 / 10), 6);
+
+ expect(ended.learning.ewcConsolidation).toBe(true);
+ });
+
+ it('SKIPS the EWC update (no synthetic gradient) when no real embedding is available', async () => {
+ (generateEmbedding as ReturnType<typeof vi.fn>).mockResolvedValueOnce({
+ embedding: [],
+ dimensions: 0,
+ model: 'none',
+ });
+
+ const started = (await hooksTrajectoryStart.handler({ task: 'task2', agent: 'coder' })) as any;
+ const trajectoryId = started.trajectoryId as string;
+ await hooksTrajectoryStep.handler({ trajectoryId, action: 'a', result: 'ok' });
+
+ const ended = (await hooksTrajectoryEnd.handler({ trajectoryId, success: true })) as any;
+
+ // No gradient recorded β€” we skip rather than inject noise.
+ expect(recordGradient).not.toHaveBeenCalled();
+ expect(ended.learning.ewcConsolidation).toBe(false);
+ });
+});
+
+describe('AUDIT FIX #5 β€” hooks_intelligence_learn actually triggers a learning cycle', () => {
+ beforeEach(() => {
+ distillLearning.mockClear();
+ runBackgroundLearning.mockClear();
+ });
+
+ it('calls the real distillLearning (DISTILL+CONSOLIDATE) path, not just stats echo', async () => {
+ const result = (await hooksIntelligenceLearn.handler({ consolidate: true })) as any;
+
+ expect(distillLearning).toHaveBeenCalledTimes(1);
+ expect(result.cycleTriggered).toBe(true);
+ expect(result.patternsDistilled).toBe(5);
+ expect(result.implementation).toBe('real-distill-consolidate');
+ expect(result.learned).toBe(true);
+ });
+});
diff --git a/v3/@claude-flow/cli/__tests__/memory-ruvector-deep.test.ts b/v3/@claude-flow/cli/__tests__/memory-ruvector-deep.test.ts
index 35745993dc..de3ef5be2b 100644
--- a/v3/@claude-flow/cli/__tests__/memory-ruvector-deep.test.ts
+++ b/v3/@claude-flow/cli/__tests__/memory-ruvector-deep.test.ts
@@ -257,6 +257,61 @@ describe('Memory Initializer', () => {
expect(status.dimensions).toBe(384);
});
});
+
+ // AUDIT #3: generateEmbedding must expose a truthful `backend` field so an
+ // operator can distinguish real ONNX semantics from the deterministic hash
+ // fallback (inverted/meaningless semantics) even when `model` reports a
+ // real-looking name.
+ describe('generateEmbedding backend field', () => {
+ const prevDisableBridge = process.env.CLAUDE_FLOW_DISABLE_BRIDGE;
+
+ afterEach(() => {
+ vi.resetModules();
+ vi.doUnmock('@huggingface/transformers');
+ vi.doUnmock('@xenova/transformers');
+ vi.doUnmock('ruvector');
+ vi.doUnmock('agentic-flow');
+ vi.doUnmock('agentic-flow/reasoningbank');
+ if (prevDisableBridge === undefined) delete process.env.CLAUDE_FLOW_DISABLE_BRIDGE;
+ else process.env.CLAUDE_FLOW_DISABLE_BRIDGE = prevDisableBridge;
+ });
+
+ it("reports backend='mock' when no real embedding model is available", async () => {
+ // Force the raw fallback path (no AgentDB bridge) and make every real
+ // embedding provider import fail so loadEmbeddingModel lands on the
+ // hash fallback (model = null).
+ process.env.CLAUDE_FLOW_DISABLE_BRIDGE = '1';
+ vi.resetModules();
+ const fail = () => { throw new Error('unavailable in test'); };
+ vi.doMock('@huggingface/transformers', fail);
+ vi.doMock('@xenova/transformers', fail);
+ vi.doMock('ruvector', fail);
+ vi.doMock('agentic-flow', fail);
+ vi.doMock('agentic-flow/reasoningbank', fail);
+
+ const { generateEmbedding } = await import('../src/memory/memory-initializer.js');
+ const result = await generateEmbedding('audit-3 mock-backend assertion');
+
+ expect(result.backend).toBe('mock');
+ expect(result.model).toBe('hash-fallback');
+ expect(Array.isArray(result.embedding)).toBe(true);
+ expect(result.embedding.length).toBe(result.dimensions);
+ });
+
+ it("always sets backend, and 'mock' implies the hash-fallback model", async () => {
+ // Invariant check that holds regardless of which providers are installed:
+ // backend is one of the two known values, and mock <=> hash-fallback.
+ process.env.CLAUDE_FLOW_DISABLE_BRIDGE = '1';
+ vi.resetModules();
+ const { generateEmbedding } = await import('../src/memory/memory-initializer.js');
+ const result = await generateEmbedding('audit-3 backend invariant');
+
+ expect(['onnx', 'mock']).toContain(result.backend);
+ if (result.backend === 'mock') {
+ expect(result.model).toBe('hash-fallback');
+ }
+ }, 60000); // real ONNX model can be slow to cold-load when installed
+ });
});
// =============================================================================
diff --git a/v3/@claude-flow/cli/package.json b/v3/@claude-flow/cli/package.json
index 119741d0d0..8b41c8d7eb 100644
--- a/v3/@claude-flow/cli/package.json
+++ b/v3/@claude-flow/cli/package.json
@@ -1,6 +1,6 @@
{
"name": "@claude-flow/cli",
- "version": "3.10.6",
+ "version": "3.10.7",
"type": "module",
"description": "Ruflo CLI - Enterprise AI agent orchestration with 60+ specialized agents, swarm coordination, MCP server, self-learning hooks, and vector memory for Claude Code",
"main": "dist/src/index.js",
diff --git a/v3/@claude-flow/cli/src/mcp-tools/hooks-tools.ts b/v3/@claude-flow/cli/src/mcp-tools/hooks-tools.ts
index 4f1c7ff9a4..feea4716be 100644
--- a/v3/@claude-flow/cli/src/mcp-tools/hooks-tools.ts
+++ b/v3/@claude-flow/cli/src/mcp-tools/hooks-tools.ts
@@ -2597,17 +2597,35 @@ export const hooksTrajectoryEnd: MCPTool = {
const ewc = await getEWCConsolidator();
if (ewc) {
try {
- // Record gradient sample for Fisher matrix update
- // Create a simple gradient from trajectory steps
- const gradients = new Array(384).fill(0).map((_, i) =>
- Math.sin(i * 0.01) * (trajectory.steps.length / 10)
- );
- ewc.recordGradient(`trajectory-${trajectoryId}`, gradients, success);
- const stats = ewc.getConsolidationStats();
- ewcResult = {
- consolidated: true,
- penalty: stats.avgPenalty,
- };
+ // AUDIT FIX #4: derive a REAL gradient from the trajectory's
+ // embedding (mirrors the DISTILL path, where step content is
+ // embedded via generateEmbedding) instead of a synthetic sine
+ // wave. The EWC library treats the embedding as the gradient
+ // proxy (see recordPatternOutcome in ewc-consolidation.ts).
+ let gradients: number[] | null = null;
+ try {
+ const { generateEmbedding } = await import('../memory/memory-initializer.js');
+ // Embed the same summary that was persisted for semantic search,
+ // so the Fisher update reflects the actual recorded trajectory.
+ const summary = `Task: ${trajectory.task} | Agent: ${trajectory.agent} | Steps: ${trajectory.steps.map(s => `${s.action}=>${s.result}`).join('; ')}${feedback ? ` | Feedback: ${feedback}` : ''}`;
+ const embeddingResult = await generateEmbedding(summary);
+ if (embeddingResult?.embedding && embeddingResult.embedding.length > 0) {
+ gradients = embeddingResult.embedding;
+ }
+ } catch {
+ // Embedding generation unavailable β€” fall through and skip EWC
+ }
+
+ if (gradients) {
+ ewc.recordGradient(`trajectory-${trajectoryId}`, gradients, success);
+ const stats = ewc.getConsolidationStats();
+ ewcResult = {
+ consolidated: true,
+ penalty: stats.avgPenalty,
+ };
+ }
+ // If no real embedding-derived gradient is available, SKIP the EWC
+ // update rather than feeding the Fisher matrix synthetic noise.
} catch {
// EWC consolidation failed, continue without it
}
@@ -3057,7 +3075,26 @@ export const hooksIntelligenceLearn: MCPTool = {
const consolidate = params.consolidate !== false;
const startTime = Date.now();
- // Get SONA statistics
+ // AUDIT FIX #5: actually TRIGGER a learning/consolidation cycle instead of
+ // only reading and echoing stats. This calls the real DISTILL path
+ // (LoRA-style confidence updates with EWC++ consolidation protection) and
+ // the background learning pass, then reports the resulting stats.
+ let distill: { patternsDistilled: number; ewcPenalty: number } | null = null;
+ let distillTriggered = false;
+ try {
+ const intelligence = await import('../memory/intelligence.js');
+ // DISTILL + CONSOLIDATE: real LoRA update with EWC++ protection
+ distill = await intelligence.distillLearning();
+ distillTriggered = distill !== null;
+ // Run background learning (ruvllm) pass as well β€” best-effort
+ try {
+ await intelligence.runBackgroundLearning();
+ } catch { /* best-effort */ }
+ } catch {
+ // intelligence layer unavailable β€” fall back to stats-only reporting
+ }
+
+ // Get SONA statistics (AFTER triggering the cycle so they reflect the update)
let sonaStats = {
totalPatterns: 0,
successfulRoutings: 0,
@@ -3077,7 +3114,7 @@ export const hooksIntelligenceLearn: MCPTool = {
};
}
- // Get EWC++ statistics and optionally trigger consolidation
+ // Get EWC++ statistics after the consolidation cycle ran
let ewcStats = {
consolidation: false,
fisherUpdated: false,
@@ -3092,17 +3129,21 @@ export const hooksIntelligenceLearn: MCPTool = {
consolidation: true,
fisherUpdated: stats.consolidationCount > 0,
forgettingPrevented: stats.highImportancePatterns,
- avgPenalty: stats.avgPenalty,
+ avgPenalty: distill?.ewcPenalty ?? stats.avgPenalty,
};
}
}
return {
- learned: sonaStats.totalPatterns > 0,
+ // "learned" now reflects whether a real distill cycle actually ran
+ learned: distillTriggered || sonaStats.totalPatterns > 0,
+ cycleTriggered: distillTriggered,
+ patternsDistilled: distill?.patternsDistilled ?? 0,
duration: Date.now() - startTime,
updates: {
trajectoriesProcessed: sonaStats.trajectoriesProcessed,
patternsLearned: sonaStats.totalPatterns,
+ patternsDistilled: distill?.patternsDistilled ?? 0,
successRate: sonaStats.trajectoriesProcessed > 0
? (sonaStats.successfulRoutings / (sonaStats.successfulRoutings + sonaStats.failedRoutings) * 100).toFixed(1) + '%'
: '0%',
@@ -3112,7 +3153,9 @@ export const hooksIntelligenceLearn: MCPTool = {
average: sonaStats.avgConfidence,
implementation: sona ? 'real-sona' : 'not-available',
},
- implementation: sona ? 'real-sona-learning' : 'placeholder',
+ implementation: distillTriggered
+ ? 'real-distill-consolidate'
+ : (sona ? 'real-sona-learning' : 'placeholder'),
};
},
};
diff --git a/v3/@claude-flow/cli/src/mcp-tools/memory-tools.ts b/v3/@claude-flow/cli/src/mcp-tools/memory-tools.ts
index 9d22a2eced..b598a1b92a 100644
--- a/v3/@claude-flow/cli/src/mcp-tools/memory-tools.ts
+++ b/v3/@claude-flow/cli/src/mcp-tools/memory-tools.ts
@@ -926,6 +926,16 @@ export const memoryTools: MCPTool[] = [
}
}
+ // AUDIT #3: report the embedding backend truthfully β€” a hash-fallback
+ // import is NOT semantically searchable, so an operator must not read
+ // "ONNX ... (384-dim)" when the vectors are mock.
+ let importBackend: 'onnx' | 'mock' | 'unknown' = 'unknown';
+ try {
+ const { generateEmbedding } = await import('../memory/memory-initializer.js');
+ const probe = await generateEmbedding('memory_import_claude backend probe');
+ importBackend = probe.backend ?? 'unknown';
+ } catch { /* probe failed β€” leave 'unknown' */ }
+
return {
success: true,
imported,
@@ -935,7 +945,8 @@ export const memoryTools: MCPTool[] = [
files: memoryFiles.length,
projects: projects.size,
namespace: ns,
- embedding: 'ONNX all-MiniLM-L6-v2 (384-dim)',
+ embedding: `all-MiniLM-L6-v2 (384-dim, backend=${importBackend})`,
+ embeddingBackend: importBackend,
};
},
},
@@ -1001,14 +1012,37 @@ export const memoryTools: MCPTool[] = [
if (stats) intelligence = { sonaEnabled: stats.sonaEnabled, patternsLearned: stats.patternsLearned, trajectoriesRecorded: stats.trajectoriesRecorded };
} catch { /* not initialized */ }
+ // AUDIT #3: probe the embedding backend so operators can tell real ONNX
+ // output from the deterministic hash fallback (which has inverted/
+ // meaningless semantics). Without this, the status string reports the
+ // model name unconditionally and mock output is indistinguishable.
+ let embeddingBackend: 'onnx' | 'mock' | 'unknown' = 'unknown';
+ try {
+ const { generateEmbedding } = await import('../memory/memory-initializer.js');
+ const probe = await generateEmbedding('memory_bridge_status backend probe');
+ embeddingBackend = probe.backend ?? 'unknown';
+ } catch { /* probe failed β€” leave 'unknown' */ }
+
+ const embeddingLabel = `all-MiniLM-L6-v2 (384-dim, backend=${embeddingBackend})`;
+
return {
claudeCode: { memoryFiles: claudeFiles, projects: claudeProjects },
- agentdb: { totalEntries: agentdbEntries, claudeMemoryEntries, namespaces: namespaceCounts, backend: 'sql.js + ONNX' },
+ agentdb: {
+ totalEntries: agentdbEntries,
+ claudeMemoryEntries,
+ namespaces: namespaceCounts,
+ backend: embeddingBackend === 'mock' ? 'sql.js + MOCK (hash fallback)' : 'sql.js + ONNX',
+ embeddingBackend,
+ },
intelligence,
// #1940: report 'connected' whenever ANY namespace has imported
// content, not just `claude-memories` β€” the bridge can be in active
// use from other import paths (e.g. plugin namespaces, task memory).
- bridge: { status: agentdbEntries > 0 ? 'connected' : 'not-synced', embedding: 'all-MiniLM-L6-v2 (384-dim)' },
+ bridge: {
+ status: agentdbEntries > 0 ? 'connected' : 'not-synced',
+ embedding: embeddingLabel,
+ embeddingBackend,
+ },
};
},
},
diff --git a/v3/@claude-flow/cli/src/memory/memory-bridge.ts b/v3/@claude-flow/cli/src/memory/memory-bridge.ts
index f339958374..5361339b96 100644
--- a/v3/@claude-flow/cli/src/memory/memory-bridge.ts
+++ b/v3/@claude-flow/cli/src/memory/memory-bridge.ts
@@ -1078,7 +1078,7 @@ export async function bridgeDeleteEntry(options: {
export async function bridgeGenerateEmbedding(
text: string,
dbPath?: string,
-): Promise<{ embedding: number[]; dimensions: number; model: string } | null> {
+): Promise<{ embedding: number[]; dimensions: number; model: string; backend?: 'onnx' | 'mock' } | null> {
const registry = await getRegistry(dbPath);
if (!registry) return null;
@@ -1090,10 +1090,16 @@ export async function bridgeGenerateEmbedding(
const emb = await embedder.embed(text);
if (!emb) return null;
+ // AUDIT #3: surface backend truthfully. AgentDB's embedder is a real ONNX
+ // model when present; if it ever exposes a mock/stub signal, honor it.
+ const isMock = (embedder as { isMock?: boolean; backend?: string }).isMock === true
+ || (embedder as { backend?: string }).backend === 'mock';
+
return {
embedding: Array.from(emb),
dimensions: emb.length,
model: 'Xenova/all-MiniLM-L6-v2',
+ backend: isMock ? 'mock' : 'onnx',
};
} catch {
return null;
diff --git a/v3/@claude-flow/cli/src/memory/memory-initializer.ts b/v3/@claude-flow/cli/src/memory/memory-initializer.ts
index 24bf4b694f..8ee4d8da46 100644
--- a/v3/@claude-flow/cli/src/memory/memory-initializer.ts
+++ b/v3/@claude-flow/cli/src/memory/memory-initializer.ts
@@ -1837,17 +1837,34 @@ export async function loadEmbeddingModel(options?: {
/**
* Generate real embedding for text
* Uses ONNX model if available, falls back to deterministic hash
+ *
+ * AUDIT #3: the `backend` field is the authoritative signal for whether the
+ * returned vector carries real ONNX semantics ('onnx') or the deterministic
+ * hash fallback ('mock'). The hash fallback produces inverted/meaningless
+ * semantics, so operators MUST be able to tell the two apart even when the
+ * `model` string reports a real model name (e.g. the AgentDB bridge always
+ * labels its output 'Xenova/all-MiniLM-L6-v2' regardless of whether AgentDB's
+ * own embedder is real or stubbed). Set `backend` truthfully by the path that
+ * actually produced the vector. Do NOT change the embedding math.
*/
export async function generateEmbedding(text: string): Promise<{
embedding: number[];
dimensions: number;
model: string;
+ backend: 'onnx' | 'mock';
}> {
// ADR-053: Try AgentDB v3 bridge first
const bridge = await getBridge();
if (bridge) {
const bridgeResult = await bridge.bridgeGenerateEmbedding(text);
- if (bridgeResult) return bridgeResult;
+ if (bridgeResult) {
+ // The bridge labels its output with a real model name unconditionally;
+ // honor the backend it reports if present, otherwise treat a real model
+ // name as ONNX (the bridge only returns when AgentDB's embedder exists).
+ const backend: 'onnx' | 'mock' =
+ (bridgeResult as { backend?: 'onnx' | 'mock' }).backend ?? 'onnx';
+ return { ...bridgeResult, backend };
+ }
}
// Ensure model is loaded
@@ -1869,7 +1886,8 @@ export async function generateEmbedding(text: string): Promise<{
return {
embedding,
dimensions: embedding.length,
- model: 'onnx'
+ model: 'onnx',
+ backend: 'onnx'
};
}
} catch {
@@ -1877,12 +1895,14 @@ export async function generateEmbedding(text: string): Promise<{
}
}
- // Deterministic hash-based fallback (for testing/demo without ONNX)
+ // Deterministic hash-based fallback (for testing/demo without ONNX).
+ // AUDIT #3: backend='mock' β€” these vectors do NOT carry real semantics.
const embedding = generateHashEmbedding(text, state.dimensions);
return {
embedding,
dimensions: state.dimensions,
- model: 'hash-fallback'
+ model: 'hash-fallback',
+ backend: 'mock'
};
}
diff --git a/v3/@claude-flow/cli/src/parser.ts b/v3/@claude-flow/cli/src/parser.ts
index 6eea07b2a1..f525b2be49 100644
--- a/v3/@claude-flow/cli/src/parser.ts
+++ b/v3/@claude-flow/cli/src/parser.ts
@@ -298,7 +298,7 @@ export class CommandParser {
if (booleanFlags.has(normalizedKey)) {
flags[normalizedKey] = true;
- } else if (nextIndex < args.length && !args[nextIndex].startsWith('-')) {
+ } else if (nextIndex < args.length && this.isFlagValue(args[nextIndex])) {
flags[normalizedKey] = this.parseValue(args[nextIndex]);
nextIndex++;
} else {
@@ -316,7 +316,7 @@ export class CommandParser {
if (booleanFlags.has(normalizedKey)) {
flags[normalizedKey] = true;
- } else if (nextIndex < args.length && !args[nextIndex].startsWith('-')) {
+ } else if (nextIndex < args.length && this.isFlagValue(args[nextIndex])) {
flags[normalizedKey] = this.parseValue(args[nextIndex]);
nextIndex++;
} else {
@@ -334,6 +334,27 @@ export class CommandParser {
return { flags, nextIndex };
}
+ /**
+ * Decide whether `arg` should be consumed as the VALUE of the preceding flag,
+ * rather than treated as the next flag.
+ *
+ * Bug fix (audit #1, follow-up to #2222): a negative numeric value such as
+ * `-1.0` starts with '-', so the old `!arg.startsWith('-')` test rejected it
+ * as a value and parsed it as a (bogus) short flag. For `route feedback
+ * -r -1.0` this silently dropped the value and coerced reward to `true` β†’ 1.0,
+ * so NEGATIVE feedback REINFORCED the agent. Only `--reward=-1.0` worked.
+ *
+ * Anything not starting with '-' is a value (unchanged). Anything that starts
+ * with '-' is a value ONLY if it is a pure negative number (e.g. `-1`, `-1.0`,
+ * `-3.14`, `-1e3`). Real flags like `-r`, `--reward`, `-abc` are never numeric
+ * after the leading dash, so they are still correctly treated as flags.
+ */
+ private isFlagValue(arg: string): boolean {
+ if (!arg.startsWith('-')) return true;
+ // Negative number: '-' followed by a parseable numeric literal.
+ return /^-\d*\.?\d+(?:[eE][+-]?\d+)?$/.test(arg);
+ }
+
private parseValue(value: string): string | number | boolean {
// Boolean
if (value.toLowerCase() === 'true') return true;
diff --git a/v3/@claude-flow/cli/src/ruvector/vector-db.ts b/v3/@claude-flow/cli/src/ruvector/vector-db.ts
index 72960b7d13..9b857808bb 100644
--- a/v3/@claude-flow/cli/src/ruvector/vector-db.ts
+++ b/v3/@claude-flow/cli/src/ruvector/vector-db.ts
@@ -11,6 +11,10 @@
* Created with love by ruv.io
*/
+import os from 'node:os';
+import path from 'node:path';
+import { randomUUID } from 'node:crypto';
+
// ============================================================================
// Types
// ============================================================================
@@ -172,7 +176,29 @@ export async function loadRuVector(): Promise<boolean> {
const VectorDBClass = ruvector.VectorDB || ruvector.VectorDb;
ruvectorModule = {
createVectorDB: async (dimensions: number): Promise<VectorDB> => {
- const db = new VectorDBClass({ dimensions });
+ // Build the HNSW graph with explicit, high-recall parameters instead
+ // of relying on the native layer's undocumented defaults. Measured on
+ // a 384-dim clustered corpus (scripts/benchmark-intelligence.mjs):
+ // - sparser graphs (e.g. m=16/efConstruction=64) are faster but
+ // drop recall@10 to ~0.68 at N=20k β€” unacceptable for a memory
+ // store, so we use the standard high-recall HNSW setting
+ // m=32/efConstruction=200, which keeps recall@10 ~0.99 while still
+ // scaling sub-linearly (log-graph traversal) above the crossover.
+ // These map onto ruvector's DbOptions.hnsw { m, efConstruction }.
+ //
+ // storagePath: the native ruvector DB takes an exclusive file lock on
+ // its storage path. Without an explicit path it defaults to a shared
+ // file in the CWD (e.g. agentdb.rvf), which is frequently already
+ // held by a running daemon/MCP server. When the lock cannot be
+ // acquired the constructor THROWS, and createVectorDB() below
+ // silently degrades to the O(n) brute-force FallbackVectorDB β€” so the
+ // "HNSW" index is never actually used. Giving each transient index a
+ // unique path avoids that contention so the real HNSW graph is built.
+ const db = new VectorDBClass({
+ dimensions,
+ hnswConfig: { m: 32, efConstruction: 200 },
+ storagePath: path.join(os.tmpdir(), `ruvector-${process.pid}-${randomUUID()}.rvf`),
+ });
// Wrap ruvector's VectorDB to match our interface
return {
insert: (embedding: Float32Array, id: string, metadata?: Record<string, unknown>) => {
@@ -248,8 +274,13 @@ export async function createVectorDB(dimensions: number = 768): Promise<VectorDB
if (ruvectorModule && typeof ruvectorModule.createVectorDB === 'function') {
try {
return await ruvectorModule.createVectorDB(dimensions);
- } catch {
- // Fall back to simple implementation
+ } catch (err) {
+ // Fall back to simple implementation, but make the degradation VISIBLE.
+ // A silent fall-through here turns HNSW into O(n) brute force without any
+ // signal β€” exactly the failure mode that masked the lock-contention bug.
+ console.warn(
+ `[vector-db] ruvector HNSW init failed, falling back to brute-force search: ${(err as Error)?.message ?? err}`
+ );
}
}
diff --git a/v3/@claude-flow/integration/src/attention-coordinator.ts b/v3/@claude-flow/integration/src/attention-coordinator.ts
index a62d81dd8b..86dded4c06 100644
--- a/v3/@claude-flow/integration/src/attention-coordinator.ts
+++ b/v3/@claude-flow/integration/src/attention-coordinator.ts
@@ -2,8 +2,9 @@
* Attention Coordinator for Flash Attention Integration
*
* Provides integration with agentic-flow's attention mechanisms,
- * including Flash Attention for 2.49x-7.47x speedup with
- * 50-75% memory reduction.
+ * including an approximate sparse Flash Attention path. Speedup and
+ * memory reduction are unverified β€” no benchmark kernel is wired here.
+ * See docs/reviews/intelligence-system-audit-2026-05-29.md
*
* Supported Mechanisms:
* - Flash Attention (fastest, recommended)
@@ -69,8 +70,11 @@ const MECHANISM_PROFILES: Record<AttentionMechanism, {
bestFor: string[];
}> = {
'flash': {
- speedupRange: [2.49, 7.47],
- memoryReduction: 0.75,
+ // Speedup is UNMEASURED β€” sentinel [0, 0] means "no verified benchmark".
+ // Do NOT restore a fabricated 2.49x-7.47x range.
+ // See docs/reviews/intelligence-system-audit-2026-05-29.md
+ speedupRange: [0, 0],
+ memoryReduction: 0,
latencyMs: [0.7, 1.5],
bestFor: ['general', 'high-throughput', 'memory-constrained'],
},
@@ -158,7 +162,8 @@ export class AttentionCoordinator extends EventEmitter {
* This implements ADR-001: Adopt agentic-flow as Core Foundation
* When a reference is provided, attention computation for sequences
* longer than 512 tokens delegates to agentic-flow's optimized
- * Flash Attention implementation for 2.49x-7.47x speedup.
+ * Flash Attention implementation (approximate sparse attention;
+ * speedup unverified β€” see docs/reviews/intelligence-system-audit-2026-05-29.md).
*
* @param attentionRef - The agentic-flow Attention interface reference
*/
@@ -496,8 +501,8 @@ export class AttentionCoordinator extends EventEmitter {
* Perform attention computation
*
* ADR-001: For sequences longer than 512 tokens, delegates to
- * agentic-flow's native Flash Attention for 2.49x-7.47x speedup
- * and 50-75% memory reduction.
+ * agentic-flow's native Flash Attention (approximate sparse attention;
+ * speedup unverified β€” see docs/reviews/intelligence-system-audit-2026-05-29.md).
*/
private async performAttention(params: {
query: number[] | Float32Array;
@@ -514,8 +519,9 @@ export class AttentionCoordinator extends EventEmitter {
// Calculate sequence length for delegation decision
const sequenceLength = qArray.length;
- // ADR-001: Delegate to agentic-flow for long sequences
- // Flash Attention provides 2.49x-7.47x speedup for sequences > 512 tokens
+ // ADR-001: Delegate to agentic-flow for long sequences.
+ // Flash Attention is an approximate sparse path; speedup unverified β€”
+ // see docs/reviews/intelligence-system-audit-2026-05-29.md
if (
this.isDelegationEnabled() &&
this.agenticFlowAttention &&
@@ -631,9 +637,15 @@ export class AttentionCoordinator extends EventEmitter {
this.metrics.throughputTps = 1000 / this.metrics.avgLatencyMs;
}
- // Calculate speedup factor based on mechanism
+ // Speedup factor based on mechanism profile.
+ // A [0, 0] speedupRange is the "unmeasured" sentinel (e.g. flash, where
+ // no benchmark kernel is wired) β€” report 0 rather than fabricate a value.
+ // See docs/reviews/intelligence-system-audit-2026-05-29.md
const profile = MECHANISM_PROFILES[this.config.mechanism];
- this.metrics.speedupFactor = (profile.speedupRange[0] + profile.speedupRange[1]) / 2;
+ const isUnmeasured = profile.speedupRange[0] === 0 && profile.speedupRange[1] === 0;
+ this.metrics.speedupFactor = isUnmeasured
+ ? 0 // unmeasured β€” no fabrication
+ : (profile.speedupRange[0] + profile.speedupRange[1]) / 2;
this.metrics.memoryEfficiency = 1 - profile.memoryReduction;
}
diff --git a/v3/@claude-flow/swarm/src/attention-coordinator.ts b/v3/@claude-flow/swarm/src/attention-coordinator.ts
index 8e1aac45b4..0c6d840dd5 100644
--- a/v3/@claude-flow/swarm/src/attention-coordinator.ts
+++ b/v3/@claude-flow/swarm/src/attention-coordinator.ts
@@ -3,15 +3,15 @@
*
* Implements attention-based coordination mechanisms from agentic-flow@alpha:
* - multi-head: Standard multi-head attention
- * - flash: 2.49x-7.47x speedup, 75% memory reduction
+ * - flash: approximate sparse attention; speedup unverified β€” see docs/reviews/intelligence-system-audit-2026-05-29.md
* - linear: For long sequences
* - hyperbolic: Hierarchical data
* - moe: Mixture of Experts routing
* - graph-rope: Graph-aware positional embeddings
*
* Performance Targets:
- * - Flash Attention: 2.49x-7.47x speedup
- * - Memory Reduction: 50-75%
+ * - Flash Attention: approximate sparse attention; speedup unverified β€” see docs/reviews/intelligence-system-audit-2026-05-29.md
+ * - Memory Reduction: unverified β€” see docs/reviews/intelligence-system-audit-2026-05-29.md
* - MoE Routing: <5ms
*
* @module v3/swarm/attention-coordinator
@@ -28,7 +28,7 @@ import { EventEmitter } from 'events';
*/
export type AttentionType =
| 'multi-head' // Standard multi-head attention
- | 'flash' // 2.49x-7.47x speedup, 75% memory reduction
+ | 'flash' // approximate sparse attention; speedup unverified β€” see docs/reviews/intelligence-system-audit-2026-05-29.md
| 'linear' // For long sequences
| 'hyperbolic' // Hierarchical data
| 'moe' // Mixture of Experts
@@ -374,7 +374,8 @@ export class AttentionCoordinator extends EventEmitter {
// ===========================================================================
/**
- * Flash Attention - 2.49x-7.47x speedup
+ * Flash Attention - approximate sparse attention; speedup unverified
+ * β€” see docs/reviews/intelligence-system-audit-2026-05-29.md
*/
private async flashAttentionCoordination(
agentOutputs: AgentOutput[]
@@ -418,8 +419,11 @@ export class AttentionCoordinator extends EventEmitter {
memoryUsed,
participatingAgents: agentOutputs.map(o => o.agentId),
metadata: {
- speedup: '2.49x-7.47x',
- memoryReduction: '75%',
+ // Speedup/memory reduction are UNMEASURED β€” no benchmark is wired.
+ // Do not advertise a made-up factor. See
+ // docs/reviews/intelligence-system-audit-2026-05-29.md
+ speedup: 'unverified',
+ memoryReduction: 'unverified',
blockSize,
},
};
@@ -967,10 +971,13 @@ export class AttentionCoordinator extends EventEmitter {
this.performanceStats.totalLatency += latency;
if (mechanism === 'flash') {
- // Track Flash Attention performance
- // In production, compare against baseline
- this.performanceStats.flashSpeedup = 2.49 + Math.random() * 4.98; // 2.49x-7.47x
- this.performanceStats.memoryReduction = 0.75;
+ // Flash Attention speedup is UNMEASURED in this build β€” no benchmark
+ // kernel is wired here. Do NOT fabricate a value. Sentinel 0 means
+ // "unmeasured"; consumers must not advertise a speedup until a real
+ // measured path (kernel benchmark vs baseline) is implemented.
+ // See docs/reviews/intelligence-system-audit-2026-05-29.md
+ this.performanceStats.flashSpeedup = 0; // 0 = unmeasured (no fabrication)
+ this.performanceStats.memoryReduction = 0; // 0 = unmeasured (no fabrication)
}
}
diff --git a/v3/CLAUDE.md b/v3/CLAUDE.md
index 6a5c02c45b..067c0c946c 100644
--- a/v3/CLAUDE.md
+++ b/v3/CLAUDE.md
@@ -31,10 +31,14 @@ npm install && npm run build && npm test
## Performance Targets
-| Metric | Target | Status |
-|--------|--------|--------|
-| HNSW Search | 150x-12,500x faster | Implemented |
-| Memory Reduction | 50-75% (Int8 quantization) | Implemented |
-| MCP Response | <100ms | Achieved |
-| CLI Startup | <500ms | Achieved |
-| Flash Attention | 2.49x-7.47x speedup | In progress |
+> Source of truth: [`docs/reviews/intelligence-system-audit-2026-05-29.md`](../docs/reviews/intelligence-system-audit-2026-05-29.md) + [`scripts/benchmark-intelligence.mjs`](../scripts/benchmark-intelligence.mjs). Numbers below are measured unless marked "target/unverified".
+
+| Metric | Measured / Target | Status |
+|--------|-------------------|--------|
+| HNSW Search | ~1.9x at N=20k, ~3.2x–4.7x at N=5k vs brute force (recall@10 ~0.99) | **Measured** (ruvector NAPI; 150x-12,500x NOT reproduced) |
+| Int8 Quantization | 3.84x compression, reconstruction cosine 0.99999 | **Measured** |
+| RaBitQ Quantization | 32x compression, 0.60ms/query | **Measured** |
+| SONA Adaptation | 0.0043ms/adapt (target <0.05ms met) | **Measured** |
+| MCP Response | <100ms | target |
+| CLI Startup | <500ms | target |
+| Flash Attention | 2.49x-7.47x | **Unverified** (no benchmark) |