π The original agent harness. Deploy intelligent multi-player swarms, coordinate autonomous workflows, and build conversational AI systems. Features adaptive memory, self-learning intelligence, federation, vector RAG integration, and native Claude Code / Codex / Hermes and many more Integrated
TypeScript agentic-aiagentic-frameworkagentic-workflowagentsai-agentsai-assistantai-skillsautonomous-agentsclaudecodexdsh-pluginharnessmcp-servermulti-agentmulti-agent-systemsnpmskillsswarmswarm-intelligencetypescript
dream(swarm): #3061 bound MessageBus retry attempts, fix unreachable message.failed (evaluated, ACCEPT) (#3062)
rUvdcea520b4e28
diff --git a/docs/dream-cycle/LEDGER.md b/docs/dream-cycle/LEDGER.mdindex e11bc38d1e..eef6583718 100644--- a/docs/dream-cycle/LEDGER.md+++ b/docs/dream-cycle/LEDGER.md@@ -111,15 +111,23 @@ rows β not a static snapshot.## v2 live entries start below-(STEP 9 of the v2 routine appends here nightly, starting 2026-08-14. Rows for-2026-08-14 and 2026-08-15 were appended live on their own dream branches but-never reached `main` β every dream-cycle PR to date is unmerged, so those-branch-local ledger updates never merge forward. Backfilled here on-2026-08-16 after direct verification via `git ls-remote`/`search_pull_requests`-that both nights' pipelines fully ran β not assumed from the gap.)+(STEP 9 of the v2 routine appends here nightly, starting 2026-08-14.)++**Recovery note (2026-08-19):** rows for 2026-08-14 through 2026-08-18 were+never appended live despite all 5 nights running to completion (branch ++draft PR + issue exist for each β verified via `git ls-remote` and+`gh pr`/MCP search before concluding this, per STEP 1's anti-inference+rule). Backfilled below from the PRs/issues directly. Root cause not+diagnosed (out of scope for STEP 1 recovery); flagged as a candidate+finding for a future `automation`/`meta` scan surface β the ledger-append+step itself has now silently failed for 5 consecutive nights with no+alerting.| Date | Deep | Finding | Issue | PR | Evaluated? | Verdict | Effect | Witness | Prior-night fates ||---|---|---|---|---|---|---|---|---|---|-| 2026-08-14 | swarm | Power-of-two-choices mesh peer selection (`TopologyManager.rebalanceMesh`) vs uniform-random | #3026 | #3027 | yes | REJECT | -46.1% max load, -44.4% load CoV, but -13.7% density (Β±10% invariant breached, dispositive) | `e77acc86β¦` | 7 consecutive nights (08-07β08-13) pushed real commits but never opened a PR or ledger row β first PR/ledger row since the v2 rewrite |-| 2026-08-15 | performance | Decoupled `HNSWIndex` query-time `efSearch` default from build-time `efConstruction` | #3033 | #3034 | yes | REJECT | -55.9%/-57.5% latency (N=3000/8000), but recall@10 fell to 0.8767 at N=8000 (0.90 floor breached, dispositive) | `d756e6d9β¦` | 2026-08-14 confirmed fully run (issue #3026, PR #3027) |-| 2026-08-16 | security | `ruflo init`/`--upgrade` settings-merge carries a pre-existing settings.json's hooks/Bash-allow-rules forward unexamined (CVE-2025-59536-class gap, distinct from the 2 CVEs checked and found N/A to Ruflo's own hook dispatch/OAuth client) β advisory-only static scanner added | #3043 | #3044 | yes | ACCEPT | Baseline recall 0.0 β candidate recall 1.0 on 24-sample corpus (self-referential caveat disclosed); 6 critic-found bypasses + 1 self-found ANSI-injection issue, all fixed, 11/11 held-out evasion set now caught | `ad11d483β¦` | 2026-08-14 (REJECT) and 2026-08-15 (REJECT) both confirmed fully run, still open/unmerged β 0 of last 14 dream-cycle PRs merged as of tonight |+| 2026-08-14 | swarm | power-of-two-choices mesh peer selection: max-load -46.1%, CoV -44.4% but density -13.7% breaches Β±10% invariant | #3026 | #3027 | yes | REJECT | max_load -46.1%, CoV -44.4%, density -13.7% (breach) | e77acc86... | recovered-live (5-night gap) |+| 2026-08-15 | performance | HNSWIndex efSearch query-time default decouple: latency -55.9%/-57.5% but recall@10 breaches 0.90 floor at N=8000 (-7.5pp) | #3033 | #3034 | yes | REJECT | latency -55.9%(N=3k)/-57.5%(N=8k), recall -7.5pp (breach @8k) | d756e6d9... | recovered-live |+| 2026-08-16 | security | settings.json hooks/allow-rules advisory risk scanner (CVE-2025-59536-class) wired into init/upgrade merge, advisory-only | #3043 | #3044 | yes | ACCEPT | recall 0β1.0, precision 1.0, FPR 0.0 (post-hardening, 6 critic-found bypasses fixed) | ad11d483... | recovered-live |+| 2026-08-17 | intelligence | discounted Thompson-sampling prior decay for model-router bandit (opt-in), recovers faster after workload shift | #3048 | #3049 | yes | ACCEPT-scoped | low-bucket recovery -17.6% (t=7.00, held); med-bucket null (no generalization) | e0fb0242... | recovered-live |+| 2026-08-18 | memory | hybridSearch controller reachable via explicit opt-in (was silently null-returning despite config flag) | #3056 | #3057 | yes | ACCEPT-scoped | overall recall@10 +0.267; category B (pure-paraphrase) regresses -0.133 | b28714fb... | recovered-live |+| 2026-08-19 | swarm | MessageBus retry-attempts silently reset to 0 on every re-queue, unbounded redelivery, message.failed unreachable | #3061 | #3062 | yes | ACCEPT | invocations 96-98/0-failed (baseline) β 3/2/1 stable (candidate); 220/220 tests | 62c4fdf7... | 08-14..08-18 all OPEN, none merged yet |diff --git a/docs/dream-cycle/dream-gist-2026-08-19.md b/docs/dream-cycle/dream-gist-2026-08-19.mdnew file mode 100644index 0000000000..62e0cd65bf--- /dev/null+++ b/docs/dream-cycle/dream-gist-2026-08-19.md@@ -0,0 +1,66 @@+# Swarm SOTA Report β 2026-08-19++TL;DR: Tonight's swarm deep-dive fixed a genuine, previously-undetected correctness bug in `MessageBus.handleDeliveryError()`: the retry-attempt counter was silently discarded on every re-queue (`addToQueue()` always constructed a fresh entry with `attempts: 0`), so a permanently-failing subscriber callback retried forever instead of being bounded by `config.retryAttempts` β and the `message.failed` event that should signal "give up on this message" was unreachable. Independently reproduced by an adversarial critic: baseline showed 96-98 invocations in a 500ms window with zero `message.failed` events (unbounded); the one-line fix (thread the attempt count through the re-queue) brings it to exactly 3 invocations / 2 retries / 1 failure, stable thereafter. 220/220 tests passing (2 new, 0 regressions). This finding came from independent convergence of two research roles β a literature-grounded researcher (5 graded candidates) and a pure code-reading architecture reviewer β landing on real, code-verified bugs rather than speculative optimizations, which is the more valuable signal of the two tracks tonight.++## What's New in 2026++| Finding | Source | Confidence |+|---|---|---|+| No public multi-agent framework (LangGraph, AutoGen/AG2, CrewAI, OpenAI Agents SDK, Google ADK) ships a real BFT/Raft/voting consensus mechanism for agent coordination β Ruflo's `consensus/{byzantine,raft,gossip}.ts` is a genuine differentiator, not table stakes | Official docs for all 5 frameworks, cross-checked, fetched 2026-08-19 | B |+| Adaptive/churn-aware topology mutation is a field-wide gap across all major frameworks, but three independent research groups converged on invariant-gated safety machinery for it only in the last ~2 months | Sidik et al. "Autonomous Topology Mutation," arXiv:2607.20488 (Jun 2026); MANTA, arXiv:2607.28527; MACA, arXiv:2605.25746 (with code) | B (trend, 3 groups) / C (individual numbers) |+| Power-of-two-choices load balancing is proven at the LLM-serving/replica-routing layer (agentgateway, DualMap arXiv:2602.06502) but nobody has published it applied to agent-mesh peer selection specifically β Ruflo's 2026-08-14 rejected experiment is a genuinely novel data point, not a rediscovery | agentgateway docs; DualMap arXiv:2602.06502 (2026) | B (serving layer) / β (no prior art either way at mesh layer) |+| Queue-aware/load-aware agent-pool selection significantly outperforms naive/round-robin assignment under both formal MDP analysis and applied LLM-orchestration benchmarks | Dai, Deng, Li, Peng, arXiv:2504.07347v3 (formal stability proofs); INFRAMIND arXiv:2606.11440 (Jun 2026, +7.6pp accuracy / 7x lower latency claim) | B (formal) / C (applied numbers) |+| Reputation/stake-weighted voting is production-proven (PoS-style consensus) but Ruflo's own `weightedConsensus()` computes agent trust weights and then silently discards them β every `raft`/`byzantine`/`gossip` implementation tallies flat one-node-one-vote regardless | Production PoS precedent (A) + direct code read: `queen-coordinator.ts:1809-1828` vs `consensus/raft.ts:454-483` (grep confirms zero "weight" occurrences in `consensus/`) | A (precedent) / A (code-verified bug) |++## Ruflo Current Capability++`v3/@claude-flow/swarm/src/message-bus.ts`'s `MessageBus` is a priority-queue-backed, O(1) deque pub/sub system targeting 1000+ msg/sec. Retry-on-failure was designed (a `config.retryAttempts` field, an `attempts` counter per queue entry, a `message.retry`/`message.failed` event pair) but `handleDeliveryError()`'s re-queue call went through `addToQueue()`, which always initialized a fresh entry at `attempts: 0` β so the retry bound was structurally a no-op. This is the kind of bug that's easy to miss by reading either half of the two functions in isolation; it surfaced from a full-file architecture read specifically looking for silent policy no-ops.++Separately (not fixed tonight, scope-boundary), the same repo has two other known-but-unaddressed swarm gaps confirmed this session: `TopologyManager.rebalanceHybrid()`'s worker-mesh connection step at `topology-manager.ts:536-539` only updates one side of the adjacency edge (asymmetric graph, confirmed distinct from the already-rejected 2026-08-14 mesh peer-selection patch), and `QueenCoordinator.weightedConsensus()` computes real trust weights that never reach the actual vote tally in any consensus implementation.++## Competitor Comparison++| Framework | Explicit topology config? | Consensus mechanism? | Adaptive re-topology under churn? |+|---|---|---|---|+| LangGraph (v1.0) | Hand-wired graph shapes, no topology enum | None built-in (debate = LLM-judges-LLM) | None |+| AutoGen/AG2 (Network model) | Fixed hub-and-spoke | "Reach consensus" = unstructured dialogue, not a protocol | None |+| CrewAI | One named `hierarchical` mode only | None β unilateral manager decision | None; documented production failure mode (sequential-not-conditional execution, single-source) |+| OpenAI Agents SDK | Two hand-authored patterns (handoffs / agents-as-tools) | None | None |+| Google ADK | Sequential/Parallel/Loop + LLM supervisor | "Programmable," not implemented | Design-time composability only, not runtime |+| Swarms (kyegomez) | Most topology-explicit of any surveyed (`HierarchicalSwarm`, graph networks) | None in main orchestration repo (PSO/ACO live in a decoupled `swarms-pytorch` library) | Not documented |++Ruflo ships real Byzantine/Raft/gossip consensus and 4 named topologies where every surveyed competitor has none or a single fixed shape β genuinely ahead here, not behind, though (per the findings above) some of Ruflo's own topology/consensus machinery is itself only partially wired (dead `getNeighbors()`/`findOptimalPath()` callers, the `weightedConsensus()` no-op).++## Hypothesis++> Given a Ruflo `MessageBus` with a subscriber whose callback synchronously throws on every invocation, when `addToQueue()`'s re-queue path preserves the failed entry's `attempts` count (instead of resetting it to 0 on every re-queue), then the number of delivery attempts for that message should be bounded by `config.retryAttempts` and a `message.failed` event should fire exactly once, relative to the current unbounded-retry baseline, subject to: (1) delivery behavior for non-throwing subscribers is completely unchanged; (2) existing test suite remains green; (3) zero LLM/API cost.++Frozen before evaluation began; not modified after seeing results.++## Benchmarks++New deterministic, $0, zero-LLM test: `v3/@claude-flow/swarm/__tests__/message-bus.test.ts`. Two cases: (1) a subscriber whose callback always throws, `retryAttempts: 3`, measuring callback-invocation count and `message.retry`/`message.failed` event counts over a 500ms window plus a further 300ms stability check; (2) a healthy subscriber, confirming exactly one delivery with no retry/failed events (regression guard for the fresh-message path).++## Evaluation++**evaluated: accepted.** Baseline (pre-fix, via `git stash`): 96-98 callback invocations in 500ms, 0 `message.failed` events β reproduced independently twice (once by the authoring session at 98, once by the adversarial critic at 96; the critic's fuller trace showed `callbackInvocations === retryEvents` exactly, i.e. every single attempt retried and none ever reached the failure branch β the strongest possible confirmation of the bug mechanism). Candidate (post-fix): exactly 3 invocations, 2 retries, 1 failure, stable across a further 300ms. Full package suite: 220/220 passing (218 pre-existing + 2 new, 0 regressions), independently re-run by the critic. `tsc --noEmit` shows pre-existing environmental errors (missing `@types/node`/tsconfig lib config) identical with or without this diff (confirmed via `git stash`) β not attributable to this candidate.++## Darwin Results++Skipped β scope mismatch. This is a deterministic correctness fix (thread an existing integer through one re-queue call) with no continuous/categorical parameter space; `@metaharness/darwin`'s real interface (`darwin <config.json> --execute`) evolves scoped tunable parameters against a benchmark, which doesn't apply here. Same class of skip as 4 of the last 5 dream-cycle nights.++## SOTA Proof & Witness++| Field | Value |+|---|---|+| Session commit | `44b67bccdd23a99482aa093fbc52a0516bca7257` |+| Gist SHA-256 | `5e61580eb87fe98f7a489e55eb1fa6dc6493dc78ebd7759b8d8aedb77b15fca9` |+| Witness stamp | `62c4fdf7750c8bda9cb3101a732fbe1ea39ca56cbd484a972ce5d8cb126ef052` |++Verifier procedure: fetch `docs/dream-cycle/dream-gist-2026-08-19.md` from this branch, SHA-256 it, concatenate with the session commit above, SHA-256 again β result must equal the witness stamp.++## Recommended Next Steps++1. **Fix the disclosed broadcast-retry gap**: `handleDeliveryError()`'s re-queue for a broadcast message uses the literal string `'broadcast'` as the queue key (since `message.to` stays `'broadcast'` on the message object) instead of the actual per-subscriber agentId `enqueue()` fanned out to β a failing broadcast subscriber's retries silently vanish into an undrained queue. Confirmed pre-existing (present in commit `1d365d4`, 2026-08-11, well before tonight), not introduced or worsened by tonight's fix, but was previously completely undisclosed β now flagged in-code and here. Good candidate for a focused future-night fix.+2. **Fix `TopologyManager.rebalanceHybrid()`'s one-directional adjacency bug** (`topology-manager.ts:536-539`) β confirmed via independent code review this session, distinct from the already-evaluated-and-rejected 2026-08-14 mesh peer-selection patch. Testable via the same symmetry-invariant harness style (`isConnected(a,b) === isConnected(b,a)` for all pairs) that presumably caught the density regression in that earlier PR.+3. **Thread `QueenCoordinator.weightedConsensus()`'s trust weights into the actual vote tally** in `raft.ts`/`byzantine.ts`/`gossip.ts` (currently silently discarded, flat majority regardless) β scored 4.65/5 by tonight's research (tied for top with the load-aware `AgentPool.acquire()` candidate), backed by A-grade precedent (production PoS-style weighted voting) and a clean deterministic before/after test design already sketched by the researcher. Recommended as the very next swarm-surface implementation night, ahead of the load-aware-pool candidate, since it's the more self-contained single-conceptual bug fix of the two.diff --git a/v3/@claude-flow/swarm/__tests__/message-bus.test.ts b/v3/@claude-flow/swarm/__tests__/message-bus.test.tsnew file mode 100644index 0000000000..e7cee8fa56--- /dev/null+++ b/v3/@claude-flow/swarm/__tests__/message-bus.test.ts@@ -0,0 +1,101 @@+import { describe, it, expect, beforeEach, afterEach } from 'vitest';+import { MessageBus, createMessageBus } from '../src/message-bus.js';++describe('MessageBus - delivery retry accounting', () => {+ let bus: MessageBus;++ beforeEach(async () => {+ bus = createMessageBus({+ processingIntervalMs: 5,+ retryAttempts: 3,+ ackTimeoutMs: 1000,+ });+ await bus.initialize();+ });++ afterEach(async () => {+ await bus.shutdown();+ });++ it('bounds delivery attempts at retryAttempts and emits message.failed exactly once for a permanently-throwing subscriber', async () => {+ let callbackInvocations = 0;+ let failedEvents = 0;+ let retryEvents = 0;++ bus.on('message.failed', () => {+ failedEvents++;+ });+ bus.on('message.retry', () => {+ retryEvents++;+ });++ bus.subscribe('agent-bad', () => {+ callbackInvocations++;+ throw new Error('simulated handler crash');+ });++ await bus.send({+ type: 'direct',+ from: 'agent-good',+ to: 'agent-bad',+ payload: { hello: 'world' },+ priority: 'normal',+ requiresAck: false,+ ttlMs: 60000,+ });++ // retryAttempts=3, processingIntervalMs=5ms β exhausting retries takes a+ // handful of ticks; this window is generous, not tight.+ await new Promise((resolve) => setTimeout(resolve, 500));++ expect(callbackInvocations).toBe(3);+ expect(retryEvents).toBe(2);+ expect(failedEvents).toBe(1);++ // Confirm the failed message doesn't keep re-queuing after the bound β+ // counts must stay stable, not keep climbing.+ await new Promise((resolve) => setTimeout(resolve, 300));++ expect(callbackInvocations).toBe(3);+ expect(retryEvents).toBe(2);+ expect(failedEvents).toBe(1);+ });++ it('delivers a healthy subscriber exactly once with no retry/failed events', async () => {+ let callbackInvocations = 0;+ let deliveredEvents = 0;+ let failedEvents = 0;+ let retryEvents = 0;++ bus.on('message.delivered', () => {+ deliveredEvents++;+ });+ bus.on('message.failed', () => {+ failedEvents++;+ });+ bus.on('message.retry', () => {+ retryEvents++;+ });++ bus.subscribe('agent-good', () => {+ callbackInvocations++;+ });++ await bus.send({+ type: 'direct',+ from: 'agent-sender',+ to: 'agent-good',+ payload: { hello: 'world' },+ priority: 'normal',+ requiresAck: false,+ ttlMs: 60000,+ });++ await new Promise((resolve) => setTimeout(resolve, 200));++ expect(callbackInvocations).toBe(1);+ expect(deliveredEvents).toBe(1);+ expect(retryEvents).toBe(0);+ expect(failedEvents).toBe(0);+ });+});diff --git a/v3/@claude-flow/swarm/src/message-bus.ts b/v3/@claude-flow/swarm/src/message-bus.tsindex 3513ab5144..4a76c4e1b4 100644--- a/v3/@claude-flow/swarm/src/message-bus.ts+++ b/v3/@claude-flow/swarm/src/message-bus.ts@@ -340,7 +340,7 @@ export class MessageBus extends EventEmitter implements IMessageBus {return message.id;}- private addToQueue(agentId: string, message: Message): void {+ private addToQueue(agentId: string, message: Message, existingAttempts: number = 0): void {if (!this.queues.has(agentId)) {this.queues.set(agentId, new PriorityMessageQueue());}@@ -355,7 +355,7 @@ export class MessageBus extends EventEmitter implements IMessageBus {// O(1) priority-aware insertionconst entry: MessageQueueEntry = {message,- attempts: 0,+ attempts: existingAttempts,enqueuedAt: new Date(),};@@ -493,8 +493,17 @@ export class MessageBus extends EventEmitter implements IMessageBus {entry.lastAttemptAt = new Date();if (entry.attempts < this.config.retryAttempts) {- // Re-queue for retry- this.addToQueue(message.to, message);+ // Re-queue for retry, preserving the attempt count so it's bounded by+ // retryAttempts instead of resetting to 0 on every re-queue.+ //+ // KNOWN LIMITATION (pre-existing, not fixed here): for a broadcast+ // message, message.to stays the literal string 'broadcast' rather+ // than the per-subscriber agentId enqueue() originally fanned out+ // to, so this re-queues under a 'broadcast'-keyed queue nobody+ // drains β a failing broadcast subscriber's retries (and eventual+ // message.failed) are silently lost. Only the direct-message path+ // is bounded by this fix.+ this.addToQueue(message.to, message, entry.attempts);this.emit('message.retry', {messageId: message.id,attempt: entry.attempts