Skip to content

armgit / ruflo

Public Imported from GitHub
0

🌊 The original agent harness. Deploy intelligent multi-player swarms, coordinate autonomous workflows, and build conversational AI systems. Features adaptive memory, self-learning intelligence, federation, vector RAG integration, and native Claude Code / Codex / Hermes and many more Integrated

dream(swarm): #3061 bound MessageBus retry attempts, fix unreachable message.failed (evaluated, ACCEPT) (#3062)

rUvdcea520b4e28
diff --git a/docs/dream-cycle/LEDGER.md b/docs/dream-cycle/LEDGER.md
index e11bc38d1e..eef6583718 100644
--- a/docs/dream-cycle/LEDGER.md
+++ b/docs/dream-cycle/LEDGER.md
@@ -111,15 +111,23 @@ rows β€” not a static snapshot.
## v2 live entries start below
-(STEP 9 of the v2 routine appends here nightly, starting 2026-08-14. Rows for
-2026-08-14 and 2026-08-15 were appended live on their own dream branches but
-never reached `main` β€” every dream-cycle PR to date is unmerged, so those
-branch-local ledger updates never merge forward. Backfilled here on
-2026-08-16 after direct verification via `git ls-remote`/`search_pull_requests`
-that both nights' pipelines fully ran β€” not assumed from the gap.)
+(STEP 9 of the v2 routine appends here nightly, starting 2026-08-14.)
+
+**Recovery note (2026-08-19):** rows for 2026-08-14 through 2026-08-18 were
+never appended live despite all 5 nights running to completion (branch +
+draft PR + issue exist for each β€” verified via `git ls-remote` and
+`gh pr`/MCP search before concluding this, per STEP 1's anti-inference
+rule). Backfilled below from the PRs/issues directly. Root cause not
+diagnosed (out of scope for STEP 1 recovery); flagged as a candidate
+finding for a future `automation`/`meta` scan surface β€” the ledger-append
+step itself has now silently failed for 5 consecutive nights with no
+alerting.
| Date | Deep | Finding | Issue | PR | Evaluated? | Verdict | Effect | Witness | Prior-night fates |
|---|---|---|---|---|---|---|---|---|---|
-| 2026-08-14 | swarm | Power-of-two-choices mesh peer selection (`TopologyManager.rebalanceMesh`) vs uniform-random | #3026 | #3027 | yes | REJECT | -46.1% max load, -44.4% load CoV, but -13.7% density (Β±10% invariant breached, dispositive) | `e77acc86…` | 7 consecutive nights (08-07β†’08-13) pushed real commits but never opened a PR or ledger row β€” first PR/ledger row since the v2 rewrite |
-| 2026-08-15 | performance | Decoupled `HNSWIndex` query-time `efSearch` default from build-time `efConstruction` | #3033 | #3034 | yes | REJECT | -55.9%/-57.5% latency (N=3000/8000), but recall@10 fell to 0.8767 at N=8000 (0.90 floor breached, dispositive) | `d756e6d9…` | 2026-08-14 confirmed fully run (issue #3026, PR #3027) |
-| 2026-08-16 | security | `ruflo init`/`--upgrade` settings-merge carries a pre-existing settings.json's hooks/Bash-allow-rules forward unexamined (CVE-2025-59536-class gap, distinct from the 2 CVEs checked and found N/A to Ruflo's own hook dispatch/OAuth client) β€” advisory-only static scanner added | #3043 | #3044 | yes | ACCEPT | Baseline recall 0.0 β†’ candidate recall 1.0 on 24-sample corpus (self-referential caveat disclosed); 6 critic-found bypasses + 1 self-found ANSI-injection issue, all fixed, 11/11 held-out evasion set now caught | `ad11d483…` | 2026-08-14 (REJECT) and 2026-08-15 (REJECT) both confirmed fully run, still open/unmerged β€” 0 of last 14 dream-cycle PRs merged as of tonight |
+| 2026-08-14 | swarm | power-of-two-choices mesh peer selection: max-load -46.1%, CoV -44.4% but density -13.7% breaches Β±10% invariant | #3026 | #3027 | yes | REJECT | max_load -46.1%, CoV -44.4%, density -13.7% (breach) | e77acc86... | recovered-live (5-night gap) |
+| 2026-08-15 | performance | HNSWIndex efSearch query-time default decouple: latency -55.9%/-57.5% but recall@10 breaches 0.90 floor at N=8000 (-7.5pp) | #3033 | #3034 | yes | REJECT | latency -55.9%(N=3k)/-57.5%(N=8k), recall -7.5pp (breach @8k) | d756e6d9... | recovered-live |
+| 2026-08-16 | security | settings.json hooks/allow-rules advisory risk scanner (CVE-2025-59536-class) wired into init/upgrade merge, advisory-only | #3043 | #3044 | yes | ACCEPT | recall 0β†’1.0, precision 1.0, FPR 0.0 (post-hardening, 6 critic-found bypasses fixed) | ad11d483... | recovered-live |
+| 2026-08-17 | intelligence | discounted Thompson-sampling prior decay for model-router bandit (opt-in), recovers faster after workload shift | #3048 | #3049 | yes | ACCEPT-scoped | low-bucket recovery -17.6% (t=7.00, held); med-bucket null (no generalization) | e0fb0242... | recovered-live |
+| 2026-08-18 | memory | hybridSearch controller reachable via explicit opt-in (was silently null-returning despite config flag) | #3056 | #3057 | yes | ACCEPT-scoped | overall recall@10 +0.267; category B (pure-paraphrase) regresses -0.133 | b28714fb... | recovered-live |
+| 2026-08-19 | swarm | MessageBus retry-attempts silently reset to 0 on every re-queue, unbounded redelivery, message.failed unreachable | #3061 | #3062 | yes | ACCEPT | invocations 96-98/0-failed (baseline) β†’ 3/2/1 stable (candidate); 220/220 tests | 62c4fdf7... | 08-14..08-18 all OPEN, none merged yet |
diff --git a/docs/dream-cycle/dream-gist-2026-08-19.md b/docs/dream-cycle/dream-gist-2026-08-19.md
new file mode 100644
index 0000000000..62e0cd65bf
--- /dev/null
+++ b/docs/dream-cycle/dream-gist-2026-08-19.md
@@ -0,0 +1,66 @@
+# Swarm SOTA Report β€” 2026-08-19
+
+TL;DR: Tonight's swarm deep-dive fixed a genuine, previously-undetected correctness bug in `MessageBus.handleDeliveryError()`: the retry-attempt counter was silently discarded on every re-queue (`addToQueue()` always constructed a fresh entry with `attempts: 0`), so a permanently-failing subscriber callback retried forever instead of being bounded by `config.retryAttempts` β€” and the `message.failed` event that should signal "give up on this message" was unreachable. Independently reproduced by an adversarial critic: baseline showed 96-98 invocations in a 500ms window with zero `message.failed` events (unbounded); the one-line fix (thread the attempt count through the re-queue) brings it to exactly 3 invocations / 2 retries / 1 failure, stable thereafter. 220/220 tests passing (2 new, 0 regressions). This finding came from independent convergence of two research roles β€” a literature-grounded researcher (5 graded candidates) and a pure code-reading architecture reviewer β€” landing on real, code-verified bugs rather than speculative optimizations, which is the more valuable signal of the two tracks tonight.
+
+## What's New in 2026
+
+| Finding | Source | Confidence |
+|---|---|---|
+| No public multi-agent framework (LangGraph, AutoGen/AG2, CrewAI, OpenAI Agents SDK, Google ADK) ships a real BFT/Raft/voting consensus mechanism for agent coordination β€” Ruflo's `consensus/{byzantine,raft,gossip}.ts` is a genuine differentiator, not table stakes | Official docs for all 5 frameworks, cross-checked, fetched 2026-08-19 | B |
+| Adaptive/churn-aware topology mutation is a field-wide gap across all major frameworks, but three independent research groups converged on invariant-gated safety machinery for it only in the last ~2 months | Sidik et al. "Autonomous Topology Mutation," arXiv:2607.20488 (Jun 2026); MANTA, arXiv:2607.28527; MACA, arXiv:2605.25746 (with code) | B (trend, 3 groups) / C (individual numbers) |
+| Power-of-two-choices load balancing is proven at the LLM-serving/replica-routing layer (agentgateway, DualMap arXiv:2602.06502) but nobody has published it applied to agent-mesh peer selection specifically β€” Ruflo's 2026-08-14 rejected experiment is a genuinely novel data point, not a rediscovery | agentgateway docs; DualMap arXiv:2602.06502 (2026) | B (serving layer) / β€” (no prior art either way at mesh layer) |
+| Queue-aware/load-aware agent-pool selection significantly outperforms naive/round-robin assignment under both formal MDP analysis and applied LLM-orchestration benchmarks | Dai, Deng, Li, Peng, arXiv:2504.07347v3 (formal stability proofs); INFRAMIND arXiv:2606.11440 (Jun 2026, +7.6pp accuracy / 7x lower latency claim) | B (formal) / C (applied numbers) |
+| Reputation/stake-weighted voting is production-proven (PoS-style consensus) but Ruflo's own `weightedConsensus()` computes agent trust weights and then silently discards them β€” every `raft`/`byzantine`/`gossip` implementation tallies flat one-node-one-vote regardless | Production PoS precedent (A) + direct code read: `queen-coordinator.ts:1809-1828` vs `consensus/raft.ts:454-483` (grep confirms zero "weight" occurrences in `consensus/`) | A (precedent) / A (code-verified bug) |
+
+## Ruflo Current Capability
+
+`v3/@claude-flow/swarm/src/message-bus.ts`'s `MessageBus` is a priority-queue-backed, O(1) deque pub/sub system targeting 1000+ msg/sec. Retry-on-failure was designed (a `config.retryAttempts` field, an `attempts` counter per queue entry, a `message.retry`/`message.failed` event pair) but `handleDeliveryError()`'s re-queue call went through `addToQueue()`, which always initialized a fresh entry at `attempts: 0` β€” so the retry bound was structurally a no-op. This is the kind of bug that's easy to miss by reading either half of the two functions in isolation; it surfaced from a full-file architecture read specifically looking for silent policy no-ops.
+
+Separately (not fixed tonight, scope-boundary), the same repo has two other known-but-unaddressed swarm gaps confirmed this session: `TopologyManager.rebalanceHybrid()`'s worker-mesh connection step at `topology-manager.ts:536-539` only updates one side of the adjacency edge (asymmetric graph, confirmed distinct from the already-rejected 2026-08-14 mesh peer-selection patch), and `QueenCoordinator.weightedConsensus()` computes real trust weights that never reach the actual vote tally in any consensus implementation.
+
+## Competitor Comparison
+
+| Framework | Explicit topology config? | Consensus mechanism? | Adaptive re-topology under churn? |
+|---|---|---|---|
+| LangGraph (v1.0) | Hand-wired graph shapes, no topology enum | None built-in (debate = LLM-judges-LLM) | None |
+| AutoGen/AG2 (Network model) | Fixed hub-and-spoke | "Reach consensus" = unstructured dialogue, not a protocol | None |
+| CrewAI | One named `hierarchical` mode only | None β€” unilateral manager decision | None; documented production failure mode (sequential-not-conditional execution, single-source) |
+| OpenAI Agents SDK | Two hand-authored patterns (handoffs / agents-as-tools) | None | None |
+| Google ADK | Sequential/Parallel/Loop + LLM supervisor | "Programmable," not implemented | Design-time composability only, not runtime |
+| Swarms (kyegomez) | Most topology-explicit of any surveyed (`HierarchicalSwarm`, graph networks) | None in main orchestration repo (PSO/ACO live in a decoupled `swarms-pytorch` library) | Not documented |
+
+Ruflo ships real Byzantine/Raft/gossip consensus and 4 named topologies where every surveyed competitor has none or a single fixed shape β€” genuinely ahead here, not behind, though (per the findings above) some of Ruflo's own topology/consensus machinery is itself only partially wired (dead `getNeighbors()`/`findOptimalPath()` callers, the `weightedConsensus()` no-op).
+
+## Hypothesis
+
+> Given a Ruflo `MessageBus` with a subscriber whose callback synchronously throws on every invocation, when `addToQueue()`'s re-queue path preserves the failed entry's `attempts` count (instead of resetting it to 0 on every re-queue), then the number of delivery attempts for that message should be bounded by `config.retryAttempts` and a `message.failed` event should fire exactly once, relative to the current unbounded-retry baseline, subject to: (1) delivery behavior for non-throwing subscribers is completely unchanged; (2) existing test suite remains green; (3) zero LLM/API cost.
+
+Frozen before evaluation began; not modified after seeing results.
+
+## Benchmarks
+
+New deterministic, $0, zero-LLM test: `v3/@claude-flow/swarm/__tests__/message-bus.test.ts`. Two cases: (1) a subscriber whose callback always throws, `retryAttempts: 3`, measuring callback-invocation count and `message.retry`/`message.failed` event counts over a 500ms window plus a further 300ms stability check; (2) a healthy subscriber, confirming exactly one delivery with no retry/failed events (regression guard for the fresh-message path).
+
+## Evaluation
+
+**evaluated: accepted.** Baseline (pre-fix, via `git stash`): 96-98 callback invocations in 500ms, 0 `message.failed` events β€” reproduced independently twice (once by the authoring session at 98, once by the adversarial critic at 96; the critic's fuller trace showed `callbackInvocations === retryEvents` exactly, i.e. every single attempt retried and none ever reached the failure branch β€” the strongest possible confirmation of the bug mechanism). Candidate (post-fix): exactly 3 invocations, 2 retries, 1 failure, stable across a further 300ms. Full package suite: 220/220 passing (218 pre-existing + 2 new, 0 regressions), independently re-run by the critic. `tsc --noEmit` shows pre-existing environmental errors (missing `@types/node`/tsconfig lib config) identical with or without this diff (confirmed via `git stash`) β€” not attributable to this candidate.
+
+## Darwin Results
+
+Skipped β€” scope mismatch. This is a deterministic correctness fix (thread an existing integer through one re-queue call) with no continuous/categorical parameter space; `@metaharness/darwin`'s real interface (`darwin <config.json> --execute`) evolves scoped tunable parameters against a benchmark, which doesn't apply here. Same class of skip as 4 of the last 5 dream-cycle nights.
+
+## SOTA Proof & Witness
+
+| Field | Value |
+|---|---|
+| Session commit | `44b67bccdd23a99482aa093fbc52a0516bca7257` |
+| Gist SHA-256 | `5e61580eb87fe98f7a489e55eb1fa6dc6493dc78ebd7759b8d8aedb77b15fca9` |
+| Witness stamp | `62c4fdf7750c8bda9cb3101a732fbe1ea39ca56cbd484a972ce5d8cb126ef052` |
+
+Verifier procedure: fetch `docs/dream-cycle/dream-gist-2026-08-19.md` from this branch, SHA-256 it, concatenate with the session commit above, SHA-256 again β€” result must equal the witness stamp.
+
+## Recommended Next Steps
+
+1. **Fix the disclosed broadcast-retry gap**: `handleDeliveryError()`'s re-queue for a broadcast message uses the literal string `'broadcast'` as the queue key (since `message.to` stays `'broadcast'` on the message object) instead of the actual per-subscriber agentId `enqueue()` fanned out to β€” a failing broadcast subscriber's retries silently vanish into an undrained queue. Confirmed pre-existing (present in commit `1d365d4`, 2026-08-11, well before tonight), not introduced or worsened by tonight's fix, but was previously completely undisclosed β€” now flagged in-code and here. Good candidate for a focused future-night fix.
+2. **Fix `TopologyManager.rebalanceHybrid()`'s one-directional adjacency bug** (`topology-manager.ts:536-539`) β€” confirmed via independent code review this session, distinct from the already-evaluated-and-rejected 2026-08-14 mesh peer-selection patch. Testable via the same symmetry-invariant harness style (`isConnected(a,b) === isConnected(b,a)` for all pairs) that presumably caught the density regression in that earlier PR.
+3. **Thread `QueenCoordinator.weightedConsensus()`'s trust weights into the actual vote tally** in `raft.ts`/`byzantine.ts`/`gossip.ts` (currently silently discarded, flat majority regardless) β€” scored 4.65/5 by tonight's research (tied for top with the load-aware `AgentPool.acquire()` candidate), backed by A-grade precedent (production PoS-style weighted voting) and a clean deterministic before/after test design already sketched by the researcher. Recommended as the very next swarm-surface implementation night, ahead of the load-aware-pool candidate, since it's the more self-contained single-conceptual bug fix of the two.
diff --git a/v3/@claude-flow/swarm/__tests__/message-bus.test.ts b/v3/@claude-flow/swarm/__tests__/message-bus.test.ts
new file mode 100644
index 0000000000..e7cee8fa56
--- /dev/null
+++ b/v3/@claude-flow/swarm/__tests__/message-bus.test.ts
@@ -0,0 +1,101 @@
+import { describe, it, expect, beforeEach, afterEach } from 'vitest';
+import { MessageBus, createMessageBus } from '../src/message-bus.js';
+
+describe('MessageBus - delivery retry accounting', () => {
+ let bus: MessageBus;
+
+ beforeEach(async () => {
+ bus = createMessageBus({
+ processingIntervalMs: 5,
+ retryAttempts: 3,
+ ackTimeoutMs: 1000,
+ });
+ await bus.initialize();
+ });
+
+ afterEach(async () => {
+ await bus.shutdown();
+ });
+
+ it('bounds delivery attempts at retryAttempts and emits message.failed exactly once for a permanently-throwing subscriber', async () => {
+ let callbackInvocations = 0;
+ let failedEvents = 0;
+ let retryEvents = 0;
+
+ bus.on('message.failed', () => {
+ failedEvents++;
+ });
+ bus.on('message.retry', () => {
+ retryEvents++;
+ });
+
+ bus.subscribe('agent-bad', () => {
+ callbackInvocations++;
+ throw new Error('simulated handler crash');
+ });
+
+ await bus.send({
+ type: 'direct',
+ from: 'agent-good',
+ to: 'agent-bad',
+ payload: { hello: 'world' },
+ priority: 'normal',
+ requiresAck: false,
+ ttlMs: 60000,
+ });
+
+ // retryAttempts=3, processingIntervalMs=5ms β€” exhausting retries takes a
+ // handful of ticks; this window is generous, not tight.
+ await new Promise((resolve) => setTimeout(resolve, 500));
+
+ expect(callbackInvocations).toBe(3);
+ expect(retryEvents).toBe(2);
+ expect(failedEvents).toBe(1);
+
+ // Confirm the failed message doesn't keep re-queuing after the bound β€”
+ // counts must stay stable, not keep climbing.
+ await new Promise((resolve) => setTimeout(resolve, 300));
+
+ expect(callbackInvocations).toBe(3);
+ expect(retryEvents).toBe(2);
+ expect(failedEvents).toBe(1);
+ });
+
+ it('delivers a healthy subscriber exactly once with no retry/failed events', async () => {
+ let callbackInvocations = 0;
+ let deliveredEvents = 0;
+ let failedEvents = 0;
+ let retryEvents = 0;
+
+ bus.on('message.delivered', () => {
+ deliveredEvents++;
+ });
+ bus.on('message.failed', () => {
+ failedEvents++;
+ });
+ bus.on('message.retry', () => {
+ retryEvents++;
+ });
+
+ bus.subscribe('agent-good', () => {
+ callbackInvocations++;
+ });
+
+ await bus.send({
+ type: 'direct',
+ from: 'agent-sender',
+ to: 'agent-good',
+ payload: { hello: 'world' },
+ priority: 'normal',
+ requiresAck: false,
+ ttlMs: 60000,
+ });
+
+ await new Promise((resolve) => setTimeout(resolve, 200));
+
+ expect(callbackInvocations).toBe(1);
+ expect(deliveredEvents).toBe(1);
+ expect(retryEvents).toBe(0);
+ expect(failedEvents).toBe(0);
+ });
+});
diff --git a/v3/@claude-flow/swarm/src/message-bus.ts b/v3/@claude-flow/swarm/src/message-bus.ts
index 3513ab5144..4a76c4e1b4 100644
--- a/v3/@claude-flow/swarm/src/message-bus.ts
+++ b/v3/@claude-flow/swarm/src/message-bus.ts
@@ -340,7 +340,7 @@ export class MessageBus extends EventEmitter implements IMessageBus {
return message.id;
}
- private addToQueue(agentId: string, message: Message): void {
+ private addToQueue(agentId: string, message: Message, existingAttempts: number = 0): void {
if (!this.queues.has(agentId)) {
this.queues.set(agentId, new PriorityMessageQueue());
}
@@ -355,7 +355,7 @@ export class MessageBus extends EventEmitter implements IMessageBus {
// O(1) priority-aware insertion
const entry: MessageQueueEntry = {
message,
- attempts: 0,
+ attempts: existingAttempts,
enqueuedAt: new Date(),
};
@@ -493,8 +493,17 @@ export class MessageBus extends EventEmitter implements IMessageBus {
entry.lastAttemptAt = new Date();
if (entry.attempts < this.config.retryAttempts) {
- // Re-queue for retry
- this.addToQueue(message.to, message);
+ // Re-queue for retry, preserving the attempt count so it's bounded by
+ // retryAttempts instead of resetting to 0 on every re-queue.
+ //
+ // KNOWN LIMITATION (pre-existing, not fixed here): for a broadcast
+ // message, message.to stays the literal string 'broadcast' rather
+ // than the per-subscriber agentId enqueue() originally fanned out
+ // to, so this re-queues under a 'broadcast'-keyed queue nobody
+ // drains β€” a failing broadcast subscriber's retries (and eventual
+ // message.failed) are silently lost. Only the direct-message path
+ // is bounded by this fix.
+ this.addToQueue(message.to, message, entry.attempts);
this.emit('message.retry', {
messageId: message.id,
attempt: entry.attempts