🌊 The original agent harness. Deploy intelligent multi-player swarms, coordinate autonomous workflows, and build conversational AI systems. Features adaptive memory, self-learning intelligence, federation, vector RAG integration, and native Claude Code / Codex / Hermes and many more Integrated
TypeScript agentic-aiagentic-frameworkagentic-workflowagentsai-agentsai-assistantai-skillsautonomous-agentsclaudecodexdsh-pluginharnessmcp-servermulti-agentmulti-agent-systemsnpmskillsswarmswarm-intelligencetypescript
docs(adr): ADR-097 — federation-wide budget circuit breaker (#1723)
Reuven62a6fc5fb0e6
diff --git a/v3/docs/adr/ADR-097-federation-budget-circuit-breaker.md b/v3/docs/adr/ADR-097-federation-budget-circuit-breaker.mdnew file mode 100644index 0000000000..5ff9dcdf77--- /dev/null+++ b/v3/docs/adr/ADR-097-federation-budget-circuit-breaker.md@@ -0,0 +1,142 @@+# ADR-097: Federation-wide Budget Circuit Breaker and Token Quotas++**Status**: Proposed+**Date**: 2026-05-04+**Version**: target v3.6.x+**Supersedes**: nothing+**Related**: ADR-086 (Agent Federation), ADR-095 (Architectural gaps — G2 includes federation transport), issues #1723 and #1724 (#1724 closed as duplicate of #1723), commit `6f495369` (G2 Ed25519 signing in federation)++## Context++Agent Federation (ADR-086) lets agents on one Ruflo node delegate tasks to peer nodes across trust boundaries. The implementation in `@claude-flow/plugin-agent-federation` already covers identity (Ed25519 keypairs), trust scoring, and `federation_send` for cross-node delegation. What it does **not** cover today:++- **Recursive delegation loops**. Node A → Node B → Node A → … If a peer wraps a received task and delegates it back, there is no hop counter to break the cycle. A pathological multi-node ring can run until process memory or the network gives out.+- **Cost cascades**. A 200-token task delegated to a peer can spawn a sub-swarm of five worker agents on the remote, each potentially calling expensive frontier models. The originator never sees the bill until the cost-tracker reconciles after the fact.+- **No backpressure on hostile / faulty peers**. A misbehaving peer that returns expensive responses cannot be muted automatically — the local trust score drifts but there's no mechanism that says "this peer is now too expensive, stop sending it work."+- **No unified cost surface**. The `ruflo-cost-tracker` plugin tracks local agent token usage but federation traffic is invisible to it.++The original issue (#1723, #1724 dup) frames this as a stability + enterprise-readiness gap. The fix is a **federated circuit breaker** layered on top of the existing federation protocol — no breaking changes to the wire format, just additive metadata.++### Existing surface to build on++| Component | Path | Today |+|---|---|---|+| Federation MCP tool | `v3/@claude-flow/plugin-agent-federation/src/mcp-tools.ts` (`federation_send`) | Sends a task to a peer; no cost / hop awareness |+| Federation node entity | `v3/@claude-flow/plugin-agent-federation/src/domain/entities/federation-node.ts` | `trustScore`, `trustLevel`, `lastSeen`; no `state: SUSPENDED` |+| Cost tracker plugin | `plugins/ruflo-cost-tracker/` | Tracks local model spend per agent / per session |+| Behavioral trust | ADR-086 §"Trust scoring" | Adjusts on protocol misbehavior; no cost-based decay yet |++## Decision++Ship four cohesive but independently-shippable parts. Each part is one iteration; the parts compose into the full circuit breaker.++### Part 1 — Budget header on `federation_send`++Extend the `federation_send` MCP tool input schema with three optional metadata fields:++```ts+{+ // existing+ peerId: string,+ taskId: string,+ payload: unknown,+ // new (all optional, defaults preserve current behavior)+ budget?: {+ maxTokens?: number, // hard cap on Σ tokens across the whole hop chain+ maxUsd?: number, // hard cap on Σ USD spend; enforced via cost-tracker+ },+ maxHops?: number, // default 8; 0 = no remote delegation allowed+}+```++These travel inside the federation envelope as a `budget` block alongside the payload. The receiving peer:++1. **Decrements `maxHops`** on receive. If it drops below 0, returns an error response (`HOP_LIMIT_EXCEEDED`) without invoking any agent. The originator gets the failure synchronously.+2. **Subtracts from the budget** as it spends. Each completed sub-task reports `tokensUsed` and `usdSpent`; the running total is checked against the cap before each subsequent action. Overshoot returns `BUDGET_EXCEEDED` and refuses further work.+3. **Propagates the remaining budget** if the peer itself federates onward. The downstream call inherits whatever budget remains, never more.++Budget defaults: when omitted, treat as `Infinity` for `maxTokens` / `maxUsd` and `8` for `maxHops`. The default hop limit alone closes the recursion-loop class without any caller change.++### Part 2 — Peer state machine: ACTIVE / SUSPENDED / EVICTED++Extend `federation-node.ts` with a `state` field driven by the breaker:++| State | When transitioned | What it means |+|---|---|---|+| `ACTIVE` | default; healthy | `federation_send` accepts deliveries to this peer |+| `SUSPENDED` | breaker tripped (cost threshold or repeated failures) | sends to this peer return `PEER_SUSPENDED` immediately; receives still accepted but ignored for trust accumulation |+| `EVICTED` | manual or post-grace-period escalation from SUSPENDED | peer removed from registry; new delivery errors are emitted as `PEER_EVICTED` |++Transition triggers:++- `ACTIVE → SUSPENDED` when **either**:+ - Trailing 24h cost from this peer > configured threshold (`peer.costSuspensionUsd`, default `$5.00`)+ - Trailing 1h failure ratio > 50% across ≥10 sends+- `SUSPENDED → ACTIVE` after a configurable cooldown (default 30 min) AND a successful health probe+- `SUSPENDED → EVICTED` after 24h continuous suspension OR explicit `federation_evict` MCP call++Cooldown + auto-recovery prevents the breaker from being a one-way door — same shape as a typical hystrix-style breaker.++### Part 3 — Cost-tracker integration++Wire the federation layer into `ruflo-cost-tracker` so federated spend appears in the same dashboards as local spend:++1. New event type `federation_spend` published to the cost-tracker bus on every `federation_send` completion: `{ peerId, taskId, tokensUsed, usdSpent, ts }`.+2. Cost-tracker aggregates per-peer rolling windows (1h / 24h / 7d) and exposes them via the existing `cost-report` skill.+3. The breaker queries cost-tracker for the per-peer 24h sum to evaluate the suspension threshold.++This is one direction (federation → cost-tracker). Cost-tracker doesn't need to mutate federation state directly; the breaker pulls.++### Part 4 — Doctor + observability++- `ruflo doctor` reports the current state of every known peer (`ACTIVE`/`SUSPENDED`/`EVICTED`) and the trailing-24h cost. A peer pinned in `SUSPENDED` for >1h shows up as a yellow warning.+- New MCP tool `federation_breaker_status` returns the same info programmatically for swarm coordinators that want to route around suspended peers.+- Structured log line on every state transition with `{prevState, newState, reason, peerId}` so post-incident triage doesn't need a debugger.++## Implementation outline++Per the user directive, the implementation team owns this section. Each phase is one iteration to keep blast radius bounded:++| Phase | Scope | Lands in |+|---|---|---|+| P1 | Budget envelope + hop counter, no peer state changes | federation plugin + new tests |+| P2 | Peer state machine (`ACTIVE`/`SUSPENDED`/`EVICTED`) + transition rules | federation-node.ts + tests |+| P3 | Cost-tracker bus event + per-peer rolling aggregation | cost-tracker plugin |+| P4 | Doctor + `federation_breaker_status` MCP tool | doctor command + mcp-tools |++Test surface:++- **Unit**: budget arithmetic, hop counter underflow, state-transition table (ACTIVE→SUSPENDED→ACTIVE→EVICTED).+- **Integration**: a synthetic peer that intentionally exceeds budget should produce `BUDGET_EXCEEDED` on the originator AND transition to `SUSPENDED` after the threshold.+- **Property**: random hop chains must always terminate at `maxHops` ≤ 8.++## Trade-offs++| Decision | Alternative | Why this |+|---|---|---|+| Optional budget fields, default `Infinity` | Required budget on every `federation_send` | Backward compatible: existing code paths keep working. Callers opt in to limits. |+| Hop counter default `8` | Default unlimited | `8` is generous for legitimate multi-hop flows (typical ≤3) and cheap insurance against the recursion-loop class. |+| Peer state in federation plugin (not cost-tracker) | Centralized in cost-tracker | Federation owns peer identity + trust; the breaker is a federation concern. Cost-tracker provides numbers; federation decides actions. |+| Trailing-24h $5 default suspension threshold | Per-peer config only | Sane default avoids "gun, foot" for new installs. Override via plugin config. |+| `EVICTED` separate from `SUSPENDED` | Single `INACTIVE` state | Eviction is operationally permanent (admin removes the peer); suspension is a soft, auto-recovering state. They have different audit/log/recovery semantics. |+| Pull from cost-tracker for breaker decisions | Push spend events into federation | Federation already speaks events to cost-tracker; reverse direction would couple in both ways. Pull-on-decision is simpler and just-in-time. |++## Risks++1. **Race between budget check and remote spend**. The receiving peer checks budget *before* invoking the agent, but the agent's actual spend is reported *after*. A budget can be modestly exceeded by the last-call's actual cost > predicted cost. Mitigation: enforce on the *predicted* cost (max_tokens × per-token-rate), then refund unused on the running total. Worst case overshoot is one model call's worth.+2. **Clock skew across peers**. The 24h rolling window is computed locally per-peer, so divergent clocks don't change the threshold semantics. Trust-score timestamps (`lastSeen`) already rely on local clocks; same threat model.+3. **Suspended peer recovery storm**. If many peers transition `SUSPENDED → ACTIVE` simultaneously after cooldown, they could all retry pending sends at once. Mitigation: jitter the cooldown by ±10% per peer.+4. **Cost-tracker plugin not installed**. If the user runs federation without cost-tracker, USD-based breaker rules degrade to "always permitted." Token-count breaker still works (federation tracks tokens itself). Document this — cost-tracker is the recommended pair, not strictly required.++## Acceptance criteria++The full implementation is done when:++- `federation_send` accepts and propagates `budget` + `maxHops` without breaking any existing test+- A synthetic recursive delegation chain (peer A → B → A → …) terminates at `maxHops` with `HOP_LIMIT_EXCEEDED` on the originator+- A peer that exceeds the cost threshold over 24h transitions to `SUSPENDED`, refuses sends for the cooldown window, and auto-recovers on a successful probe+- `ruflo doctor` shows peer states + trailing-24h cost+- `federation_breaker_status` MCP tool returns structured state per peer+- Cost-tracker `cost-report` shows federated spend grouped by peer+- New tests cover budget arithmetic, hop counter, state transitions, and cost-tracker bus events+- Full vitest suite stays green; no regressions in the 1917-test baseline