Skip to content

armgit / Piik

Public Imported from GitHub
0

Free, open-source screen sharing for private live streams with friends. Watch together in a browser or self-host Piik. 免费开源的私密屏幕共享,支持游戏直播、一起看电影与自部署。

feat(media): reselect healthy SFU roots [skip ci]

AAA9e3ebe26ce7d
diff --git a/docs/adr/0005-automatic-hybrid-media-routing.md b/docs/adr/0005-automatic-hybrid-media-routing.md
index 8c278667..5e7bcff2 100644
--- a/docs/adr/0005-automatic-hybrid-media-routing.md
+++ b/docs/adr/0005-automatic-hybrid-media-routing.md
@@ -459,20 +459,72 @@ mobile, or resource evidence. Chrome reported `candidateType=prflx` for the same
local candidate; the standards-bounded interpretation and raw-counter summary
are retained in the linked research rather than normalized away.
-### Deferred Optimization
+### Deferred Optimization And Source-Only Follow-Up
-These items are not current runtime behavior:
+These items are not deployed runtime behavior; source-only status is called out
+explicitly:
- A room in which the Host and every possible root are restricted may require
several server-fed exceptional edges. Any such extension requires a per-room
selected-relay and central-egress admission cap; it must wait or fail at that
cap rather than become unbounded server fanout.
-- ICE restart and connection rebuild already recover failed edges after Wi-Fi,
- cellular, or similar network changes. A healthy fallback path remains sticky:
- the current controller does not proactively move it back when a new Viewer
- offers a better peer route or the old network recovers. Any later preference
- migration must be triggered by a discrete event, observe cooldown, and move
- only the affected Viewer-rooted subtree.
+- Bounded healthy SFU-to-peer reselection is accepted below. The first
+ source-only slice is implemented for one SFU root and its existing baseline
+ parent; it is not deployed or browser-canary evidence.
+
+#### Bounded Healthy SFU Reselection
+
+Production release `6634cb9` was observed on 2026-08-21 to retain an SFU route
+after network recovery or a short page refresh. A same-tab refresh retains the
+room-scoped `clientId`, while the five-second disconnect grace retains peer and
+route identity, so reauthentication legitimately receives the sticky SFU
+assignment. Grace expiry removes that root and lets a later join start from the
+peer baseline.
+
+Healthy reselection is event-driven and room-bounded. It must not poll
+`navigator.connection`, run a periodic route score, or globally rebalance a
+healthy room. Only these discrete events open one opportunity:
+
+1. An active SFU root receives an authoritative SFU assignment on a new
+ signaling session or reports LiveKit recovery, followed by two bounded stats
+ windows with positive current-generation RTP and decoded-frame progress.
+2. A connected Viewer advertises relay capacity transitioning from zero to one.
+
+One room owns at most one peer migration attempt: either one healthy probe or
+one pending/answered `peer-selected` attempt. It may overlap active media only
+when `Host peer children + active SFU publication + probe <= 2` and `browser
+relay children + probe <= 1`. Without a free slot the opportunity is consumed
+and the active route is left unchanged. The probe is bound to the current
+share/publication, active and pending revisions, root, parent and Host sessions,
+and one pinned connection ID. Signaling from any other pair or generation fails
+closed.
+
+During prepare the Viewer keeps rendering SFU while the existing peer parent
+creates one provisional edge. ICE `connected` is insufficient. The Viewer
+requires a live video track, positive inbound RTP and positive decoded frames,
+then revalidates all of that immediately before promotion. The controller
+commits that root/subtree atomically and retires SFU only after proof. A probe
+failure sent before commit aborts without adding failed-parent state; if the
+proven edge fails after server commit, the existing active-edge recovery path
+runs rather than leaving server and Viewer on different routes.
+
+Timeout, stale generation, participant/session replacement, room stop or
+deletion aborts the provisional route. Abort keeps the active SFU assignment
+and descendants, deletes the provisional connection identity, and does not
+consume an ordinary route-failure intent. The Viewer disposes the provisional
+peer connection and buffered signals before accepting rollback. Active-SFU loss
+first aborts the probe instead of being swallowed. If the active Viewer
+subscriber or Host publisher reports a real pending-revision `prepare`
+failure, only the exact current SFU owner may abort. Ordinary transport uses a
+null connection; selected Host ingress must match its activated connection
+identity. The rollback active revision then drives the existing one-refresh
+recovery. Success and abort share the room's 30-second migration cooldown.
+
+The implemented source slice deliberately handles exactly one active SFU root
+and only its parent retained in the deterministic peer baseline. The accepted
+capacity `0 -> 1` trigger and deterministic selection among multiple SFU roots
+remain follow-ups. No timer/polling trigger, global score, new dependency,
+ordinary-peer TURN grant, or second route controller was added.
The bounded cost model and privacy-safe ICE fields
live in [Low-Server-Cost Media Routes](../research/low-server-media-routes.md).
diff --git a/docs/project-memory.md b/docs/project-memory.md
index 4119420b..03adaf2f 100644
--- a/docs/project-memory.md
+++ b/docs/project-memory.md
@@ -27,7 +27,7 @@ Last updated: 2026-08-21
- Production Web prefers H.264 through standard ordering, keeps the browser fallback list, and requests SFU H.264 with `backupCodec=false`; VP8 is used only when H.264 is unavailable. This is not a second encoder stack or shared-encode proof. ADR-0006's browser bridge remains no-go.
- Mobile endpoints are Web Viewer-only. Current Android/iOS browsers do not expose Web Host capture; AirPlay/system mirroring is Viewer-local output.
- ADR-0005: direct/peer UDP -> bounded SFU roots -> selected-edge TURN. The deployed controller caps pending/answered `peer-selected` at one per room; Host ingress is independent. A pre-`ecc794d` exact-source/production-media canary proves active Host ingress only. Ordinary peers stay STUN-only.
-- The controller is process-enabled, not room-allowlisted: `PEER_ASSISTED_MEDIA=true` gives every normal room the same direct/peer -> SFU -> selected-edge path with per-room state. Room `1` is historical smoke. Preserve sticky P2P, one active upstream and break-before-make; recovery gets one attempt per layer (ICE restart, same-parent rebuild, alternate peer, then SFU), never three identical retries.
+- `PEER_ASSISTED_MEDIA=true` enables isolated all-room controllers; room `1` is historical. Keep sticky P2P and one recovery attempt per layer. Source-only healthy-SFU MBB covers one root and retained parent after reauth/recovery, gated by media proof, Host2/browser1 overlap and cooldown. It excludes `peer-selected`; active-SFU loss aborts into normal recovery. Capacity `0 -> 1`, multi-root and browser gates remain open.
- C+B quality reparenting is edge-local and cooldown-bound; no score, timer, or global parent penalty.
- Flagship media is UDP; HTTPS/WSS stays TLS/TCP; the old release remains rollback-only.
- Treat settings as ceilings and degradation as unclassified. Use correlated Host A+B/Viewer C and one-variable evidence; never force AV1, infer by UA, or create a composite score.
diff --git a/docs/research/advanced-peer-distribution.md b/docs/research/advanced-peer-distribution.md
index 6e4ef844..49a03cd7 100644
--- a/docs/research/advanced-peer-distribution.md
+++ b/docs/research/advanced-peer-distribution.md
@@ -1,10 +1,10 @@
# Advanced Peer Distribution
-- Research date: 2026-08-19
+- Research date: 2026-08-21
- Scope: at most eight trusted viewers, sub-second interactive media, endpoint
downstream fanout at most two, and minimal central-server media egress
-- Status: bounded C+B local reparenting is deployed for one exact room and the
- deterministic admission-rescue slice is implemented source-only; advanced
+- Status: bounded C+B local reparenting and admission rescue are deployed;
+ one-root healthy SFU reselection is implemented source-only; advanced
encoded-media routes remain isolated candidates
## Terms
@@ -364,10 +364,10 @@ retain bounded peer descendants. A viewer that cannot attach behind any healthy
root may be separately admitted only under the explicit `E`/central-egress cap;
this is an exceptional compatibility budget, not unbounded whole-room fanout.
The SFU/UDP and optional selected-edge TURN transport accounting lives in
-[Low-Server-Cost Media Routes](./low-server-media-routes.md). ADR-0005's current
-controller remains failure-only, while its accepted target also handles
-admission with no eligible peer path; neither changes the sticky local
-reparenting candidate above.
+[Low-Server-Cost Media Routes](./low-server-media-routes.md). ADR-0005's deployed
+controller handles failure and bounded admission; the current source also has
+the one-root healthy-reselection slice described below. None changes the sticky
+local reparenting candidate above.
The controller must be automatic and invisible. It uses explicit capability
bits, route revisions, media generations, edge budgets, and discrete failures;
@@ -375,6 +375,23 @@ it does not expose topology choices to host or viewers. Initial hard triggers
are capacity, maximum depth, ICE/media failure, unsupported codec, and bounded
send-queue overflow. Measurements may later justify additional triggers.
+For healthy SFU-to-peer reselection, W3C inbound RTP counters expose the needed
+bounded observations: `packetsReceived` counts received RTP packets and
+`framesDecoded` counts successfully decoded video frames. Neither ICE state nor
+a track object alone proves decodable progress. Pinned LiveKit client 2.22.0
+already exposes reconnecting/reconnected room events, so recovery is a discrete
+trigger; no network-type poll or new timer is required. The current source arms
+the same two-window gate when a refreshed signaling session receives an
+authoritative sticky SFU assignment. This avoids immediately reversing an
+ordinary peer-to-SFU fallback, whose initial authenticated assignment was peer.
+
+The overlap edge is physical: an active Host SFU publication consumes one of
+the Host's two downstream media edges while the provisional peer offer consumes
+the other. A browser parent remains limited to one downstream edge during this
+probe even if its normal advertised relay capacity is two. Failed or stale
+probes restore the prior controller revision without writing ordinary
+failed-parent state.
+
Use make-before-break when a downstream slot is free. With all two slots in
use, strict fanout means break-before-make; zero interruption, no standby, and a
hard two-edge limit cannot all be guaranteed simultaneously. Sub-second recovery
@@ -383,7 +400,7 @@ also requires a media/data heartbeat near 100-200 ms rather than the current
## Sources And License Boundary
-Sources checked on 2026-08-20:
+Sources checked on 2026-08-20 and 2026-08-21:
- [WebRTC SVC](https://www.w3.org/TR/webrtc-svc/),
[Encoded Transform](https://www.w3.org/TR/webrtc-encoded-transform/),
@@ -421,6 +438,8 @@ Sources checked on 2026-08-20:
[ICE, RFC 8445](https://www.rfc-editor.org/rfc/rfc8445.html), and
[LiveKit SFU](https://docs.livekit.io/reference/internals/livekit-sfu/) -
observable path boundaries and the centralized low-latency alternative.
+- [LiveKit client 2.22 room events](https://github.com/livekit/client-sdk-js/blob/v2.22.0/src/room/Room.ts)
+ - pinned reconnect/reconnected behavior; no implementation source was copied.
- [TURN, RFC 8656](https://www.rfc-editor.org/rfc/rfc8656.html) - per-edge relay
transport rather than a room topology.
- [MOQT draft](https://datatracker.ietf.org/doc/draft-ietf-moq-transport/) and
diff --git a/docs/status.md b/docs/status.md
index de4a1145..44672d71 100644
--- a/docs/status.md
+++ b/docs/status.md
@@ -19,7 +19,7 @@ Parallel flagship work; minimum relevant checks and rollback proof. Research plu
- Production `screener-v2` admits private fragment grants directly. Code-only Viewers need site access, then public rooms accept the code while private rooms also require their password. `SITE_ACCESS_PASSWORD`, `/api/site-access`, and its cookie are deployed atomically; raw credentials are never stored.
- Source UI uses adjacent nickname editing, distinct site/room prompts, and a shared room-code form. Status and details stay neutral before media proof; current evidence labels green P2P/yellow `SFU fallback`, while relay evidence adds `TURN` plus only a locally observed protocol.
- Native v2 is source-only (memory rooms, 300s reclaim); it follows authenticated direct-child assignments up to two and fails unsupported SFU/selected ingress boundedly.
-- Production is roots <=2/Web relay <=2; every Web Viewer keeps one upstream and bounded sticky recovery, while child3 is rejected. Room `1` is historical; HTTPS/WSS is TLS/TCP.
+- Production is roots/Web relay <=2; every Viewer has one upstream and rejects child3. One-root healthy-SFU MBB is source-only, not deployed; capacity `0 -> 1`, multi-root and browser gates remain open. Room `1` is historical; HTTPS/WSS is TLS/TCP.
- After an answer, a generation-bound 15s initial-connect deadline enters ICE restart; success/replacement/disposal cancels it. It is deployed but not mobile-verified.
- Room `1` C+B reparenting remains uncalibrated. Deployed admission rescue promotes an unassigned relay over the oldest childless zero-capacity Host leaf, with no scores or periodic optimization.
@@ -35,7 +35,7 @@ Parallel flagship work; minimum relevant checks and rollback proof. Research plu
- Local gate proves pre-DOM fragment clearing, room-scoped session isolation, no site/grant transport-log hits, and no raw room-password SQLite sentinel; rotate/revoke tests pass. Headless loopback only.
- Native wire/session tests enforce at most two authoritative children, no presence-created edge, stale-revision ignore, and one bounded unsupported-route failure per revision.
- Production caps pending/answered `peer-selected` at one per room; Host ingress is independent. Focused admission/release/rollback/STUN-only tests pass; earlier Chrome proves active Host-ingress function only.
-- Exact `1331fbdd` passed fresh typecheck, builds, hygiene and 32 files/473 tests. Bounded runtime, UTF-8 artifact, inode and deploy gates preserved config/nft/DB hashes and four zero-restart services.
+- Exact `1331fbdd` passed typecheck/build/hygiene and 32/473; deploy gates preserved runtime, artifacts, config/nft/DB hashes and four zero-restart services. Healthy-reselection source passed the same local gates and 32/489; no browser/deploy ran.
## Unverified Boundaries
@@ -46,15 +46,15 @@ Parallel flagship work; minimum relevant checks and rollback proof. Research plu
- ADR-0006 has one Viewer proof; two-edge/FIFO, hardware, endurance, public/downloaded-package, and browser-diversity proof remain open.
- Production fanout is Host2/Viewer2 and rejects child3. Relay resource behavior remains open.
- Active SFU already publishes `q,h`; only BWE/resource/zero-child/root gates are unverified. No local per-leaf UDP shaper; resume with Linux `tc` or public canary, never CDP. Web P2P/SVC shortcuts remain no-go.
-- Access remains unverified headfully and in production: fragment consumption, invitation rotation/revoke, and exact request/nginx/journal/SQLite leak inspection. The source gate is loopback-only; accounts stay out.
-- Structure debt: split `HybridMediaRouter`, `HostPage`, `SignalingServer`, and `ViewerPage` only at proven consumer boundaries, never by file length.
+- Access lacks headful/production proof for fragment consumption, rotate/revoke and request/nginx/journal/SQLite leakage; its source gate is loopback-only. No accounts.
+- Split large modules only at proven consumer boundaries, never by file length.
## Next Milestone
Next: representative production-room UDP/frame/cap/stop canary; room `1` is history. TURN performance follows later.
Recovery uses one ICE restart, one same-parent rebuild, one alternate peer, then SFU. Next gate exactly-two/Dynacast-off BWE on a zero-child SFU leaf, then root evacuation.
Native WGC/MF passed one-Viewer hardware; package download, Viewer2/FIFO and game A/V remain. Use representative target-device or production evidence for performance, not an ordinary-PC synthetic gate.
-Ordinary peers stay STUN-only; initial ingress and `peer-selected` still need forced-relay proof.
+Ordinary peers stay STUN-only; initial ingress, `peer-selected`, and healthy-reselection browser gates remain open.
ADR-0004 still needs its resource/quality matrix; the 20-viewer gate precedes a default change. Mobile uses the same capacity and is a compatibility observation.
## Blockers And Decisions
diff --git "a/docs/\351\234\200\346\261\202\347\220\206\350\247\243.md" "b/docs/\351\234\200\346\261\202\347\220\206\350\247\243.md"
index 8eb96985..05839d6a 100644
--- "a/docs/\351\234\200\346\261\202\347\220\206\350\247\243.md"
+++ "b/docs/\351\234\200\346\261\202\347\220\206\350\247\243.md"
@@ -152,6 +152,8 @@
- Given 受控 RTT/丢包测试条件,When 连续分享高动态游戏画面 30 分钟,Then 延迟、帧率与丢帧达到上述目标,且没有不可恢复的冻结。
- Given 部署版出现游戏低帧、持续 `bandwidth` 限制或只有 Host 刷新才能恢复,When 归因该报告,Then 用同一机器、浏览器、驱动、场景和单个有线 direct Viewer 对照精确 release `769de201f7cc` 与最新 `main`,依次比较受影响状态、只重建该 `HostPeer`、只替换 capture、完整 Host 重建和本地预览开/关;记录实际 codec/encoder、A+B/C、游戏 FPS、CPU 与 GPU encode 使用,不能把部署版报告直接归为最新代码缺陷。
- Given 分享过程中切换 Wi-Fi/有线网络,When ICE 连接失败,Then 客户端触发 ICE restart,并在成功或超时后给出明确状态。
+- Given 一个 active SFU root 在新 signaling session 上复用权威 SFU assignment、信令重认证或 LiveKit 恢复后持续产出 current-generation RTP/解码帧,或一个新 Viewer 的 relay capacity 从 0 变为 1,When cooldown 已结束且候选 parent 有满足瞬时 fanout 上限的重叠空槽,Then 控制器只为一个确定性 root/subtree 建立一条 provisional P2P edge,保留原 SFU 媒体直到新路径出现首个可解码视频帧和正向 RTP 进展后再原子提交,不因 ICE `connected` 单独切换。
+- Given SFU 健康回迁没有重叠空槽,或 provisional edge 失败、超时、代次过期、参与者变化、房间停止、权限撤销或 session 被替换,When 本次机会结束,Then 原 SFU root、descendants 和未受影响 branches 继续原路且不黑屏,临时 edge 被销毁,不写入 `failedParentPeerIdsByViewer`、不消耗普通 route-failure intent,并在 cooldown 前不因同一事件重试。
- Given 浏览器未返回系统音轨,When 分享开始,Then 分享者在启播前看到“当前来源没有可共享音频”的提示。
- Given 分享者选择标签页或窗口,When 浏览器支持对应音频捕获,Then picker 提供该来源音频而不扩大为系统混音。Given 分享者选择整个屏幕,When 浏览器支持系统音频,Then picker 提供由用户决定的系统音频开关。应用同时请求 `windowAudio: "window"` 和 `systemAudio: "include"`,不根据 track 存在性伪造来源标签。
- Given 浏览器返回分享音轨,When 开始发布,Then 应用设置标准 `contentHint = "music"` 作为内容意图元数据;该值可读回,但不作为 Opus、码率、声道、采样率、FEC、DTX 或立体声的质量证据。
diff --git a/src/client/components/status-badge-model.ts b/src/client/components/status-badge-model.ts
index 46df396c..ce7bf7a3 100644
--- a/src/client/components/status-badge-model.ts
+++ b/src/client/components/status-badge-model.ts
@@ -75,10 +75,12 @@ export function viewerRouteEvidence(
if (upstream.kind === "peer") {
const matchingPeer = peer?.peerId === upstream.peerId ? peer : null;
const hasMatchingPeerEvidence = hasPeerRouteEvidence(matchingPeer);
- return {
- route: hasMatchingPeerEvidence ? "p2p" : null,
- evidence: hasMatchingPeerEvidence ? matchingPeer : null,
- };
+ if (hasMatchingPeerEvidence) {
+ return { route: "p2p", evidence: matchingPeer };
+ }
+ return sfu
+ ? { route: "sfu", evidence: sfu }
+ : { route: null, evidence: null };
}
return { route: null, evidence: null };
}
diff --git a/src/client/media/host-sfu-route.ts b/src/client/media/host-sfu-route.ts
index 87b79553..5ff7cb95 100644
--- a/src/client/media/host-sfu-route.ts
+++ b/src/client/media/host-sfu-route.ts
@@ -97,6 +97,9 @@ export class HostSfuRoute {
}
if (update.assignment.sfuPublicationGeneration === null) {
this.clearPending();
+ if (this.route.getMediaAssignment()?.sfuPublicationGeneration) {
+ this.events.reconcileChildren(update.assignment.childPeerIds);
+ }
}
return result;
}
diff --git a/src/client/media/viewer-sfu-route.ts b/src/client/media/viewer-sfu-route.ts
index d2322117..de530723 100644
--- a/src/client/media/viewer-sfu-route.ts
+++ b/src/client/media/viewer-sfu-route.ts
@@ -34,8 +34,9 @@ interface ViewerSubscriberSlot {
interface ViewerSfuRouteEvents {
activatePeer: (
assignment: ParticipantRouteAssignment,
- ) => void | Promise<void>;
- preparePeer?: (assignment: ParticipantRouteAssignment | null) => void;
+ revision?: number,
+ ) => boolean | void | Promise<boolean | void>;
+ preparePeer?: (assignment: ParticipantRouteAssignment | null, revision?: number) => void;
resetMedia?: () => void;
reconcileSfuChildren: (childPeerIds: string[]) => void;
onSfuStream: (
@@ -46,6 +47,7 @@ interface ViewerSfuRouteEvents {
onSfuVideoAvailability?: (available: boolean) => void;
onSfuUpdate?: (metrics: ConnectionMetrics | null) => void;
onSfuState?: (state: "connected" | "reconnecting") => void;
+ onHealthySfu?: (revision: number) => void;
send: (message: ClientMessage) => boolean;
createSubscriber?: (
events: {
@@ -63,6 +65,8 @@ export class ViewerSfuRoute {
private pending: ViewerSubscriberSlot | null = null;
private active: ViewerSubscriberSlot | null = null;
private recovery: { revision: number; refreshed: boolean } | null = null;
+ private peerProbeRevision: number | null = null;
+ private healthySfuWindows: number | null = null;
private transitionTail: Promise<void> = Promise.resolve();
private resyncGeneration = 0;
private resyncing = false;
@@ -81,8 +85,20 @@ export class ViewerSfuRoute {
}
if (previousRevision !== update.revision) {
this.recovery = null;
+ this.peerProbeRevision = null;
+ this.healthySfuWindows = null;
}
if (update.phase === "prepare") {
+ if (
+ update.assignment.upstream.kind === "peer" &&
+ this.route.getMediaAssignment()?.upstream.kind === "sfu"
+ ) {
+ this.peerProbeRevision = update.revision;
+ this.events.preparePeer?.(update.assignment, update.revision);
+ }
+ if (update.assignment.upstream.kind !== "sfu") {
+ this.events.reconcileSfuChildren(update.assignment.childPeerIds);
+ }
if (this.pending?.revision !== update.revision) {
this.clearPending();
}
@@ -96,7 +112,9 @@ export class ViewerSfuRoute {
if (!token) {
return result;
}
- this.preparePeer(update.assignment);
+ if (this.peerProbeRevision !== update.revision) {
+ this.preparePeer(update.assignment);
+ }
if (this.resyncing) {
return result;
}
@@ -116,6 +134,8 @@ export class ViewerSfuRoute {
this.resyncing = true;
this.route.reset();
this.recovery = null;
+ this.peerProbeRevision = null;
+ this.healthySfuWindows = null;
const pending = this.pending;
const active = this.active;
this.pending = null;
@@ -197,11 +217,27 @@ export class ViewerSfuRoute {
onStats: (metrics: ConnectionMetrics) => {
if (this.active === slot && !slot.failed) {
this.events.onSfuUpdate?.(metrics);
+ if (this.healthySfuWindows !== null && this.healthySfuWindows >= 0) {
+ if (
+ (metrics.intervalPacketsReceived ?? 0) <= 0 ||
+ (metrics.intervalFramesDecoded ?? 0) <= 0
+ ) {
+ this.healthySfuWindows = null;
+ } else if (++this.healthySfuWindows === 2) {
+ this.healthySfuWindows = null;
+ this.events.onHealthySfu?.(slot.revision);
+ }
+ }
}
},
onState: (state: "connected" | "reconnecting") => {
if (this.active === slot && !slot.failed) {
this.events.onSfuState?.(state);
+ if (state === "reconnecting") {
+ this.healthySfuWindows = -1;
+ } else if (this.healthySfuWindows === -1) {
+ this.healthySfuWindows = 0;
+ }
}
},
onDisconnected: () => this.handleFailure(slot),
@@ -255,6 +291,15 @@ export class ViewerSfuRoute {
);
}
+ armHealthySfuReselection(revision: number): void {
+ if (
+ this.route.getRevision() === revision &&
+ this.route.getPlannedAssignment()?.upstream.kind === "sfu"
+ ) {
+ this.healthySfuWindows = 0;
+ }
+ }
+
async disconnect(): Promise<void> {
if (this.closed) {
return;
@@ -264,6 +309,8 @@ export class ViewerSfuRoute {
this.resyncing = false;
this.route.reset();
this.recovery = null;
+ this.peerProbeRevision = null;
+ this.healthySfuWindows = null;
this.events.preparePeer?.(null);
await this.queueTransition(async () => {
const pending = this.pending;
@@ -322,6 +369,15 @@ export class ViewerSfuRoute {
return;
}
+ const probeRevision = this.peerProbeRevision;
+ const promoted =
+ probeRevision === token.revision
+ ? await this.events.activatePeer(assignment, token.revision)
+ : false;
+ if (probeRevision === token.revision && promoted !== true) {
+ return;
+ }
+ this.peerProbeRevision = null;
this.clearPending();
if (this.active) {
await this.retireActive();
@@ -329,7 +385,9 @@ export class ViewerSfuRoute {
if (!this.route.owns(token, "active")) {
return;
}
- await this.events.activatePeer(assignment);
+ if (!promoted) {
+ await this.events.activatePeer(assignment);
+ }
if (!this.route.owns(token, "active")) {
return;
}
diff --git a/src/client/pages/ViewerPage.tsx b/src/client/pages/ViewerPage.tsx
index 9300be7c..b6b19386 100644
--- a/src/client/pages/ViewerPage.tsx
+++ b/src/client/pages/ViewerPage.tsx
@@ -89,6 +89,13 @@ interface SfuUpstreamState {
connectionState: "connected" | "reconnecting";
metrics: ConnectionMetrics | null;
}
+interface PeerProbe {
+ revision: number;
+ parentPeerId: string;
+ ready: boolean;
+ stream: MediaStream | null;
+ snapshot: PeerSnapshot | null;
+}
export function ViewerPage({ roomId, viewerGrant }: ViewerPageProps) {
const [accessState, setAccessState] = useState<
@@ -186,6 +193,7 @@ export function ViewerPage({ roomId, viewerGrant }: ViewerPageProps) {
let preparedParentSignals: Array<
Extract<ServerMessage, { type: "signal" }>
> = [];
+ let peerProbe: PeerProbe | null = null;
let retiredUpstream: {
parentPeerId: string;
connectionId: string;
@@ -364,16 +372,92 @@ export function ViewerPage({ roomId, viewerGrant }: ViewerPageProps) {
ensureViewerRelay()?.setChildren(nextChildPeerIds);
}
+ function discardPeerProbe(): void {
+ const hadPeerProbe = peerProbe !== null;
+ peerProbe = null;
+ prepareParent(null);
+ if (!hadPeerProbe) {
+ return;
+ }
+ peerRef.current?.dispose();
+ peerRef.current = null;
+ }
+
+ function peerProbeHasCurrentMedia(
+ probe: PeerProbe,
+ ): probe is PeerProbe & { stream: MediaStream; snapshot: PeerSnapshot } {
+ const { snapshot, stream } = probe;
+ return (
+ snapshot !== null &&
+ stream !== null &&
+ snapshot.peerId === probe.parentPeerId &&
+ peerRef.current?.hasConnectionId(snapshot.connectionId) === true &&
+ snapshot.connectionState === "connected" &&
+ (snapshot.metrics.intervalPacketsReceived ?? 0) > 0 &&
+ (snapshot.metrics.intervalFramesDecoded ?? 0) > 0 &&
+ stream.getVideoTracks().some((track) => track.readyState === "live")
+ );
+ }
+
+ function provePeerProbe(): void {
+ const probe = peerProbe;
+ if (!probe || probe.ready || !peerProbeHasCurrentMedia(probe)) {
+ return;
+ }
+ probe.ready = signal.send({
+ type: "route-ready",
+ revision: probe.revision,
+ phase: "prepare",
+ });
+ }
+
function ensureViewerSfuRoute(): ViewerSfuRoute {
if (viewerSfuRoute) {
return viewerSfuRoute;
}
let route: ViewerSfuRoute;
route = new ViewerSfuRoute({
- activatePeer: (assignment) => {
+ activatePeer: async (assignment, revision) => {
if (!active || viewerSfuRoute !== route) {
return;
}
+ if (revision !== undefined) {
+ const probe = peerProbe;
+ if (
+ !probe ||
+ !probe.ready ||
+ probe.revision !== revision ||
+ assignment.upstream.kind !== "peer" ||
+ assignment.upstream.peerId !== probe.parentPeerId ||
+ !peerProbeHasCurrentMedia(probe)
+ ) {
+ if (probe?.ready && probe.snapshot) {
+ signal.send({
+ type: "route-failed",
+ revision,
+ phase: "active",
+ connectionId: probe.snapshot.connectionId,
+ });
+ }
+ discardPeerProbe();
+ return false;
+ }
+ peerProbe = null;
+ prepareParent(null);
+ applyMediaAssignment(
+ {
+ parentPeerId: probe.parentPeerId,
+ childPeerIds: assignment.childPeerIds,
+ },
+ true,
+ );
+ ensureViewerRelay()?.setStream(probe.stream);
+ setRemoteStream(probe.stream);
+ setPeerSnapshot(probe.snapshot);
+ setSfuUpstream(null);
+ setStatusText("正在播放");
+ return true;
+ }
applyMediaAssignment({
parentPeerId:
assignment.upstream.kind === "peer"
@@ -382,13 +466,23 @@ export function ViewerPage({ roomId, viewerGrant }: ViewerPageProps) {
childPeerIds: assignment.childPeerIds,
});
if (assignment.upstream.kind === "peer") {
- return drainPreparedParentSignals(assignment.upstream.peerId);
+ await drainPreparedParentSignals(assignment.upstream.peerId);
}
},
- preparePeer: (assignment) => {
+ preparePeer: (assignment, revision) => {
if (!active || viewerSfuRoute !== route) {
return;
}
+ discardPeerProbe();
+ if (revision !== undefined && assignment?.upstream.kind === "peer") {
+ peerProbe = {
+ revision,
+ parentPeerId: assignment.upstream.peerId,
+ ready: false,
+ stream: null,
+ snapshot: null,
+ };
+ }
prepareParent(
assignment?.upstream.kind === "peer"
? assignment.upstream.peerId
@@ -433,6 +527,8 @@ export function ViewerPage({ roomId, viewerGrant }: ViewerPageProps) {
);
}
},
+ onHealthySfu: (revision) =>
+ signal.send({ type: "sfu-reselection-ready", revision }),
onSfuVideoAvailability: (available) => {
if (!active || viewerSfuRoute !== route || available) {
return;
@@ -462,9 +558,11 @@ export function ViewerPage({ roomId, viewerGrant }: ViewerPageProps) {
);
setStatusText("正在播放");
if (initialVideoStream) {
- peerRef.current?.dispose();
- peerRef.current = null;
- setPeerSnapshot(null);
+ if (!peerProbe) {
+ peerRef.current?.dispose();
+ peerRef.current = null;
+ setPeerSnapshot(null);
+ }
}
},
send: (message) =>
@@ -476,9 +574,9 @@ export function ViewerPage({ roomId, viewerGrant }: ViewerPageProps) {
function clearViewerSfuRoute(): void {
const route = viewerSfuRoute;
+ discardPeerProbe();
viewerSfuRoute = null;
setSfuUpstream(null);
- prepareParent(null);
void route?.disconnect();
}
@@ -527,7 +625,7 @@ export function ViewerPage({ roomId, viewerGrant }: ViewerPageProps) {
viewerRelay?.stop();
}
- function applyMediaAssignment(assignment: MediaAssignment): void {
+ function applyMediaAssignment(assignment: MediaAssignment, preserveUpstream = false): void {
const nextAssignment = limitMediaAssignment(
assignment,
MAX_VIEWER_MEDIA_CHILDREN,
@@ -536,7 +634,7 @@ export function ViewerPage({ roomId, viewerGrant }: ViewerPageProps) {
const previousChildPeerIds = currentAssignment.childPeerIds;
currentAssignment = nextAssignment;
- if (previousParentId !== nextAssignment.parentPeerId) {
+ if (!preserveUpstream && previousParentId !== nextAssignment.parentPeerId) {
clearUpstreamState();
setStatusText("正在恢复连接");
}
@@ -560,7 +658,7 @@ export function ViewerPage({ roomId, viewerGrant }: ViewerPageProps) {
viewerSignalMessage(peerAssisted, targetPeerId, payload),
),
sendRestartRequest: (targetPeerId, connectionId, rebuild) =>
- signal.send(
+ !peerProbe && signal.send(
viewerRestartMessage(
peerAssisted,
targetPeerId,
@@ -570,6 +668,11 @@ export function ViewerPage({ roomId, viewerGrant }: ViewerPageProps) {
),
onStream: (nextStream) => {
if (active) {
+ if (peerProbe) {
+ peerProbe.stream = nextStream;
+ provePeerProbe();
+ return;
+ }
setRemoteStream(nextStream);
setStatusText("正在播放");
// Audio and video can arrive as separate track events on the
@@ -579,6 +682,11 @@ export function ViewerPage({ roomId, viewerGrant }: ViewerPageProps) {
},
onUpdate: (snapshot) => {
if (active) {
+ if (peerProbe) {
+ peerProbe.snapshot = snapshot;
+ provePeerProbe();
+ return;
+ }
qualityEvidenceReporter.offer(snapshot, currentRouteRevision);
if (
!currentHostOnline &&
@@ -600,6 +708,25 @@ export function ViewerPage({ roomId, viewerGrant }: ViewerPageProps) {
}
},
onRecoveryExhausted: (parentPeerId, connectionId) => {
+ if (
+ peerProbe?.parentPeerId === parentPeerId &&
+ peerProbe.snapshot?.connectionId === connectionId
+ ) {
+ signal.send({
+ type: "route-failed",
+ revision: peerProbe.revision,
+ phase: "prepare",
+ connectionId,
+ });
+ signal.send({
+ type: "route-failed",
+ revision: peerProbe.revision,
+ phase: "active",
+ connectionId,
+ });
+ discardPeerProbe();
+ return true;
+ }
if (
selectedEdgeTurn?.parentPeerId === parentPeerId &&
selectedEdgeTurn.newConnectionId === connectionId
@@ -692,6 +819,9 @@ export function ViewerPage({ roomId, viewerGrant }: ViewerPageProps) {
if (!active || !messageAuthority.owns(authorityToken)) {
return;
}
+ if (message.routeAssignment.upstream.kind === "sfu") {
+ ensureViewerSfuRoute().armHealthySfuReselection(message.routeRevision);
+ }
}
if (
!message.hostOnline &&
@@ -817,6 +947,10 @@ export function ViewerPage({ roomId, viewerGrant }: ViewerPageProps) {
return;
}
if (message.type === "signal") {
+ if (peerAssisted && peerProbe?.parentPeerId === message.fromPeerId) {
+ await ensurePeer()?.acceptSignal(message.fromPeerId, message.payload);
+ return;
+ }
if (
peerAssisted &&
preparedParentPeerId === message.fromPeerId
diff --git a/src/server/hybrid-media-router.ts b/src/server/hybrid-media-router.ts
index 5953ab69..19a1497c 100644
--- a/src/server/hybrid-media-router.ts
+++ b/src/server/hybrid-media-router.ts
@@ -46,6 +46,7 @@ interface PendingRoutePreparation {
expectedSessionIds: ReadonlyMap<string, string>;
grantsIssued: boolean;
hostSfuIngressTurnAttempted: boolean;
+ healthyReselection: HealthySfuReselectionProbe | null;
timer: NodeJS.Timeout;
}
@@ -86,6 +87,22 @@ interface SelectedSfuIngressAttempt {
timer: NodeJS.Timeout;
}
+interface ActiveSelectedSfuIngress {
+ shareGeneration: string;
+ hostPeerId: string;
+ hostSessionId: string;
+ publicationGeneration: string;
+ connectionId: string;
+}
+
+interface HealthySfuReselectionProbe {
+ activeRevision: number;
+ shareGeneration: string;
+ publicationGeneration: string;
+ rootPeerId: string;
+ parentPeerId: string;
+}
+
interface QualityRouteIntentGuard {
viewerSessionId: string;
connectionId: string;
@@ -189,6 +206,10 @@ export class HybridMediaRouter {
string,
SelectedSfuIngressAttempt
>();
+ private readonly activeSelectedSfuIngresses = new Map<
+ string,
+ ActiveSelectedSfuIngress
+ >();
private readonly relayCapacitySessionsByRoom = new Map<
string,
Map<string, string>
@@ -237,6 +258,7 @@ export class HybridMediaRouter {
clearTimeout(attempt.timer);
}
this.selectedSfuIngressAttempts.clear();
+ this.activeSelectedSfuIngresses.clear();
this.mediaRouteControllers.clear();
this.failedParentPeerIdsByViewer.clear();
this.viewerRouteIntentsByRoom.clear();
@@ -356,7 +378,24 @@ export class HybridMediaRouter {
? this.selectedEdgeTurnAttempts.get(targetKey)
: undefined;
if (!attempt) {
- return undefined;
+ const pending = this.pendingRoutePreparations.get(input.roomId);
+ const probe = pending?.healthyReselection;
+ const parentToRoot = probe?.parentPeerId === input.sourcePeerId &&
+ probe.rootPeerId === input.targetPeerId;
+ const rootToParent = probe?.rootPeerId === input.sourcePeerId &&
+ probe.parentPeerId === input.targetPeerId;
+ if (!pending || !probe || (!parentToRoot && !rootToParent)) {
+ return undefined;
+ }
+ if (!this.healthySfuReselectionIsCurrent(input.roomId, pending)) {
+ return false;
+ }
+ const connectionId = this.options.getConnectionId(input.roomId, probe.rootPeerId);
+ return parentToRoot && input.descriptionType === "offer"
+ ? connectionId === undefined
+ : connectionId === input.connectionId &&
+ (input.signalKind === "candidate" ||
+ (rootToParent && input.descriptionType === "answer"));
}
const parentToViewer = input.sourcePeerId === attempt.parentPeerId;
const currentRoute = this.mediaRouteControllers.get(input.roomId)?.getActiveRoute();
@@ -677,6 +716,131 @@ export class HybridMediaRouter {
this.drainViewerRouteIntents(participant.roomId);
}
+ handleHealthySfuReselection(
+ participant: AuthenticatedRouteParticipant,
+ revision: number,
+ ): void {
+ const roomId = participant.roomId;
+ const controller = this.mediaRouteControllers.get(roomId);
+ const active = controller?.getActiveRoute();
+ const publicationGeneration = active?.sfu.publicationGeneration;
+ const shareGeneration = this.options.getShareGeneration(roomId);
+ if (
+ participant.role !== "viewer" ||
+ !controller ||
+ controller.getPendingRoute() ||
+ this.roomPeerMigrationAttempt(roomId) !== null ||
+ active?.revision !== revision ||
+ active.sfu.rootPeerIds.length !== 1 ||
+ active.sfu.rootPeerIds[0] !== participant.peerId ||
+ !publicationGeneration ||
+ !shareGeneration
+ ) {
+ return;
+ }
+ const now = this.options.now?.() ?? Date.now();
+ if (
+ !Number.isFinite(now) ||
+ (this.roomQualityMigrationCooldownUntilMs.get(roomId) ?? 0) > now
+ ) {
+ return;
+ }
+ this.startRoomQualityMigrationCooldown(roomId);
+ const hostPeerId = this.peerRelayTopology.getHostPeerId(roomId);
+ const parentPeerId = this.peerRelayTopology.getAssignment(
+ roomId,
+ participant.peerId,
+ )?.parentPeerId;
+ if (!hostPeerId || !parentPeerId) {
+ return;
+ }
+ const host = this.connectedPeer(roomId, hostPeerId);
+ const parent = this.connectedPeer(roomId, parentPeerId);
+ const activeParent = active.assignments.get(parentPeerId);
+ const capacity = Math.min(
+ this.peerRelayTopology.getDownstreamCapacity(roomId, parentPeerId),
+ parentPeerId === hostPeerId ? 2 : 1,
+ );
+ if (
+ !host ||
+ !parent ||
+ !activeParent ||
+ activeParent.childPeerIds.length +
+ (activeParent.sfuPublicationGeneration ? 1 : 0) +
+ 1 >
+ capacity ||
+ (parentPeerId !== hostPeerId &&
+ this.relayCapacitySessionsByRoom.get(roomId)?.get(parentPeerId) !==
+ parent.sessionId)
+ ) {
+ return;
+ }
+ const assignments = participantRoutesFromTopology(
+ this.peerRelayTopology.getAssignments(roomId),
+ hostPeerId,
+ null,
+ [],
+ );
+ const rootAssignment = assignments.get(participant.peerId)!;
+ const parentAssignment = assignments.get(parentPeerId)!;
+ let pendingRevision: number | undefined;
+ try {
+ pendingRevision = controller.prepare({
+ assignments,
+ expectedParticipantIds: new Set([participant.peerId]),
+ });
+ } catch {
+ return;
+ }
+ if (pendingRevision === undefined) {
+ return;
+ }
+ const timer = setTimeout(
+ () => this.abortPendingRoute(roomId),
+ this.options.sfuFallback?.prepareTimeoutMs ??
+ DEFAULT_SFU_PREPARE_TIMEOUT_MS,
+ );
+ timer.unref();
+ this.options.deleteConnectionId(roomId, participant.peerId);
+ this.pendingRoutePreparations.set(roomId, {
+ revision: pendingRevision,
+ intentPeerId: participant.peerId,
+ routeIntent: {
+ failedParentPeerId: null,
+ sessionId: participant.sessionId,
+ sfuAttempts: 0,
+ unavailableReported: false,
+ },
+ qualityIntent: null,
+ expectedSessionIds: new Map([
+ [participant.peerId, participant.sessionId],
+ [parentPeerId, parent.sessionId],
+ [hostPeerId, host.sessionId],
+ ]),
+ grantsIssued: true,
+ hostSfuIngressTurnAttempted: false,
+ healthyReselection: {
+ activeRevision: revision,
+ shareGeneration,
+ publicationGeneration,
+ rootPeerId: participant.peerId,
+ parentPeerId,
+ },
+ timer,
+ });
+ for (const [sessionId, assignment] of [
+ [participant.sessionId, rootAssignment],
+ [parent.sessionId, parentAssignment],
+ ] as const) {
+ this.options.sendToSession(sessionId, {
+ type: "route-update",
+ revision: pendingRevision,
+ phase: "prepare",
+ assignment,
+ });
+ }
+ }
+
handleRouteReady(
participant: AuthenticatedRouteParticipant,
message: Extract<ClientMessage, { type: "route-ready" }>,
@@ -715,6 +879,20 @@ export class HybridMediaRouter {
) {
return;
}
+ const healthy = pending.healthyReselection;
+ if (healthy) {
+ const current = this.healthySfuReselectionIsCurrent(participant.roomId, pending);
+ if (
+ participant.peerId !== healthy.rootPeerId ||
+ !this.options.getConnectionId(participant.roomId, healthy.rootPeerId) ||
+ !current
+ ) {
+ if (!current) {
+ this.abortPendingRoute(participant.roomId);
+ }
+ return;
+ }
+ }
const expectedSessionId = pending.expectedSessionIds.get(participant.peerId);
if (!expectedSessionId) {
this.sendError(
@@ -745,6 +923,63 @@ export class HybridMediaRouter {
participant: AuthenticatedRouteParticipant,
message: Extract<ClientMessage, { type: "route-failed" }>,
): void {
+ const healthyPending = this.pendingRoutePreparations.get(participant.roomId);
+ const healthy = healthyPending?.healthyReselection;
+ const ownsHealthyActiveSfu = healthyPending
+ ? this.ownsActiveSfuForHealthyReselection(participant, healthyPending)
+ : false;
+ if (
+ message.phase === "active" &&
+ healthy?.activeRevision === message.revision &&
+ ownsHealthyActiveSfu
+ ) {
+ this.abortPendingRoute(participant.roomId);
+ const rollbackRevision = this.mediaRouteControllers
+ .get(participant.roomId)
+ ?.getActiveRoute().revision;
+ if (rollbackRevision === undefined) {
+ return;
+ }
+ message = { ...message, revision: rollbackRevision };
+ }
+ if (
+ message.phase === "prepare" &&
+ healthyPending?.revision === message.revision &&
+ healthy
+ ) {
+ if (!ownsHealthyActiveSfu) {
+ return;
+ }
+ const activeIngress = this.activeSelectedSfuIngresses.get(
+ participant.roomId,
+ );
+ const hasCurrentSelectedHostIngress =
+ participant.role === "host" &&
+ activeIngress?.hostPeerId === participant.peerId &&
+ activeIngress.hostSessionId === participant.sessionId &&
+ activeIngress.shareGeneration === healthy.shareGeneration &&
+ activeIngress.publicationGeneration ===
+ healthy.publicationGeneration;
+ const activeSfuTransportFailed = hasCurrentSelectedHostIngress
+ ? activeIngress.connectionId === message.connectionId
+ : message.connectionId === null;
+ if (activeSfuTransportFailed) {
+ if (participant.role === "host") {
+ this.activeSelectedSfuIngresses.delete(participant.roomId);
+ }
+ this.abortPendingRoute(participant.roomId, undefined, false);
+ return;
+ }
+ if (
+ participant.role === "viewer" &&
+ participant.peerId === healthy.rootPeerId &&
+ message.connectionId ===
+ this.options.getConnectionId(participant.roomId, healthy.rootPeerId)
+ ) {
+ this.abortPendingRoute(participant.roomId);
+ }
+ return;
+ }
if (message.phase === "prepare") {
this.handlePrepareRouteFailed(participant, message);
return;
@@ -993,6 +1228,19 @@ export class HybridMediaRouter {
this.abortPendingRoute(participant.roomId);
}
+ private roomPeerMigrationAttempt(
+ roomId: string,
+ ): "healthy" | "selected" | null {
+ if (this.pendingRoutePreparations.get(roomId)?.healthyReselection) {
+ return "healthy";
+ }
+ return [...this.selectedEdgeTurnAttempts.values()].some(
+ (attempt) => attempt.roomId === roomId,
+ )
+ ? "selected"
+ : null;
+ }
+
private startSelectedEdgeTurn(
roomId: string,
viewerPeerId: string,
@@ -1003,14 +1251,15 @@ export class HybridMediaRouter {
const active = controller?.getActiveRoute();
const intent = this.viewerRouteIntentsByRoom.get(roomId)?.get(viewerPeerId);
const key = viewerConnectionKey(roomId, viewerPeerId);
+ const roomAttempt = this.roomPeerMigrationAttempt(roomId);
+ if (roomAttempt === "healthy") {
+ // The healthy probe already owns recovery, so suppress active-route failback.
+ return true;
+ }
if (this.selectedEdgeTurnAttempts.has(key)) {
return true;
}
- if (
- [...this.selectedEdgeTurnAttempts.values()].some(
- (attempt) => attempt.roomId === roomId,
- )
- ) {
+ if (roomAttempt === "selected") {
return false;
}
if (
@@ -1256,6 +1505,7 @@ export class HybridMediaRouter {
pending.timer = timer;
pending.hostSfuIngressTurnAttempted = true;
}
+ this.activeSelectedSfuIngresses.delete(roomId);
this.selectedSfuIngressAttempts.set(key, attempt);
this.options.sendToSession(hostSessionId, {
type: "selected-edge-turn",
@@ -1291,6 +1541,13 @@ export class HybridMediaRouter {
clearTimeout(attempt.timer);
}
this.selectedSfuIngressAttempts.delete(roomId);
+ this.activeSelectedSfuIngresses.set(roomId, {
+ shareGeneration: attempt.shareGeneration,
+ hostPeerId: attempt.hostPeerId,
+ hostSessionId: attempt.hostSessionId,
+ publicationGeneration: attempt.publicationGeneration,
+ connectionId: attempt.newConnectionId,
+ });
}
private finishSelectedEdgeTurnFailure(
@@ -1881,6 +2138,7 @@ export class HybridMediaRouter {
expectedSessionIds,
grantsIssued: false,
hostSfuIngressTurnAttempted: false,
+ healthyReselection: null,
timer,
});
void this.issueSfuPrepareGrants(
@@ -1995,11 +2253,69 @@ export class HybridMediaRouter {
clearTimeout(pending.timer);
this.pendingRoutePreparations.delete(roomId);
const active = controller.getActiveRoute();
- this.clearChangedRouteConnectionIds(roomId, before, active);
+ const healthy = pending.healthyReselection;
+ if (healthy) {
+ this.failedParentPeerIdsByViewer.delete(
+ viewerConnectionKey(roomId, healthy.rootPeerId),
+ );
+ this.viewerRouteIntentsByRoom.get(roomId)?.delete(healthy.rootPeerId);
+ this.consumedSfuRefreshesByRoom.delete(roomId);
+ } else {
+ this.clearChangedRouteConnectionIds(roomId, before, active);
+ }
this.broadcastActiveRoute(roomId, active);
this.drainViewerRouteIntents(roomId);
}
+ private healthySfuReselectionIsCurrent(
+ roomId: string,
+ pending: PendingRoutePreparation,
+ ): boolean {
+ const probe = pending.healthyReselection;
+ const controller = this.mediaRouteControllers.get(roomId);
+ const active = controller?.getActiveRoute();
+ return (
+ probe !== null &&
+ this.pendingRoutePreparations.get(roomId) === pending &&
+ active?.revision === probe.activeRevision &&
+ active.sfu.publicationGeneration === probe.publicationGeneration &&
+ controller?.getPendingRoute()?.route.revision === pending.revision &&
+ this.options.getShareGeneration(roomId) === probe.shareGeneration &&
+ this.pendingSessionsAreCurrent(roomId, pending)
+ );
+ }
+
+ private ownsActiveSfuForHealthyReselection(
+ participant: AuthenticatedRouteParticipant,
+ pending: PendingRoutePreparation,
+ ): boolean {
+ const probe = pending.healthyReselection;
+ if (
+ !probe ||
+ pending.expectedSessionIds.get(participant.peerId) !==
+ participant.sessionId ||
+ !this.healthySfuReselectionIsCurrent(participant.roomId, pending)
+ ) {
+ return false;
+ }
+ const active = this.mediaRouteControllers
+ .get(participant.roomId)
+ ?.getActiveRoute();
+ const assignment = active?.assignments.get(participant.peerId);
+ if (participant.role === "viewer") {
+ return (
+ participant.peerId === probe.rootPeerId &&
+ assignment?.upstream.kind === "sfu" &&
+ active?.sfu.rootPeerIds.includes(participant.peerId) === true
+ );
+ }
+ return (
+ participant.peerId ===
+ this.peerRelayTopology.getHostPeerId(participant.roomId) &&
+ assignment?.sfuPublicationGeneration === probe.publicationGeneration
+ );
+ }
+
private abortPendingRoute(
roomId: string,
excludedPeerId?: string,
@@ -2012,6 +2328,9 @@ export class HybridMediaRouter {
}
clearTimeout(pending.timer);
this.pendingRoutePreparations.delete(roomId);
+ if (pending.healthyReselection) {
+ this.options.deleteConnectionId(roomId, pending.healthyReselection.rootPeerId);
+ }
const ingress = this.selectedSfuIngressAttempts.get(roomId);
if (ingress?.revision === pending.revision) {
clearTimeout(ingress.timer);
@@ -2170,6 +2489,17 @@ export class HybridMediaRouter {
return;
}
consumed.add(refreshKey);
+ const activeIngress = this.activeSelectedSfuIngresses.get(roomId);
+ if (
+ activeIngress?.hostPeerId === peerId &&
+ activeIngress.hostSessionId === sessionId &&
+ activeIngress.publicationGeneration ===
+ active.sfu.publicationGeneration &&
+ activeIngress.shareGeneration ===
+ this.options.getShareGeneration(roomId)
+ ) {
+ this.activeSelectedSfuIngresses.delete(roomId);
+ }
}
try {
const token = await fallback.tokenIssuer.issueToken({
@@ -2322,6 +2652,20 @@ export class HybridMediaRouter {
this.selectedSfuIngressAttempts.delete(key);
}
}
+ const activeIngress = this.activeSelectedSfuIngresses.get(roomId);
+ if (
+ activeIngress &&
+ (route.sfu.publicationGeneration !==
+ activeIngress.publicationGeneration ||
+ route.assignments.get(activeIngress.hostPeerId)
+ ?.sfuPublicationGeneration !== activeIngress.publicationGeneration ||
+ this.connectedPeer(roomId, activeIngress.hostPeerId)?.sessionId !==
+ activeIngress.hostSessionId ||
+ this.options.getShareGeneration(roomId) !==
+ activeIngress.shareGeneration)
+ ) {
+ this.activeSelectedSfuIngresses.delete(roomId);
+ }
this.clearRoomViewerQualityEvidenceStates(roomId);
for (const [peerId, assignment] of route.assignments) {
if (peerId === excludedPeerId) {
@@ -2468,6 +2812,7 @@ export class HybridMediaRouter {
clearTimeout(ingress.timer);
this.selectedSfuIngressAttempts.delete(roomId);
}
+ this.activeSelectedSfuIngresses.delete(roomId);
this.consumedSfuRefreshesByRoom.delete(roomId);
this.sfuDisabledRoomIds.delete(roomId);
this.clearRoomViewerQualityEvidenceStates(roomId);
@@ -2492,6 +2837,9 @@ export class HybridMediaRouter {
clearTimeout(ingress.timer);
this.selectedSfuIngressAttempts.delete(roomId);
}
+ if (this.activeSelectedSfuIngresses.get(roomId)?.hostPeerId === peerId) {
+ this.activeSelectedSfuIngresses.delete(roomId);
+ }
}
private clearSfuRefreshesForPeer(roomId: string, peerId: string): void {
diff --git a/src/server/signaling.ts b/src/server/signaling.ts
index 8c4c6748..54594270 100644
--- a/src/server/signaling.ts
+++ b/src/server/signaling.ts
@@ -653,6 +653,16 @@ export class SignalingServer {
message.downstreamEdges,
);
return;
+ case "sfu-reselection-ready":
+ if (!this.isHybridMediaEnabled()) {
+ this.sendError(socket, "FORBIDDEN", "Media routes are not enabled");
+ return;
+ }
+ this.hybridMediaRouter!.handleHealthySfuReselection(
+ { ...authenticated, sessionId: this.socketStates.get(socket)!.sessionId },
+ message.revision,
+ );
+ return;
case "route-ready":
if (!this.isHybridMediaEnabled()) {
this.sendError(socket, "FORBIDDEN", "Media routes are not enabled");
diff --git a/src/shared/protocol.ts b/src/shared/protocol.ts
index b2c47596..e3cbf8fc 100644
--- a/src/shared/protocol.ts
+++ b/src/shared/protocol.ts
@@ -533,6 +533,12 @@ export const clientMessageSchema = z.union([
downstreamEdges: relayDownstreamEdgesSchema,
})
.strict(),
+ z
+ .object({
+ type: z.literal("sfu-reselection-ready"),
+ revision: mediaRouteRevisionSchema,
+ })
+ .strict(),
z
.object({
type: z.literal("route-ready"),
diff --git a/tests/protocol.test.ts b/tests/protocol.test.ts
index ea4cbfaa..67ec18af 100644
--- a/tests/protocol.test.ts
+++ b/tests/protocol.test.ts
@@ -608,7 +608,13 @@ describe("client signaling protocol", () => {
},
);
- it("accepts bounded route acknowledgements, failures, and SFU refreshes", () => {
+ it("accepts bounded route acknowledgements, failures, and SFU recovery events", () => {
+ expect(
+ clientMessageSchema.safeParse({
+ type: "sfu-reselection-ready",
+ revision: 7,
+ }).success,
+ ).toBe(true);
expect(
clientMessageSchema.safeParse({
type: "route-ready",
@@ -651,6 +657,12 @@ describe("client signaling protocol", () => {
revision: 1.5,
}).success,
).toBe(false);
+ expect(
+ clientMessageSchema.safeParse({
+ type: "sfu-reselection-ready",
+ revision: -1,
+ }).success,
+ ).toBe(false);
});
});
diff --git a/tests/route-transition.test.ts b/tests/route-transition.test.ts
index 082cd456..ce4b3abd 100644
--- a/tests/route-transition.test.ts
+++ b/tests/route-transition.test.ts
@@ -119,6 +119,26 @@ function createFakeSubscriber(
};
}
+async function activateViewerSfuRoute(
+ events: ConstructorParameters<typeof ViewerSfuRoute>[0],
+ log: string[] = [],
+) {
+ let subscriber!: ReturnType<typeof createFakeSubscriber>;
+ const route = new ViewerSfuRoute({
+ ...events,
+ createSubscriber: (subscriberEvents) => {
+ subscriber = createFakeSubscriber(subscriberEvents, log, "subscriber");
+ return subscriber;
+ },
+ });
+ route.accept({ revision: 1, phase: "prepare", assignment: sfuAssignment() });
+ await route.acceptConfig(sfuConfig(1));
+ route.accept({ revision: 1, phase: "active", assignment: sfuAssignment() });
+ await vi.waitFor(() => expect(subscriber.activate).toHaveBeenCalledOnce());
+ subscriber.events.onStream({} as MediaStream);
+ return { route, subscriber };
+}
+
describe("MediaRouteTransition", () => {
it("keeps current media through prepare and until an active SFU video track", () => {
const route = new MediaRouteTransition();
@@ -270,6 +290,99 @@ describe("HostSfuRoute", () => {
]);
});
+ it("reports the pending peer revision when the active publisher fails", async () => {
+ const messages: ClientMessage[] = [];
+ const publisher = createFakePublisher([], "publisher");
+ let disconnectActive!: () => void;
+ const route = new HostSfuRoute({
+ getStream: () => ({}) as MediaStream,
+ getProfile: () => QUALITY_PROFILES["720p30"],
+ reconcileChildren: () => undefined,
+ send: (message) => {
+ messages.push(message);
+ return true;
+ },
+ createPublisher: (onDisconnected) => {
+ disconnectActive = onDisconnected;
+ return publisher;
+ },
+ });
+ const active = hostAssignment("generation-a");
+ route.accept({ revision: 1, phase: "prepare", assignment: active });
+ await route.acceptConfig(sfuConfig(1));
+ route.accept({ revision: 1, phase: "active", assignment: active });
+ await vi.waitFor(() => expect(publisher.activate).toHaveBeenCalledOnce());
+
+ messages.length = 0;
+ route.accept({
+ revision: 2,
+ phase: "prepare",
+ assignment: hostAssignment(null, ["viewer-root"]),
+ });
+ disconnectActive();
+
+ expect(messages).toEqual([{
+ type: "route-failed",
+ revision: 2,
+ phase: "prepare",
+ connectionId: null,
+ }]);
+ });
+
+ it("reports the selected ingress connection when it fails during peer prepare", async () => {
+ const messages: ClientMessage[] = [];
+ const publishers: ReturnType<typeof createFakePublisher>[] = [];
+ const failures: Array<() => void> = [];
+ const route = new HostSfuRoute({
+ getStream: () => ({}) as MediaStream,
+ getProfile: () => QUALITY_PROFILES["720p30"],
+ reconcileChildren: () => undefined,
+ send: (message) => {
+ messages.push(message);
+ return true;
+ },
+ createPublisher: (onDisconnected) => {
+ const publisher = createFakePublisher(
+ [],
+ `publisher-${publishers.length + 1}`,
+ );
+ publishers.push(publisher);
+ failures.push(onDisconnected);
+ return publisher;
+ },
+ });
+ const active = hostAssignment("generation-a");
+ route.accept({ revision: 1, phase: "prepare", assignment: active });
+ await route.acceptConfig(sfuConfig(1));
+ route.accept({ revision: 1, phase: "active", assignment: active });
+ await vi.waitFor(() => expect(publishers[0]?.activate).toHaveBeenCalledOnce());
+ failures[0]();
+ messages.length = 0;
+
+ expect(route.startSelectedEdgeTurn(hostSfuIngressGrant())).toBe(true);
+ await vi.waitFor(() =>
+ expect(messages).toContainEqual({
+ type: "route-ready",
+ revision: 1,
+ phase: "active",
+ }),
+ );
+ messages.length = 0;
+ route.accept({
+ revision: 2,
+ phase: "prepare",
+ assignment: hostAssignment(null, ["viewer-root"]),
+ });
+ failures[1]();
+
+ expect(messages).toEqual([{
+ type: "route-failed",
+ revision: 2,
+ phase: "prepare",
+ connectionId: "selected-connection-new",
+ }]);
+ });
+
it("applies a host-SFU selected grant only to a relay-only publisher", async () => {
const messages: ClientMessage[] = [];
const publishers: ReturnType<typeof createFakePublisher>[] = [];
@@ -862,6 +975,34 @@ describe("ViewerSfuRoute", () => {
]);
});
+ it("reports the pending peer revision when the active subscriber fails", async () => {
+ const messages: ClientMessage[] = [];
+ const { route, subscriber } = await activateViewerSfuRoute({
+ activatePeer: () => undefined,
+ reconcileSfuChildren: () => undefined,
+ onSfuStream: () => undefined,
+ send: (message) => {
+ messages.push(message);
+ return true;
+ },
+ });
+
+ messages.length = 0;
+ route.accept({
+ revision: 2,
+ phase: "prepare",
+ assignment: peerAssignment("host-peer"),
+ });
+ subscriber.events.onDisconnected();
+
+ expect(messages).toEqual([{
+ type: "route-failed",
+ revision: 2,
+ phase: "prepare",
+ connectionId: null,
+ }]);
+ });
+
it("buffers an assigned parent offer until SFU fallback retirement completes", async () => {
const log: string[] = [];
const bufferedSignals: string[] = [];
@@ -945,6 +1086,125 @@ describe("ViewerSfuRoute", () => {
expect(deliveredSignals).toEqual(["early-offer"]);
});
+ it("promotes a proven peer probe before retiring active SFU media", async () => {
+ const log: string[] = [];
+ let probeReady = false;
+ const prepared: Array<{ parent: string | null; revision?: number }> = [];
+ const activatePeer = vi.fn(
+ async (assignment: ParticipantRouteAssignment, revision?: number) => {
+ const parent = assignment.upstream.kind === "peer"
+ ? assignment.upstream.peerId
+ : assignment.upstream.kind;
+ log.push(`peer:${parent}:${revision ?? "active"}`);
+ return revision === undefined ? undefined : probeReady;
+ },
+ );
+ const { route, subscriber } = await activateViewerSfuRoute({
+ activatePeer,
+ preparePeer: (assignment, revision) => prepared.push({
+ parent: assignment?.upstream.kind === "peer"
+ ? assignment.upstream.peerId
+ : null,
+ ...(revision === undefined ? {} : { revision }),
+ }),
+ reconcileSfuChildren: (children) => log.push(`children:${children.join(",")}`),
+ onSfuStream: () => undefined,
+ send: () => true,
+ }, log);
+ log.length = 0;
+
+ const peer = peerAssignment("host-peer", ["relay-child"]);
+ route.accept({ revision: 2, phase: "prepare", assignment: peer });
+ expect(prepared).toContainEqual({ parent: "host-peer", revision: 2 });
+ expect(log).toContain("children:relay-child");
+ expect(subscriber.deactivate).not.toHaveBeenCalled();
+
+ route.accept({ revision: 2, phase: "active", assignment: peer });
+ await vi.waitFor(() => expect(activatePeer).toHaveBeenCalledWith(peer, 2));
+ expect(subscriber.deactivate).not.toHaveBeenCalled();
+ expect(subscriber.disconnect).not.toHaveBeenCalled();
+
+ probeReady = true;
+ route.accept({ revision: 2, phase: "active", assignment: peer });
+ await vi.waitFor(() => expect(subscriber.disconnect).toHaveBeenCalledOnce());
+ expect(log.indexOf("peer:host-peer:2")).toBeLessThan(
+ log.indexOf("subscriber:deactivate"),
+ );
+ expect(activatePeer).not.toHaveBeenCalledWith(peer);
+ });
+
+ it("requests provisional peer teardown on a newer SFU rollback", async () => {
+ const prepared: Array<{ parent: string | null; revision?: number }> = [];
+ const { route, subscriber } = await activateViewerSfuRoute({
+ activatePeer: () => undefined,
+ preparePeer: (assignment, revision) => prepared.push({
+ parent: assignment?.upstream.kind === "peer"
+ ? assignment.upstream.peerId
+ : null,
+ ...(revision === undefined ? {} : { revision }),
+ }),
+ reconcileSfuChildren: () => undefined,
+ onSfuStream: () => undefined,
+ send: () => true,
+ });
+
+ route.accept({
+ revision: 2,
+ phase: "prepare",
+ assignment: peerAssignment("host-peer"),
+ });
+ expect(prepared.at(-1)).toEqual({ parent: "host-peer", revision: 2 });
+ route.accept({ revision: 3, phase: "active", assignment: sfuAssignment() });
+ expect(prepared.at(-1)).toEqual({ parent: null });
+ expect(subscriber.deactivate).not.toHaveBeenCalled();
+ });
+
+ it("retries a sticky authoritative SFU route only after stable decoded RTP", async () => {
+ const healthy: number[] = [];
+ let subscriber!: ReturnType<typeof createFakeSubscriber>;
+ const route = new ViewerSfuRoute({
+ activatePeer: () => undefined,
+ reconcileSfuChildren: () => undefined,
+ onSfuStream: () => undefined,
+ onHealthySfu: (revision) => healthy.push(revision),
+ send: () => true,
+ createSubscriber: (events) => {
+ subscriber = createFakeSubscriber(events, [], "subscriber");
+ return subscriber;
+ },
+ });
+
+ await route.resyncAuthoritative({
+ revision: 7,
+ phase: "active",
+ assignment: sfuAssignment(),
+ });
+ route.armHealthySfuReselection(7);
+ await route.acceptConfig(sfuConfig(7));
+ await vi.waitFor(() => expect(subscriber.activate).toHaveBeenCalledOnce());
+ subscriber.events.onStream({} as MediaStream);
+ subscriber.events.onStats?.({
+ intervalPacketsReceived: 4,
+ intervalFramesDecoded: 0,
+ } as ConnectionMetrics);
+ subscriber.events.onStats?.({
+ intervalPacketsReceived: 4,
+ intervalFramesDecoded: 3,
+ } as ConnectionMetrics);
+ expect(healthy).toEqual([]);
+
+ route.armHealthySfuReselection(7);
+ subscriber.events.onStats?.({
+ intervalPacketsReceived: 4,
+ intervalFramesDecoded: 3,
+ } as ConnectionMetrics);
+ subscriber.events.onStats?.({
+ intervalPacketsReceived: 2,
+ intervalFramesDecoded: 1,
+ } as ConnectionMetrics);
+ expect(healthy).toEqual([7]);
+ });
+
it("switches on first SFU video and forwards loss across SFU revision reuse", async () => {
const log: string[] = [];
const messages: ClientMessage[] = [];
diff --git a/tests/server-signal.test.ts b/tests/server-signal.test.ts
index 77b99dac..368a976a 100644
--- a/tests/server-signal.test.ts
+++ b/tests/server-signal.test.ts
@@ -864,6 +864,13 @@ async function failActiveSfuRoot(client: TestClient, revision: number) {
}));
}
+function requestHealthyReselection(client: TestClient, revision: number): void {
+ client.socket.send(JSON.stringify({
+ type: "sfu-reselection-ready",
+ revision,
+ }));
+}
+
async function startPeerSelectedTurn(
viewer: TestClient,
parent: TestClient,
@@ -4140,7 +4147,458 @@ describe("WebSocket signaling", () => {
expect(viewerGrant.oldConnectionId).toBe(direct.oldConnectionId);
});
- it("uses one selected TURN edge after a Viewer SFU refresh is exhausted", async () => {
+ it("commits one healthy SFU root probe only after the peer edge is proven", async () => {
+ const harness = await startSfuHarness({
+ tokenIssuer: { async issueToken({ peerId }) { return `token-${peerId}`; } },
+ maxRoots: 1,
+ });
+ const active = await activateSingleViewerSfu(
+ harness.webSocketUrl,
+ harness.room,
+ "healthy-commit",
+ );
+
+ requestHealthyReselection(active.viewer, active.revision - 1);
+ await expect(active.viewer.inbox.next("route-update", 40)).rejects.toThrow("Timed out");
+ requestHealthyReselection(active.viewer, active.revision);
+ const [rootPrepare, hostPrepare] = await Promise.all([
+ nextPreparedRoute(active.viewer),
+ nextPreparedRoute(active.host),
+ ]);
+ expect(rootPrepare.assignment.upstream).toEqual({
+ kind: "peer",
+ peerId: active.hostAuth.peerId,
+ });
+ expect(hostPrepare.assignment).toMatchObject({
+ childPeerIds: [active.viewerAuth.peerId],
+ sfuPublicationGeneration: null,
+ });
+
+ requestHealthyReselection(active.viewer, active.revision);
+ active.viewer.socket.send(JSON.stringify({
+ type: "route-ready",
+ revision: rootPrepare.revision,
+ phase: "prepare",
+ }));
+ await expect(active.viewer.inbox.next("route-update", 40)).rejects.toThrow("Timed out");
+
+ const connectionId = "healthy-commit-probe";
+ active.host.socket.send(JSON.stringify({
+ type: "signal",
+ targetPeerId: active.viewerAuth.peerId,
+ payload: {
+ kind: "description",
+ connectionId,
+ description: { type: "offer", sdp: "v=0\r\n" },
+ },
+ }));
+ expect(await active.viewer.inbox.next("signal")).toMatchObject({
+ fromPeerId: active.hostAuth.peerId,
+ payload: { connectionId },
+ });
+ active.viewer.socket.send(JSON.stringify({
+ type: "signal",
+ targetPeerId: active.hostAuth.peerId,
+ payload: {
+ kind: "description",
+ connectionId: `${connectionId}-stale`,
+ description: { type: "answer", sdp: "v=0\r\n" },
+ },
+ }));
+ expect((await active.viewer.inbox.next("error")).code).toBe("FORBIDDEN");
+ active.viewer.socket.send(JSON.stringify({
+ type: "signal",
+ targetPeerId: active.hostAuth.peerId,
+ payload: {
+ kind: "description",
+ connectionId,
+ description: { type: "answer", sdp: "v=0\r\n" },
+ },
+ }));
+ await active.host.inbox.next("signal");
+ active.viewer.socket.send(JSON.stringify({
+ type: "route-ready",
+ revision: rootPrepare.revision,
+ phase: "prepare",
+ }));
+ const [rootActive, hostActive] = await Promise.all([
+ nextActiveRouteRevision(active.viewer, rootPrepare.revision),
+ nextActiveRouteRevision(active.host, hostPrepare.revision),
+ ]);
+ expect(rootActive.assignment.upstream).toEqual({
+ kind: "peer",
+ peerId: active.hostAuth.peerId,
+ });
+ expect(hostActive.assignment.sfuPublicationGeneration).toBeNull();
+
+ active.viewer.socket.send(JSON.stringify({
+ type: "route-failed",
+ revision: rootActive.revision,
+ phase: "active",
+ connectionId,
+ }));
+ const [rootRecovery, hostRecovery] = await Promise.all([
+ nextPreparedRoute(active.viewer),
+ nextPreparedRoute(active.host),
+ ]);
+ expect(rootRecovery.assignment.upstream).toEqual({ kind: "sfu" });
+ expect(hostRecovery.assignment.sfuPublicationGeneration).toBeTruthy();
+ });
+
+ it("rolls a failed healthy probe back to SFU and applies room cooldown", async () => {
+ let now = 50_000;
+ const harness = await startSfuHarness({
+ tokenIssuer: { async issueToken({ peerId }) { return `token-${peerId}`; } },
+ prepareTimeoutMs: 200,
+ now: () => now,
+ });
+ const active = await activateSingleViewerSfu(
+ harness.webSocketUrl,
+ harness.room,
+ "healthy-rollback",
+ );
+ requestHealthyReselection(active.viewer, active.revision);
+ const [rootPrepare] = await Promise.all([
+ nextPreparedRoute(active.viewer),
+ nextPreparedRoute(active.host),
+ ]);
+ const connectionId = "healthy-rollback-probe";
+ active.host.socket.send(JSON.stringify({
+ type: "signal",
+ targetPeerId: active.viewerAuth.peerId,
+ payload: {
+ kind: "description",
+ connectionId,
+ description: { type: "offer", sdp: "v=0\r\n" },
+ },
+ }));
+ await active.viewer.inbox.next("signal");
+ active.viewer.socket.send(JSON.stringify({
+ type: "route-failed",
+ revision: rootPrepare.revision,
+ phase: "prepare",
+ connectionId,
+ }));
+ const [rootRollback, hostRollback] = await Promise.all([
+ nextActiveRouteAfter(active.viewer, rootPrepare.revision),
+ nextActiveRouteAfter(active.host, rootPrepare.revision),
+ ]);
+ expect(rootRollback.assignment.upstream).toEqual({ kind: "sfu" });
+ expect(hostRollback.assignment.sfuPublicationGeneration).toBeTruthy();
+
+ requestHealthyReselection(active.viewer, rootRollback.revision);
+ await expect(active.viewer.inbox.next("route-update", 40)).rejects.toThrow("Timed out");
+ now += 30_001;
+ requestHealthyReselection(active.viewer, rootRollback.revision);
+ const retried = await nextPreparedRoute(active.viewer);
+ expect(retried.assignment.upstream).toEqual({
+ kind: "peer",
+ peerId: active.hostAuth.peerId,
+ });
+ const timedOut = await nextActiveRouteAfter(active.viewer, retried.revision);
+ expect(timedOut.assignment.upstream).toEqual({ kind: "sfu" });
+ });
+
+ it("does not swallow active SFU loss while a healthy probe is pending", async () => {
+ const harness = await startSfuHarness({
+ tokenIssuer: { async issueToken({ peerId }) { return `token-${peerId}`; } },
+ });
+ const active = await activateSingleViewerSfu(
+ harness.webSocketUrl,
+ harness.room,
+ "healthy-active-loss",
+ );
+ requestHealthyReselection(active.viewer, active.revision);
+ const pending = await nextPreparedRoute(active.viewer);
+ await nextPreparedRoute(active.host);
+ active.viewer.socket.send(JSON.stringify({
+ type: "route-failed",
+ revision: active.revision,
+ phase: "active",
+ connectionId: null,
+ }));
+ let recovered = await nextActiveRouteAfter(active.viewer, pending.revision);
+ if (recovered.assignment.upstream.kind === "sfu") {
+ recovered = await nextActiveRouteAfter(active.viewer, recovered.revision);
+ }
+ expect(recovered.assignment.upstream).toEqual({
+ kind: "peer",
+ peerId: active.hostAuth.peerId,
+ });
+ });
+
+ it.each(["viewer", "host"] as const)(
+ "aborts a healthy probe when the active %s SFU transport sends its prepare failure",
+ async (failedRole) => {
+ const harness = await startSfuHarness({
+ tokenIssuer: {
+ async issueToken({ peerId }) { return `token-${peerId}`; },
+ },
+ });
+ const active = await activateSingleViewerSfu(
+ harness.webSocketUrl,
+ harness.room,
+ `healthy-${failedRole}-prepare-loss`,
+ );
+ requestHealthyReselection(active.viewer, active.revision);
+ const [rootPrepare, hostPrepare] = await Promise.all([
+ nextPreparedRoute(active.viewer),
+ nextPreparedRoute(active.host),
+ ]);
+ const failedClient = failedRole === "viewer" ? active.viewer : active.host;
+ const failedPrepare = failedRole === "viewer" ? rootPrepare : hostPrepare;
+
+ failedClient.socket.send(JSON.stringify({
+ type: "route-failed",
+ revision: active.revision,
+ phase: "prepare",
+ connectionId: null,
+ }));
+ await Promise.all([
+ expect(
+ active.viewer.inbox.next("route-update", 40),
+ ).rejects.toThrow("Timed out"),
+ expect(
+ active.host.inbox.next("route-update", 40),
+ ).rejects.toThrow("Timed out"),
+ ]);
+
+ failedClient.socket.send(JSON.stringify({
+ type: "route-failed",
+ revision: failedPrepare.revision,
+ phase: "prepare",
+ connectionId: null,
+ }));
+ const [rootRollback, hostRollback] = await Promise.all([
+ nextActiveRouteAfter(active.viewer, failedPrepare.revision),
+ nextActiveRouteAfter(active.host, failedPrepare.revision),
+ ]);
+ expect(rootRollback.assignment.upstream).toEqual({ kind: "sfu" });
+ expect(hostRollback.assignment.sfuPublicationGeneration).toBeTruthy();
+
+ failedClient.socket.send(JSON.stringify({
+ type: "refresh-sfu",
+ revision: rootRollback.revision,
+ }));
+ expect(await failedClient.inbox.next("sfu-config")).toMatchObject({
+ revision: rootRollback.revision,
+ });
+ },
+ );
+
+ it("recovers when active selected Host ingress fails during healthy prepare", async () => {
+ const harness = await startSfuHarness({
+ tokenIssuer: {
+ async issueToken({ peerId }) { return `token-${peerId}`; },
+ },
+ selectedEdgeTurn: true,
+ });
+ const active = await activateSingleViewerSfu(
+ harness.webSocketUrl,
+ harness.room,
+ "healthy-selected-host-loss",
+ );
+ active.host.socket.send(JSON.stringify({
+ type: "route-failed",
+ revision: active.revision,
+ phase: "active",
+ connectionId: null,
+ }));
+ const ingress = await active.host.inbox.next("selected-edge-turn");
+ expect(ingress.edgeKind).toBe("host-sfu-ingress");
+ active.host.socket.send(JSON.stringify({
+ type: "route-ready",
+ revision: active.revision,
+ phase: "active",
+ }));
+
+ requestHealthyReselection(active.viewer, active.revision);
+ const [rootPrepare, hostPrepare] = await Promise.all([
+ nextPreparedRoute(active.viewer),
+ nextPreparedRoute(active.host),
+ ]);
+ active.host.socket.send(JSON.stringify({
+ type: "route-failed",
+ revision: active.revision,
+ phase: "prepare",
+ connectionId: ingress.newConnectionId,
+ }));
+ active.host.socket.send(JSON.stringify({
+ type: "route-failed",
+ revision: hostPrepare.revision,
+ phase: "prepare",
+ connectionId: `${ingress.newConnectionId}-wrong`,
+ }));
+ active.host.socket.send(JSON.stringify({
+ type: "route-failed",
+ revision: hostPrepare.revision,
+ phase: "prepare",
+ connectionId: null,
+ }));
+ await Promise.all([
+ expect(
+ active.viewer.inbox.next("route-update", 40),
+ ).rejects.toThrow("Timed out"),
+ expect(
+ active.host.inbox.next("route-update", 40),
+ ).rejects.toThrow("Timed out"),
+ ]);
+
+ active.host.socket.send(JSON.stringify({
+ type: "route-failed",
+ revision: hostPrepare.revision,
+ phase: "prepare",
+ connectionId: ingress.newConnectionId,
+ }));
+ const [rootRollback, hostRollback] = await Promise.all([
+ nextActiveRouteAfter(active.viewer, rootPrepare.revision),
+ nextActiveRouteAfter(active.host, hostPrepare.revision),
+ ]);
+ expect(rootRollback.assignment.upstream).toEqual({ kind: "sfu" });
+ expect(hostRollback.assignment.sfuPublicationGeneration).toBeTruthy();
+
+ active.host.socket.send(JSON.stringify({
+ type: "refresh-sfu",
+ revision: hostRollback.revision,
+ }));
+ expect(await active.host.inbox.next("sfu-config")).toMatchObject({
+ revision: hostRollback.revision,
+ });
+ });
+
+ it("keeps ordinary intent when topology abort fills the overlap slot", async () => {
+ const reselections = vi.spyOn(
+ HybridMediaRouter.prototype,
+ "handleHealthySfuReselection",
+ );
+ const harness = await startSfuHarness({
+ tokenIssuer: { async issueToken({ peerId }) { return `token-${peerId}`; } },
+ });
+ const active = await activateSingleViewerSfu(
+ harness.webSocketUrl,
+ harness.room,
+ "healthy-no-slot",
+ );
+ requestHealthyReselection(active.viewer, active.revision);
+ await nextPreparedRoute(active.viewer);
+ await nextPreparedRoute(active.host);
+ const leaf = await openClient(harness.webSocketUrl);
+ const leafAuth = peerAssisted(await authenticate(
+ leaf,
+ harness.room,
+ "viewer",
+ "healthy-no-slot-leaf",
+ 0,
+ ));
+ expect(leafAuth.routeAssignment.upstream).toEqual({
+ kind: "peer",
+ peerId: active.hostAuth.peerId,
+ });
+ const rootActive = await nextActiveRouteRevision(
+ active.viewer,
+ leafAuth.routeRevision,
+ );
+ await nextActiveRouteRevision(active.host, leafAuth.routeRevision);
+ const router = reselections.mock.instances.at(-1) as unknown as {
+ viewerRouteIntentsByRoom: Map<string, Map<string, { sfuAttempts: number }>>;
+ };
+ const intent = router.viewerRouteIntentsByRoom
+ .get(harness.room.roomId)
+ ?.get(active.viewerAuth.peerId);
+ expect(intent?.sfuAttempts).toBe(1);
+ reselections.mockRestore();
+ requestHealthyReselection(active.viewer, rootActive.revision);
+ await expect(active.viewer.inbox.next("route-update", 40)).rejects.toThrow("Timed out");
+ await expect(active.host.inbox.next("route-update", 40)).rejects.toThrow("Timed out");
+ });
+
+ it("keeps a browser parent to one downstream edge during overlap", async () => {
+ const harness = await startSfuHarness({
+ tokenIssuer: { async issueToken({ peerId }) { return `token-${peerId}`; } },
+ });
+ const peers = await prepareFallbackForTwoViewers(
+ harness.webSocketUrl,
+ harness.room,
+ "healthy-relay-budget",
+ undefined,
+ { viewerParentSibling: true },
+ );
+ for (const [client, revision] of [
+ [peers.host, peers.hostPrepare.revision],
+ [peers.failedViewer, peers.rootPrepare.revision],
+ ] as const) client.socket.send(JSON.stringify({
+ type: "route-ready", revision, phase: "prepare",
+ }));
+ const [activeRoot, activeParent] = await Promise.all([
+ nextActiveRouteRevision(peers.failedViewer, peers.rootPrepare.revision),
+ nextActiveRouteRevision(peers.rootViewer, peers.rootPrepare.revision),
+ ]);
+ expect(activeParent.assignment.childPeerIds).toEqual([
+ peers.rootSiblingAuth!.peerId,
+ ]);
+ expect(activeRoot.assignment.upstream).toEqual({ kind: "sfu" });
+ requestHealthyReselection(peers.failedViewer, activeRoot.revision);
+ await expect(peers.failedViewer.inbox.next("route-update", 40)).rejects.toThrow("Timed out");
+ });
+
+ it("cleans a healthy probe across Viewer session replacement and stop", async () => {
+ let now = 90_000;
+ const harness = await startSfuHarness({
+ tokenIssuer: { async issueToken({ peerId }) { return `token-${peerId}`; } },
+ now: () => now,
+ });
+ const active = await activateSingleViewerSfu(
+ harness.webSocketUrl,
+ harness.room,
+ "healthy-session",
+ );
+ requestHealthyReselection(active.viewer, active.revision);
+ const [oldPrepare] = await Promise.all([
+ nextPreparedRoute(active.viewer),
+ nextPreparedRoute(active.host),
+ ]);
+ const oldClosed = new Promise<void>((resolve) =>
+ active.viewer.socket.once("close", () => resolve()),
+ );
+ const replacement = await openClient(harness.webSocketUrl);
+ const replacementAuth = peerAssisted(await authenticate(
+ replacement,
+ harness.room,
+ "viewer",
+ "healthy-session-viewer",
+ ));
+ await oldClosed;
+ expect(replacementAuth.routeRevision).toBeGreaterThan(oldPrepare.revision);
+ expect(replacementAuth.routeAssignment.upstream).toEqual({ kind: "sfu" });
+ const hostRollback = await nextActiveRouteAfter(active.host, oldPrepare.revision);
+ expect(hostRollback.assignment.sfuPublicationGeneration).toBeTruthy();
+
+ replacement.socket.send(JSON.stringify({
+ type: "route-ready",
+ revision: oldPrepare.revision,
+ phase: "prepare",
+ }));
+ requestHealthyReselection(replacement, replacementAuth.routeRevision);
+ await expect(replacement.inbox.next("route-update", 40)).rejects.toThrow("Timed out");
+ now += 30_001;
+ requestHealthyReselection(replacement, replacementAuth.routeRevision);
+ const replacementPrepare = await nextPreparedRoute(replacement);
+ await nextPreparedRoute(active.host);
+
+ active.host.socket.send(JSON.stringify({
+ type: "stop-sharing",
+ shareGeneration: "healthy-session-share",
+ }));
+ await replacement.inbox.next("sharing-stopped");
+ replacement.socket.send(JSON.stringify({
+ type: "route-ready",
+ revision: replacementPrepare.revision,
+ phase: "prepare",
+ }));
+ await expect(replacement.inbox.next("route-update", 40)).rejects.toThrow("Timed out");
+ });
+
+ it("keeps selected TURN exclusive with healthy probes until failure", async () => {
const harness = await startSfuHarness({
tokenIssuer: { async issueToken({ peerId }) { return `token-${peerId}`; } },
selectedEdgeTurn: true,
@@ -4175,6 +4633,10 @@ describe("WebSocket signaling", () => {
iceServer: { urls: ["turn:turn.example.test:3478?transport=udp"] },
});
expect(viewerGrant.newConnectionId).not.toBe(direct.oldConnectionId);
+ requestHealthyReselection(direct.viewer, direct.revision);
+ await expect(
+ direct.viewer.inbox.next("route-update", 40),
+ ).rejects.toThrow("Timed out");
direct.host.socket.send(JSON.stringify({
type: "signal",
targetPeerId: direct.viewerAuth.peerId,
@@ -4198,6 +4660,10 @@ describe("WebSocket signaling", () => {
},
}));
await direct.host.inbox.next("signal");
+ requestHealthyReselection(direct.viewer, direct.revision);
+ await expect(
+ direct.viewer.inbox.next("route-update", 40),
+ ).rejects.toThrow("Timed out");
direct.viewer.socket.send(JSON.stringify({
type: "route-failed",
revision: direct.revision,
@@ -4210,6 +4676,54 @@ describe("WebSocket signaling", () => {
).rejects.toThrow("Timed out");
});
+ it("keeps a healthy probe exclusive when an SFU refresh token fails", async () => {
+ let failRefresh = false;
+ let failedRefreshes = 0;
+ const harness = await startSfuHarness({
+ tokenIssuer: {
+ async issueToken({ peerId }) {
+ if (failRefresh) {
+ failedRefreshes += 1;
+ throw new Error("refresh failed");
+ }
+ return `token-${peerId}`;
+ },
+ },
+ selectedEdgeTurn: true,
+ });
+ const active = await activateSingleViewerSfu(
+ harness.webSocketUrl,
+ harness.room,
+ "healthy-refresh-failure",
+ );
+ requestHealthyReselection(active.viewer, active.revision);
+ await Promise.all([
+ nextPreparedRoute(active.viewer),
+ nextPreparedRoute(active.host),
+ ]);
+
+ failRefresh = true;
+ active.viewer.socket.send(JSON.stringify({
+ type: "refresh-sfu",
+ revision: active.revision,
+ }));
+ await vi.waitFor(() => expect(failedRefreshes).toBe(1));
+ await Promise.all([
+ expect(
+ active.viewer.inbox.next("selected-edge-turn", 40),
+ ).rejects.toThrow("Timed out"),
+ expect(
+ active.host.inbox.next("selected-edge-turn", 40),
+ ).rejects.toThrow("Timed out"),
+ expect(
+ active.viewer.inbox.next("route-update", 40),
+ ).rejects.toThrow("Timed out"),
+ expect(
+ active.host.inbox.next("route-update", 40),
+ ).rejects.toThrow("Timed out"),
+ ]);
+ });
+
it("admits only one pending peer-selected TURN attempt per room", async () => {
const harness = await startSfuHarness({
tokenIssuer: { async issueToken({ peerId }) { return `token-${peerId}`; } },
diff --git a/tests/status-badge.test.ts b/tests/status-badge.test.ts
index 50ddf264..069f25eb 100644
--- a/tests/status-badge.test.ts
+++ b/tests/status-badge.test.ts
@@ -56,6 +56,13 @@ describe("route status badges", () => {
route: "sfu",
evidence: sfu,
});
+ expect(
+ viewerRouteEvidence(
+ { kind: "peer", peerId: "new-parent" },
+ peerSnapshot,
+ sfu,
+ ),
+ ).toEqual({ route: "sfu", evidence: sfu });
});
it("requires a connected peer or an observed selected path", () => {