Speed up downloads by combining multiple network connections in parallel
TypeScript bandwidth-aggregationcross-platformdesktop-appdownloaderdownloadmanagerlinuxmacosmulticonnectionparallel-downloadswindows
Simpler stream count and quicker backups for slow blocks (#49)
Anmol Kapilf9d1d725794f
diff --git a/README.md b/README.mdindex 41facfc..1025f7b 100644--- a/README.md+++ b/README.md@@ -67,7 +67,7 @@ File ──→ Split ─────┤ ├─## Features-- 🚀 **Multi-interface, multi-connection downloads** — splits files into chunks of up to 8 MB and fans them out across worker connections bound to specific network interfaces, each kept open from one chunk to the next. How many connections each interface gets is worked out while downloading: as many as keep making it faster, up to 16.+- 🚀 **Multi-interface, multi-connection downloads** — splits files into chunks of up to 8 MB and fans them out across worker connections bound to specific network interfaces, each kept open from one chunk to the next. Each interface starts with 8 connections and doubles once they are all receiving, up to 32; if the server refuses some of them (503, 429, 403) while serving the rest, the refused ones close and that interface stays at the ones left; a server refusing everything (busy, or an expired link) doesn't lower it.- 🔌 **Hardware interface detection** — queries Windows adapters via PowerShell `Get-NetAdapter` and macOS hardware ports via `networksetup` so Wi-Fi, Ethernet, tethered iPhones, and Thunderbolt bridges are labeled by real device names instead of bare BSD names (`en0`, `en6`).- ⚖️ **Dynamic work-stealing queue** — chunks are leased from a shared pending queue; faster networks pull more chunks instead of waiting for slower connections to finish.- ⏸️ **Resumable downloads** — cleanly pause and resume downloads with progress saved in a destination-side staging file.@@ -144,7 +144,7 @@ Instead, Plexo uses a **dynamic work-stealing queue**:3. A pool of worker connections continuously lease the next chunk from the queue as soon as they become free. Each keeps its one connection to the server from chunk to chunk, so it pays for the handshake and TCP's slow start once, not per chunk. Connections start interleaved across networks, so each network is served before any is served twice.- **How many.** Each network starts with 4. More help only when something limits each connection on its own (a server capping per-connection speed, or a long, lossy route), not once the network itself is full, so Plexo measures instead of guessing. Once speeds have settled, it doubles one network's connections, waits for them to settle again, and keeps them only if that network got at least 15% faster, net of anything the other networks lost beyond their usual ups and downs (two networks behind one router share its uplink). Otherwise the new connections close, and their chunks go back to the queue to carry on from where they got to. So do extra connections the server turns away. Up to 16 per network, and never more than there are chunks for.4. Faster interfaces finish chunks quicker and immediately pick up new ones; slower interfaces pull fewer chunks.-5. **Racing the tail.** Once no chunk is left waiting, a free connection can start a second attempt at a chunk another connection is fetching too slowly (one that still needs as long again as it has already taken, at least 5 seconds), picking up from where the first had got to. Whichever finishes first wins and the other is dropped. It costs a few bytes fetched twice at the very end, and it means one slow connection — or one slow network — can no longer hold the whole download back. A stream doing this is marked **BACKUP** in the streams table.+5. **Racing the tail.** Once no chunk is left waiting, a free connection can start a second attempt at a chunk another connection is fetching too slowly (as soon as the free connection would finish it in under half the time the chunk still needs; a chunk whose second attempt is stuck too can get one more), picking up from where the first had got to. Whichever finishes first wins and the other is dropped. It costs a few bytes fetched twice at the very end, and it means one slow connection — or one slow network — can no longer hold the whole download back. A stream doing this is marked **BACKUP** in the streams table.```textShared Pending Queue: [Chunk #4] [Chunk #5] [Chunk #6] [Chunk #7] [Chunk #8] ...diff --git a/e2e/chaos.spec.ts b/e2e/chaos.spec.tsindex 65b275d..d62eaaa 100644--- a/e2e/chaos.spec.ts+++ b/e2e/chaos.spec.ts@@ -149,12 +149,8 @@ test('any sequence of pauses, crashes and faults still ends in the exact file @cseed: fileSeed,bytesPerSecond: 512 * 1024}).start()- // Short measuring windows, so streams come and go within the run; kept automatic across- // the relaunches a crash or a quit brings.- const app = new PlexoApp(- dirs,- connections === 'auto' ? { PLEXO_E2E_STREAMS: '', PLEXO_E2E_PROBE_MS: '500' } : {}- )+ // Kept automatic across the relaunches a crash or a quit brings.+ const app = new PlexoApp(dirs, connections === 'auto' ? { PLEXO_E2E_STREAMS: '' } : {})const real: Real = { app, origin, id: '', faults: [] }origin.setRule(({ range }) =>range && !(range.start === 0 && range.end === 0) ? real.faults.shift() : undefineddiff --git a/e2e/concurrency.spec.ts b/e2e/concurrency.spec.tsindex f01e28e..7514573 100644--- a/e2e/concurrency.spec.ts+++ b/e2e/concurrency.spec.ts@@ -1,352 +1,134 @@import { expect, test } from '@playwright/test'-import {- ConcurrencyController,- type Action,- type ConcurrencyPolicy,- type Snapshot-} from '../src/main/download/concurrency'--// M. How many streams each network runs. The controller is pure, so it's driven here against-// models of what limits a download — the server, the link, an uplink two networks share — with-// the noise and ramp-up of real connections, and judged by where it settles.--const MB = 1e6-const POLICY: ConcurrencyPolicy = {- maxPerNetwork: 16,- windowMs: 2000,- warmupMs: 2000,- minGain: 0.15,- maxWindows: 4-}-const TICK_MS = 500-const SEEDS = 200--/** Bytes per second each network settles at, given how many streams each runs. */-type Model = (streams: Map<string, number>) => Map<string, number>--interface Options {- seconds?: number- spareWork?: number- /** Each tick's delivery is off by up to this share, either way. */- noise?: number- seed?: number- /** Seconds a network takes to get most of the way to a new speed (TCP ramping up). */- ramp?: number- /** Streams past this many on a network are turned away by the server. */- accepts?: number- /** Ticks a retired stream takes to leave. */- retireTicks?: number-}--interface Run {- streams: Map<string, number>- /** Most streams each network had at once. */- peak: Map<string, number>- actions: Action[]- /** What the controller saw whenever it acted. */- seen: Snapshot[]-}--/** A seeded generator, so a failing case can be run again exactly. */-function random(seed: number): () => number {- return () => {- seed = (seed + 0x6d2b79f5) | 0- let t = Math.imul(seed ^ (seed >>> 15), 1 | seed)- t = (t + Math.imul(t ^ (t >>> 7), 61 | t)) ^ t- return ((t ^ (t >>> 14)) >>> 0) / 4294967296- }-}--function simulate(model: Model, networks: string[], options: Options = {}): Run {- const { seconds = 120, spareWork = 1000, noise = 0, seed = 1, ramp = 0 } = options- const { accepts = Infinity, retireTicks = 1 } = options- const next = random(seed)- const controller = new ConcurrencyController(POLICY)- const streams = new Map(networks.map((id) => [id, 4]))- const peak = new Map(streams)- const received = new Map(networks.map((id) => [id, 0]))- let rates = new Map(networks.map((id) => [id, 0]))- const leaving: { networkId: string; count: number; at: number }[] = []- const run: Run = { streams, peak, actions: [], seen: [] }-- for (let now = 0; now <= seconds * 1000; now += TICK_MS) {- const target = model(streams)- const lag = ramp ? Math.exp(-TICK_MS / 1000 / ramp) : 0- rates = new Map(- networks.map((id) => [id, target.get(id)! + (rates.get(id)! - target.get(id)!) * lag])- )- for (const id of networks) {- const jitter = 1 + noise * (next() * 2 - 1)- received.set(id, received.get(id)! + (rates.get(id)! * jitter * TICK_MS) / 1000)- }- while (leaving.length > 0 && leaving[0].at <= now) {- const gone = leaving.shift()!- streams.set(gone.networkId, streams.get(gone.networkId)! - gone.count)- }- const snapshot: Snapshot = {- now,- networks: networks.map((id) => ({- id,- streams:- streams.get(id)! -- leaving- .filter((entry) => entry.networkId === id)- .reduce((sum, entry) => sum + entry.count, 0),- received: received.get(id)!,- rejected: Math.max(0, streams.get(id)! - accepts)- })),- spareWork,- retiring: leaving.reduce((sum, entry) => sum + entry.count, 0)- }- const action = controller.tick(snapshot)- if (!action) continue- run.actions.push(action)- run.seen.push(snapshot)- if (action.kind === 'add') {- streams.set(action.networkId, streams.get(action.networkId)! + action.count)- peak.set(- action.networkId,- Math.max(peak.get(action.networkId)!, streams.get(action.networkId)!)- )- } else {- leaving.push({ ...action, at: now + retireTicks * TICK_MS })- }- }- return run-}--/** The share of seeds for which `holds` is true. */-function share(holds: (seed: number) => boolean): number {- let count = 0- for (let seed = 1; seed <= SEEDS; seed++) if (holds(seed)) count++- return count / SEEDS-}--/** Each connection gets at most `perStream`, and the link at most `link`. */-const capped =- (perStream: number, link: number): Model =>- (streams) =>- new Map([...streams].map(([id, count]) => [id, Math.min(count * perStream, link)]))--/** Networks behind one uplink of `link`, each connection good for `perStream`. */-const sharedUplink =- (perStream: number, link: number): Model =>- (streams) => {- const total = [...streams.values()].reduce((sum, count) => sum + count, 0)- const delivered = Math.min(total * perStream, link)- return new Map([...streams].map(([id, count]) => [id, (delivered * count) / total]))- }--/** Ethernet full at 100 MB/s beside a cellular link, which more streams speed up or don't. */-const besideEthernet =- (cellularGains: boolean): Model =>- (streams) =>- new Map([- ['ethernet', Math.min(streams.get('ethernet')! * 50 * MB, 100 * MB)],- ['cellular', cellularGains ? streams.get('cellular')! * 0.25 * MB : 1 * MB]- ])+import { ConcurrencyController, type NetworkSnapshot } from '../src/main/download/concurrency'++// M. How many streams each network runs. The controller is pure, so each rule is checked against+// the exact situation it is for.++const MAX = 32++const network = (+ id: string,+ streams: number,+ over: Partial<NetworkSnapshot> = {}+): NetworkSnapshot => ({+ id,+ streams,+ answered: streams,+ refused: 0,+ served: streams,+ ...over+})test.describe('stream count', () => {- test('a server that caps each connection gets more of them, up to the limit', () => {- const { streams, actions } = simulate(capped(1 * MB, 1000 * MB), ['a'])- expect(streams.get('a')).toBe(16)- // Doubling: two steps, not twelve.- expect(actions).toEqual([- { kind: 'add', networkId: 'a', count: 4 },- { kind: 'add', networkId: 'a', count: 8 }- ])- expect(- share(- (seed) =>- simulate(capped(1 * MB, 1000 * MB), ['a'], { noise: 0.1, seed }).streams.get('a') === 16- )- ).toBe(1)+ test('doubles once every stream is receiving, up to the limit', () => {+ const controller = new ConcurrencyController(MAX)+ const counts: number[] = []+ for (let streams = 8; streams < MAX;) {+ const actions = controller.tick({ networks: [network('a', streams)], spareWork: 1000 })+ expect(actions).toHaveLength(1)+ streams += actions[0].count+ counts.push(streams)+ }+ expect(counts).toEqual([16, 32])+ expect(controller.tick({ networks: [network('a', MAX)], spareWork: 1000 })).toEqual([])})- test('a full link keeps what it started with: the extra streams are tried, then retired', () => {- const { streams, peak } = simulate(capped(10 * MB, 20 * MB), ['a'])- expect(peak.get('a')).toBe(8)- expect(streams.get('a')).toBe(4)+ test('waits while any stream has yet to receive anything', () => {+ const controller = new ConcurrencyController(MAX)expect(- share(- (seed) =>- simulate(capped(10 * MB, 20 * MB), ['a'], { noise: 0.1, seed }).streams.get('a') === 4- )- ).toBe(1)+ controller.tick({ networks: [network('a', 8, { answered: 7 })], spareWork: 1000 })+ ).toEqual([])})- test('grows while it pays and stops where it stops paying', () => {- // 4 streams: 4 MB/s. 8: 8 MB/s. 16 would be the 10 MB/s link: +25%, which still pays.- expect(simulate(capped(1 * MB, 10 * MB), ['a']).streams.get('a')).toBe(16)- // A 9 MB/s link: 16 streams would add 1 MB/s to 8, under the 15% a step has to earn.- expect(simulate(capped(1 * MB, 9 * MB), ['a']).streams.get('a')).toBe(8)+ test('adds no more streams than the waiting blocks can keep busy', () => {+ const controller = new ConcurrencyController(MAX)+ expect(controller.tick({ networks: [network('a', 8)], spareWork: 3 })).toEqual([+ { kind: 'add', networkId: 'a', count: 3 }+ ])+ expect(controller.tick({ networks: [network('a', 8)], spareWork: 0 })).toEqual([])+ // The waiting blocks are shared: the first network takes what it can use.+ expect(+ controller.tick({ networks: [network('a', 8), network('b', 8)], spareWork: 10 })+ ).toEqual([+ { kind: 'add', networkId: 'a', count: 8 },+ { kind: 'add', networkId: 'b', count: 2 }+ ])})- test('a slow network is judged by its own gain, not lost in a fast one’s noise', () => {- // Cellular's doubling is worth 1 MB/s; Ethernet's everyday ±10% is worth ±10 MB/s.- const grows = share(- (seed) =>- simulate(besideEthernet(true), ['ethernet', 'cellular'], { noise: 0.1, seed }).streams.get(- 'cellular'- ) === 16- )- const staysPut = share(- (seed) =>- simulate(besideEthernet(false), ['ethernet', 'cellular'], { noise: 0.1, seed }).streams.get(- 'cellular'- ) === 4- )- expect(grows).toBeGreaterThanOrEqual(0.95)- expect(staysPut).toBeGreaterThanOrEqual(0.95)+ test('streams the server refuses close, and it never grows past the ones left again', () => {+ const controller = new ConcurrencyController(MAX)+ expect(+ controller.tick({+ networks: [network('a', 16, { served: 10, refused: 6 })],+ spareWork: 1000+ })+ ).toEqual([{ kind: 'retire', networkId: 'a', count: 6 }])+ // Later, all well again: still no growth past 10.+ expect(controller.tick({ networks: [network('a', 10)], spareWork: 1000 })).toEqual([])+ // A stream lost some other way is made up for, back to the limit and no further.+ expect(controller.tick({ networks: [network('a', 7)], spareWork: 1000 })).toEqual([+ { kind: 'add', networkId: 'a', count: 3 }+ ])})- test('streams that only take speed from a network behind the same uplink are retired', () => {- const { streams, peak } = simulate(sharedUplink(1 * MB, 8 * MB), ['wifi', 'ethernet'])- expect(peak.get('wifi')).toBe(8)- expect(peak.get('ethernet')).toBe(8)- expect(streams).toEqual(- new Map([- ['wifi', 4],- ['ethernet', 4]- ])- )- const bothStay = share((seed) => {- const run = simulate(sharedUplink(1 * MB, 8 * MB), ['wifi', 'ethernet'], { noise: 0.1, seed })- return run.streams.get('wifi') === 4 && run.streams.get('ethernet') === 4- })- expect(bothStay).toBeGreaterThanOrEqual(0.95)+ test('streams that have yet to receive anything are not taken for refused ones', () => {+ // One stream refused just after starting, while the other seven were still connecting: the+ // limit is the seven, not the none that had answered yet.+ const controller = new ConcurrencyController(MAX)+ expect(+ controller.tick({+ networks: [network('a', 8, { answered: 0, served: 0, refused: 1 })],+ spareWork: 1000+ })+ ).toEqual([])+ expect(+ controller.tick({ networks: [network('a', 8, { served: 7, refused: 1 })], spareWork: 1000 })+ ).toEqual([{ kind: 'retire', networkId: 'a', count: 1 }])+ expect(controller.tick({ networks: [network('a', 7)], spareWork: 1000 })).toEqual([])})- test('speeds still ramping up are waited out, not mistaken for a step’s gain', () => {- // A full link whose connections take seconds to reach speed: measuring them halfway up would- // make the first doubling look like it paid.- for (const ramp of [1, 2, 4]) {- const staysPut = share(- (seed) =>- simulate(capped(10 * MB, 20 * MB), ['a'], { ramp, noise: 0.05, seed }).streams.get(- 'a'- ) === 4- )- expect(staysPut, `ramping over ${ramp} s`).toBeGreaterThanOrEqual(0.95)+ test('a server refusing while it sends nothing is not limiting connections: the count stays', () => {+ // Busy, or the link expired: waited out or reported elsewhere, not a reason to close streams.+ // Refusals spread over a few looks don't add up to a limit either.+ const controller = new ConcurrencyController(MAX)+ for (const refused of [16, 5, 11]) {+ expect(+ controller.tick({+ networks: [network('a', 16, { served: 0, refused })],+ spareWork: 1000+ })+ ).toEqual([])}- })-- test('new streams the server turns away are retired, and the network grows no further', () => {- const { streams, actions } = simulate(capped(1 * MB, 1000 * MB), ['a'], { accepts: 4 })- expect(streams.get('a')).toBe(4)- expect(actions).toEqual([- { kind: 'add', networkId: 'a', count: 4 },- { kind: 'retire', networkId: 'a', count: 4 }+ // Once it serves again, the network grows as before.+ expect(controller.tick({ networks: [network('a', 16)], spareWork: 1000 })).toEqual([+ { kind: 'add', networkId: 'a', count: 16 }])})- test('nothing is measured while retired streams are still leaving', () => {- // Two networks behind one uplink: each step is undone, and the next network's is only taken- // once the last one's streams have gone.- const run = simulate(sharedUplink(1 * MB, 8 * MB), ['wifi', 'ethernet'], { retireTicks: 6 })- expect(run.actions.map((action) => action.kind)).toEqual(['add', 'retire', 'add', 'retire'])- for (const [index, action] of run.actions.entries()) {- if (action.kind === 'add') expect(run.seen[index].retiring).toBe(0)- }- })-- test('no growth without waiting blocks for the new streams to take', () => {- expect(simulate(capped(1 * MB, 1000 * MB), ['a'], { spareWork: 0 }).actions).toEqual([])- // Room for two more: the step is cut to fit.- expect(simulate(capped(1 * MB, 1000 * MB), ['a'], { spareWork: 2 }).actions[0]).toEqual({- kind: 'add',- networkId: 'a',- count: 2- })- })-- test('a network joining or leaving mid-step undoes the step, and a network that rejoins can grow again', () => {- const controller = new ConcurrencyController(POLICY)- const received = new Map([- ['a', 0],- ['b', 0]- ])- const streams = new Map([- ['a', 4],- ['b', 4]+ test('a limit only ever comes down', () => {+ const controller = new ConcurrencyController(MAX)+ controller.tick({ networks: [network('a', 16, { served: 10, refused: 6 })], spareWork: 1000 })+ // A later partial refusal on fewer streams lowers it again; it never rises.+ expect(+ controller.tick({ networks: [network('a', 10, { served: 8, refused: 2 })], spareWork: 1000 })+ ).toEqual([{ kind: 'retire', networkId: 'a', count: 2 }])+ expect(controller.tick({ networks: [network('a', 6)], spareWork: 1000 })).toEqual([+ { kind: 'add', networkId: 'a', count: 2 }])- let now = 0- const tick = (ids: string[]): Action | undefined => {- // A server that caps each connection at 0.5 MB/s: another stream always pays.- for (const id of ids) received.set(id, received.get(id)! + streams.get(id)! * 0.25 * MB)- now += TICK_MS- return controller.tick({- now,- networks: ids.map((id) => ({- id,- streams: streams.get(id)!,- received: received.get(id)!,- rejected: 0- })),- spareWork: 1000,- retiring: 0- })- }- let action: Action | undefined- while (!(action = tick(['a', 'b']))) expect(now).toBeLessThan(60_000)- expect(action.kind).toBe('add')- const grown = action.networkId- const other = grown === 'a' ? 'b' : 'a'- streams.set(grown, streams.get(grown)! + action.count)-- // The other network drops out before the step is judged: its speed going says nothing about- // the step, so the step is undone rather than judged against it.- expect(tick([grown])).toEqual({ kind: 'retire', networkId: grown, count: action.count })- streams.set(grown, 4)-- // Back again, it is grown like any network that has just joined.- for (let added = false; !added;) {- const next = tick(['a', 'b'])- if (next?.kind === 'add' && next.networkId === other) added = true- else if (next?.kind === 'add')- streams.set(next.networkId, streams.get(next.networkId)! + next.count)- expect(now).toBeLessThan(120_000)- }})- test('a pause mid-step gives back the untried streams; what was settled stays settled', () => {- const controller = new ConcurrencyController(POLICY)- const tick = (now: number, streams: number, received: number): Action | undefined =>+ test('each network is decided on its own', () => {+ const controller = new ConcurrencyController(MAX)+ // b is refused and one of a's streams hasn't answered: only b changes, and a is untouched.+ expect(controller.tick({- now,- networks: [{ id: 'a', streams, received, rejected: 0 }],- spareWork: 1000,- retiring: 0+ networks: [network('a', 8, { answered: 7 }), network('b', 16, { served: 8, refused: 8 })],+ spareWork: 1000})- // 1 MB/s whatever the stream count: a full link.- let received = 0- let now = 0- let streams = 4- const until = (done: (action: Action) => boolean): Action => {- for (; now < 60_000; now += TICK_MS) {- received += 0.5 * MB- const action = tick(now, streams, received)- if (action && done(action)) return action- }- throw new Error('never happened')- }- until((action) => action.kind === 'add')- streams = 8- now += TICK_MS- // Paused partway through judging the step.- expect(controller.interrupt()).toEqual({ kind: 'retire', networkId: 'a', count: 4 })- streams = 4- // Resumed: the step is tried again from scratch, found not to pay, and not tried a third time.- expect(until((action) => action.kind === 'retire')).toEqual({- kind: 'retire',- networkId: 'a',- count: 4- })- streams = 4- expect(controller.interrupt()).toBeUndefined()- for (let end = now + 30_000; now < end; now += TICK_MS) {- received += 0.5 * MB- expect(tick(now, streams, received)).toBeUndefined()- }+ ).toEqual([{ kind: 'retire', networkId: 'b', count: 8 }])+ // A network that left and came back keeps its limit, and one that never refused grows.+ expect(+ controller.tick({ networks: [network('a', 8), network('b', 8)], spareWork: 1000 })+ ).toEqual([{ kind: 'add', networkId: 'a', count: 8 }])})})diff --git a/e2e/fixtures.ts b/e2e/fixtures.tsindex f6fedf9..119f308 100644--- a/e2e/fixtures.ts+++ b/e2e/fixtures.ts@@ -312,7 +312,7 @@ export function checkEvents(sessions: DownloadState[][]): void {}if (state.status === 'downloading') {// A stream holds a block exactly while it is fetching it. A block has at most one stream- // fetching it for real and one racing it as a hedge, and is in flight whenever the first+ // fetching it for real and two racing it as hedges, and is in flight whenever the first// is there. What the stream rows show is only as true as this.const primaries = new Map<number, number>()const hedges = new Map<number, number>()@@ -328,8 +328,11 @@ export function checkEvents(sessions: DownloadState[][]): void {const tally = chunk.hedge ? hedges : primariestally.set(chunk.currentBlockIndex, (tally.get(chunk.currentBlockIndex) ?? 0) + 1)}- for (const [index, count] of [...primaries, ...hedges]) {- expect(count, `${label}: block ${index} has one stream of each kind`).toBe(1)+ for (const [index, count] of primaries) {+ expect(count, `${label}: block ${index} has one stream fetching it`).toBe(1)+ }+ for (const [index, count] of hedges) {+ expect(count, `${label}: block ${index} has at most two hedges`).toBeLessThanOrEqual(2)}for (const index of primaries.keys()) {expect(state.blocks?.[index]?.status, `${label}: held block ${index} is in flight`).toBe(diff --git a/e2e/plan.spec.ts b/e2e/plan.spec.tsindex 73210cf..c22dbb6 100644--- a/e2e/plan.spec.ts+++ b/e2e/plan.spec.ts@@ -65,9 +65,9 @@ test.describe('download plan', () => {expect(plan.blockSizeBytes).toBe(DEFAULT_MAX_BLOCK_BYTES)})- test('each network starts with four streams, with blocks enough to grow to its limit', () => {+ test('each network starts with eight streams, with blocks enough to grow to its limit', () => {const plan = planDownload({ totalBytes: 512 * MIB, splittable: true, networkCount: 2 })- expect(startingStreams(plan.blockCount, 2)).toBe(4)+ expect(startingStreams(plan.blockCount, 2)).toBe(8)// Streams added later need waiting blocks to take: two each, at the most every network can have.expect(plan.blockCount).toBeGreaterThanOrEqual(2 * MAX_STREAMS_PER_NETWORK * 2)})diff --git a/e2e/scheduler.spec.ts b/e2e/scheduler.spec.tsindex d50de21..391abf6 100644--- a/e2e/scheduler.spec.ts+++ b/e2e/scheduler.spec.ts@@ -11,7 +11,7 @@ import type { BlockState, BlockStatus, ChunkStatus } from '../src/shared/types'// J. Who fetches what. Pure, so the races that decide it (which stream asked first, how far// along the holder is) are pinned down as inputs instead of left to timing.-const POLICY: SchedulerPolicy = { hedgeAfterMs: 5000, maxHedgesPerBlock: 2 }+const POLICY: SchedulerPolicy = { hedgeAfterMs: 5000, maxHedgesPerBlock: 2, startupMs: 1000 }const NOW = 100_000const LENGTH = 1000@@ -54,8 +54,13 @@ function state(parts: {}}-const pick = (s: SchedulerState, id: number, networkId: string): string | undefined => {- const work = pickWork(s, { id, networkId }, NOW, POLICY)+const pick = (+ s: SchedulerState,+ id: number,+ networkId: string,+ speedBytesPerSec?: number+): string | undefined => {+ const work = pickWork(s, { id, networkId, speedBytesPerSec }, NOW, POLICY)return work && `${work.kind}:${work.block.index}`}@@ -193,11 +198,48 @@ test.describe('scheduler: racing a slow block', () => {expect(pick(avoided, 2, 'a')).toBe('primary:1')})- test('one hedge at a time, and at most a few over a block’s life', () => {- const hedged = laggard(0, { attempts: [[0, [attempt(0, 'a'), attempt(1, 'a', 'hedge')]]] })- expect(pick(hedged, 2, 'a')).toBeUndefined()+ test('a block whose hedge is stuck too is raced again, up to a few over its life', () => {+ const streams = [+ stream(0, 'a', 'downloading', 0),+ stream(1, 'a', 'downloading', 0),+ stream(2, 'a', 'pending')+ ]+ const hedged = laggard(0, {+ streams,+ attempts: [[0, [attempt(0, 'a'), attempt(1, 'a', 'hedge')]]],+ hedgesUsed: [[0, 1]]+ })+ expect(pick(hedged, 2, 'a')).toBe('hedge:0')expect(pick(laggard(0, { hedgesUsed: [[0, 2]] }), 1, 'a')).toBeUndefined()- expect(pick(laggard(0, { hedgesUsed: [[0, 1]] }), 1, 'a')).toBe('hedge:0')+ })++ test('…but not while its hedge is doing fine', () => {+ // The hedge will be done in 0.8 s: nobody could beat that.+ const hedged = laggard(0, {+ streams: [+ stream(0, 'a', 'downloading', 0),+ stream(1, 'a', 'downloading', 1000),+ stream(2, 'a', 'pending')+ ],+ attempts: [[0, [attempt(0, 'a'), attempt(1, 'a', 'hedge')]]],+ hedgesUsed: [[0, 1]]+ })+ expect(pick(hedged, 2, 'a')).toBeUndefined()+ })++ test('a free stream that has shown its speed races as soon as it would clearly finish first', () => {+ // 800 bytes left at 200 B/s: 4 s. Under the 5 s a stream of unknown speed waits for, but a+ // stream that fetched its last block at 800 B/s would be done in 1 s + 1 s startup.+ expect(pick(laggard(200), 1, 'a')).toBeUndefined()+ expect(pick(laggard(200), 1, 'a', 800)).toBe('hedge:0')+ // Not when it would only finish a little sooner: at 400 B/s it needs 3 s, over half of 4 s.+ expect(pick(laggard(200), 1, 'a', 400)).toBeUndefined()+ })++ test('a free stream of unknown speed races a block that needs at least hedgeAfterMs more', () => {+ // 800 bytes left at 160 B/s: exactly 5 s. At 200 B/s, 4 s: left alone.+ expect(pick(laggard(160), 1, 'a')).toBe('hedge:0')+ expect(pick(laggard(200), 1, 'a')).toBeUndefined()})test('a stream never races its own block', () => {@@ -263,7 +305,7 @@ test.describe('scheduler: racing a slow block', () => {expect(pick(s, 1, 'a')).toBeUndefined()})- test('whatever is chosen is real work: a waiting block, or a lone primary’s block', () => {+ test('whatever is chosen is real work: a waiting block, or a block still being fetched', () => {const networks = ['a', 'b']fc.assert(fc.property(@@ -301,7 +343,9 @@ test.describe('scheduler: racing a slow block', () => {} else {expect(blocks.some((b) => b.status === 'pending')).toBe(false)expect(work.block.status).toBe('downloading')- expect(s.attempts.get(work.block.index)).toHaveLength(1)+ const attempts = s.attempts.get(work.block.index) ?? []+ expect(attempts.some((a) => a.kind === 'primary')).toBe(true)+ expect(attempts.some((a) => a.streamId === 99)).toBe(false)}})diff --git a/e2e/streams.spec.ts b/e2e/streams.spec.tsindex 5f21720..080076a 100644--- a/e2e/streams.spec.ts+++ b/e2e/streams.spec.ts@@ -4,9 +4,6 @@ import { BLOCK, expect, test } from './fixtures'// server that limits it in each of the ways that matter.test.describe('automatic stream count', () => {- // Two ticks to a measuring window, so no one tick decides anything.- test.use({ appEnv: { PLEXO_E2E_PROBE_MS: '1000' } })-const peakStreams = (plexo: { sessions: { chunks: unknown[] }[][] }): number =>Math.max(...plexo.sessions.at(-1)!.map((state) => state.chunks.length))@@ -31,4 +28,28 @@ test.describe('automatic stream count', () => {expect(state.chunks).toHaveLength(4)expect(origin.log.some((request) => request.status === 503)).toBe(true)})++ test.describe('waited out', () => {+ test.use({ appEnv: { PLEXO_E2E_SERVER_BUSY_MS: '60000' } })++ test('a server busy for everyone at first is waited out, not taken as a connection limit', async ({+ plexo,+ serve+ }) => {+ const origin = await serve({ size: 96 * BLOCK, bytesPerSecond: 256 * 1024 })+ let busyUntil = 0+ origin.setRule(({ range }) => {+ if (range?.start === 0 && range.end === 0) return undefined // the probe+ busyUntil ||= Date.now() + 1500+ return Date.now() < busyUntil ? { status: 503 } : undefined+ })++ await plexo.start(origin.url(), origin.sha256, { connections: 'auto' })+ const state = await plexo.waitForStatus('completed', 40_000)+ expect(origin.log.some((request) => request.status === 503)).toBe(true)+ // Every stream was turned away alike, so none was closed for it, and the count still grew.+ expect(peakStreams(plexo), 'the count grew once the server served').toBeGreaterThan(8)+ expect(state.chunks.length).toBeGreaterThan(8)+ })+ })})diff --git a/src/main/download/concurrency.ts b/src/main/download/concurrency.tsindex cd68e12..bae2c0b 100644--- a/src/main/download/concurrency.ts+++ b/src/main/download/concurrency.ts@@ -1,25 +1,20 @@-// How many streams each network runs, decided while the download runs from what it measures.+// How many streams each network runs. Deliberately simple, after Gopeed and aria2: no timers and+// no speed comparisons, so noise on a flaky network can't move it.//-// Another connection only helps when something limits each connection on its own — a server-// that caps a connection's speed, or TCP on a long or lossy path — and not once the network-// itself is full. Nothing outside says which, so the controller finds out the one way it can:-// it adds streams to a network and keeps them if the download got faster.-//-// - Speeds are only compared once they have settled: a measurement is taken window after window-// until no network is still speeding up. New connections ramp up, and a speed caught halfway-// would make whatever comes next look like a gain.-// - One network at a time, doubling its streams (4 → 8 → 16): the answer in a couple of steps.-// - A step is kept when it made the download faster by at least `minGain` of that network's-// speed before it: the network's own gain, less what the other networks lost. Their loss only-// counts once it is more than `minGain` of their own speed — that much is beyond their-// everyday noise — so a slow link's real gain isn't lost in a fast one's ups and downs, while-// streams that only take speed from a network behind the same uplink still gain nothing.-// - A step that doesn't pay is undone and that network grows no further. So is one whose new-// streams are turned away: failing before they receive anything is how a server says it wants-// fewer connections.-// - Networks join and leave while a download runs. Speeds measured before say nothing about-// the networks there are now, so a step under way is then undone unjudged and measuring starts-// over; a network that joins (or comes back) is new to it, whatever was found out before.+// - A network starts with START_STREAMS_PER_NETWORK (see reconcile in downloadManager.ts).+// - Once every one of its streams has received data, it doubles, up to the limit. Another+// connection costs little when the link is already full, and a server that caps each+// connection's speed is only outrun by more of them.+// - A server that refuses some of a network's streams (503, 429, 403) while it is sending data+// down others is saying it wants fewer connections from this address: the refused streams+// close, and the network never goes above the ones left again for this download. Only the+// streams refused count against it: one that failed some other way, or hasn't been answered+// yet, isn't a refusal. As in Gopeed, where a refused connection stops and the others carry on.+// - A server that refuses while sending nothing isn't limiting connections: it is busy, or the+// link has expired or been denied. That is waited out or reported elsewhere (see finishFailed+// in downloadManager.ts), and the count stays. The network doesn't grow meanwhile.+// - Each network is decided on its own, so one that drops out or comes back never disturbs the+// others.//// No I/O and no clock of its own: it reads a snapshot and says what to do, so every rule can be// checked against exact situations.@@ -28,217 +23,50 @@ export interface NetworkSnapshot {id: string/** Its streams, not counting any retiring. */streams: number- /** Everything its streams have received, ever. */- received: number- /** Its streams that have failed without ever receiving anything. */- rejected: number+ /** Its streams that have received data. */+ answered: number+ /** Its streams the server refused a request from since the last snapshot. */+ refused: number+ /** Its streams that received data since the last snapshot. */+ served: number}export interface Snapshot {- now: number/** The networks in use: the ones it may grow. */networks: readonly NetworkSnapshot[]/** How many more streams the waiting blocks could keep busy. */spareWork: number- /** Streams on their way out after a step that didn't pay. */- retiring: number}export type Action = { kind: 'add' | 'retire'; networkId: string; count: number }-export interface ConcurrencyPolicy {- maxPerNetwork: number- /** How long each measurement window runs. */- windowMs: number- /** How long streams get to connect before they're measured at all. */- warmupMs: number- /** The least a step must gain, as a share of its network's speed before it. */- minGain: number- /** Windows a measurement may take to settle before it is taken as it stands. */- maxWindows: number-}--/** Bytes per second, by network id. */-type Rates = Map<string, number>--/** Measuring window after window until the speeds settle. */-interface Measurement {- /** Nothing is measured before this. */- startAt: number- /** Where the current window began, once it has. */- from: { at: number; received: Map<string, number> } | null- /** The last window's speeds. */- last: Rates | null- windows: number-}--type Phase =- /** Measuring the streams as they are. */- | { kind: 'measure'; measurement: Measurement }- /** Streams were just added to `networkId`; measuring whether they paid. */- | {- kind: 'trial'- networkId: string- added: number- before: Rates- rejectedBefore: number- measurement: Measurement- }- /** No network can grow any more. */- | { kind: 'done' }--const sum = (rates: Rates, include: (id: string) => boolean): number => {- let total = 0- for (const [id, rate] of rates) if (include(id)) total += rate- return total-}-export class ConcurrencyController {- private phase: Phase | null = null- /** Networks that grow no further. */- private readonly settled = new Set<string>()- /** Where the next look for a network to grow starts, so each gets its turn. */- private next = 0- /** The networks the last tick saw. */- private members: Set<string> | null = null-- constructor(private readonly policy: ConcurrencyPolicy) {}-- tick(snapshot: Snapshot): Action | undefined {- const known = this.members- const members = (this.members = new Set(snapshot.networks.map((network) => network.id)))- if (known && (known.size !== members.size || [...members].some((id) => !known.has(id)))) {- return this.regroup(snapshot, known)- }-- const phase = (this.phase ??= { kind: 'measure', measurement: this.measurement(snapshot) })- switch (phase.kind) {- case 'measure': {- // Streams on their way out would count towards a speed that's about to drop.- if (snapshot.retiring > 0) {- phase.measurement = this.measurement(snapshot)- return undefined- }- const rates = this.measure(phase.measurement, snapshot)- return rates && this.grow(snapshot, rates)+ /** The most streams each network may run: lowered for good when its server refuses one. */+ private readonly ceilings = new Map<string, number>()++ constructor(private readonly maxPerNetwork: number) {}++ tick(snapshot: Snapshot): Action[] {+ const actions: Action[] = []+ let spare = snapshot.spareWork+ for (const network of snapshot.networks) {+ const { id, streams } = network+ if (network.refused > 0 && network.served > 0) {+ const ceiling = Math.max(1, streams - network.refused)+ this.ceilings.set(id, Math.min(ceiling, this.ceilings.get(id) ?? Infinity))}-- case 'trial': {- const network = snapshot.networks.find((entry) => entry.id === phase.networkId)- if (!network || network.rejected > phase.rejectedBefore) return this.undo(snapshot, phase)- const after = this.measure(phase.measurement, snapshot)- if (!after) return undefined- const { minGain } = this.policy- const own = phase.networkId- const ownBefore = phase.before.get(own) ?? 0- const othersBefore = sum(phase.before, (id) => id !== own)- const othersLoss = othersBefore - sum(after, (id) => id !== own)- const loss = othersLoss > minGain * othersBefore ? othersLoss : 0- if ((after.get(own) ?? 0) - ownBefore - loss < minGain * ownBefore) {- return this.undo(snapshot, phase)- }- // Kept, and measured settled: that is where the next step starts from.- return this.grow(snapshot, after)+ const ceiling = Math.min(this.maxPerNetwork, this.ceilings.get(id) ?? Infinity)+ if (streams > ceiling) {+ actions.push({ kind: 'retire', networkId: id, count: streams - ceiling })+ continue}-- case 'done':- return undefined+ // Not while the server is turning requests away.+ if (network.refused > 0 || streams === 0 || network.answered < streams) continue+ const count = Math.min(streams, ceiling - streams, spare)+ if (count < 1) continue+ spare -= count+ actions.push({ kind: 'add', networkId: id, count })}- }-- /** The download stopped (a pause): streams whose step was never judged are to go, and- * measuring starts over when it runs again. What was settled stays settled. */- interrupt(): Action | undefined {- const phase = this.phase- this.phase = phase?.kind === 'done' ? phase : null- return phase?.kind === 'trial'- ? { kind: 'retire', networkId: phase.networkId, count: phase.added }- : undefined- }-- /** Networks joined or left: see the top of this file. */- private regroup(snapshot: Snapshot, known: Set<string>): Action | undefined {- for (const { id } of snapshot.networks) if (!known.has(id)) this.settled.delete(id)- const phase = this.phase- this.phase = { kind: 'measure', measurement: this.measurement(snapshot) }- return phase?.kind === 'trial'- ? { kind: 'retire', networkId: phase.networkId, count: phase.added }- : undefined- }-- private measurement(snapshot: Snapshot): Measurement {- return { startAt: snapshot.now + this.policy.warmupMs, from: null, last: null, windows: 0 }- }-- /** Advances a measurement; its speeds once they have settled, or once it has run out of- * windows to settle in. */- private measure(measurement: Measurement, { now, networks }: Snapshot): Rates | undefined {- if (now < measurement.startAt) return undefined- const received = new Map(networks.map((network) => [network.id, network.received]))- const { from } = measurement- if (!from) {- measurement.from = { at: now, received }- return undefined- }- if (now - from.at < this.policy.windowMs) return undefined-- const seconds = (now - from.at) / 1000- const rates: Rates = new Map(- networks.map((network) => [- network.id,- (network.received - (from.received.get(network.id) ?? 0)) / seconds- ])- )- const { last } = measurement- measurement.from = { at: now, received }- measurement.last = rates- measurement.windows++- // Settled once no network is faster than in the last window by half what a step must gain:- // any more, and what's still to come of a ramp could pass for a step's gain.- const settled =- last !== null &&- [...rates].every(([id, rate]) => rate <= (last.get(id) ?? 0) * (1 + this.policy.minGain / 2))- return settled || measurement.windows >= this.policy.maxWindows ? rates : undefined- }-- /** Adds streams to the next network that can take them, or measures again if none can yet. */- private grow(snapshot: Snapshot, before: Rates): Action | undefined {- const { networks, spareWork } = snapshot- let canGrowLater = false- for (let i = 0; i < networks.length; i++) {- const index = (this.next + i) % networks.length- const network = networks[index]- if (this.settled.has(network.id) || network.streams >= this.policy.maxPerNetwork) continue- canGrowLater = true- const count = Math.min(- network.streams,- this.policy.maxPerNetwork - network.streams,- spareWork- )- // A network delivering nothing has no speed to improve on, and one already turning- // streams away wants no more of them.- if (count < 1 || network.rejected > 0 || !((before.get(network.id) ?? 0) > 0)) continue-- this.next = (index + 1) % networks.length- this.phase = {- kind: 'trial',- networkId: network.id,- added: count,- before,- rejectedBefore: network.rejected,- measurement: this.measurement(snapshot)- }- return { kind: 'add', networkId: network.id, count }- }- this.phase = canGrowLater- ? { kind: 'measure', measurement: this.measurement(snapshot) }- : { kind: 'done' }- return undefined- }-- private undo(snapshot: Snapshot, phase: Extract<Phase, { kind: 'trial' }>): Action {- this.settled.add(phase.networkId)- this.phase = { kind: 'measure', measurement: this.measurement(snapshot) }- return { kind: 'retire', networkId: phase.networkId, count: phase.added }+ return actions}}diff --git a/src/main/download/downloadManager.ts b/src/main/download/downloadManager.tsindex 17e9af7..8f5d1c5 100644--- a/src/main/download/downloadManager.ts+++ b/src/main/download/downloadManager.ts@@ -17,12 +17,7 @@ import type {} from '../../shared/types'import { testKnobs, testStreamsPerNetwork } from '../testKnobs'import { advanceBlock, retractBlock } from './blockProgress'-import {- ConcurrencyController,- type Action,- type ConcurrencyPolicy,- type Snapshot-} from './concurrency'+import { ConcurrencyController, type Action, type Snapshot } from './concurrency'import { downloadChunk, fetchRange, HttpStatusError, RemoteChangedError } from './chunkDownloader'import { DownloadFile } from './downloadFile'import { compareVersion, type FileVersion } from './fileVersion'@@ -119,6 +114,11 @@ interface ChunkRuntime {/** When the server first answered busy (see HttpStatusError.transient) since this stream last* made progress: a busy server is waited out for SERVER_BUSY_FOR_MS, not MAX_CHUNK_RETRIES. */busySince: number | null+ /** Since the stream count last looked: whether the server turned one of its requests away+ * (403, 429, 503), and whether it received anything. What a connection limit is judged by (see+ * concurrency.ts). */+ refused: boolean+ served: boolean/** Stopped for good, and to leave the list once its worker has (see retireStreams). */retiring: boolean}@@ -165,9 +165,8 @@ interface DownloadRuntime {attempts: Map<number, Attempt[]>/** Hedges started per block index, capped at SCHEDULER_POLICY.maxHedgesPerBlock. */hedgesByBlock: Map<number, number>- /** By network id: everything its streams have received, which is what the stream count is- * judged by (see concurrency.ts), and when it last showed it was alive — received something,- * or came into use. */+ /** By network id: everything its streams have received, and when it last showed it was alive —+ * received something, or came into use. */traffic: Map<string, { received: number; aliveAt: number }>/** By network id: until when the server asked (Retry-After) not to be sent new requests over* it. */@@ -283,14 +282,8 @@ const SILENT_AFTER_MS = testKnobs.silentAfterMsconst MAX_REFRESHES_PER_BLOCK = 2const SCHEDULER_POLICY: SchedulerPolicy = {hedgeAfterMs: testKnobs.hedgeAfterMs,- maxHedgesPerBlock: 2-}-const CONCURRENCY_POLICY: ConcurrencyPolicy = {- maxPerNetwork: MAX_STREAMS_PER_NETWORK,- windowMs: testKnobs.probeWindowMs,- warmupMs: testKnobs.probeWindowMs,- minGain: 0.15,- maxWindows: 4+ maxHedgesPerBlock: 2,+ startupMs: 1000}/** Whether the file can be fetched in parts. Otherwise one request has to carry all of it: one@@ -303,7 +296,7 @@ function splittable(request: StartDownloadRequest): boolean {* and unless a test fixes it. */function concurrencyFor(request: StartDownloadRequest): ConcurrencyController | null {return splittable(request) && testStreamsPerNetwork() === null- ? new ConcurrencyController(CONCURRENCY_POLICY)+ ? new ConcurrencyController(MAX_STREAMS_PER_NETWORK): null}/** How often a stream with nothing to do looks for work again. */@@ -858,8 +851,7 @@ export class DownloadManager {/** The computer woke from sleep. Its sockets are likely dead, though nothing will say so until* a stall watchdog runs out, and every judgement made by the clock (a silent network, a- * crawling connection, a step of the stream-count controller) spans the sleep. All of it starts- * over. */+ * crawling connection) spans the sleep. All of it starts over. */systemResumed(): void {const now = Date.now()for (const runtime of this.runtimes.values()) {@@ -869,7 +861,6 @@ export class DownloadManager {self.warmSince = nowself.slowSince = null}- this.adjustStreams(runtime, runtime.concurrency?.interrupt())this.wake(runtime,() => true,@@ -993,15 +984,16 @@ export class DownloadManager {if (runtime.state.speedBytesPerSec !== speed) this.scheduleUpdate(runtime)this.refreshStuckConnections(runtime, now)this.reconcile(runtime)- this.adjustStreams(runtime, runtime.concurrency?.tick(this.concurrencySnapshot(runtime, now)))+ this.adjustStreams(runtime, runtime.concurrency?.tick(this.concurrencySnapshot(runtime)))+ // Each refusal is counted once.+ for (const self of runtime.chunkRuntimes.values()) {+ self.refused = false+ self.served = false+ }}// The last blocks are in, or the run was stopped: its streams wind down.runtime.stop.abort()await Promise.all(runtime.workers.values())- // Stopped partway through a step: streams it added were never judged, so they don't stay.- if (runtime.state.status !== 'downloading') {- this.adjustStreams(runtime, runtime.concurrency?.interrupt())- }if (runtime.state.status !== 'downloading') {// Paused, errored, or cancelled — nothing left to do right now. An error keeps what it has@@ -1220,11 +1212,13 @@ export class DownloadManager {if (chunks.length > 0) this.scheduleUpdate(runtime)}- private adjustStreams(runtime: DownloadRuntime, action: Action | undefined): void {- if (action?.kind === 'add') {- this.startStreams(runtime, this.addStreams(runtime, action.networkId, action.count))- } else if (action?.kind === 'retire') {- this.retireStreams(runtime, action.networkId, action.count)+ private adjustStreams(runtime: DownloadRuntime, actions: Action[] | undefined): void {+ for (const action of actions ?? []) {+ if (action.kind === 'add') {+ this.startStreams(runtime, this.addStreams(runtime, action.networkId, action.count))+ } else {+ this.retireStreams(runtime, action.networkId, action.count)+ }}}@@ -1503,6 +1497,7 @@ export class DownloadManager {): void {const now = Date.now()self.receivedBytes += deltaBytes+ self.served = trueconst traffic = this.traffic(runtime, chunk.interfaceId)traffic.received += deltaBytestraffic.aliveAt = now@@ -1719,6 +1714,11 @@ export class DownloadManager {// that answered busy is waited out for SERVER_BUSY_FOR_MS as well; then this stream gives up,// and once all its network's have, so does the network.const now = Date.now()+ // The server turning a request away: the stream count takes it as a limit (see+ // concurrency.ts).+ if (error instanceof HttpStatusError && [403, 429, 503].includes(error.status)) {+ self.refused = true+ }const busy = error instanceof HttpStatusError && error.transientif (busy) self.busySince ??= nowconst waitingOut = busy && now - (self.busySince ?? now) < SERVER_BUSY_FOR_MS@@ -1778,23 +1778,25 @@ export class DownloadManager {this.goIdle(chunk)}- private concurrencySnapshot(runtime: DownloadRuntime, now: number): Snapshot {+ private concurrencySnapshot(runtime: DownloadRuntime): Snapshot {const inUse = runtime.state.networks.filter((network) => network.status === 'on')const networks = inUse.map(({ id }) => {let streams = 0- let rejected = 0+ let answered = 0+ let refused = 0+ let served = 0for (const chunk of this.liveStreams(runtime, id)) {const self = runtime.chunkRuntimes.get(chunk.id)streams++- if (self && self.failures > 0 && self.receivedBytes === 0) rejected+++ if (self && self.receivedBytes > 0) answered+++ if (self?.refused) refused+++ if (self?.served) served++}- return { id, streams, rejected, received: this.traffic(runtime, id).received }+ return { id, streams, answered, refused, served }})- let retiring = 0- for (const self of runtime.chunkRuntimes.values()) if (self.retiring) retiring+++ // A new stream takes a waiting block the moment it starts.const waiting = runtime.blocks.filter((block) => block.status === 'pending').length- // Two waiting blocks per new stream, so it neither idles nor strands a slow network's share.- return { now, networks, spareWork: Math.floor(waiting / 2), retiring }+ return { networks, spareWork: waiting }}/** New streams on `networkId`, put on the download's list for the caller to start. */@@ -1811,11 +1813,20 @@ export class DownloadManager {return added}- /** Stops the newest `count` streams on `networkId` for good. Stopping partway through a block+ /** Stops `count` streams on `networkId` for good. Stopping partway through a block* costs nothing: what it wrote stays, and whoever takes the block next resumes from there. */private retireStreams(runtime: DownloadRuntime, networkId: string, count: number): void {if (count < 1) return- for (const chunk of this.liveStreams(runtime, networkId).slice(-count)) {+ // Streams the server just refused go first (see concurrency.ts), then ones whose last+ // request failed, then the newest.+ const rank = (chunk: ChunkState): number => {+ const self = runtime.chunkRuntimes.get(chunk.id)+ return self?.refused ? 2 : (self?.failures ?? 0) > 0 ? 1 : 0+ }+ const order = this.liveStreams(runtime, networkId).sort(+ (a, b) => rank(b) - rank(a) || b.id - a.id+ )+ for (const chunk of order.slice(0, count)) {const self = runtime.chunkRuntimes.get(chunk.id)if (self && runtime.workers.has(chunk.id)) {// Its worker takes it off the list once it has stopped.@@ -1856,6 +1867,8 @@ export class DownloadManager {failures: 0,strikes: 0,busySince: null,+ refused: false,+ served: false,retiring: false}runtime.chunkRuntimes.set(chunk.id, self)@@ -1884,7 +1897,11 @@ export class DownloadManager {avoid: runtime.avoidNetworkByBlock,hedgesUsed: runtime.hedgesByBlock},- { id: chunk.id, networkId: chunk.interfaceId },+ {+ id: chunk.id,+ networkId: chunk.interfaceId,+ speedBytesPerSec: self.lastBlockSpeed || undefined+ },Date.now(),SCHEDULER_POLICY)diff --git a/src/main/download/plan.ts b/src/main/download/plan.tsindex 8aa2fbc..39c1cbe 100644--- a/src/main/download/plan.ts+++ b/src/main/download/plan.ts@@ -16,13 +16,13 @@ export const DEFAULT_MAX_BLOCK_BYTES = 8 * MIB* megabyte the round trips cost more than splitting the file saves. */export const MIN_BLOCK_BYTES = MIB-/** Streams each network starts with. Servers are built for browsers, which open about six- * connections to a host, so four troubles none of them. */-export const START_STREAMS_PER_NETWORK = 4+/** Streams each network starts with, as download managers do (IDM, XDM: 8). From there it+ * doubles once they are all receiving (see concurrency.ts). */+export const START_STREAMS_PER_NETWORK = 8-/** The most streams a network can grow to. A safety net for noisy speed readings rather than- * the real limit, which is whether another stream still helps (see concurrency.ts). */-export const MAX_STREAMS_PER_NETWORK = 16+/** The most streams a network can grow to (IDM's limit). Enough for a far server, where TCP holds+ * each connection to a megabyte or so a second; a server that wants fewer says so by refusing. */+export const MAX_STREAMS_PER_NETWORK = 32/** Blocks planned per stream a network could grow to. Streams pull blocks as they free up, so a* fast network takes more of them — but only if there are more blocks than streams. With onediff --git a/src/main/download/scheduler.ts b/src/main/download/scheduler.tsindex aa17c6c..55b5d7e 100644--- a/src/main/download/scheduler.ts+++ b/src/main/download/scheduler.ts@@ -10,7 +10,8 @@ import type { BlockState, ChunkState } from '../../shared/types'// no block is left waiting, so it can never take bandwidth from work that still needs// doing: what a hedge costs is bytes fetched twice at the very end, and what it buys is// that one slow connection can no longer hold the whole download back. Whichever attempt-// finishes first wins; the other is dropped.+// finishes first wins; the others are dropped. It is handed out as soon as it would+// clearly finish first, and a block whose hedge is stuck too can be raced again.export interface AttemptView {kind: 'primary' | 'hedge'@@ -32,16 +33,22 @@ export interface SchedulerState {}export interface SchedulerPolicy {- /** A block is hedged once its holder has been at it this long and still needs at least as long- * again. Long enough to have measured a real speed, and far more than a new connection costs. */+ /** A block is only raced once every attempt at it has been going this long: long enough to+ * have measured a real speed. */hedgeAfterMs: number/** Bounds the duplicate work on a block whose hedges keep failing. */maxHedgesPerBlock: number+ /** What a new request costs before its first byte arrives (connecting, the server answering),+ * counted into how long a hedge would take. */+ startupMs: number}export interface Requester {id: numbernetworkId: string+ /** How fast this stream fetched its last block, if it has finished one. Without it, a block is+ * only raced once its attempts need hedgeAfterMs more. */+ speedBytesPerSec?: number}export interface Work {@@ -66,7 +73,8 @@ function nextWaitingBlock(state: SchedulerState, networkId: string): BlockState)}-/** The block most worth a second attempt: the one whose holder will be longest yet. */+/** The block most worth another attempt: the one that will be longest yet, if the requester+ * would clearly beat it. A block is as slow as its fastest attempt. */function nextHedgeTarget(state: SchedulerState,who: Requester,@@ -76,38 +84,50 @@ function nextHedgeTarget(// Only when everything left is already being fetched.if (state.blocks.some((block) => block.status === 'pending')) return undefined+ const speedOf = (streamId: number): number =>+ state.streams.find((stream) => stream.id === streamId)?.speedBytesPerSec ?? 0+let target: BlockState | undefinedlet latest = 0for (const [index, attempts] of state.attempts) {const block = state.blocks[index]if (block?.index !== index || block.rangeEnd === null || block.status !== 'downloading')continue- // Lone primary only: one hedge at a time, and nothing to race if the holder let go.- if (attempts.length !== 1 || attempts[0].kind !== 'primary') continue- const holder = attempts[0]-- if (holder.streamId === who.id) continue+ // Nothing to race if every attempt let go: the block goes back to the queue instead.+ if (attempts.length === 0 || !attempts.some((attempt) => attempt.kind === 'primary')) continue+ if (attempts.some((attempt) => attempt.streamId === who.id)) continueif ((state.hedgesUsed.get(index) ?? 0) >= policy.maxHedgesPerBlock) continue// A network that got nothing from this block won't do better on a second try.if (state.avoid.get(index) === who.networkId) continue- if (now - holder.startedAt < policy.hedgeAfterMs) continue+ // Speeds only mean something once every attempt has been going a while.+ if (attempts.some((attempt) => now - attempt.startedAt < policy.hedgeAfterMs)) continueconst remaining = block.rangeEnd - block.rangeStart + 1 - block.bytesDownloadedif (remaining <= 0) continue- const speed = state.streams.find((stream) => stream.id === holder.streamId)?.speedBytesPerSec// A holder that has gone quiet has no finish time at all.- const eta = speed ? (remaining / speed) * 1000 : Infinity- if (eta < policy.hedgeAfterMs) continue+ const eta = Math.min(+ ...attempts.map((attempt) => {+ const speed = speedOf(attempt.streamId)+ return speed ? (remaining / speed) * 1000 : Infinity+ })+ )+ // Worth it only if the requester would finish in under half that time. One whose speed isn't+ // known yet races a block that needs at least hedgeAfterMs more.+ const worthIt = who.speedBytesPerSec+ ? eta >= 2 * (policy.startupMs + (remaining / who.speedBytesPerSec) * 1000)+ : eta >= policy.hedgeAfterMs+ if (!worthIt) continue- // The holder's network may be what is slow, so a stream on another network gets the first+ // The holders' network may be what is slow, so a stream on another network gets the first// go — unless none of those would take it either.+ const holders = new Set(attempts.map((attempt) => attempt.networkId))const otherNetworkFree = state.streams.some((stream) =>- stream.interfaceId !== holder.networkId &&+ !holders.has(stream.interfaceId) &&stream.status === 'pending' &&state.avoid.get(index) !== stream.interfaceId)- if (holder.networkId === who.networkId && otherNetworkFree) continue+ if (holders.has(who.networkId) && otherNetworkFree) continueif (eta > latest) {target = blockdiff --git a/src/main/testKnobs.ts b/src/main/testKnobs.tsindex 1ba61b2..33f28ce 100644--- a/src/main/testKnobs.ts+++ b/src/main/testKnobs.ts@@ -25,9 +25,7 @@ export const testKnobs = {slowWarmupMs: positiveNumber('PLEXO_E2E_SLOW_WARMUP_MS', 5_000),slowForMs: positiveNumber('PLEXO_E2E_SLOW_FOR_MS', 10_000),silentAfterMs: positiveNumber('PLEXO_E2E_SILENT_MS', 5_000),- hedgeAfterMs: positiveNumber('PLEXO_E2E_HEDGE_MS', 5_000),- /** How long each measurement of the stream-count controller runs (see concurrency.ts). */- probeWindowMs: positiveNumber('PLEXO_E2E_PROBE_MS', 2_000),+ hedgeAfterMs: positiveNumber('PLEXO_E2E_HEDGE_MS', 2_000),/** Skips the real GitHub check and pretends this version is available, for exercising the* update banner without needing an actual newer release published. */forceUpdateVersion: env['PLEXO_FORCE_UPDATE_VERSION']