Speed up downloads by combining multiple network connections in parallel
TypeScript bandwidth-aggregationcross-platformdesktop-appdownloaderdownloadmanagerlinuxmacosmulticonnectionparallel-downloadswindows
Close streams a server leaves unanswered (#53)
Anmol Kapil8dae93ff852a
diff --git a/README.md b/README.mdindex 5874f8a..cd29526 100644--- a/README.md+++ b/README.md@@ -67,7 +67,7 @@ File ──→ Split ─────┤ ├─## Features-- 🚀 **Multi-interface, multi-connection downloads** — splits files into chunks of up to 8 MB and fans them out across worker connections bound to specific network interfaces, each kept open from one chunk to the next. Each interface starts with 8 connections and doubles once they are all receiving, up to 32. If the server refuses some (503, 429, 403), that interface drops to the ones it accepted, then gets one more back each minute without a refusal; a server refusing everything (busy, or an expired link) doesn't lower it. You can also pick a fixed 4, 8, 16 or 32 per interface on the start screen instead of Auto.+- 🚀 **Multi-interface, multi-connection downloads** — splits files into chunks of up to 8 MB and fans them out across worker connections bound to specific network interfaces, each kept open from one chunk to the next. Each interface starts with 8 connections and doubles once they are all receiving, up to 32. If the server refuses some (503, 429, 403, or leaves them unanswered), that interface drops to the ones it accepted, then gets one more back each minute without a refusal; a server refusing everything (busy, or an expired link) doesn't lower it. You can also pick a fixed 4, 8, 16 or 32 per interface on the start screen instead of Auto.- 🔌 **Hardware interface detection** — queries Windows adapters via PowerShell `Get-NetAdapter` and macOS hardware ports via `networksetup` so Wi-Fi, Ethernet, tethered iPhones, and Thunderbolt bridges are labeled by real device names instead of bare BSD names (`en0`, `en6`).- ⚖️ **Dynamic work-stealing queue** — chunks are leased from a shared pending queue; faster networks pull more chunks instead of waiting for slower connections to finish.- ⏸️ **Resumable downloads** — cleanly pause and resume downloads with progress saved in a destination-side staging file.@@ -142,7 +142,7 @@ Instead, Plexo uses a **dynamic work-stealing queue**:1. The file is split into **chunks of up to 8 MB** (smaller for small files, so every network gets a share).2. All chunks enter a centralized pending queue.3. A pool of worker connections continuously lease the next chunk from the queue as soon as they become free. Each keeps its one connection to the server from chunk to chunk, so it pays for the handshake and TCP's slow start once, not per chunk. Connections start interleaved across networks, so each network is served before any is served twice.- - **How many.** Each network starts with 8 connections, and once every one of them is receiving it doubles, up to 32, never more than there are chunks for. Another connection costs little when the link is already full, and a server that caps each connection's speed is only outrun by more of them. A server that refuses some of a network's connections (503, 429, 403) while still sending data down the others is asking for fewer: the refused ones close, their chunks go back to the queue to carry on from where they got to, and that network stays at the ones left. After a minute without another refusal it may have one more, and so on back up. A server that refuses everything (busy, or an expired link) isn't limiting connections, so the count stays. On the start screen you can pick a fixed 4, 8, 16 or 32 per network instead of Auto.+ - **How many.** Each network starts with 8 connections, and once every one of them is receiving it doubles, up to 32, never more than there are chunks for. Another connection costs little when the link is already full, and a server that caps each connection's speed is only outrun by more of them. A server that refuses some of a network's connections (503, 429, 403, or leaves them unanswered for 5 seconds) while still sending data down the others is asking for fewer: the refused ones close, their chunks go back to the queue to carry on from where they got to, and that network stays at the ones left. After a minute without another refusal it may have one more, and so on back up. A server that refuses everything (busy, or an expired link) isn't limiting connections, so the count stays. On the start screen you can pick a fixed 4, 8, 16 or 32 per network instead of Auto.4. Faster interfaces finish chunks quicker and immediately pick up new ones; slower interfaces pull fewer chunks.5. **Racing the tail.** Once no chunk is left waiting, a free connection can start a second attempt at a chunk another connection is fetching too slowly (as soon as the free connection would finish it in under half the time the chunk still needs; a chunk whose second attempt is stuck too can get one more), picking up from where the first had got to. Whichever finishes first wins and the other is dropped. It costs a few bytes fetched twice at the very end, and it means one slow connection — or one slow network — can no longer hold the whole download back. A stream doing this is marked **BACKUP** in the streams table.diff --git a/e2e/disk.spec.ts b/e2e/disk.spec.tsindex a40934c..fc88fa3 100644--- a/e2e/disk.spec.ts+++ b/e2e/disk.spec.ts@@ -83,9 +83,10 @@ test.describe('disk space @disk', () => {test('a file that fits once but not twice completes on the destination volume', async ({serve}) => {- const volume = await smallVolume(12)+ const volume = await smallVolume(40)test.skip(!volume, 'cannot create a small volume on this machine')// Both metadata and destination are on the small volume; the payload must occupy only one copy.+ // Room is left for Chromium's own caches in userData, which take several MB at launch.const app = new PlexoApp({userData: join(volume!.path, 'userData'),dest: join(volume!.path, 'dest')@@ -94,7 +95,7 @@ test.describe('disk space @disk', () => {await mkdir(app.dirs.userData)await mkdir(app.dirs.dest)await app.launch()- const origin = await serve({ size: 112 * BLOCK }) // 7 MB: fits once, not twice+ const origin = await serve({ size: 384 * BLOCK }) // 24 MB: fits once, not twiceawait app.start(origin.url(), origin.sha256)const done = await app.waitForStatus('completed')expect(await readdir(app.dirs.dest)).toEqual([done.fileName])diff --git a/e2e/streams.spec.ts b/e2e/streams.spec.tsindex 52c22d7..b1cad56 100644--- a/e2e/streams.spec.ts+++ b/e2e/streams.spec.ts@@ -29,6 +29,32 @@ test.describe('automatic stream count', () => {expect(origin.log.some((request) => request.status === 503)).toBe(true)})+ test.describe('left unanswered', () => {+ // As in the app, a request with no answer is noticed as silent before it times out.+ test.use({ appEnv: { PLEXO_E2E_STALL_MS: '20000' } })++ test('a server that leaves extra connections unanswered keeps the ones it answers', async ({+ plexo,+ serve+ }) => {+ const origin = await serve({ size: 256 * BLOCK, bytesPerSecond: 256 * 1024 })+ const accepted = new Set<number>()+ origin.setRule(({ connection, range }) => {+ if (range?.start === 0 && range.end === 0) return undefined // the probe+ if (accepted.has(connection) || accepted.size < 4) {+ accepted.add(connection)+ return undefined+ }+ return 'stallHeaders'+ })++ await plexo.start(origin.url(), origin.sha256, { connections: 'auto' })+ const state = await plexo.waitForStatus('completed', 40_000)+ expect(peakStreams(plexo), 'more streams were tried').toBe(8)+ expect(state.chunks).toHaveLength(4)+ })+ })+test.describe('waited out', () => {test.use({ appEnv: { PLEXO_E2E_SERVER_BUSY_MS: '60000' } })diff --git a/src/main/download/concurrency.ts b/src/main/download/concurrency.tsindex 228c1bd..ab8c15c 100644--- a/src/main/download/concurrency.ts+++ b/src/main/download/concurrency.ts@@ -6,14 +6,15 @@// - Once every one of its streams has received data, it doubles, up to the limit. Another// connection costs little when the link is already full, and a server that caps each// connection's speed is only outrun by more of them.-// - A server that refuses some of a network's streams (503, 429, 403) while it is sending data-// down others is saying it wants fewer connections from this address: the refused streams-// close, and the network stays at the ones left. After RECOVER_MS without another refusal it-// may have one more, and so on back up to the limit, as in Surge: a server that was only-// briefly unwell doesn't cap the rest of a long download, and one that meant it costs one-// refused request a minute. Only the-// streams refused count against it: one that failed some other way, or hasn't been answered-// yet, isn't a refusal. As in Gopeed, where a refused connection stops and the others carry on.+// - A server that refuses some of a network's streams (503, 429, 403, or leaves them unanswered+// for SILENT_AFTER_MS, see downloadManager.ts) while it is sending data down others is saying+// it wants fewer connections from this address: the refused streams close, and the network+// stays at the ones left. After RECOVER_MS without another refusal it may have one more, and+// so on back up to the limit, as in Surge: a server that was only briefly unwell doesn't cap+// the rest of a long download, and one that meant it costs one refused request a minute. Only+// the streams refused count against it: one that failed some other way, or hasn't been+// answered yet, isn't a refusal. As in Gopeed, where a refused connection stops and the others+// carry on.// - A server that refuses while sending nothing isn't limiting connections: it is busy, or the// link has expired or been denied. That is waited out or reported elsewhere (see finishFailed// in downloadManager.ts), and the count stays. The network doesn't grow meanwhile.diff --git a/src/main/download/downloadManager.ts b/src/main/download/downloadManager.tsindex aec0fd7..b65ac7c 100644--- a/src/main/download/downloadManager.ts+++ b/src/main/download/downloadManager.ts@@ -115,8 +115,8 @@ interface ChunkRuntime {* made progress: a busy server is waited out for SERVER_BUSY_FOR_MS, not MAX_CHUNK_RETRIES. */busySince: number | null/** Since the stream count last looked: whether the server turned one of its requests away- * (403, 429, 503), and whether it received anything. What a connection limit is judged by (see- * concurrency.ts). */+ * (403, 429, 503, or left it unanswered), and whether it received anything. What a connection+ * limit is judged by (see concurrency.ts). */refused: booleanserved: boolean/** Stopped for good, and to leave the list once its worker has (see retireStreams). */@@ -1299,6 +1299,9 @@ export class DownloadManager {if (silent && this.markUnreachable(runtime, this.network(runtime, attempt.networkId), now)) {unreachable = true}+ // Silent while its network's other connections are served: the server took the connection+ // and left it waiting, which is how some turn away one too many (see concurrency.ts).+ if (silent) self.refused = trueconst index = attempt.block.indexconst refreshes = runtime.refreshesByBlock.get(index) ?? 0