Skip to content

armgit / plexo

Public Imported from GitHub
0

Speed up downloads by combining multiple network connections in parallel

Resume on a network that's still there; README and Auto hint match the new stream count (#52)

Anmol Kapil49fde70bac51
diff --git a/README.md b/README.md
index 5830059..5874f8a 100644
--- a/README.md
+++ b/README.md
@@ -142,7 +142,7 @@ Instead, Plexo uses a **dynamic work-stealing queue**:
1. The file is split into **chunks of up to 8 MB** (smaller for small files, so every network gets a share).
2. All chunks enter a centralized pending queue.
3. A pool of worker connections continuously lease the next chunk from the queue as soon as they become free. Each keeps its one connection to the server from chunk to chunk, so it pays for the handshake and TCP's slow start once, not per chunk. Connections start interleaved across networks, so each network is served before any is served twice.
- - **How many.** Each network starts with 4. More help only when something limits each connection on its own (a server capping per-connection speed, or a long, lossy route), not once the network itself is full, so Plexo measures instead of guessing. Once speeds have settled, it doubles one network's connections, waits for them to settle again, and keeps them only if that network got at least 15% faster, net of anything the other networks lost beyond their usual ups and downs (two networks behind one router share its uplink). Otherwise the new connections close, and their chunks go back to the queue to carry on from where they got to. So do extra connections the server turns away. Up to 16 per network, and never more than there are chunks for.
+ - **How many.** Each network starts with 8 connections, and once every one of them is receiving it doubles, up to 32, never more than there are chunks for. Another connection costs little when the link is already full, and a server that caps each connection's speed is only outrun by more of them. A server that refuses some of a network's connections (503, 429, 403) while still sending data down the others is asking for fewer: the refused ones close, their chunks go back to the queue to carry on from where they got to, and that network stays at the ones left. After a minute without another refusal it may have one more, and so on back up. A server that refuses everything (busy, or an expired link) isn't limiting connections, so the count stays. On the start screen you can pick a fixed 4, 8, 16 or 32 per network instead of Auto.
4. Faster interfaces finish chunks quicker and immediately pick up new ones; slower interfaces pull fewer chunks.
5. **Racing the tail.** Once no chunk is left waiting, a free connection can start a second attempt at a chunk another connection is fetching too slowly (as soon as the free connection would finish it in under half the time the chunk still needs; a chunk whose second attempt is stuck too can get one more), picking up from where the first had got to. Whichever finishes first wins and the other is dropped. It costs a few bytes fetched twice at the very end, and it means one slow connection — or one slow network — can no longer hold the whole download back. A stream doing this is marked **BACKUP** in the streams table.
diff --git a/e2e/liveNetworks.spec.ts b/e2e/liveNetworks.spec.ts
index 5ac5aa7..7fee177 100644
--- a/e2e/liveNetworks.spec.ts
+++ b/e2e/liveNetworks.spec.ts
@@ -114,6 +114,56 @@ test.describe('two networks @smoke', () => {
const resumed = await plexo.waitUntil((state) => state.status === 'downloading')
expect(network(resumed, 'a')?.enabled).toBe(true)
})
+
+ test('switching the last network off and on quickly ends up where it was left', async ({
+ plexo,
+ serve
+ }) => {
+ const origin = await serve({ size: 128 * BLOCK, bytesPerSecond: 256 * 1024 })
+ const id = await plexo.start(origin.url(), origin.sha256, { connections: 2 })
+ await plexo.waitUntil((state) => state.bytesDownloaded > 0)
+
+ // Back on before the pause has wound down: it resumes.
+ const off = plexo.api.setDownloadNetwork(id, 'a', false)
+ await plexo.api.setDownloadNetwork(id, 'a', true)
+ await off
+ await plexo.waitUntil((state) => state.status === 'downloading' && streamsOn(state, 'a') > 0)
+
+ // Off, on and off again before the resume has got going: it stays paused.
+ await plexo.api.setDownloadNetwork(id, 'a', false)
+ await plexo.waitUntil((state) => state.status === 'paused')
+ await plexo.api.setDownloadNetwork(id, 'a', true)
+ await plexo.api.setDownloadNetwork(id, 'a', false)
+ await settle(1000)
+ const left = (await plexo.current())!
+ expect(left.status).toBe('paused')
+ expect(network(left, 'a')?.enabled).toBe(false)
+
+ // …and switching it on once more still resumes it.
+ await plexo.api.setDownloadNetwork(id, 'a', true)
+ await plexo.waitForStatus('completed')
+ })
+
+ test('resumed with none on, it picks a network that is still there', async ({ plexo, serve }) => {
+ const origin = await serve({ size: 128 * BLOCK, bytesPerSecond: 256 * 1024 })
+ const id = await plexo.start(origin.url(), origin.sha256, {
+ networks: ['a', 'b'],
+ connections: 2
+ })
+ await plexo.waitUntil((state) => state.bytesDownloaded > 0)
+
+ await plexo.api.setDownloadNetwork(id, 'a', false)
+ await plexo.api.setDownloadNetwork(id, 'b', false)
+ await plexo.waitUntil((state) => state.status === 'paused')
+ // b, the one switched off last, is unplugged while paused.
+ await setNetworks(plexo, { a: NETWORKS['a'] })
+
+ await plexo.api.resumeDownload(id)
+ const resumed = await plexo.waitUntil((state) => state.status === 'downloading')
+ expect(network(resumed, 'a')?.enabled).toBe(true)
+ expect(network(resumed, 'b')?.enabled).toBe(false)
+ await plexo.waitForStatus('completed')
+ })
})
test.describe('a network that can’t reach the server @smoke', () => {
diff --git a/src/main/download/downloadManager.ts b/src/main/download/downloadManager.ts
index d3e912e..aec0fd7 100644
--- a/src/main/download/downloadManager.ts
+++ b/src/main/download/downloadManager.ts
@@ -758,14 +758,15 @@ export class DownloadManager {
const { status, resumable } = runtime.state
if (status !== 'paused' && !(status === 'error' && resumable !== false)) return
- void this.resumeAfterVerifying(runtime)
+ void this.resumeAfterVerifying(runtime, false)
}
// A file that changed on the server while this download was paused is caught by the first
// chunk request after resuming: its response is checked against the version the download
// started on (see runWorker), which can tell a real change from a relabelled server. Whether a
// network is there to resume on is the run's business: with none, it waits for one.
- private async resumeAfterVerifying(runtime: DownloadRuntime): Promise<void> {
+ // `byNetwork`: resumed by switching a network back on, rather than by Resume.
+ private async resumeAfterVerifying(runtime: DownloadRuntime, byNetwork: boolean): Promise<void> {
// The paused run can still be winding down: a writer closing, a sample check in flight. A new
// one must not start beside it — both would go on to publish, and act on each other's streams.
await runtime.runPromise
@@ -779,18 +780,22 @@ export class DownloadManager {
return
}
if (runtime.state.status !== 'paused' && runtime.state.status !== 'error') return
+ const { networks } = runtime.state
+ // Switched back on, then off again while the paused run wound down: it stays paused.
+ if (byNetwork && !networks.some((network) => network.enabled)) return
runtime.state.status = 'downloading'
runtime.state.error = undefined
runtime.state.resumable = undefined
runtime.pausedForNoNetwork = undefined
// Paused by switching off every network: resuming switches back on the one switched off
- // last, or, after a restart, the first one present.
- const { networks } = runtime.state
+ // last if it is still there, or else the first one present.
if (!networks.some((network) => network.enabled)) {
+ const present = (network: { id: string }): boolean => !!this.networks.find(network.id)
const again =
+ networks.find((network) => network.id === runtime.lastSwitchedOff && present(network)) ??
+ networks.find(present) ??
networks.find((network) => network.id === runtime.lastSwitchedOff) ??
- networks.find((network) => this.networks.find(network.id)) ??
networks[0]
if (again) again.enabled = true
}
@@ -843,11 +848,16 @@ export class DownloadManager {
if (wasLast && status === 'downloading') {
runtime.pausedForNoNetwork = true
await this.pause(id)
+ // Not paused after all (it was already publishing): nothing for a network to resume.
+ if (runtime.state.status !== 'paused') runtime.pausedForNoNetwork = undefined
return
}
this.reconcile(runtime)
this.pushUpdate(runtime)
- if (enabled && status === 'paused' && runtime.pausedForNoNetwork) this.resume(id)
+ // Switched back on while the pause is still winding down, it resumes once that is done.
+ if (enabled && status === 'paused' && runtime.pausedForNoNetwork) {
+ void this.resumeAfterVerifying(runtime, true)
+ }
}
/** The computer's networks changed (see NetworkMonitor). A network whose addresses changed
diff --git a/src/renderer/src/screens/IdleScreen.tsx b/src/renderer/src/screens/IdleScreen.tsx
index 9bfce14..ea66fb8 100644
--- a/src/renderer/src/screens/IdleScreen.tsx
+++ b/src/renderer/src/screens/IdleScreen.tsx
@@ -350,7 +350,7 @@ export function IdleScreen(): React.JSX.Element {
{isSingleStreamOnly
? '1 stream'
: streamsChoice === 'auto'
- ? 'per network, as many as help'
+ ? 'per network, 8 up to 32'
: 'per network'}
</div>
</div>