Skip to content

armgit / plexo

Public Imported from GitHub
0

Speed up downloads by combining multiple network connections in parallel

feat: decide each network's stream count while downloading

Anmol Kapila32314947094
diff --git a/README.md b/README.md
index 2a78504..e86531f 100644
--- a/README.md
+++ b/README.md
@@ -67,7 +67,7 @@ File ──→ Split ─────┤ ├─
## Features
-- 🚀 **Multi-interface, multi-connection downloads** — splits files into chunks of up to 8 MB and fans them out across worker connections bound to specific network interfaces (up to 8 parallel connections per interface, 32 total).
+- 🚀 **Multi-interface, multi-connection downloads** — splits files into chunks of up to 8 MB and fans them out across worker connections bound to specific network interfaces, each kept open from one chunk to the next. How many connections each interface gets is worked out while downloading: as many as keep making it faster, up to 16.
- 🔌 **Hardware interface detection** — queries Windows adapters via PowerShell `Get-NetAdapter` and macOS hardware ports via `networksetup` so Wi-Fi, Ethernet, tethered iPhones, and Thunderbolt bridges are labeled by real device names instead of bare BSD names (`en0`, `en6`).
- ⚖️ **Dynamic work-stealing queue** — chunks are leased from a shared pending queue; faster networks pull more chunks instead of waiting for slower connections to finish.
- ⏸️ **Resumable downloads** — cleanly pause and resume downloads with progress saved in a destination-side staging file.
@@ -140,7 +140,8 @@ Instead, Plexo uses a **dynamic work-stealing queue**:
1. The file is split into **chunks of up to 8 MB** (smaller for small files, so every network gets a share).
2. All chunks enter a centralized pending queue.
-3. A pool of worker connections (up to 8 per interface, 32 total, never more than there are chunks to work on) continuously lease the next chunk from the queue as soon as they become free. Connections start interleaved across networks, so each network is served before any is served twice.
+3. A pool of worker connections continuously lease the next chunk from the queue as soon as they become free. Each keeps its one connection to the server from chunk to chunk, so it pays for the handshake and TCP's slow start once, not per chunk. Connections start interleaved across networks, so each network is served before any is served twice.
+ - **How many.** Each network starts with 4. More help only when something limits each connection on its own (a server capping per-connection speed, or a long, lossy route), not once the network itself is full, so Plexo measures instead of guessing: it doubles one network's connections and keeps them only if the whole download got at least 15% of that network's speed faster. Otherwise they finish their chunk and close. So do extra connections the server turns away. Up to 16 per network, and never more than there are chunks for.
4. Faster interfaces finish chunks quicker and immediately pick up new ones; slower interfaces pull fewer chunks.
5. **Racing the tail.** Once no chunk is left waiting, a free connection can start a second attempt at a chunk another connection is fetching too slowly (one that still needs as long again as it has already taken, at least 5 seconds), picking up from where the first had got to. Whichever finishes first wins and the other is dropped. It costs a few bytes fetched twice at the very end, and it means one slow connection — or one slow network — can no longer hold the whole download back. A stream doing this is marked **BACKUP** in the streams table.
diff --git a/e2e/concurrency.spec.ts b/e2e/concurrency.spec.ts
new file mode 100644
index 0000000..b39e950
--- /dev/null
+++ b/e2e/concurrency.spec.ts
@@ -0,0 +1,168 @@
+import { expect, test } from '@playwright/test'
+import {
+ ConcurrencyController,
+ type Action,
+ type ConcurrencyPolicy
+} from '../src/main/download/concurrency'
+
+// M. How many streams each network runs. The controller is pure, so it's driven here against
+// models of what limits a download — the server, the link, an uplink two networks share — and
+// judged by where it settles.
+
+const MB = 1e6
+const POLICY: ConcurrencyPolicy = {
+ maxPerNetwork: 16,
+ windowMs: 2000,
+ warmupMs: 2000,
+ minGain: 0.15
+}
+const TICK_MS = 500
+
+/** Bytes per second each network delivers, given how many streams each runs. */
+type Model = (streams: Map<string, number>) => Map<string, number>
+
+interface Run {
+ streams: Map<string, number>
+ /** Most streams each network had at once. */
+ peak: Map<string, number>
+ actions: Action[]
+}
+
+function simulate(
+ model: Model,
+ networks: string[],
+ options: {
+ seconds?: number
+ spareWork?: number
+ failing?: (id: string, streams: number) => boolean
+ } = {}
+): Run {
+ const { seconds = 120, spareWork = 1000, failing = () => false } = options
+ const controller = new ConcurrencyController(POLICY)
+ const streams = new Map(networks.map((id) => [id, 4]))
+ const peak = new Map(streams)
+ const received = new Map(networks.map((id) => [id, 0]))
+ const actions: Action[] = []
+ let retiring = 0
+ for (let now = 0; now <= seconds * 1000; now += TICK_MS) {
+ const rates = model(streams)
+ for (const id of networks)
+ received.set(id, received.get(id)! + (rates.get(id)! * TICK_MS) / 1000)
+ const action = controller.tick({
+ now,
+ networks: networks.map((id) => ({
+ id,
+ streams: streams.get(id)!,
+ received: received.get(id)!,
+ failing: failing(id, streams.get(id)!)
+ })),
+ spareWork,
+ retiring
+ })
+ // Retired streams finish their block and are gone by the next look.
+ retiring = 0
+ if (!action) continue
+ actions.push(action)
+ const change = action.kind === 'add' ? action.count : -action.count
+ streams.set(action.networkId, streams.get(action.networkId)! + change)
+ peak.set(
+ action.networkId,
+ Math.max(peak.get(action.networkId)!, streams.get(action.networkId)!)
+ )
+ if (action.kind === 'retire') retiring = action.count
+ }
+ return { streams, peak, actions }
+}
+
+/** Each connection gets at most `perStream`, and the link at most `link`. */
+const capped =
+ (perStream: number, link: number): Model =>
+ (streams) =>
+ new Map([...streams].map(([id, count]) => [id, Math.min(count * perStream, link)]))
+
+test.describe('stream count', () => {
+ test('a server that caps each connection gets more of them, up to the limit', () => {
+ const { streams, actions } = simulate(capped(1 * MB, 1000 * MB), ['a'])
+ expect(streams.get('a')).toBe(16)
+ // Doubling: two steps, not twelve.
+ expect(actions).toEqual([
+ { kind: 'add', networkId: 'a', count: 4 },
+ { kind: 'add', networkId: 'a', count: 8 }
+ ])
+ })
+
+ test('a full link keeps what it started with: the extra streams are tried, then retired', () => {
+ const { streams, peak } = simulate(capped(10 * MB, 20 * MB), ['a'])
+ expect(peak.get('a')).toBe(8)
+ expect(streams.get('a')).toBe(4)
+ })
+
+ test('grows while it pays and stops where it stops paying', () => {
+ // 4 streams: 4 MB/s. 8: 8 MB/s. 16 would be the 10 MB/s link: +25%, which still pays.
+ expect(simulate(capped(1 * MB, 10 * MB), ['a']).streams.get('a')).toBe(16)
+ // A 9 MB/s link: 16 streams would add 1 MB/s to 8, under the 15% a step has to earn.
+ expect(simulate(capped(1 * MB, 9 * MB), ['a']).streams.get('a')).toBe(8)
+ })
+
+ test('streams that only take speed from a network sharing the same uplink are retired', () => {
+ // Two networks behind one 8 MB/s uplink, each connection good for 1 MB/s.
+ const sharedUplink: Model = (streams) => {
+ const total = [...streams.values()].reduce((sum, count) => sum + count, 0)
+ const delivered = Math.min(total * MB, 8 * MB)
+ return new Map([...streams].map(([id, count]) => [id, (delivered * count) / total]))
+ }
+ const { streams, peak } = simulate(sharedUplink, ['wifi', 'ethernet'])
+ expect(peak.get('wifi')).toBe(8)
+ expect(peak.get('ethernet')).toBe(8)
+ expect(streams).toEqual(
+ new Map([
+ ['wifi', 4],
+ ['ethernet', 4]
+ ])
+ )
+ })
+
+ test("a slow network's real gain counts, however fast the network beside it", () => {
+ const model: Model = (streams) =>
+ new Map([
+ ['ethernet', Math.min(streams.get('ethernet')! * 50 * MB, 100 * MB)],
+ ['cellular', streams.get('cellular')! * 0.25 * MB]
+ ])
+ const { streams } = simulate(model, ['ethernet', 'cellular'])
+ // Cellular's doubling is a 1 MB/s gain against 100 MB/s in total, and it is kept all the same.
+ expect(streams.get('cellular')).toBe(16)
+ expect(streams.get('ethernet')).toBe(4)
+ })
+
+ test('networks take turns', () => {
+ const { actions } = simulate(capped(1 * MB, 1000 * MB), ['a', 'b'])
+ expect(actions.map((action) => action.networkId)).toEqual(['a', 'b', 'a', 'b'])
+ })
+
+ test('new streams that fail are retired at once, and the network grows no further', () => {
+ // The server turns away anything past the first four connections.
+ const { streams, actions } = simulate(capped(1 * MB, 1000 * MB), ['a'], {
+ failing: (_id, count) => count > 4
+ })
+ expect(streams.get('a')).toBe(4)
+ expect(actions).toEqual([
+ { kind: 'add', networkId: 'a', count: 4 },
+ { kind: 'retire', networkId: 'a', count: 4 }
+ ])
+ })
+
+ test('no growth without waiting blocks for the new streams to take', () => {
+ expect(simulate(capped(1 * MB, 1000 * MB), ['a'], { spareWork: 0 }).actions).toEqual([])
+ // Room for two more: the step is cut to fit.
+ expect(simulate(capped(1 * MB, 1000 * MB), ['a'], { spareWork: 2 }).actions[0]).toEqual({
+ kind: 'add',
+ networkId: 'a',
+ count: 2
+ })
+ })
+
+ test('a network delivering nothing is not grown', () => {
+ const { actions } = simulate(() => new Map([['a', 0]]), ['a'])
+ expect(actions).toEqual([])
+ })
+})
diff --git a/e2e/fixtures.ts b/e2e/fixtures.ts
index 2547dff..20c8778 100644
--- a/e2e/fixtures.ts
+++ b/e2e/fixtures.ts
@@ -47,7 +47,8 @@ type Api = {
interface StartOptions {
networks?: string[]
- connections?: number
+ /** Streams per network, fixed so a test can count requests; 'auto' lets the app decide. */
+ connections?: number | 'auto'
fileName?: string
destinationDir?: string
}
@@ -94,6 +95,8 @@ export class PlexoApp {
PLEXO_E2E_STALL_MS: '1500',
// Off unless a test asks for it: a hedge is an extra request, and most tests count them.
PLEXO_E2E_HEDGE_MS: '600000',
+ // Fixed for the same reason, and for downloads started through the UI.
+ PLEXO_E2E_STREAMS: '2',
PLEXO_E2E_INTERFACES: interfacesEnv(NETWORKS),
...this.extraEnv
}
@@ -162,8 +165,20 @@ export class PlexoApp {
return this.electronApp.evaluate(fn as never, arg) as Promise<R>
}
+ /** Sets how many streams the next download runs per network (see StartOptions). */
+ private async pinStreams(connections: number | 'auto' = 2): Promise<void> {
+ await this.evaluateMain(
+ (_electron, value) => {
+ if (value === null) delete process.env.PLEXO_E2E_STREAMS
+ else process.env.PLEXO_E2E_STREAMS = value
+ },
+ connections === 'auto' ? null : String(connections)
+ )
+ }
+
/** Probes and starts `url` exactly the way IdleScreen's Start button does. */
async start(url: string, expectedSha: string, options: StartOptions = {}): Promise<string> {
+ await this.pinStreams(options.connections)
await this.api.listInterfaces()
const probe = await this.api.probeUrl(url)
const multiChunk = probe.supportsRanges && probe.totalBytes !== null
@@ -178,8 +193,6 @@ export class PlexoApp {
totalBytes: probe.totalBytes ?? 0,
supportsRanges: multiChunk,
interfaceIds: multiChunk ? networks : networks.slice(0, 1),
- chunkCount: multiChunk ? networks.length * (options.connections ?? 2) : 1,
- connectionsPerNetwork: multiChunk ? (options.connections ?? 2) : 1,
etag: probe.etag,
lastModified: probe.lastModified
})
@@ -202,8 +215,10 @@ export class PlexoApp {
/** Starts a dev-tool simulated download of a local file (see simDownload.ts). */
async startSimulated(
request: Omit<IpcContract['startSimulatedDownload']['args'][0], 'destinationDir'>,
- expectedSha: string
+ expectedSha: string,
+ options: Pick<StartOptions, 'connections'> = {}
): Promise<string> {
+ await this.pinStreams(options.connections)
const destBefore = await readdir(this.dirs.dest)
const id = await this.api.startSimulatedDownload({ ...request, destinationDir: this.dirs.dest })
this.tracked.set(id, { expectedSha, destBefore, destinationDir: this.dirs.dest })
diff --git a/e2e/origin.ts b/e2e/origin.ts
index efc6946..69557f9 100644
--- a/e2e/origin.ts
+++ b/e2e/origin.ts
@@ -64,6 +64,8 @@ export interface OriginOptions {
contentDisposition?: string
/** Per-response speed limit, so a download lasts long enough to be interrupted. */
bytesPerSecond?: number
+ /** A speed limit every response shares, as on a full link: another connection gets nothing. */
+ sharedBytesPerSecond?: number
}
/** Deterministic bytes, so a failing seed reproduces the exact same file. */
@@ -109,6 +111,8 @@ export class Origin {
private sockets = new Set<Socket>()
private connectionIds = new WeakMap<Socket, number>()
private connections = 0
+ /** When the shared limit next has room (see sharedBytesPerSecond). */
+ private sharedBusyUntil = 0
private server = createServer((req, res) => void this.handle(req, res))
private port = 0
@@ -315,6 +319,12 @@ export class Origin {
if (rate) {
await new Promise((resolve) => setTimeout(resolve, (piece.length / rate) * 1000))
}
+ const shared = this.options.sharedBytesPerSecond
+ if (shared) {
+ const now = Date.now()
+ this.sharedBusyUntil = Math.max(now, this.sharedBusyUntil) + (piece.length / shared) * 1000
+ await new Promise((resolve) => setTimeout(resolve, this.sharedBusyUntil - now))
+ }
}
if (cutAfter !== null) {
diff --git a/e2e/plan.spec.ts b/e2e/plan.spec.ts
index 346e96f..5f153cd 100644
--- a/e2e/plan.spec.ts
+++ b/e2e/plan.spec.ts
@@ -4,11 +4,11 @@ import {
DEFAULT_MAX_BLOCK_BYTES,
interleave,
MAX_BLOCKS,
- MAX_STREAMS,
MAX_STREAMS_PER_NETWORK,
MIN_BLOCK_BYTES,
- planDownload
-} from '../src/shared/plan'
+ planDownload,
+ START_STREAMS_PER_NETWORK
+} from '../src/main/download/plan'
// I. How a download is cut into blocks and streams. Pure, so it's checked over the whole input
// space instead of hand-picked sizes.
@@ -19,7 +19,7 @@ const splittable = fc.record({
totalBytes: fc.integer({ min: 1, max: 2 ** 42 }),
splittable: fc.constant(true),
networkCount: fc.integer({ min: 1, max: 6 }),
- streamsPerNetwork: fc.integer({ min: 1, max: 12 })
+ streamsPerNetwork: fc.option(fc.integer({ min: 1, max: 20 }), { nil: undefined })
})
test.describe('download plan', () => {
@@ -48,9 +48,8 @@ test.describe('download plan', () => {
perNetwork.set(network, (perNetwork.get(network) ?? 0) + 1)
}
expect(perNetwork.size).toBe(request.networkCount)
- expect(streamNetworks.length).toBeLessThanOrEqual(MAX_STREAMS)
for (const count of perNetwork.values()) {
- expect(count).toBeLessThanOrEqual(request.streamsPerNetwork)
+ expect(count).toBeLessThanOrEqual(request.streamsPerNetwork ?? START_STREAMS_PER_NETWORK)
expect(count).toBeLessThanOrEqual(MAX_STREAMS_PER_NETWORK)
// Streams beyond the block count would idle — except the one each network keeps.
expect(count).toBeLessThanOrEqual(
@@ -96,21 +95,22 @@ test.describe('download plan', () => {
})
test('a large file keeps 8 MB blocks', () => {
- const plan = planDownload({
- totalBytes: 4 * 1024 * MIB,
- splittable: true,
- networkCount: 2,
- streamsPerNetwork: 4
- })
+ const plan = planDownload({ totalBytes: 4 * 1024 * MIB, splittable: true, networkCount: 2 })
expect(plan.blockSizeBytes).toBe(DEFAULT_MAX_BLOCK_BYTES)
})
+ test('each network starts with four streams, with blocks enough to grow to its limit', () => {
+ const plan = planDownload({ totalBytes: 512 * MIB, splittable: true, networkCount: 2 })
+ expect(plan.streamNetworks).toEqual([0, 1, 0, 1, 0, 1, 0, 1])
+ // Streams added later need waiting blocks to take: two each, at the most every network can have.
+ expect(plan.blockCount).toBeGreaterThanOrEqual(2 * MAX_STREAMS_PER_NETWORK * 2)
+ })
+
test('a mid-size file is cut so a fast network can out-pull a slow one', () => {
const plan = planDownload({
totalBytes: 12 * MIB,
splittable: true,
- networkCount: 2,
- streamsPerNetwork: 2
+ networkCount: 2
})
// Two 8 MB-ish blocks would pin a third of the file to whichever network is slower.
expect(plan.blockCount).toBeGreaterThanOrEqual(8)
diff --git a/e2e/sim.spec.ts b/e2e/sim.spec.ts
index c9548e8..92fe477 100644
--- a/e2e/sim.spec.ts
+++ b/e2e/sim.spec.ts
@@ -33,9 +33,7 @@ test.describe('simulated downloads @smoke', () => {
await plexo.startSimulated(
{
sourceFilePath: source.path,
- networks: [network('one', 30), network('two', 30), network('three', 30)],
- chunkCount: 6,
- connectionsPerNetwork: 2
+ networks: [network('one', 30), network('two', 30), network('three', 30)]
},
source.sha
)
@@ -55,11 +53,10 @@ test.describe('simulated downloads @smoke', () => {
await plexo.startSimulated(
{
sourceFilePath: path,
- networks: [network('one'), network('two')],
- chunkCount: 16,
- connectionsPerNetwork: 8
+ networks: [network('one'), network('two')]
},
- sha256(bytes)
+ sha256(bytes),
+ { connections: 8 }
)
const state = await plexo.waitForStatus('completed')
@@ -87,9 +84,7 @@ test.describe('simulated downloads @smoke', () => {
const id = await plexo.startSimulated(
{
sourceFilePath: source.path,
- networks: [{ ...network('one'), speedBytesPerSec: 200_000 }],
- chunkCount: 2,
- connectionsPerNetwork: 2
+ networks: [{ ...network('one'), speedBytesPerSec: 200_000 }]
},
source.sha
)
@@ -116,9 +111,7 @@ test.describe('simulated downloads @smoke', () => {
const id = await plexo.startSimulated(
{
sourceFilePath: source.path,
- networks: [{ ...network('one'), speedBytesPerSec: 200_000 }],
- chunkCount: 2,
- connectionsPerNetwork: 2
+ networks: [{ ...network('one'), speedBytesPerSec: 200_000 }]
},
source.sha
)
@@ -148,9 +141,7 @@ test.describe('a slow simulated network', () => {
networks: [
{ kind: 'usb', label: 'slow', speedBytesPerSec: 10_000, faultRatePercent: 0 },
network('fast')
- ],
- chunkCount: 4,
- connectionsPerNetwork: 2
+ ]
},
source.sha
)
diff --git a/e2e/streams.spec.ts b/e2e/streams.spec.ts
new file mode 100644
index 0000000..4a3a210
--- /dev/null
+++ b/e2e/streams.spec.ts
@@ -0,0 +1,54 @@
+import { BLOCK, expect, test } from './fixtures'
+
+// N. How many streams a download runs, decided while it runs (see concurrency.ts), against a
+// server that limits it in each of the ways that matter.
+
+test.describe('automatic stream count', () => {
+ test.use({ appEnv: { PLEXO_E2E_PROBE_MS: '400' } })
+
+ const peakStreams = (plexo: { sessions: { chunks: unknown[] }[][] }): number =>
+ Math.max(...plexo.sessions.at(-1)!.map((state) => state.chunks.length))
+
+ test('a server that caps each connection gets more of them, up to the limit', async ({
+ plexo,
+ serve
+ }) => {
+ const origin = await serve({ size: 192 * BLOCK, bytesPerSecond: 256 * 1024 })
+ await plexo.start(origin.url(), origin.sha256, { connections: 'auto' })
+ const state = await plexo.waitForStatus('completed', 30_000)
+ expect(state.chunks).toHaveLength(16)
+ })
+
+ test('a full link keeps the streams it started with: more are tried, then retired', async ({
+ plexo,
+ serve
+ }) => {
+ const origin = await serve({ size: 64 * BLOCK, sharedBytesPerSecond: 1_000_000 })
+ await plexo.start(origin.url(), origin.sha256, { connections: 'auto' })
+ const state = await plexo.waitForStatus('completed', 30_000)
+ expect(peakStreams(plexo), 'more streams were tried').toBe(8)
+ expect(state.chunks).toHaveLength(4)
+ })
+
+ test('a server that turns extra connections away keeps the ones it accepted', async ({
+ plexo,
+ serve
+ }) => {
+ const origin = await serve({ size: 64 * BLOCK, bytesPerSecond: 256 * 1024 })
+ const accepted = new Set<number>()
+ origin.setRule(({ connection, range }) => {
+ if (range?.start === 0 && range.end === 0) return undefined // the probe
+ if (accepted.has(connection) || accepted.size < 4) {
+ accepted.add(connection)
+ return undefined
+ }
+ return { status: 503 }
+ })
+
+ await plexo.start(origin.url(), origin.sha256, { connections: 'auto' })
+ const state = await plexo.waitForStatus('completed', 30_000)
+ expect(peakStreams(plexo), 'more streams were tried').toBe(8)
+ expect(state.chunks).toHaveLength(4)
+ expect(origin.log.some((request) => request.status === 503)).toBe(true)
+ })
+})
diff --git a/e2e/ui.spec.ts b/e2e/ui.spec.ts
index de629df..15f237c 100644
--- a/e2e/ui.spec.ts
+++ b/e2e/ui.spec.ts
@@ -126,8 +126,6 @@ test.describe('settings @smoke', () => {
: 'Switch to dark theme'
await themeToggle.click()
- await page().getByRole('button', { name: '4×' }).click()
-
await stubNativeUi(plexo, dirs.dest)
await page().getByRole('button', { name: 'Browse…' }).click()
@@ -142,10 +140,6 @@ test.describe('settings @smoke', () => {
await expect(page().getByRole('link', { name: 'Update available: 9.9.9' })).toBeVisible()
await expect(page().getByRole('alertdialog')).toBeHidden()
await expect(page().getByRole('button', { name: labelAfterSwitch })).toBeVisible()
- await expect(page().getByRole('button', { name: '4×' })).toHaveAttribute(
- 'aria-pressed',
- 'true'
- )
await expect(page().getByText(dirs.dest)).toBeVisible()
await expect(page().getByText('Office fibre')).toBeVisible()
await page().getByRole('button', { name: 'Edit network' }).first().click()
@@ -169,25 +163,19 @@ test.describe('settings @smoke', () => {
dirs
}) => {
const settingsPath = join(dirs.userData, 'app-settings.json')
- // The streams picker and the TO row, as the user sees them — compared whole, so no
- // platform's idea of the default folder is baked into the test.
- const choices = (): Promise<string[]> =>
- Promise.all([
- plexo.page.getByRole('button', { name: /^\d×$/, pressed: true }).innerText(),
- plexo.page.getByText('TO', { exact: true }).locator('..').innerText()
- ])
+ // The TO row, as the user sees it — compared whole, so no platform's idea of the default
+ // folder is baked into the test.
+ const choices = (): Promise<string> =>
+ plexo.page.getByText('TO', { exact: true }).locator('..').innerText()
const fresh = await choices()
await plexo.quit()
- await writeFile(settingsPath, '{"streamsPerNetwork": 4,')
+ await writeFile(settingsPath, '{"destinationDir": "/')
await plexo.launch()
expect(await choices()).toEqual(fresh)
await plexo.quit()
- await writeFile(
- settingsPath,
- JSON.stringify({ streamsPerNetwork: 3, destinationDir: join(dirs.dest, 'unplugged') })
- )
+ await writeFile(settingsPath, JSON.stringify({ destinationDir: join(dirs.dest, 'unplugged') }))
await plexo.launch()
expect(await choices()).toEqual(fresh)
})
diff --git a/src/main/download/concurrency.ts b/src/main/download/concurrency.ts
new file mode 100644
index 0000000..7256cfa
--- /dev/null
+++ b/src/main/download/concurrency.ts
@@ -0,0 +1,182 @@
+// How many streams each network runs, decided while the download runs from what it measures.
+//
+// Another connection only helps when something limits each connection on its own — a server
+// that caps a connection's speed, or TCP on a long or lossy path — and not once the network
+// itself is full. Nothing outside says which, so the controller finds out the one way it can:
+// it adds streams to a network and keeps them if the download got faster.
+//
+// - One network at a time, doubling its streams (4 → 8 → 16): the answer in a couple of steps.
+// - A step is kept when total speed rose by at least `minGain` of what that network was doing
+// before it. Total, because two networks can share one uplink, and streams that only take
+// speed from the other network gain nothing. The network's own speed as the yardstick,
+// because a slow cellular link's real gain would otherwise vanish in a fast Ethernet's noise.
+// - A step that doesn't pay is undone — its streams retire once their block is done — and that
+// network grows no further. Nor does one whose new streams fail: the server's way of saying
+// it wants fewer connections.
+//
+// No I/O and no clock of its own: it reads a snapshot and says what to do, so every rule can be
+// checked against exact situations.
+
+export interface NetworkSnapshot {
+ id: string
+ /** Its streams, not counting any retiring. */
+ streams: number
+ /** Everything its streams have received, ever. */
+ received: number
+ /** One of its streams is failing: retrying, or given up. */
+ failing: boolean
+}
+
+export interface Snapshot {
+ now: number
+ networks: readonly NetworkSnapshot[]
+ /** How many more streams the waiting blocks could keep busy. */
+ spareWork: number
+ /** Streams still winding down after a step that didn't pay. */
+ retiring: number
+}
+
+export type Action = { kind: 'add' | 'retire'; networkId: string; count: number }
+
+export interface ConcurrencyPolicy {
+ maxPerNetwork: number
+ /** How long each measurement runs. */
+ windowMs: number
+ /** How long streams get to connect and reach speed before they're measured. */
+ warmupMs: number
+ /** The least a step must add to total speed, as a share of its network's speed before it. */
+ minGain: number
+}
+
+/** Bytes per second, overall and per network. */
+interface Rates {
+ total: number
+ byNetwork: Map<string, number>
+}
+
+interface Mark {
+ at: number
+ received: Map<string, number>
+}
+
+type Phase =
+ /** Letting streams reach speed before anything is measured. */
+ | { kind: 'wait'; until: number }
+ /** Measuring the streams as they are. */
+ | { kind: 'measure'; from: Mark }
+ /** Streams were just added to `networkId`; letting them connect and reach speed. */
+ | { kind: 'warmup'; networkId: string; added: number; before: Rates; until: number }
+ /** Measuring whether they paid. */
+ | { kind: 'trial'; networkId: string; added: number; before: Rates; from: Mark }
+ /** No network can grow any more. */
+ | { kind: 'done' }
+
+const mark = ({ now, networks }: Snapshot): Mark => ({
+ at: now,
+ received: new Map(networks.map((network) => [network.id, network.received]))
+})
+
+function ratesSince(from: Mark, { now, networks }: Snapshot): Rates {
+ const seconds = Math.max(now - from.at, 1) / 1000
+ const byNetwork = new Map<string, number>()
+ let total = 0
+ for (const network of networks) {
+ const rate = (network.received - (from.received.get(network.id) ?? 0)) / seconds
+ byNetwork.set(network.id, rate)
+ total += rate
+ }
+ return { total, byNetwork }
+}
+
+export class ConcurrencyController {
+ private phase: Phase | null = null
+ /** Networks that grow no further. */
+ private readonly settled = new Set<string>()
+ /** Where the next look for a network to grow starts, so each gets its turn. */
+ private next = 0
+
+ constructor(private readonly policy: ConcurrencyPolicy) {}
+
+ tick(snapshot: Snapshot): Action | undefined {
+ const { policy } = this
+ const phase = (this.phase ??= { kind: 'wait', until: snapshot.now + policy.warmupMs })
+ switch (phase.kind) {
+ case 'wait':
+ if (snapshot.now >= phase.until) this.phase = { kind: 'measure', from: mark(snapshot) }
+ return undefined
+
+ case 'measure':
+ // Streams on their way out would count towards a speed that's about to drop.
+ if (snapshot.retiring > 0) {
+ this.phase = { kind: 'measure', from: mark(snapshot) }
+ return undefined
+ }
+ if (snapshot.now - phase.from.at < policy.windowMs) return undefined
+ return this.grow(snapshot, ratesSince(phase.from, snapshot))
+
+ case 'warmup':
+ if (this.failing(snapshot, phase.networkId)) return this.undo(snapshot, phase)
+ if (snapshot.now >= phase.until) {
+ const { networkId, added, before } = phase
+ this.phase = { kind: 'trial', networkId, added, before, from: mark(snapshot) }
+ }
+ return undefined
+
+ case 'trial': {
+ if (this.failing(snapshot, phase.networkId)) return this.undo(snapshot, phase)
+ if (snapshot.now - phase.from.at < policy.windowMs) return undefined
+ const after = ratesSince(phase.from, snapshot)
+ const networkBefore = phase.before.byNetwork.get(phase.networkId) ?? 0
+ if (after.total - phase.before.total < policy.minGain * networkBefore) {
+ return this.undo(snapshot, phase)
+ }
+ // Kept: what was just measured is where the next step starts from.
+ return this.grow(snapshot, after)
+ }
+
+ case 'done':
+ return undefined
+ }
+ }
+
+ /** Adds streams to the next network that can take them, or measures again if none can yet. */
+ private grow(snapshot: Snapshot, before: Rates): Action | undefined {
+ const { networks, spareWork } = snapshot
+ let canGrowLater = false
+ for (let i = 0; i < networks.length; i++) {
+ const index = (this.next + i) % networks.length
+ const network = networks[index]
+ if (this.settled.has(network.id) || network.streams >= this.policy.maxPerNetwork) continue
+ canGrowLater = true
+ const count = Math.min(
+ network.streams,
+ this.policy.maxPerNetwork - network.streams,
+ spareWork
+ )
+ // A network delivering nothing has no speed to improve on.
+ if (network.failing || count < 1 || !((before.byNetwork.get(network.id) ?? 0) > 0)) continue
+
+ this.next = (index + 1) % networks.length
+ this.phase = {
+ kind: 'warmup',
+ networkId: network.id,
+ added: count,
+ before,
+ until: snapshot.now + this.policy.warmupMs
+ }
+ return { kind: 'add', networkId: network.id, count }
+ }
+ this.phase = canGrowLater ? { kind: 'measure', from: mark(snapshot) } : { kind: 'done' }
+ return undefined
+ }
+
+ private undo(snapshot: Snapshot, phase: Extract<Phase, { kind: 'warmup' | 'trial' }>): Action {
+ this.settled.add(phase.networkId)
+ this.phase = { kind: 'measure', from: mark(snapshot) }
+ return { kind: 'retire', networkId: phase.networkId, count: phase.added }
+ }
+
+ private failing(snapshot: Snapshot, networkId: string): boolean {
+ return snapshot.networks.find((network) => network.id === networkId)?.failing ?? true
+ }
+}
diff --git a/src/main/download/downloadManager.ts b/src/main/download/downloadManager.ts
index 32a623a..f92ef96 100644
--- a/src/main/download/downloadManager.ts
+++ b/src/main/download/downloadManager.ts
@@ -13,13 +13,14 @@ import type {
StartDownloadRequest,
StartSimulatedDownloadRequest
} from '../../shared/types'
-import { interleave, planDownload } from '../../shared/plan'
-import { testKnobs } from '../testKnobs'
+import { testKnobs, testStreamsPerNetwork } from '../testKnobs'
import { advanceBlock, retractBlock } from './blockProgress'
+import { ConcurrencyController, type ConcurrencyPolicy, type Snapshot } from './concurrency'
import { downloadChunk, fetchRange, RemoteChangedError } from './chunkDownloader'
import { DownloadFile } from './downloadFile'
import { compareVersion, type FileVersion } from './fileVersion'
import { ensureDirectory, reserveDestinationPath } from './paths'
+import { interleave, MAX_STREAMS_PER_NETWORK, planDownload } from './plan'
import { pickWork, type SchedulerPolicy, type Work } from './scheduler'
import {
compatibleInterfaces,
@@ -96,6 +97,8 @@ interface ChunkRuntime {
receivedBytes: number
/** Failed attempts in a row, for backoff. */
failures: number
+ /** Stops for good after the block it's on, and leaves the list (see retireStreams). */
+ retiring: boolean
}
interface SpeedSample {
@@ -133,6 +136,9 @@ interface DownloadRuntime {
attempts: Map<number, Attempt[]>
/** Hedges started per block index, capped at SCHEDULER_POLICY.maxHedgesPerBlock. */
hedgesByBlock: Map<number, number>
+ /** Everything each network's streams have received, by network id: what the stream count is
+ * judged by (see concurrency.ts). */
+ receivedByNetwork: Map<string, number>
}
interface PersistedDownload {
@@ -217,6 +223,12 @@ const SCHEDULER_POLICY: SchedulerPolicy = {
hedgeAfterMs: testKnobs.hedgeAfterMs,
maxHedgesPerBlock: 2
}
+const CONCURRENCY_POLICY: ConcurrencyPolicy = {
+ maxPerNetwork: MAX_STREAMS_PER_NETWORK,
+ windowMs: testKnobs.probeWindowMs,
+ warmupMs: testKnobs.probeWindowMs,
+ minGain: 0.15
+}
// Idle connections look for work every 250 ms, so waiting a bit longer hands a refreshed block
// to a connection that is already proven fast, if there is one.
const REFRESH_HANDOFF_MS = 300
@@ -259,6 +271,21 @@ function delay(ms: number, signal: AbortSignal): Promise<void> {
})
}
+function newStream(id: number, iface: NetworkInterfaceInfo): ChunkState {
+ return {
+ id,
+ interfaceId: iface.id,
+ interfaceLabel: iface.displayName,
+ interfaceKind: iface.kind,
+ rangeStart: 0,
+ rangeEnd: null,
+ bytesDownloaded: 0,
+ speedBytesPerSec: 0,
+ status: 'pending',
+ retryCount: 0
+ }
+}
+
function requestedVersion(request: StartDownloadRequest): FileVersion {
return { etag: request.etag, lastModified: request.lastModified, totalBytes: request.totalBytes }
}
@@ -422,7 +449,8 @@ export class DownloadManager {
refreshesByBlock: new Map(),
avoidNetworkByBlock: new Map(),
attempts: new Map(),
- hedgesByBlock: new Map()
+ hedgesByBlock: new Map(),
+ receivedByNetwork: new Map()
})
} catch {
// Ignore incomplete or corrupt manifests; other downloads can still be restored.
@@ -531,8 +559,6 @@ export class DownloadManager {
totalBytes,
supportsRanges: true,
interfaceIds: interfaces.map((iface) => iface.id),
- chunkCount: payload.chunkCount,
- connectionsPerNetwork: payload.connectionsPerNetwork,
etag: null,
lastModified: null
}
@@ -556,9 +582,7 @@ export class DownloadManager {
totalBytes: requestPayload.totalBytes,
splittable: requestPayload.supportsRanges,
networkCount: interfaces.length,
- streamsPerNetwork:
- requestPayload.connectionsPerNetwork ??
- Math.round(requestPayload.chunkCount / interfaces.length),
+ streamsPerNetwork: testStreamsPerNetwork() ?? undefined,
maxBlockBytes: testKnobs.blockBytes // 8 MB outside tests
})
@@ -605,21 +629,9 @@ export class DownloadManager {
})
}
- const chunks: ChunkState[] = plan.streamNetworks.map((networkIndex, id) => {
- const iface = interfaces[networkIndex]
- return {
- id,
- interfaceId: iface.id,
- interfaceLabel: iface.displayName,
- interfaceKind: iface.kind,
- rangeStart: 0,
- rangeEnd: null,
- bytesDownloaded: 0,
- speedBytesPerSec: 0,
- status: 'pending',
- retryCount: 0
- }
- })
+ const chunks = plan.streamNetworks.map((networkIndex, id) =>
+ newStream(id, interfaces[networkIndex])
+ )
const activeInterfaces = [...new Set(plan.streamNetworks)].map((index) => interfaces[index])
const state: DownloadState = {
@@ -655,7 +667,8 @@ export class DownloadManager {
refreshesByBlock: new Map(),
avoidNetworkByBlock: new Map(),
attempts: new Map(),
- hedgesByBlock: new Map()
+ hedgesByBlock: new Map(),
+ receivedByNetwork: new Map()
}
this.runtimes.set(id, runtime)
await this.persistNow(runtime)
@@ -849,18 +862,33 @@ export class DownloadManager {
)
}
- /** Runs (or resumes) fixed worker streams in parallel, leasing blocks until all are completed. */
+ /** Runs (or resumes) the streams in parallel, leasing blocks until all are completed, and adds
+ * or retires streams as measurements say (see concurrency.ts). */
private async runChunksToCompletion(
runtime: DownloadRuntime,
chunks: ChunkState[]
): Promise<void> {
- const active = new Map<number, Promise<number>>()
- for (const chunk of chunks) {
+ /** Running workers, by stream id. */
+ const active = new Map<number, Promise<void>>()
+ const startWorker = (chunk: ChunkState): void => {
active.set(
chunk.id,
- this.runWorker(runtime, chunk).then(() => chunk.id)
+ this.runWorker(runtime, chunk).finally(() => {
+ active.delete(chunk.id)
+ if (runtime.chunkRuntimes.get(chunk.id)?.retiring) this.removeStream(runtime, chunk)
+ })
)
}
+ for (const chunk of chunks) startWorker(chunk)
+
+ // Streams are only added where there is more than one to have: a download without ranges or
+ // a known size is a single request.
+ const concurrency =
+ runtime.requestPayload.supportsRanges &&
+ runtime.state.totalBytes > 0 &&
+ testStreamsPerNetwork() === null
+ ? new ConcurrencyController(CONCURRENCY_POLICY)
+ : null
const speedTicker = setInterval(() => {
// Only the `finally` below stops this ticker — self-clearing on status here would let a
@@ -875,14 +903,23 @@ export class DownloadManager {
this.scheduleUpdate(runtime)
}
this.refreshStuckConnections(runtime, now)
+
+ const action = concurrency?.tick(this.concurrencySnapshot(runtime, now))
+ if (action?.kind === 'add') {
+ for (const chunk of this.addStreams(runtime, action.networkId, action.count)) {
+ startWorker(chunk)
+ }
+ } else if (action?.kind === 'retire') {
+ for (const chunk of this.retireStreams(runtime, action.networkId, action.count)) {
+ // One that already stopped (it failed for good) has no worker left to remove it.
+ if (!active.has(chunk.id)) this.removeStream(runtime, chunk)
+ }
+ }
}, 500)
speedTicker.unref()
try {
- while (active.size > 0) {
- const finishedId = await Promise.race(active.values())
- active.delete(finishedId)
- }
+ while (active.size > 0) await Promise.race(active.values())
} finally {
clearInterval(speedTicker)
}
@@ -1202,6 +1239,10 @@ export class DownloadManager {
): void {
const now = Date.now()
self.receivedBytes += deltaBytes
+ runtime.receivedByNetwork.set(
+ chunk.interfaceId,
+ (runtime.receivedByNetwork.get(chunk.interfaceId) ?? 0) + deltaBytes
+ )
let samples = runtime.speedSamplesByChunk.get(chunk.id)
if (!samples) {
samples = []
@@ -1452,6 +1493,75 @@ export class DownloadManager {
this.goIdle(chunk)
}
+ private concurrencySnapshot(runtime: DownloadRuntime, now: number): Snapshot {
+ const networks = runtime.activeInterfaces.map((iface) => {
+ let streams = 0
+ let failing = false
+ for (const chunk of runtime.state.chunks) {
+ if (chunk.interfaceId !== iface.id || runtime.chunkRuntimes.get(chunk.id)?.retiring)
+ continue
+ streams++
+ if (chunk.status === 'retrying' || chunk.status === 'error') failing = true
+ }
+ return {
+ id: iface.id,
+ streams,
+ failing,
+ received: runtime.receivedByNetwork.get(iface.id) ?? 0
+ }
+ })
+ let retiring = 0
+ for (const self of runtime.chunkRuntimes.values()) if (self.retiring) retiring++
+ const waiting = runtime.blocks.filter((block) => block.status === 'pending').length
+ // Two waiting blocks per new stream, so it neither idles nor strands a slow network's share.
+ return { now, networks, spareWork: Math.floor(waiting / 2), retiring }
+ }
+
+ /** New streams on `networkId`, put on the download's list for the caller to start. */
+ private addStreams(runtime: DownloadRuntime, networkId: string, count: number): ChunkState[] {
+ const iface = runtime.activeInterfaces.find((entry) => entry.id === networkId)
+ if (!iface) return []
+ let id = Math.max(-1, ...runtime.state.chunks.map((chunk) => chunk.id))
+ const added = Array.from({ length: count }, () => newStream(++id, iface))
+ runtime.state.chunks.push(...added)
+ debug('streams', { network: iface.displayName, added: count })
+ this.scheduleUpdate(runtime)
+ return added
+ }
+
+ /** Winds down the newest `count` streams on `networkId`. One between blocks — idle, or waiting
+ * out a retry — stops now; one partway through a block finishes it first. */
+ private retireStreams(runtime: DownloadRuntime, networkId: string, count: number): ChunkState[] {
+ const live = runtime.state.chunks.filter(
+ (chunk) => chunk.interfaceId === networkId && !runtime.chunkRuntimes.get(chunk.id)?.retiring
+ )
+ const retired = live.slice(-count)
+ for (const chunk of retired) {
+ const self = runtime.chunkRuntimes.get(chunk.id)
+ if (!self) continue
+ self.retiring = true
+ if (!self.attempt) self.controller.abort()
+ }
+ debug('streams', { network: networkId, retired: count })
+ return retired
+ }
+
+ /** Takes a retired stream off the list. What it downloaded and its retries pass to a stream
+ * still on its network, since the per-network totals on screen are summed from the streams. */
+ private removeStream(runtime: DownloadRuntime, chunk: ChunkState): void {
+ runtime.chunkRuntimes.delete(chunk.id)
+ runtime.speedSamplesByChunk.delete(chunk.id)
+ const index = runtime.state.chunks.indexOf(chunk)
+ if (index < 0) return
+ runtime.state.chunks.splice(index, 1)
+ const heir = runtime.state.chunks.find((entry) => entry.interfaceId === chunk.interfaceId)
+ if (heir) {
+ heir.bytesDownloaded += chunk.bytesDownloaded
+ heir.retryCount += chunk.retryCount
+ }
+ this.scheduleUpdate(runtime)
+ }
+
private async runWorker(runtime: DownloadRuntime, chunk: ChunkState): Promise<void> {
const iface =
runtime.activeInterfaces.find((i) => i.id === chunk.interfaceId) ??
@@ -1474,13 +1584,14 @@ export class DownloadManager {
slowSince: null,
lastBlockSpeed: 0,
receivedBytes: 0,
- failures: 0
+ failures: 0,
+ retiring: false
}
runtime.chunkRuntimes.set(chunk.id, self)
try {
while (runtime.state.status === 'downloading') {
- if (controller.signal.aborted) break
+ if (controller.signal.aborted || self.retiring) break
// Taken atomically: nothing between choosing the work and registering it can yield.
const work = pickWork(
diff --git a/src/shared/plan.ts b/src/main/download/plan.ts
similarity index 71%
rename from src/shared/plan.ts
rename to src/main/download/plan.ts
index f6e1c8d..5227c94 100644
--- a/src/shared/plan.ts
+++ b/src/main/download/plan.ts
@@ -1,6 +1,6 @@
-// How a download is cut into blocks and how many streams work on them. Pure, and shared, so the
-// main process (which does the cutting) and the start screen (which previews the stream count)
-// cannot disagree about it.
+// How a download is cut into blocks and how many streams start on them. Pure, so every rule here
+// can be checked against exact situations. How many streams a network ends up with is decided
+// while the download runs (see concurrency.ts); the plan only has to leave them enough blocks.
const MIB = 1024 * 1024
@@ -15,17 +15,18 @@ export const MIN_BLOCK_BYTES = MIB
* blocks grow instead of multiplying. */
export const MAX_BLOCKS = 4096
-export const MAX_STREAMS_PER_NETWORK = 8
-export const MAX_STREAMS = 32
-/** The per-network stream counts offered in the UI. */
-export const PRESET_STREAMS = [1, 2, 4, 8] as const
+/** Streams each network starts with. Servers are built for browsers, which open about six
+ * connections to a host, so four troubles none of them. */
+export const START_STREAMS_PER_NETWORK = 4
-/** Blocks planned per stream, at least. Streams pull blocks as they free up, so a fast network
- * takes more of them — but only if there are more blocks than streams to begin with. With one
+/** The most streams a network can grow to. A safety net for noisy speed readings rather than
+ * the real limit, which is whether another stream still helps (see concurrency.ts). */
+export const MAX_STREAMS_PER_NETWORK = 16
+
+/** Blocks planned per stream a network could grow to. Streams pull blocks as they free up, so a
+ * fast network takes more of them — but only if there are more blocks than streams. With one
* block each, a stream on a slow network would hold its block while the rest sat idle. */
const BLOCKS_PER_STREAM = 2
-/** …and never fewer than this many per network, whatever the stream count. */
-const MIN_BLOCKS_PER_NETWORK = 4
export interface DownloadPlan {
/** Bytes per block; the last block holds the remainder. */
@@ -42,8 +43,8 @@ export interface PlanRequest {
/** false when the server can't serve byte ranges. */
splittable: boolean
networkCount: number
- /** Streams wanted on each network. */
- streamsPerNetwork: number
+ /** Streams to start on each network; START_STREAMS_PER_NETWORK unless a test pins it. */
+ streamsPerNetwork?: number
maxBlockBytes?: number
}
@@ -79,10 +80,13 @@ export function planDownload(request: PlanRequest): DownloadPlan {
}
const maxBlockBytes = request.maxBlockBytes ?? DEFAULT_MAX_BLOCK_BYTES
- const requested = clamp(Math.floor(request.streamsPerNetwork), 1, MAX_STREAMS_PER_NETWORK)
+ const requested = clamp(
+ Math.floor(request.streamsPerNetwork ?? START_STREAMS_PER_NETWORK),
+ 1,
+ MAX_STREAMS_PER_NETWORK
+ )
- const targetBlocks =
- networkCount * Math.max(MIN_BLOCKS_PER_NETWORK, requested * BLOCKS_PER_STREAM)
+ const targetBlocks = networkCount * MAX_STREAMS_PER_NETWORK * BLOCKS_PER_STREAM
const blockSizeBytes = Math.max(
clamp(
Math.ceil(totalBytes / targetBlocks),
@@ -99,7 +103,7 @@ export function planDownload(request: PlanRequest): DownloadPlan {
const streamNetworks = interleave(
Array.from({ length: networkCount * perNetwork }, (_, i) => Math.floor(i / perNetwork)),
(network) => network
- ).slice(0, MAX_STREAMS)
+ )
return { blockSizeBytes, blockCount, streamNetworks }
}
diff --git a/src/main/download/simDownload.ts b/src/main/download/simDownload.ts
index b1a375f..2dc7a3b 100644
--- a/src/main/download/simDownload.ts
+++ b/src/main/download/simDownload.ts
@@ -18,6 +18,9 @@ interface SimSession {
sourcePath: string
/** Keyed by the synthetic NetworkInterfaceInfo.id assigned to each simulated network. */
networks: Map<string, SimulatedNetworkConfig>
+ /** When each network is next free to deliver, by the same id. A network's speed is shared by
+ * all of its streams, as a real link's is: a stream more doesn't make it any faster. */
+ busyUntil: Map<string, number>
}
const sessions = new Map<string, SimSession>()
@@ -30,7 +33,7 @@ function registerSimSession(
networks: Map<string, SimulatedNetworkConfig>
): string {
const token = randomUUID()
- sessions.set(token, { sourcePath, networks })
+ sessions.set(token, { sourcePath, networks, busyUntil: new Map() })
return token
}
@@ -137,7 +140,12 @@ export function downloadChunkSimulated(options: ChunkDownloadOptions): Promise<v
// real: a real download is limited by the network, so the block grid and speed readout
// should move at a rate this slow, not spike and then sit idle.
input.pause()
- const delayMs = Math.max(1, (buffer.length / speedBytesPerSec) * 1000)
+ const now = Date.now()
+ const deliveredAt =
+ Math.max(now, session.busyUntil.get(connection.iface.id) ?? 0) +
+ (buffer.length / speedBytesPerSec) * 1000
+ session.busyUntil.set(connection.iface.id, deliveredAt)
+ const delayMs = Math.max(1, deliveredAt - now)
setTimeout(() => {
if (settled) return
bytesWritten += buffer.length
diff --git a/src/main/ipc/handlers.ts b/src/main/ipc/handlers.ts
index df716f9..8f0b9df 100644
--- a/src/main/ipc/handlers.ts
+++ b/src/main/ipc/handlers.ts
@@ -116,7 +116,6 @@ export function registerIpcHandlers(getWindow: () => BrowserWindow | null): Down
isDev: is.dev,
themeSource: currentThemeSource(),
networkPreferences: settings.networkPreferences ?? {},
- streamsPerNetwork: settings.streamsPerNetwork,
destinationDir: destinationExists ? destinationDir : undefined
} satisfies InitialState
} catch (error) {
diff --git a/src/main/settings.ts b/src/main/settings.ts
index 0d48600..ec2534b 100644
--- a/src/main/settings.ts
+++ b/src/main/settings.ts
@@ -1,7 +1,6 @@
import { rm } from 'node:fs/promises'
import { isAbsolute, join } from 'node:path'
import { app, nativeTheme } from 'electron'
-import { PRESET_STREAMS } from '../shared/plan'
import type {
AppSettings,
NetworkPreference,
@@ -42,16 +41,13 @@ function sanitizeNetworkPreferences(parsed: unknown): NetworkPreferences {
function sanitizeSettings(parsed: unknown): AppSettings {
if (!isRecord(parsed)) return {}
- const { themeSource, dismissedUpdateVersion, streamsPerNetwork, destinationDir } = parsed
+ const { themeSource, dismissedUpdateVersion, destinationDir } = parsed
const settings: AppSettings = {}
// 'system' was once an option — dropping it falls back to the OS appearance (loadThemeSource).
if (themeSource === 'light' || themeSource === 'dark') settings.themeSource = themeSource
if (typeof dismissedUpdateVersion === 'string') {
settings.dismissedUpdateVersion = dismissedUpdateVersion
}
- if ((PRESET_STREAMS as readonly unknown[]).includes(streamsPerNetwork)) {
- settings.streamsPerNetwork = streamsPerNetwork as number
- }
if (typeof destinationDir === 'string' && isAbsolute(destinationDir)) {
settings.destinationDir = destinationDir
}
diff --git a/src/main/testKnobs.ts b/src/main/testKnobs.ts
index 8072ba1..6c3e8e4 100644
--- a/src/main/testKnobs.ts
+++ b/src/main/testKnobs.ts
@@ -22,11 +22,22 @@ export const testKnobs = {
slowForMs: positiveNumber('PLEXO_E2E_SLOW_FOR_MS', 10_000),
silentAfterMs: positiveNumber('PLEXO_E2E_SILENT_MS', 5_000),
hedgeAfterMs: positiveNumber('PLEXO_E2E_HEDGE_MS', 5_000),
+ /** How long each measurement of the stream-count controller runs (see concurrency.ts). */
+ probeWindowMs: positiveNumber('PLEXO_E2E_PROBE_MS', 2_000),
/** Skips the real GitHub check and pretends this version is available, for exercising the
* update banner without needing an actual newer release published. */
forceUpdateVersion: env['PLEXO_FORCE_UPDATE_VERSION']
}
+/** `PLEXO_E2E_STREAMS=2` fixes how many streams each network runs and turns the automatic
+ * sizing off, so a test can count requests. Read on every call rather than once, so a test can
+ * change it between downloads. */
+export function testStreamsPerNetwork(): number | null {
+ if (app.isPackaged) return null
+ const value = Number(process.env['PLEXO_E2E_STREAMS'])
+ return value > 0 ? value : null
+}
+
/** `PLEXO_E2E_INTERFACES=a=127.0.0.1,b=192.168.1.5` replaces the real interface list. Read on
* every call rather than once, so a test can make a network "disappear" mid-download by
* rewriting process.env in the main process. */
diff --git a/src/renderer/src/components/DevToolsPanel.tsx b/src/renderer/src/components/DevToolsPanel.tsx
index 55b0913..31a7802 100644
--- a/src/renderer/src/components/DevToolsPanel.tsx
+++ b/src/renderer/src/components/DevToolsPanel.tsx
@@ -5,11 +5,9 @@ import { describeError, toDisplayPath } from '../utils/format'
import { TruncatedText } from './TruncatedText'
import { Button } from './ui/button'
import { Dialog, DialogClose, DialogContent, DialogTitle } from './ui/dialog'
-import { ToggleGroup, ToggleGroupItem } from './ui/toggle-group'
import { Tooltip, TooltipContent, TooltipTrigger } from './ui/tooltip'
const KIND_OPTIONS: NetworkInterfaceKind[] = ['wifi', 'usb', 'ethernet', 'bridge', 'other']
-const PRESET_CONNECTIONS = [1, 2, 4, 8] as const
const MAX_SIM_NETWORKS = 4
const MBPS_TO_BYTES_PER_SEC = (1024 * 1024) / 8
@@ -56,7 +54,6 @@ export function DevToolsPanel(): React.JSX.Element | null {
const [sourceFilePath, setSourceFilePath] = useState<string | null>(null)
const [destinationDir, setDestinationDir] = useState('')
const [networks, setNetworks] = useState<SimNetworkDraft[]>(DEFAULT_NETWORKS)
- const [connectionsPerNetwork, setConnectionsPerNetwork] = useState(2)
const [starting, setStarting] = useState(false)
const [error, setError] = useState<string | null>(null)
@@ -113,9 +110,7 @@ export function DevToolsPanel(): React.JSX.Element | null {
await window.plexo.startSimulatedDownload({
sourceFilePath,
destinationDir: effectiveDestinationDir,
- networks: simulatedNetworks,
- chunkCount: networks.length * connectionsPerNetwork,
- connectionsPerNetwork
+ networks: simulatedNetworks
})
setOpen(false)
} catch (err) {
@@ -298,29 +293,6 @@ export function DevToolsPanel(): React.JSX.Element | null {
))}
</div>
- <div className="flex flex-col gap-[5px]">
- <div id="devtools-connections-label" className={fieldLabelClass}>
- CONNECTIONS PER NETWORK
- </div>
- <ToggleGroup
- value={[String(connectionsPerNetwork)]}
- onValueChange={(values) => {
- if (values.length === 0) return
- setConnectionsPerNetwork(Number(values[0]))
- }}
- aria-labelledby="devtools-connections-label"
- variant="pill"
- size="xs"
- spacing={1}
- >
- {PRESET_CONNECTIONS.map((preset) => (
- <ToggleGroupItem key={preset} value={String(preset)}>
- {preset}×
- </ToggleGroupItem>
- ))}
- </ToggleGroup>
- </div>
-
{error && (
<div className="font-sans text-[11.5px] leading-[1.4] text-destructive">⚠ {error}</div>
)}
diff --git a/src/renderer/src/screens/IdleScreen.tsx b/src/renderer/src/screens/IdleScreen.tsx
index 6a52835..014df84 100644
--- a/src/renderer/src/screens/IdleScreen.tsx
+++ b/src/renderer/src/screens/IdleScreen.tsx
@@ -1,4 +1,3 @@
-import { PRESET_STREAMS, planDownload } from '@shared/plan'
import type { ProbeResult } from '@shared/types'
import { cn } from 'cn'
import { AlertTriangle, ClipboardPaste, Info } from 'lucide-react'
@@ -7,7 +6,6 @@ import { NetworkCard } from '../components/NetworkCard'
import { ScreenFooter } from '../components/ScreenFooter'
import { Alert, AlertDescription, AlertTitle } from '../components/ui/alert'
import { Button } from '../components/ui/button'
-import { ToggleGroup, ToggleGroupItem } from '../components/ui/toggle-group'
import { useNetworkPolling } from '../hooks/useNetworkPolling'
import { useAppStore } from '../store/useAppStore'
import { describeError, formatBytes, toDisplayPath } from '../utils/format'
@@ -66,8 +64,6 @@ export function IdleScreen(): React.JSX.Element {
const setUrl = useAppStore((store) => store.setDraftUrl)
const destinationDir = useAppStore((store) => store.destinationDir)
const setDestinationDir = useAppStore((store) => store.setDestinationDir)
- const chunksPerNetwork = useAppStore((store) => store.streamsPerNetwork)
- const setChunksPerNetwork = useAppStore((store) => store.setStreamsPerNetwork)
const [probe, setProbe] = useState<ProbeState>({ status: 'idle' })
// Tracks deselections rather than selections, so a newly-detected interface starts selected.
@@ -113,18 +109,6 @@ export function IdleScreen(): React.JSX.Element {
const enabledIds = detectedIds.filter((id) => !deselectedInterfaceIds.includes(id))
const selectedInterfaceIds = isSingleStreamOnly ? enabledIds.slice(0, 1) : enabledIds
- const connectionsPerNetwork = isSingleStreamOnly ? 1 : chunksPerNetwork
- // A small file gets fewer streams than asked for — one with no block to claim would only
- // idle — so once the file's size is known the count comes from the same plan the download
- // will use.
- const totalChunks = ready
- ? planDownload({
- totalBytes: ready.totalBytes ?? 0,
- splittable: multiChunkAllowed,
- networkCount: selectedInterfaceIds.length,
- streamsPerNetwork: chunksPerNetwork
- }).streamNetworks.length
- : selectedInterfaceIds.length * chunksPerNetwork
const startLabel = starting ? 'Starting…' : probe.status === 'probing' ? 'Checking…' : 'Start'
const canStart =
probe.status === 'ready' &&
@@ -134,9 +118,9 @@ export function IdleScreen(): React.JSX.Element {
const footerParts = [
`${selectedInterfaceIds.length} ${selectedInterfaceIds.length === 1 ? 'network' : 'networks'} selected`
]
- if (selectedInterfaceIds.length > 0) {
- footerParts.push(`${totalChunks} ${totalChunks === 1 ? 'stream' : 'parallel streams'}`)
- }
+ // How many streams each network gets is worked out during the download; the one case worth
+ // saying up front is a server that can't split the file at all.
+ if (isSingleStreamOnly) footerParts.push('1 stream: the server can’t split this file')
if (ready && ready.totalBytes !== null) footerParts.push(formatBytes(ready.totalBytes))
let subnetConflict: { subnet: string; names: string[] } | null = null
@@ -203,8 +187,6 @@ export function IdleScreen(): React.JSX.Element {
totalBytes: probe.result.totalBytes ?? 0,
supportsRanges: multiChunkAllowed,
interfaceIds: selectedInterfaceIds,
- chunkCount: totalChunks,
- connectionsPerNetwork,
etag: probe.result.etag,
lastModified: probe.result.lastModified
})
@@ -315,61 +297,6 @@ export function IdleScreen(): React.JSX.Element {
</Button>
</div>
- <div
- className={cn(
- 'flex min-h-9 items-center justify-between gap-3 rounded-[9px] border border-border px-3 py-1.5',
- isSingleStreamOnly && 'opacity-60'
- )}
- >
- <div className="flex flex-wrap items-center gap-2">
- <div id="idle-streams-label" className={fieldLabelClass}>
- PARALLEL STREAMS
- </div>
- <ToggleGroup
- value={[String(chunksPerNetwork)]}
- onValueChange={(values) => {
- if (values.length === 0) return
- setChunksPerNetwork(Number(values[0]))
- }}
- disabled={isSingleStreamOnly}
- aria-labelledby="idle-streams-label"
- variant="pill"
- size="xs"
- spacing={1}
- >
- {PRESET_STREAMS.map((preset) => (
- // h-6/min-w-6: WCAG 2.5.8's 24px floor — the xs toggle size is 20px, and this is
- // the primary "how many parallel connections" control.
- <ToggleGroupItem key={preset} value={String(preset)} className="h-6 min-w-6">
- {preset}×
- </ToggleGroupItem>
- ))}
- </ToggleGroup>
- </div>
-
- <div
- className={cn(
- 'text-right font-mono text-[11px] whitespace-nowrap',
- isSingleStreamOnly ? 'text-muted-foreground' : 'text-[var(--text-secondary)]'
- )}
- >
- {isSingleStreamOnly ? (
- '1 stream (server does not support ranges)'
- ) : (
- <>
- <span className="font-semibold text-foreground">{chunksPerNetwork}</span> / network
- {selectedInterfaceIds.length > 0 && (
- <>
- {' · '}
- <span className="font-semibold text-foreground">{totalChunks}</span> total
- parallel {totalChunks === 1 ? 'stream' : 'streams'}
- </>
- )}
- </>
- )}
- </div>
- </div>
-
{startError && <ErrorAlert message={startError} />}
</div>
diff --git a/src/renderer/src/store/useAppStore.ts b/src/renderer/src/store/useAppStore.ts
index 1fe00ba..5abc928 100644
--- a/src/renderer/src/store/useAppStore.ts
+++ b/src/renderer/src/store/useAppStore.ts
@@ -29,7 +29,6 @@ interface AppStore {
/** Persisted in the main process alongside nativeTheme.themeSource. */
themeSource: ThemeSource
- streamsPerNetwork: number
/** Null until the one-time startup check resolves, or if it found nothing worth showing
* (already up to date, already dismissed, or the check failed). */
@@ -59,7 +58,6 @@ interface AppStore {
refreshLatencies: () => Promise<void>
setNetworkPreference: (id: string, patch: NetworkPreference) => void
setThemeSource: (source: ThemeSource) => void
- setStreamsPerNetwork: (streamsPerNetwork: number) => void
checkForUpdate: () => Promise<void>
dismissUpdate: () => void
setCurrentDownload: (state: DownloadState) => void
@@ -85,7 +83,6 @@ export const useAppStore = create<AppStore>((set, get) => ({
latencies: {},
networkPreferences: initial.networkPreferences,
themeSource: initial.themeSource,
- streamsPerNetwork: initial.streamsPerNetwork ?? 2,
availableUpdate: null,
homeDir: initial.homeDir,
@@ -141,11 +138,6 @@ export const useAppStore = create<AppStore>((set, get) => ({
persist({ themeSource })
},
- setStreamsPerNetwork: (streamsPerNetwork) => {
- set({ streamsPerNetwork })
- persist({ streamsPerNetwork })
- },
-
checkForUpdate: async () => {
try {
const availableUpdate = await window.plexo.checkForUpdate()
diff --git a/src/shared/types.ts b/src/shared/types.ts
index ab73bd7..b60dbf5 100644
--- a/src/shared/types.ts
+++ b/src/shared/types.ts
@@ -135,8 +135,6 @@ export interface StartSimulatedDownloadRequest {
sourceFilePath: string
destinationDir: string
networks: SimulatedNetworkConfig[]
- chunkCount: number
- connectionsPerNetwork?: number
}
export interface UpdateInfo {
@@ -153,7 +151,6 @@ export interface UpdateInfo {
export interface AppSettings {
themeSource?: ThemeSource
dismissedUpdateVersion?: string
- streamsPerNetwork?: number
/** The last destination folder picked. */
destinationDir?: string
/** User customizations (name/color) per network interface id. */
@@ -169,7 +166,6 @@ export interface InitialState {
isDev: boolean
themeSource: ThemeSource
networkPreferences: NetworkPreferences
- streamsPerNetwork?: number
/** The last folder picked, if it still exists — otherwise the renderer uses downloadsDir. */
destinationDir?: string
}
@@ -182,10 +178,6 @@ export interface StartDownloadRequest {
totalBytes: number
supportsRanges: boolean
interfaceIds: string[]
- /** Total chunks to split the download into across interfaceIds. */
- chunkCount: number
- /** Number of parallel connections allocated per physical network. */
- connectionsPerNetwork?: number
etag: string | null
lastModified: string | null
}