Bulk asynchronous downloader for multiple file hosts
attempts
| Type | Default |
|---|---|
PositiveInt |
2 |
The number of download attempts per file. Some conditions are never retried (such as a 404 HTTP status)
downloads:
attempts: 2
back_pressure
| Type | Default |
|---|---|
bool |
True |
Throttle scrape requests if too many downloads for the same site are queued.
This works by setting a limit of max queued downloads of a single site to the value of <concurrency> x10 (capped at 50). If the number of pending downloads
for that site reaches the limit, all scraping for the site will be paused until the queue goes below it
downloads:
back_pressure: True
{% hint style="warning" %}
Disabling back_pressure will cause high memory usage and performance degradation if the number of URLs pending to be scrapped is high (1K+)
{% endhint %}
delay
| Type | Default |
|---|---|
NonNegativeFloat |
0.0 |
Number of seconds to wait in between downloads to the same domain.
Some domains have internal limits set by the program, which can not be modified:
bunkr: 0.5nhentai.net: 1
downloads:
delay: 0.0
speed_limit
| Type | Default |
|---|---|
ByteSize |
0 |
{% hint style="info" %}
This setting specifies speed and it's interpreted as <value> / second. ex: 8MB means 8MB/s
{% endhint %}
This is the max rate of downloading in bytes (per second) for all downloads combined. Set to 0 to disable
downloads:
speed_limit: 0B
concurrent_segments
| Type | Default |
|---|---|
PositiveInt |
10 |
Allow up to <N> HLS segments to be downloaded concurrently.
downloads:
concurrent_segments: 10
jitter
| Type | Default |
|---|---|
NonNegativeFloat |
0.0 |
Additional number of seconds to wait in between downloads. CDL will wait an additional random number of seconds in between 0 and the jitter value.
downloads:
jitter: 0.0
concurrency
| Type | Default |
|---|---|
PositiveInt |
15 |
This is the maximum number of files that can be downloaded simultaneously.
downloads:
concurrency: 15
concurrency_per_domain
| Type | Default |
|---|---|
PositiveInt |
5 |
This is the maximum number of files that can be downloaded from a single domain simultaneously.
Some domains have internal limits set by the program, which can not be modified:
bunkr: 1 per unique servercyberfile: 1noodlemagazine: 24chan: 1pixeldrain: 2xxxbunker: 2
downloads:
concurrency_per_domain: 5
skip_and_mark_completed
| Type | Default |
|---|---|
bool |
false |
Skip the download process for every file and mark them as downloaded in the database.
Use this to make CDL permanently skip a set of URLs. See Retries and Download History.
downloads:
skip_and_mark_completed: false
slow_speed
| Type | Default |
|---|---|
ByteSize |
0 |
{% hint style="info" %}
This setting specifies speed and it's interpreted as <value> / second. ex: 8MB means 8MB/s
{% endhint %}
Downloads with a speed lower than this value for more than 10 seconds will be skipped. Set to 0 to disable
downloads:
slow_speed: 0B