Skip to content

Bulk asynchronous downloader for multiple file hosts

3.5 KB
Raw Blame

attempts

Type Default
PositiveInt 2

The number of download attempts per file. Some conditions are never retried (such as a 404 HTTP status)

downloads:
  attempts: 2

back_pressure

Type Default
bool True

Throttle scrape requests if too many downloads for the same site are queued.

This works by setting a limit of max queued downloads of a single site to the value of <concurrency> x10 (capped at 50). If the number of pending downloads for that site reaches the limit, all scraping for the site will be paused until the queue goes below it

downloads:
  back_pressure: True

{% hint style="warning" %} Disabling back_pressure will cause high memory usage and performance degradation if the number of URLs pending to be scrapped is high (1K+) {% endhint %}

delay

Type Default
NonNegativeFloat 0.0

Number of seconds to wait in between downloads to the same domain.

Some domains have internal limits set by the program, which can not be modified:

  • bunkr: 0.5
  • nhentai.net: 1
downloads:
  delay: 0.0

speed_limit

Type Default
ByteSize 0

{% hint style="info" %} This setting specifies speed and it's interpreted as <value> / second. ex: 8MB means 8MB/s {% endhint %}

This is the max rate of downloading in bytes (per second) for all downloads combined. Set to 0 to disable

downloads:
  speed_limit: 0B

concurrent_segments

Type Default
PositiveInt 10

Allow up to <N> HLS segments to be downloaded concurrently.

downloads:
  concurrent_segments: 10

jitter

Type Default
NonNegativeFloat 0.0

Additional number of seconds to wait in between downloads. CDL will wait an additional random number of seconds in between 0 and the jitter value.

downloads:
  jitter: 0.0

concurrency

Type Default
PositiveInt 15

This is the maximum number of files that can be downloaded simultaneously.

downloads:
  concurrency: 15

concurrency_per_domain

Type Default
PositiveInt 5

This is the maximum number of files that can be downloaded from a single domain simultaneously.

Some domains have internal limits set by the program, which can not be modified:

  • bunkr: 1 per unique server
  • cyberfile: 1
  • noodlemagazine: 2
  • 4chan: 1
  • pixeldrain: 2
  • xxxbunker: 2
downloads:
  concurrency_per_domain: 5

skip_and_mark_completed

Type Default
bool false

Skip the download process for every file and mark them as downloaded in the database.

Use this to make CDL permanently skip a set of URLs. See Retries and Download History.

downloads:
  skip_and_mark_completed: false

slow_speed

Type Default
ByteSize 0

{% hint style="info" %} This setting specifies speed and it's interpreted as <value> / second. ex: 8MB means 8MB/s {% endhint %}

Downloads with a speed lower than this value for more than 10 seconds will be skipped. Set to 0 to disable

downloads:
  slow_speed: 0B