Bulk asynchronous downloader for multiple file hosts
disabled
| Type | Default | Additional Info |
|---|---|---|
list[NonEmptyStr] |
[] |
This is an AdditiveArg |
You can supply a list of crawlers to disable for the current run. This will make CDL completely ignore the crawler, as if the site was not supported. However, links from the site will still be processed by Real-Debrid (if enabled), JDownloader (If enabled) and the Generic crawler (If enabled), in that order.
The list should be valid crawlers names. The name of the crawler is the name of the primary site they support. ex: 4Chan, Mega.nz, Dropbox
You can get the crawler's name from the supported sites page The name of the crawer is the title of their section in the page (in bold).
crawlers:
disabled: []
generic
Generic crawlers are designed to work on any site that uses a specific framework. Users can supply a list of sites to map to these crawlers,
and CDL will then be able to download from them. The URL in the list should be the primary URL of the site. ex: https://forums.docker.com/
Supported generic crawlers:
chevereto: This works on any site that uses Chevereto.discourse: This works on any forum that uses Discourse.kvs: Works on any tube site using Kernel Video Sharing. Basically, any site that looks like one of these: https://www.kernel-video-sharing.com/en/themes/, ex: https://www.kvs-demo.com/wordpress_media: This crawler should work on any WordPress site where content primarily consists of images or galleries. The images need to be hosted on the site itself. It requires sites to have a public WordPress REST API.wordpress_html: This works on any WordPress site. It scrapes the actual HTML of the site, which means it works on sites that have embedded third-party media like videos or links to hosting sites. It is always slower thanwordpress_media.
chevereto
| Type | Default |
|---|---|
list[HttpURL] |
[] |
discourse
| Type | Default |
|---|---|
list[HttpURL] |
[] |
kvs
| Type | Default |
|---|---|
list[HttpURL] |
[] |
wordpress_media
| Type | Default |
|---|---|
list[HttpURL] |
[] |
wordpress_html
| Type | Default |
|---|---|
list[HttpURL] |
[] |
crawlers:
generic:
chevereto: []
discourse: []
kvs: []
wordpress_html: []
wordpress_media: []
Bandcamp
formats
| Type | Default |
|---|---|
List of mp3-320, mp3, aac-hi, wav, flac, vorbis, aiff or alas |
[mp3-320, mp3, aac-hi, wav, flac, vorbis, aiff, alas] |
Format to choose for downloads (if available), ordered by preference.
crawlers:
bandcamp:
formats:
- mp3-320
- mp3
- aac-hi
- wav
- flac
- vorbis
- aiff
- alas
bluesky
crawlers:
bluesky:
content_urls: true
external: true
reposts: false
threads: true
external
| Type | Default |
|---|---|
Bool |
true |
Parse and download embeds from third-party sites
content_urls
| Type | Default |
|---|---|
Bool |
true |
Parse and try to download any URL found inside the text of a post.
threads
| Type | Default |
|---|---|
Bool |
true |
Downloads all posts in a thread (All direct replies from OP to their own post)
reposts
| Type | Default |
|---|---|
Bool |
false |
Download media from reposts in the user's timeline
Clyp.it
crawlers:
clypit:
prefer_mp3: false
prefer_mp3
| Type | Default |
|---|---|
Bool |
false |
Download audios as .mp3 files even if WAV (high quality) versions are available
Google Drive
crawlers:
google_drive:
default_formats:
docs: docx
sheets: xslx
slides: pptx
default_formats
docs
| Type | Default |
|---|---|
docx, odt, rtf, txt, epub, pdf, md or zip |
docx |
Default format for documents (can be overridden per URL with the 'format' query param)
sheets
| Type | Default |
|---|---|
xslx, ods, html, csv or tsv |
xslx |
Default format for spreedsheets (can be overridden per URL with the 'format' query param)
slides
| Type | Default |
|---|---|
pptx or odp |
pptx |
Default format for presentations (can be overridden per URL with the 'format' query param)
OnePace
prefer_dub
| Type | Default |
|---|---|
Bool |
false |
Download episodes with english audio tracks instead of japanese (if available)
Pawchive / OnlyHaven
file
| Type | Default |
|---|---|
Bool |
true |
Download the main file in a post (if any)
attachments
| Type | Default |
|---|---|
Bool |
true |
Download all attachments in a post (may or may not include file)
content_urls
| Type | Default |
|---|---|
Bool |
true |
Scan the description (text) in a post and download any URL found
{% hint style="warning" %} This option slows down scraping significally (50x requests in total) {% endhint %}
expand_posts
| Type | Default |
|---|---|
Bool |
false |
When crawling profiles/favorites/searchs results, make an additional request per post to get the original filenames of files and the content/text in the post. To download deferred big files, this option must be enabled
{% hint style="warning" %} This option slows down scraping significally (100x requests in total) {% endhint %}
embed
| Type | Default |
|---|---|
Bool |
true |
Download the embedded file from third party sites (if any)(mega.nz, pcloud, dropbox, etc..)
Embedded files show up as clickable boxes on the website:

crawlers:
pawchive:
attachments: true
content_urls: true
embed: true
file: true
PornHub
crawlers:
pornhub:
profile_paths:
- "photos/public"
- "gifs/public"
- "videos"
- "videos/upload"
profile_paths
| Type | Default |
|---|---|
list of NonEmptyStr |
[photos/public, gifs/public, videos, videos/upload] |
Subpaths to scrape when an input URL is a profile's homepage. ex: https://www.pornhub.com/model/<model_name>
reddit:
api: arctic_shift
content_urls: true
api
| Type | Default |
|---|---|
arctic_shift |
arctic_shift |
Backend API to use to connect to Reddit. The only currently supported API is arctic_shift, which allows retrieving deleted posts and subreddit-specific
posts with no authentication.
content_urls
| Type | Default |
|---|---|
Bool |
true |
Scan the description (text) in a post and download any URL found
Tiktok
crawlers:
tiktok:
original: false
original
| Type | Default |
|---|---|
Bool |
false |
By default, CDL will download the "optimized for streaming" version of tiktok videos. Setting this option to True will download videos in original (source) quality.
_original will be added as a suffix to their filename.
{% hint style="warning" %} This will make video downloads several times slower
When it is set to false (the default) CDL can download 50 videos with a single request.
When it is set to true , CDL needs to make at least 3 requests per video to download them.
There's also a daily limit of the API CDL uses: 5000 requests per day per IP
Setting this option to true will consume the daily limit faster
{% endhint %}
crawlers:
twitter:
articles:
cover: true
media: true
cards: true
content_urls: true
image_size: orig
retweets: false
threads: true
cards
| Type | Default |
|---|---|
Bool |
true |
Parse and download cards in a post (embeds from third-party sites)
content_urls
| Type | Default |
|---|---|
Bool |
true |
Parse and try to download any URL found inside the text of a tweet
image_size
| Type | Default |
|---|---|
orig, 4096x4096, large, medium, small, thumb |
orig |
Resolution used for image downloads
{% hint style="info" %}
orig is original quality but it's not always available. You may get 404 - Not Found or 403 - Forbidden errors. The same applies to 4096x4096.
large, medium, or small are always available.
{% endhint %}
threads
| Type | Default |
|---|---|
Bool |
true |
Downloads media from all direct replies of the tweet's author to their own tweet
retweets
| Type | Default |
|---|---|
Bool |
false |
Download media from retweets in the user's timeline
articles
Controls which content to download when CDL finds an article (longer tweets from premium users)
{% hint style="info" %} Scanning the content (text) of articles for URLs is not supported {% endhint %}
cover
Download the cover image of articles
media
Download media files (attachments) in the article