Skip to content

Bulk asynchronous downloader for multiple file hosts

10.4 KB
Raw Blame

disabled

Type Default Additional Info
list[NonEmptyStr] [] This is an AdditiveArg

You can supply a list of crawlers to disable for the current run. This will make CDL completely ignore the crawler, as if the site was not supported. However, links from the site will still be processed by Real-Debrid (if enabled), JDownloader (If enabled) and the Generic crawler (If enabled), in that order.

The list should be valid crawlers names. The name of the crawler is the name of the primary site they support. ex: 4Chan, Mega.nz, Dropbox

You can get the crawler's name from the supported sites page The name of the crawer is the title of their section in the page (in bold).

crawlers:
  disabled: []

generic

Generic crawlers are designed to work on any site that uses a specific framework. Users can supply a list of sites to map to these crawlers, and CDL will then be able to download from them. The URL in the list should be the primary URL of the site. ex: https://forums.docker.com/

Supported generic crawlers:

  • chevereto: This works on any site that uses Chevereto.

  • discourse: This works on any forum that uses Discourse.

  • kvs: Works on any tube site using Kernel Video Sharing. Basically, any site that looks like one of these: https://www.kernel-video-sharing.com/en/themes/, ex: https://www.kvs-demo.com/

  • wordpress_media: This crawler should work on any WordPress site where content primarily consists of images or galleries. The images need to be hosted on the site itself. It requires sites to have a public WordPress REST API.

  • wordpress_html: This works on any WordPress site. It scrapes the actual HTML of the site, which means it works on sites that have embedded third-party media like videos or links to hosting sites. It is always slower than wordpress_media.

chevereto

Type Default
list[HttpURL] []

discourse

Type Default
list[HttpURL] []

kvs

Type Default
list[HttpURL] []

wordpress_media

Type Default
list[HttpURL] []

wordpress_html

Type Default
list[HttpURL] []
crawlers:
  generic:
    chevereto: []
    discourse: []
    kvs: []
    wordpress_html: []
    wordpress_media: []

Bandcamp

formats

Type Default
List of mp3-320, mp3, aac-hi, wav, flac, vorbis, aiff or alas [mp3-320, mp3, aac-hi, wav, flac, vorbis, aiff, alas]

Format to choose for downloads (if available), ordered by preference.

crawlers:
  bandcamp:
    formats:
      - mp3-320
      - mp3
      - aac-hi
      - wav
      - flac
      - vorbis
      - aiff
      - alas

bluesky

crawlers:
  bluesky:
    content_urls: true
    external: true
    reposts: false
    threads: true

external

Type Default
Bool true

Parse and download embeds from third-party sites

content_urls

Type Default
Bool true

Parse and try to download any URL found inside the text of a post.

threads

Type Default
Bool true

Downloads all posts in a thread (All direct replies from OP to their own post)

reposts

Type Default
Bool false

Download media from reposts in the user's timeline

Clyp.it

crawlers:
  clypit:
    prefer_mp3: false

prefer_mp3

Type Default
Bool false

Download audios as .mp3 files even if WAV (high quality) versions are available

Google Drive

crawlers:
  google_drive:
    default_formats:
      docs: docx
      sheets: xslx
      slides: pptx

default_formats

docs

Type Default
docx, odt, rtf, txt, epub, pdf, md or zip docx

Default format for documents (can be overridden per URL with the 'format' query param)

sheets

Type Default
xslx, ods, html, csv or tsv xslx

Default format for spreedsheets (can be overridden per URL with the 'format' query param)

slides

Type Default
pptx or odp pptx

Default format for presentations (can be overridden per URL with the 'format' query param)

OnePace

prefer_dub

Type Default
Bool false

Download episodes with english audio tracks instead of japanese (if available)

Pawchive / OnlyHaven

file

Type Default
Bool true

Download the main file in a post (if any)

attachments

Type Default
Bool true

Download all attachments in a post (may or may not include file)

content_urls

Type Default
Bool true

Scan the description (text) in a post and download any URL found

{% hint style="warning" %} This option slows down scraping significally (50x requests in total) {% endhint %}

expand_posts

Type Default
Bool false

When crawling profiles/favorites/searchs results, make an additional request per post to get the original filenames of files and the content/text in the post. To download deferred big files, this option must be enabled

{% hint style="warning" %} This option slows down scraping significally (100x requests in total) {% endhint %}

embed

Type Default
Bool true

Download the embedded file from third party sites (if any)(mega.nz, pcloud, dropbox, etc..)

Embedded files show up as clickable boxes on the website:

embed preview

crawlers:
  pawchive:
    attachments: true
    content_urls: true
    embed: true
    file: true

PornHub

crawlers:
  pornhub:
    profile_paths:
      - "photos/public"
      - "gifs/public"
      - "videos"
      - "videos/upload"

profile_paths

Type Default
list of NonEmptyStr [photos/public, gifs/public, videos, videos/upload]

Subpaths to scrape when an input URL is a profile's homepage. ex: https://www.pornhub.com/model/<model_name>

Reddit

reddit:
  api: arctic_shift
  content_urls: true

api

Type Default
arctic_shift arctic_shift

Backend API to use to connect to Reddit. The only currently supported API is arctic_shift, which allows retrieving deleted posts and subreddit-specific posts with no authentication.

content_urls

Type Default
Bool true

Scan the description (text) in a post and download any URL found

Tiktok

crawlers:
  tiktok:
    original: false

original

Type Default
Bool false

By default, CDL will download the "optimized for streaming" version of tiktok videos. Setting this option to True will download videos in original (source) quality.

_original will be added as a suffix to their filename.

{% hint style="warning" %} This will make video downloads several times slower

When it is set to false (the default) CDL can download 50 videos with a single request. When it is set to true , CDL needs to make at least 3 requests per video to download them.

There's also a daily limit of the API CDL uses: 5000 requests per day per IP

Setting this option to true will consume the daily limit faster {% endhint %}

Twitter

crawlers:
  twitter:
    articles:
      cover: true
      media: true
    cards: true
    content_urls: true
    image_size: orig
    retweets: false
    threads: true

cards

Type Default
Bool true

Parse and download cards in a post (embeds from third-party sites)

content_urls

Type Default
Bool true

Parse and try to download any URL found inside the text of a tweet

image_size

Type Default
orig, 4096x4096, large, medium, small, thumb orig

Resolution used for image downloads

{% hint style="info" %} orig is original quality but it's not always available. You may get 404 - Not Found or 403 - Forbidden errors. The same applies to 4096x4096.

large, medium, or small are always available. {% endhint %}

threads

Type Default
Bool true

Downloads media from all direct replies of the tweet's author to their own tweet

retweets

Type Default
Bool false

Download media from retweets in the user's timeline

articles

Controls which content to download when CDL finds an article (longer tweets from premium users)

{% hint style="info" %} Scanning the content (text) of articles for URLs is not supported {% endhint %}

cover

Download the cover image of articles

media

Download media files (attachments) in the article