Skip to content

Bulk asynchronous downloader for multiple file hosts

docs: changelog and wiki updates for v7.0.0 release (#1069)

NTFSvolume4fdfe8006480
diff --git a/CHANGELOG.md b/CHANGELOG.md
index 4b481fd6d..67772a2ae 100644
--- a/CHANGELOG.md
+++ b/CHANGELOG.md
@@ -22,6 +22,79 @@ All notable changes to this project will be documented here. For more details, v
The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.1.0/),
and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0.html).
+## [7.0.0] - 2025-07-04
+
+### Added
+
+- Camwhores.tv support
+- Flugel-anime support
+- Missav support
+- Beeg.com support
+- efukt support
+- Vipergirls support
+- Initial HLS support (requires ffmpeg)
+- Support for any WordPress media site. See: <https://script-ware.gitbook.io/cyberdrop-dl/reference/configuration-options/global-settings/generic_crawlers_instances>
+- Support for any Discourse site See: <https://script-ware.gitbook.io/cyberdrop-dl/reference/configuration-options/global-settings/generic_crawlers_instances>
+- Add a config option to disable specific crawlers: <https://script-ware.gitbook.io/cyberdrop-dl/reference/configuration-options/global-settings/general#disable_crawlers>
+- Add new `ssl_context` config option: <https://script-ware.gitbook.io/cyberdrop-dl/reference/configuration-options/global-settings/general#ssl_context>
+- Add a config option to map unknown URLs to generic crawlers: <https://script-ware.gitbook.io/cyberdrop-dl/reference/configuration-options/global-settings/generic_crawlers_instances>
+
+### Changed
+
+- Date parsing errors will be logged as warnings, not errors
+- Use an asynchronous DNS resolver
+- Use native system certificates by default, with fallback to `certifi`. See: <https://script-ware.gitbook.io/cyberdrop-dl/reference/configuration-options/global-settings/general#ssl_context>
+- Some crawlers have a new logic to create a filename. The name format for newly downloaded files may not match previous versions.
+- Config options that take format strings will be validated at startup
+- URLs will only be processed once, ignoring duplicates, even if they come from different crawlers. ex: If you have a dropbox URL in `URLs.txt` and CDL finds that same URL while crawling a forum, it will silently ignore the URL from the forum.
+- General performance improvements
+- New blocked domains:
+
+> gfycat
+> ko-fi.com
+> paypal.me
+> amazon.com
+> throne.com
+> youtu.be
+> youtube.com
+> linktr.ee
+> beacons.page
+> beacons.ai
+> allmylinks.com
+
+### Deprecated
+
+- Support for 32-bit Python and 32-bit operating systems in general is deprecated. On a future version, the start scripts will explicitly fail when trying to install on 32-bit systems. This deprecation only applies to the start scripts. 32-bit versions can still be installed manually via `pip`, but the user may need to compile any missing wheels.
+
+### Removed
+
+- `no-ui` option
+- `allow_insecure_connections` option
+- Importing cookies from multi browsers is no longer supported. CDL will only try to extract cookies from a single browser. The `browsers` option was renamed `browser`
+
+### Fixed
+
+- `ImportError: cannot import name 'MethodType'` on startup
+- Handle errors while processing confirmation links (Xenforo)
+- Redgifs not being downloaded (Xenforo)
+- Attachments not being downloaded (Xenforo)
+- Posts not being fully scraped if a single link in them failed (Xenforo)
+- Download from user profiles (xHamster)
+- Folder downloads (Cyberfile)
+- Do not crash when using the `Create file hashes` option from the main menu
+- Do not download "related" or "favorites" videos from a profile (PornHub)
+- `KeyError` when using the `--separate-posts` option
+- Using Real Debrid no longer causes CDL to crash
+- Image downloads (HotPic)
+- Do not try to download emojis (twitter_images)
+- Fix albums downloads (saint)
+- Handle all possible response errors (reddit)
+- Do not use reddit's `user-agent` for every site
+- Fix `unclose client session` error at the end of a run
+- Fix parsing of dropbox folders URLS
+- Handle embeded URLS (box.com)
+- [Regression] Fixed free space check when using network drives (windows)
+
## [6.10.1] - 2025-06-15
### Changed
@@ -39,7 +112,6 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
- Remove invalid characters from filenames (Cyberfile)
- Discord server name (Kemono)
-
## [6.10.0] - 2025-05-30
### Added
diff --git a/docs/SUMMARY.md b/docs/SUMMARY.md
index 0319a45f0..f2ac79812 100644
--- a/docs/SUMMARY.md
+++ b/docs/SUMMARY.md
@@ -8,7 +8,7 @@
* [macOS](getting-started/python-install/macos.md)
* [Cyberdrop-DL Install](getting-started/cyberdrop-dl-install.md)
* [Running CDL on Android](getting-started/cyberdrop-dl-on-android.md)
-* [Transition to V6](transition-to-v6.md)
+* [Transition to V7](transition-to-v6.md)
* [Frequently Asked Questions](frequently-asked-questions.md)
* [Migration to Cyberdrop-DL-Patched](migration-to-cyberdrop-dl-patched.md)
* [SimpCity Support Dropped](simpcity-support-dropped.md)
@@ -34,6 +34,7 @@
* [General](reference/configuration-options/global-settings/general.md)
* [Rate Limiting Options](reference/configuration-options/global-settings/rate-limiting-options.md)
* [UI Options](reference/configuration-options/global-settings/ui-options.md)
+ * [Generic Crawlers](reference/configuration-options/global-settings/generic_crawlers_instances.md)
* [CLI Arguments](reference/cli-arguments.md)
* [Notifications](reference/notifications.md)
* [Changelog](https://github.com/jbsparrow/CyberDropDownloader/blob/master/CHANGELOG.md)
diff --git a/docs/frequently-asked-questions.md b/docs/frequently-asked-questions.md
index 3227bc86a..9f7ba6747 100644
--- a/docs/frequently-asked-questions.md
+++ b/docs/frequently-asked-questions.md
@@ -38,9 +38,16 @@ if you are on macOS you may need to do the following:
pip3 install --upgrade cyberdrop-dl-patched
```
+</details>
+<details>
+<summary> Why do i get DDoS-Guard Error downloading from <X> sites? </summary>
+
+You may need to import cookies. Follow the instructions here: [How to extract cookies (DDoSGuard or login errors) #839](https://github.com/jbsparrow/CyberDropDownloader/discussions/839)
+
</details>
<details>
+
<summary> Where is the downloader.log file? </summary>
If you are running using one of the new start scripts it'll be in `./AppData/configs/<config>/logs/`
diff --git a/docs/getting-started/README.md b/docs/getting-started/README.md
index 9dc3b3ad2..551974bb4 100644
--- a/docs/getting-started/README.md
+++ b/docs/getting-started/README.md
@@ -16,6 +16,10 @@ layout:
# Getting Started
+{% hint style="warning" %}
+Cyberdrop-DL requires python 3.11, 3.12 or python 3.13
+{% endhint %}
+
## Installing Python
Cyberdrop-DL is written in Python. You'll need Python installed on your system to be able to run it. If you're using Linux or macOS, Python is likely already included by default. However, if you're on Windows or need a different version, you can easily download and install Python from the official website
diff --git a/docs/migration-to-cyberdrop-dl-patched.md b/docs/migration-to-cyberdrop-dl-patched.md
index d87ddf5b2..6e1fb4bcf 100644
--- a/docs/migration-to-cyberdrop-dl-patched.md
+++ b/docs/migration-to-cyberdrop-dl-patched.md
@@ -14,9 +14,16 @@ layout:
visible: true
---
-# Migration to Cyberdrop-DL-Patched
+# Move to Cyberdrop-DL-Patched
+
+If you used the original version of the package (`cyberdrop-dl` instead of `cyberdrop-dl-patched`), you can follow the steps below to migrate.
+
+{% hint style="warning" %}
+You may need to adjust your config to make sure it's compatible with the newer versions.
+
+See: [Transition to V7](https://script-ware.gitbook.io/cyberdrop-dl/transition-to-v7)
+{% endhint %}
-To migrate to the patched version of Cyberdrop-DL, you can follow the steps below.
## If you installed on your own using pip,
diff --git a/docs/reference/cli-arguments.md b/docs/reference/cli-arguments.md
index 1de47f7f0..2053e5945 100644
--- a/docs/reference/cli-arguments.md
+++ b/docs/reference/cli-arguments.md
@@ -17,16 +17,16 @@ layout:
# CLI Arguments
{% hint style="info" %}
-Anything input as a CLI argument will take priority over config values.
+CLI inputs always take priority over config values.
{% endhint %}
{% hint style="info" %}
-Use `-` instead of `_` to separate words in an option name when using it as a CLI argument: Ex: `auto-dedupe` instead of `auto_dedupe`
+Use `-` instead of `_` to separate words in an config option name when using it as a CLI argument: Ex: `auto_dedupe` needs to be `auto-dedupe` when using it via the CLI
{% endhint %}
-You can pass any of the **Config Settings** and **Global Settings** options as a cli argument for the program
+You can pass any of the **Config Settings** and **Global Settings** options as a cli argument for the program.
-For items not explained below, you can find their counterparts in the configuration options to see what they do
+For items not explained below, you can find their counterparts in the configuration options to see what they do.
## CLI only arguments
@@ -36,7 +36,7 @@ For items not explained below, you can find their counterparts in the configurat
| ------ | ----------------------------- |
| `Path` | `<Current Working Directory>` |
-Folder where Cyberdrop-DL will store its data files.
+Folder where Cyberdrop-DL will store it's database, cache and config files.
### `completed-after`
@@ -119,6 +119,7 @@ Download TikTok audios from posts and save them as separate files
Max number of links to retry. Using `0` means no limit
+<!-- This needs textual
### `no-textual-ui`
| Type | Default | Action |
@@ -126,6 +127,7 @@ Max number of links to retry. Using `0` means no limit
| `BoolFlag` | `False` | `store_true` |
Disable CDL's textual UI (TUI with buttons and mouse support)
+-->
### `portrait`
@@ -171,7 +173,7 @@ Shows a list of all supported sites and exits
| Type | Default |
| ---------- | ------- |
-| `StrEnum` | `FULLSCREEN` |
+| `CaseInsensitiveStrEnum` | `FULLSCREEN` |
UI can have 1 of these values:
diff --git a/docs/reference/configuration-options/authentication.md b/docs/reference/configuration-options/authentication.md
index 06a5ec407..404ddebc6 100644
--- a/docs/reference/configuration-options/authentication.md
+++ b/docs/reference/configuration-options/authentication.md
@@ -27,25 +27,13 @@ Once you have put your `session` cookie into the authentication file, you can ad
<summary>Forums</summary>
-In order to scrape links/content from forums, you need to provide Cyberdrop-DL with your login details so it can access the website. This section also includes cookies for the support forums.
+{% hint style="warning" %}
+Logging to forums with `Authentification` settings was deprecated in v6.7.0
-If you use the cookie extractor to load the `XF_User` cookies into the program, you don't need to provide these credentials. If you ever log out of the forum in your browser though, you will need to use the cookie extractor again to get new cookies.
+You need to use cookie files.
-It is best to leave the authentication parameter for SimpCity blank, as they have made their forum public and have asked users scraping the website not to use logged in users.
-
-In order to set specific authentication values for a config instead of the global values, make an `authentication.yaml` file in the config folder.
-
-## `xf_user_cookie`
-
-This is the value for the `XF_User` cookie mentioned above. If you want to only use credentials, you can leave this blank.
-
-## `username`
-
-This is your username for the forum. Again, if you use cookies, you don't need to provide this.
-
-## `password`
-
-This is your password for the forum. Again, if you use cookies, you don't need to provide this.
+See: [How to extract cookies (DDoSGuard or login errors) #839](https://github.com/jbsparrow/CyberDropDownloader/discussions/839)
+{% endhint %}
</details>
diff --git a/docs/reference/configuration-options/global-settings/general.md b/docs/reference/configuration-options/global-settings/general.md
index 2f30be6f8..92283018e 100644
--- a/docs/reference/configuration-options/global-settings/general.md
+++ b/docs/reference/configuration-options/global-settings/general.md
@@ -3,23 +3,22 @@ description: These are some general settings that will be used regardless of whi
---
# General
-## `allow_insecure_connections`
+## `disable_crawlers`
-| Type | Default |
-| ------ | ------- |
-| `bool` | `false` |
+| Type | Default | Additional Info |
+| ------------------- | ------- | -------------------------------------------------------------------- |
+| `list[NonEmptyStr]` | `[]` | This is an [`AdditiveArg`](../special_setting_types.md#additiveargs) |
-Setting this to `true` will allow the program to connect to websites without SSL encryption (insecurely).
+You can supply a list of crawlers to disable for the current run. This will make CDL completely ignore the crawler, as if the site was not supported. However, links from the site will still be processed by Real-Debrid (if enabled), Jdownloader (If enabled) and the Generic crawler (If enabled), in that order.
-{% hint style="danger" %}
-This will make the connection insecure, and sensitive data may be exposed. You should only enable this option if you know what you are doing. For your safety, is recommended to always use a secure HTTPS connection to protect your privacy.
-{% endhint %}
+The list should be valid crawlers names. The name of the crawler is the name of the primary site they support. ex: `4Chan`, `Bunkrr`, `Dropbox`
+Crawlers names correspond to the column `site` in the [supported sites page](https://script-ware.gitbook.io/cyberdrop-dl/reference/supported-websites#supported-sites).
## `enable_generic_crawler`
| Type | Default |
| ------ | ------- |
-| `bool` | `true` |
+| `bool` | `true` |
CDl has a generic crawler that will try to download from unsupported sites. Setting this to `false` will disable it.
@@ -33,10 +32,15 @@ CDL will still try to download from unsupported URLs if the last part of the URL
| ------------------- | ------- |
| `HttpURL` or `null` | `null` |
-[FlareSolverr](https://github.com/FlareSolverr/FlareSolverr) is a proxy server to bypass Cloudflare and DDoS-GUARD protection. The provided value must be a valid `http` URL of an existing flaresolverr instance. Ex: `http://ip:port`
+[FlareSolverr](https://github.com/FlareSolverr/FlareSolverr) is a proxy server to bypass Cloudflare and `DDoS-Guard` protection. The provided value must be a valid `http` URL of an existing flaresolverr instance. Ex: `http://192.168.1.44:4000`
+
+{% hint style="info" %}
+`0.0.0.0` is NOT a valid IP address. To set up a flaresolverr instance running on the same machine as CDL, use `127.0.0.1` as the IP
+{% endhint %}
{% hint style="warning" %}
-This wiki does not covert flaresolverr setup process. If you need help, refer to their documentation. Please refrain from opening issues related to flaresolverr.
+This wiki does not cover flaresolverr setup process. If you need help, refer to their documentation. Please do not open issues related to flaresolverr or `DDoS-Guard`.
+See: [How to extract cookies (DDoSGuard or login errors) #839](https://github.com/jbsparrow/CyberDropDownloader/discussions/839) for alternatives using cookies
{% endhint %}
## `max_file_name_length`
@@ -45,7 +49,7 @@ This wiki does not covert flaresolverr setup process. If you need help, refer to
| ------------- | ------- |
| `PositiveInt` | `95` |
-This is the maximum number of characters allowable in a filename.
+This is the maximum number of characters filename should have. CDL will truncate filenames longer that this.
## `max_folder_name_length`
@@ -53,8 +57,9 @@ This is the maximum number of characters allowable in a filename.
| ------------- | ------- |
| `PositiveInt` | `95` |
-This is the maximum number of characters allowable in a folder name.
+This is the maximum number of characters a folder should have. CDL will truncate folders longer that this.
+<!-- This needs Textual
## `pause_on_insufficient_space`
| Type | Default |
@@ -66,6 +71,7 @@ Setting this to `true` will make CDL pause if there not enough free space availa
{% hint style="info" %}
CDL will only pause once. After the user resumes, every `InsufficientFreeSpaceError` will be propagated
{% endhint %}
+-->
## `proxy`
@@ -73,7 +79,7 @@ CDL will only pause once. After the user resumes, every `InsufficientFreeSpaceEr
| ------------------- | ------- |
| `HttpURL` or `null` | `null` |
-The proxy you want CDL to use. Only `http` proxies are supported. Ex: `https://user:pass@ip:port`
+The proxy you want CDL to use. Only `http` proxies are supported. Ex: `https://user:password@ip:port`
## `required_free_space`
@@ -83,14 +89,41 @@ The proxy you want CDL to use. Only `http` proxies are supported. Ex: `https://u
This is the minimum amount of free space require to start new downloads.
+{% hint style="info" %}
+If you set a value lower than `512MB`, CDL will override it with `512MB`
+{% endhint %}
+
+## `ssl_context`
+
+| Type | Default |
+| --------------------- | -------------------- |
+| `NonEmptyStr` or None | `truststore+certifi` |
+
+Context that will used to verify SSL connections. Valid values are:
+
+- `truststore`: Will use certificates already included with the OS
+
+- `certifi`: Will use certificates bundled with the `certifi` version available at the release of the current CDL version
+
+- `truststore+certifi`: Will use certificates already included with the OS, with a fallback to `certifi` for missing certificates
+
+- `None`: Will completly disable SSL verification, allowing secure connections.
+
+Setting this to `None` will allow the program to connect to websites without SSL encryption (insecurely).
+
+{% hint style="danger" %}
+Sensitive data may be exposed using an insecure connection. For your safety, is recommended to always use a secure HTTPS connection.
+{% endhint %}
+
## `user_agent`
| Type | Default |
| ------------- | ---------------------------------------------------------------------------------- |
| `NonEmptyStr` | `Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:133.0) Gecko/20100101 Firefox/133.0` |
-The user agent is the signature of your browser, determining how it is presented to the websites you visit.. You can google "what is my user agent" to see what yours may be.
+The user agent is the signature of your browser. Some sites use it to identify if the requests come from a human or a robot.
+You can google "what is my user agent" to get yours.
{% hint style="info" %}
-If you use flaresolverr, this value must match with flaresolverr user agent for its cookies to work
+If you use flaresolverr, this value MUST match with flaresolverr's user agent. Otherwise, flaresolverr cookies won't work
{% endhint %}
diff --git a/docs/reference/configuration-options/global-settings/generic_crawlers_instances.md b/docs/reference/configuration-options/global-settings/generic_crawlers_instances.md
new file mode 100644
index 000000000..cb65efae9
--- /dev/null
+++ b/docs/reference/configuration-options/global-settings/generic_crawlers_instances.md
@@ -0,0 +1,34 @@
+---
+description: These are the options to define generic crawlers.
+
+---
+
+CDL has some "generic" crawlers disabled by default. Generic crawlers are designed to work on any site that uses a **specific** framework. Users can supply a list of sites to map to these crawlers, and CDL will then be able to download from them. The URL in the list should be the primary URL of the site. ex: `https://forums.docker.com/`
+
+Currently, there are three generic crawlers:
+
+- `wordpress_media`: This crawler should work on any [WordPress](https://wordpress.com/) site where content primarily consists of images or galleries. The images need to be hosted on the site itself. It requires sites to have a public WordPress REST API.
+
+- `wordpress_html`: This works on any WordPress site. It scrapes the actual HTML of the site, which means it works even on sites that have embedded third-party media like videos or links to hosting sites. It is always slower than `wordpress_media`.
+
+- `discourse`: This works on any forum that uses [Discourse](https://www.discourse.org/).
+
+# generic_crawlers_instances
+
+## `wordpress_media`
+
+| Type | Default |
+| --------------- | ------- |
+| `list[HttpURL]` | `[]` |
+
+## `wordpress_html`
+
+| Type | Default |
+| --------------- | ------- |
+| `list[HttpURL]` | `[]` |
+
+## `discourse`
+
+| Type | Default |
+| --------------- | ------- |
+| `list[HttpURL]` | `[]` |
diff --git a/docs/reference/configuration-options/global-settings/rate-limiting-options.md b/docs/reference/configuration-options/global-settings/rate-limiting-options.md
index c3f5f0b7c..72a6996e0 100644
--- a/docs/reference/configuration-options/global-settings/rate-limiting-options.md
+++ b/docs/reference/configuration-options/global-settings/rate-limiting-options.md
@@ -20,7 +20,7 @@ The number of seconds to wait while connecting to a website before timing out
| ------------- | ------- |
| `PositiveInt` | `5` |
-The number of download attempts per file. Regardless of this value, some conditions (such as a 404 HTTP status) will cause a file to not be retried at all.
+The number of download attempts per file. Some conditions are never retried (such as a `404` HTTP status)
## `download_delay`
@@ -32,9 +32,11 @@ This is the number of seconds to wait between downloads to the same domain.
Some domains have internal limits set by the program, which can not be modified:
-- `bunkrr`: 1
-- `cyberfile.me`: 1
-- `pixeldrain` : 2
+- `bunkr`: 0.5,
+- `coomer`: 0.5,
+- `kemono`: 0.5,
+- `nhentai.net`: 1,
+
## `download_speed_limit`
@@ -110,7 +112,15 @@ This is the maximum number of files that can be downloaded simultaneously.
This is the maximum number of files that can be downloaded from a single domain simultaneously.
-Some domains have internal limits set by the program, such as `bunkrr`, `cyberfile.me`, etc.
+Some domains have internal limits set by the program, which can not be modified:
+
+- `bunkr`: 1
+- `cyberdrop`: 1
+- `cyberfile`: 1
+- `noodlemagazine`: 2
+- `4chan`: 1
+- `pixeldrain`: 2
+- `xxxbunker`: 2
## `rate_limit`
@@ -130,4 +140,4 @@ This is the maximum number of requests that can be made by the program per secon
| ------------- | ------- |
| `PositiveInt` | `300` |
-The number of seconds to wait while reading data from a website before timing out. If it's a download, it will be retried and won't count against the `download_attempts` limit.
+The number of seconds to wait while reading data from a website before timing out.
diff --git a/docs/reference/configuration-options/settings/browser_cookies.md b/docs/reference/configuration-options/settings/browser_cookies.md
index 4cbeb8483..87c6ca02a 100644
--- a/docs/reference/configuration-options/settings/browser_cookies.md
+++ b/docs/reference/configuration-options/settings/browser_cookies.md
@@ -14,13 +14,13 @@ The `user-agent` config value **MUST** match the `user-agent` of the browser fro
Toggles automatic import of cookies at the start of each run
-## `browsers`
+## `browser`
| Type | Default |
| ---------------- | ---------- |
-| `list[BROWSERS]` | `[chrome]` |
+| `BROWSER` | `firefox` |
-List of browsers to use for extraction. Each item must be a supported browser's name, separated by commas
+A browser to use for extraction. Browser must be a supported browser's name.
### Supported Browsers
@@ -40,13 +40,6 @@ List of browsers to use for extraction. Each item must be a supported browser's
| Vivaldi | :x: | :white_check_mark: | :white_check_mark: |
| W3M | :x: | :white_check_mark: | :x: |
-{% hint style="info" %}
-If cookies exists on multiple selected browsers, the cookies from the last browser in the list will have priority
-{% endhint %}
-
-{% hint style="info" %}
-If the value entered is `null` or an empty list, no cookies will be extracted from any browser
-{% endhint %}
## `sites`
@@ -60,6 +53,8 @@ List of domains to extract cookies from. You can put any domain on the list, eve
If cookie extraction fails, you can manually extract the cookies from your browser using tools like [cookie-editor](https://cookie-editor.com) and save them at `AppData/Cookies/<site_name>.txt`. The file must be a Netscape formatted cookie file. You can use any name for the file as long as it has a `.txt` extension.
+See: [How to extract cookies (DDoSGuard or login errors) #839](https://github.com/jbsparrow/CyberDropDownloader/discussions/839) for detailed instructions
+
{% hint style="info" %}
Multiple cookie files are supported. You could have a `SocialMediaGirls.txt` file and a `cyberdrop.txt` file, for example
{% endhint %}
diff --git a/docs/reference/configuration-options/settings/download_options.md b/docs/reference/configuration-options/settings/download_options.md
index 732634374..a90239f9a 100644
--- a/docs/reference/configuration-options/settings/download_options.md
+++ b/docs/reference/configuration-options/settings/download_options.md
@@ -30,7 +30,7 @@ However, to make sure the program will not run endlessly, there are certain situ
| ------ | ------- |
| `bool` | `false` |
-By default the program will do it's absolute best to try and find when a file was uploaded. It'll then set the `last modified`, `last accessed` and `created` dates on the file to match.
+By default the program will do it's absolute best to try and find the upload date of a file. It'll then set the `last modified` and `last accessed` dates on the file to match. On Windows and macOS, it will also try to set the `created` date.
Setting this to `true` will disable this function, and the dates for those metadata entries will be the date the file was downloaded.
@@ -75,6 +75,7 @@ Limit **FORUM** scrape to 15 posts max, grab all links and media within those po
--maximum-number-of-children 15 0 10
```
+
{% endtab %}
{% tab title="example 2" %}
@@ -88,16 +89,20 @@ Only grab the first link from each post in a forum, but that link will have no `
{% tab title="example 3" %}
Only grab the first **POST** / **ALBUM** from a **FILE_HOST_PROFILE**
+
```shell
--maximum-number-of-children 0 0 1
```
+
{% endtab %}
{% tab title="example 4" %}
No **FORUM** limit, no **FORUM_POST** limit, no **FILE_HOST_PROFILE** limit, maximum of 20 items from any **FILE_HOST_ALBUM**:
+
```shell
--maximum-number-of-children 0 0 0 20
```
+
{% endtab %}
{% endtabs %}
@@ -115,7 +120,9 @@ Setting this to `true` will remove the "(DOMAIN)" portion of folder names on new
| ------ | ------- |
| `bool` | `false` |
-Setting this to `true` will remove the alphanumeric ID added to the end of filenames by some websites like `cyberdrop.me`.
+Setting this to `true` will remove the alphanumeric ID added to the end of filenames by some websites.
+
+This option only works for URLs from `cyberdrop.me` at the moment.
Multipart archive filenames will be corrected to follow the proper naming pattern for their format.
@@ -127,7 +134,16 @@ Supported formats: `.rar` `.7z` `.tar` `.gz` `.bz2` `.zip`
| ------ | ------- |
| `bool` | `false` |
-Setting this to `true` will prevent Cyberdrop-DL to scrape entire thread if an individual post link was provided as input.
+Setting this to `true` will prevent Cyberdrop-DL from scraping an entire thread if the input URL had an specific post in it.
+
+CDL will only download files within that post.
+
+For most forum sites, the post id is part of the fragment in the URL.
+
+ex: `/thread/iphone-16-16e-16-plus-16-pro-16-promax.256047/page-64#post-7512404` has a post id of `7512404`
+
+If `scrape_single_forum_post` is `false`, CDL will download all post in the thread, from post `7512404` until the last post
+If `scrape_single_forum_post` is `true`, CDL will only download file within post `7512404` itself and stop.
## `separate_posts`
@@ -135,7 +151,19 @@ Setting this to `true` will prevent Cyberdrop-DL to scrape entire thread if an i
| ------ | ------- |
| `bool` | `false` |
-Setting this to `true` will separate content from forum and site posts into separate folders. Only affects sites which have 'posts': `Forums`, `reddit`, `coomer`, `kemono` and `nekohouse`.
+Setting this to `true` will separate content from forum and site posts into separate folders.
+
+This option only works with sites that have 'posts':
+
+- `Forums`
+- `Discourse`
+- `reddit`
+- `coomer`, `kemono` and `nekohouse`.
+
+For some sites, this value is hardcorded to `true` because each post is always an individual page:
+
+- `Wordpress`
+- `eFukt`
## `separate_posts_format`
@@ -147,25 +175,40 @@ This is the format for the directory created when using `--separate-posts`.
Unique Path Flags:
-> `date`: date of the post
+> `date`: date of the post. This is a python `datetime` object
>
-> `number`: post number
+> `id`: The post id. This is always a `string`, even if some sites use numbers
>
-> `id`: same as `number`
+> `number`: This no longer means anything. Currently, it always has the same value as `id`
>
-> `title`: post title
+> `title`: post title. This is a `string`
{% hint style="warning" %}
Not all sites support all possible flags. Ex: Posts from reddit only support the `title` flag
+If you use a format with a field that the site does not support, CDL will replace it with `UNKNOWN_<FIELD_NAME>`
+
+ex: using the format `reddit post #{number}` -> `reddit post #UNKNOWN_ID`
{% endhint %}
Setting it to `{default}` will use the default format, which is different for each crawler:
-| Site | Default Format |
-| --------------------------------- | ------------------ |
-| `Coomer`, `Kemono` an `Nekohouse` | `{date} - {title}` |
-| `Forums` | `post-{number}` |
-| `Reddit` | `{title}` |
+| Site | Default Format |
+| ------------------------------------- | ---------------------------------- |
+| `Coomer`, `Kemono` and `Nekohouse` | `{date} - {title}` |
+| `Forums (Xenforo/vBulletin/Invision)` | `{date} - {id} - {title}` |
+| `Discourse` | `{date} - {id} - {title}` |
+| `Reddit` | `{title}` |
+| `WordPress` | `{date:%Y-%m-%d} - {id} - {title}` |
+| `eFukt` | `{date:%Y-%m-%d} {title}` |
+
+A date without a `format_spec` defaults to ISO 8601 format
+
+You can use any valid format string supported by python, with the following restrictions:
+
+- You can not have positional arguments in the format string. ex: `post {0} from date {1}`
+- You can not have unnamed fields in the format string. ex: `post {} from date {}`
+- You can not perform operations within the format string. ex: `post {id + 1} from date {date}`
+- All the fields named in the format string must be valid fields for that format option. CDL will validate this at startup
## `skip_download_mark_completed`
@@ -183,29 +226,34 @@ Setting this to `true` will skip the download process for every file and mark th
Setting this to `true` will skip downloading files from any referer that have been scraped before. The file (s) will always be skipped, regardless of whether the referer was successfully scraped or not
-
## `maximum_thread_depth`
-| Type | Default |
-|----------------|----------|
-| `NonNegativeInt` | 0 |
+| Type | Default |
+| ---------------- | ------- |
+| `NonNegativeInt` | 0 |
{% hint style="warning" %}
-It is not recommended to set this above the default value of 0, as there is a high chance of infinite nesting in certain cases.
+It is not recommended to set this above the default value of `0`, as there is a high chance of infinite nesting in certain cases.
For example, when dealing with Megathreads, if a Megathread is linked to another Megathread, you could end up scraping an undesirable amount of data.
{% endhint %}
Restricts how many levels deep the scraper is allowed to go while scraping a thread
-Values
-0: No nesting allowed, only the top level thread is allowed
-None: unlimited parents
-1>: limits to the value given
+Values:
+
+- `0`: No nesting allowed, only the top level thread is allowed
+- `1+`: limits to the value given
+
+{% hint style="info" %}
+This setting is hardcoded to `false` for Discourse sites
+{% endhint %}
### Example
+
Consider CDL finds the following sub-threads while scraping an input URL:
-\```
+
+```bash
└── thread_01
├── thread_02
├── thread_03
@@ -218,7 +266,8 @@ Consider CDL finds the following sub-threads while scraping an input URL:
├── thread_07
│ └── thread_12
└── thread_08
-\```
+```
+
- With `maximum_thread_depth` = 0, CDL will only download files in `thread_01`, all the other threads will be ignored
-- With `maximum_thread_depth` = 1, CDL will only download files in `thread_01` to `thread_08`. `thread_09` to `thread_12` will be ignored
+- With `maximum_thread_depth` = 1, CDL will only download files in `thread_01` to `thread_08`. All threads from `thread_09` to `thread_12` will be ignored
- With `maximum_thread_depth` >= 2, CDL will download files from all the threads in this case
diff --git a/docs/reference/configuration-options/settings/dupe_cleanup_options.md b/docs/reference/configuration-options/settings/dupe_cleanup_options.md
index 9743f6492..bc3fb7a23 100644
--- a/docs/reference/configuration-options/settings/dupe_cleanup_options.md
+++ b/docs/reference/configuration-options/settings/dupe_cleanup_options.md
@@ -17,7 +17,7 @@ Cyberdrop-DL maintains an internal database of all downloaded files, indexed by
If enabled, calculates the `md5` hash for every downloaded file and store it in the internal database. This allows the program to match files with sites that provide hash information before downloading them, skipping the download altogether if a match is found
{% hint style="warning" %}
-`md5` was the default hashing algorithm of Cyberdrop-DL V5. If you have a V5 database that you'd like to import into V6, it's recommended to enable `md5` hashing to ensure new downloads match previously hashed files.
+`md5` was the default hashing algorithm of Cyberdrop-DL V5. If you have a V5 database that you'd like to import into a newer version, it's recommended to enable `md5` hashing to ensure new downloads match previously hashed files.
{% endhint %}
## `add_sha256_hash`
@@ -36,7 +36,7 @@ If enabled, calculates the `sha256` hash for every downloaded file and store it
Enables deduping files functionality. Needs `hashing` to be enabled
-The auto_dedupe feature scans the database for files with matching hashes and sizes, automatically deleting any duplicates and retaining only the oldest copy.
+The `auto_dedupe` feature scans the database for files with matching hashes and sizes, automatically deleting any duplicates and retaining only the oldest copy.
Deletion will only occur if two or more matching files are found during the database search.
diff --git a/docs/reference/configuration-options/settings/file_size_limits.md b/docs/reference/configuration-options/settings/file_size_limits.md
index bb319db37..2923de7e5 100644
--- a/docs/reference/configuration-options/settings/file_size_limits.md
+++ b/docs/reference/configuration-options/settings/file_size_limits.md
@@ -2,7 +2,7 @@
You can provide the maximum and minimum file size for each file "type".
-All options on this category take a `ByteSize` input ([more info here](../special_setting_types.md)).
+All options on this category take a `ByteSize` input ([more info here](../special_setting_types.mds#bytesize)).
| Type | Default |
| ---------- | ------- |
diff --git a/docs/reference/configuration-options/settings/logs.md b/docs/reference/configuration-options/settings/logs.md
index fbb06b537..cdc5073cb 100644
--- a/docs/reference/configuration-options/settings/logs.md
+++ b/docs/reference/configuration-options/settings/logs.md
@@ -42,7 +42,7 @@ Line width to use when writing to the main log file
| --------------------- | ------- |
| `timedelta` or `null` | `null` |
-With `rotate_logs` enabled, this setting specifies the retention period for log files before they are deleted
+With `rotate_logs` enabled, this setting specifies the retention period for log files before they are deleted.
- A `timedelta` input is expected to be a valid ISO 8601 timespan, ex: `P10DT2H30M10S`
- An `int` input is assumed to be the number of days
diff --git a/docs/reference/configuration-options/settings/runtime_options.md b/docs/reference/configuration-options/settings/runtime_options.md
index 734e986b3..2b5efd647 100644
--- a/docs/reference/configuration-options/settings/runtime_options.md
+++ b/docs/reference/configuration-options/settings/runtime_options.md
@@ -11,7 +11,7 @@ These are higher level options that effect the overarching functions of the prog
Same as `log_level` but it controls which messages are shown on the console.
{% hint style="info" %}
-This option is ignored unless CDL is running with `--no-ui`
+This option is ignored unless CDL is running with `--ui DISABLED`
{% endhint %}
## `deep_scrape`
@@ -36,9 +36,10 @@ For example, scraping an album normally takes one single request. However, with
| ------ | ------- |
| `bool` | `false` |
-The program will leave partial files alone as they will be used to resume downloads on subsequent runs.
+Files downloaded by CDL have a `.part` extension. CDL only changes the extension to the original one after a successful download.
+This allows CDL to resume downloads on subsequent runs.
-Setting this to `true` will remove any partial downloads from the download folder.
+Setting this to `true` will delete any `.part` files in the download folder.
## `ignore_history`
@@ -46,9 +47,9 @@ Setting this to `true` will remove any partial downloads from the download folde
| ------ | ------- |
| `bool` | `false` |
-By default, the program tracks your downloads to prevent downloading the same files multiple times, helping to save time and reduce strain on the servers you're downloading from.
+By default, the program tracks your downloads in a database to prevent downloading the same files multiple times, to save time and reduce strain on the servers you're downloading from.
-Setting this to `true` will cause the program to ignore the history, and will allow you to re-download files.
+Setting this to `true` will cause the program to ignore the database, and will allow you to re-download files.
## `jdownloader_autostart`
@@ -58,7 +59,9 @@ Setting this to `true` will cause the program to ignore the history, and will al
Setting this to `true` will make jdownloader start downloads as soon as they are sent.
+{% hint style="info" %}
This option has no effect unless `send_unsupported_to_jdownloader` is `true`
+{% endhint %}
## `jdownloader_download_dir`
@@ -68,7 +71,9 @@ This option has no effect unless `send_unsupported_to_jdownloader` is `true`
The `download_dir` jdownloader will use. A `null` value (the default) will make jdownloader use the same `download_dir` as Cyberdrop-DL. Use this option as path mapping when jdownloader is running on a different host / docker.
+{% hint style="info" %}
This option has no effect unless `send_unsupported_to_jdownloader` is `true`
+{% endhint %}
## `jdownloader_whitelist`
@@ -76,9 +81,11 @@ This option has no effect unless `send_unsupported_to_jdownloader` is `true`
| ------------------- | ------- |
| `list[NonEmptyStr]` | `[]` |
-List of domain names. An unsupported URL will only be sent to jdownloader if its host is found on the list. An empty whitelist (the default) will disable this functionality, sending any unsupported URL to jdownloader
+List of domain names. An unsupported URL will only be sent to jdownloader if its host is found on the list. An empty whitelist (the default) will disable this functionality, sending any unsupported URL to jdownloader.
+{% hint style="info" %}
This option has no effect unless `send_unsupported_to_jdownloader` is `true`
+{% endhint %}
## `log_level`
@@ -98,7 +105,21 @@ Defines the logging level for messages, according to [Python logging levels](htt
| `CRITICAL` | 50 | Fatal error that causes Cyberdrop-DL to exit immediately |
{% hint style="info" %}
-Using anything other that `DEBUG` makes troubleshooting issues harder. Practically speaking, this should only be adjusted if you expect CDL to run for an extended period (with a large number of input URLs) to minimize the log files sizes
+Using anything other that `DEBUG` makes troubleshooting issues harder. Practically speaking, you should _only_ change this if you expect CDL to run for an extended period (with a large number of input URLs) to minimize the log files sizes
+{% endhint %}
+
+{% hint style="info" %}
+There is more verbose level of logs: `DEVELOPER`
+
+It includes pager logs, posts logs, individual requests logs (their URLs and their response code), requests cache hits and misses, file system locks logs and local variable values from within a traceback. It's not exposed as a config option becuase it is too verbose.
+
+You can set the environment variable `CDL_DEBUG_LOG_FOLDER` to any valid folder and CDL will create an additional log file with the level set to `DEVELOPER` inside it.
+{% endhint %}
+
+{% hint style="critical" %}
+As the name suggest, the `DEVELOPER` log level is only meant for developers or, more generally, people that want and have the "know how"
+to troubleshoot CDL issues on their own. You should never share those logs with _anyone_ as it could potentially have raw information in
+the locals informaion of the tracebacks like cookies, auth credentials, IPs, etc.
{% endhint %}
## `send_unsupported_to_jdownloader`
diff --git a/docs/reference/configuration-options/settings/sorting.md b/docs/reference/configuration-options/settings/sorting.md
index 87c6cebb4..51758dfd7 100644
--- a/docs/reference/configuration-options/settings/sorting.md
+++ b/docs/reference/configuration-options/settings/sorting.md
@@ -1,16 +1,17 @@
# Sorting
-Cyberdrop-DL has a file sorter built in, but you have to enable it to use it.
+Cyberdrop-DL has a file sorter built in, but it's disabled by default
-You can use the shared path flags below in any part of the sorting schemas. You can also use essentially none of them and have a hard coded path. However, `filename` and `ext` must always be used.
+You can use the field names bellow to create a custom path format. You can also use essentially none of them and have a hard coded path.
+However, `filename` and `ext` must always be used.
-Shared path flags:
+Common fields for sorting format options (supported for `audio`, `videos`, `images` and `other`):
-> `sort_dir`: The same path as `sort_folder`
+> `sort_dir`: This represent the same path as `sort_folder` from the download options
>
> `base_dir`: the name of highest level folder inside `scan_folder` ex: (model name / thread name)
>
-> `parent_dir`: the name of the folder where the file is. Ex: (album name)
+> `parent_dir`: the name of the folder where the file is located at. Ex: (album name)
>
> `filename`: the file's name (stem)
>
@@ -24,9 +25,9 @@ Shared path flags:
Sets the starting point for the file scan
-Each direct child of the `scan_folder` is recursively scanned, and files are moved based on your settings
+Each direct child of the `scan_folder` is recursively scanned, and files are moved based on your settings.
-If this is set to `null` (the default), `download_dir` is used instead
+If this is set to `null` (the default), the value of `download_dir` from the download options is used.
## `sort_downloads`
@@ -45,7 +46,7 @@ Setting this to `true` will allow Cyberdrop-DL to sort downloads after a run is
This is the path to the folder you'd like sorted downloads to be stored in.
{% hint style="warning" %}
-Setting `sort_folder` to the same value as `scan_folder` is not officiality supported and will lead to undefined results
+Setting `sort_folder` to the same value as `scan_folder` is not supported and will lead to expected results
{% endhint %}
## `sort_incrementer_format`
@@ -66,13 +67,13 @@ You can modify the format as needed, but it must include `{i}` to specify where
| ----------------------- | --------------------------------------------- |
| `NonEmptyStr` or `null` | `{sort_dir}/{base_dir}/Audio/{filename}{ext}` |
-This is the format for the directory structure and naming scheme for audio files. Set to `null` skip sorting audio files
+This is the format for the directory structure and naming scheme for audio files. Set to `null` to skip sorting audio files
-Unique Path Flags:
+In addition to the common sorting format fields, this option supports:
> `bitrate`: file bit rate
>
-> `duration`: audio runtime
+> `duration`: audio total runtime
>
> `length`: same as `duration`
>
@@ -84,9 +85,9 @@ Unique Path Flags:
| ----------------------- | ---------------------------------------------- |
| `NonEmptyStr` or `null` | `{sort_dir}/{base_dir}/Images/{filename}{ext}` |
-This is the format for the directory structure and naming scheme for image files. Set to `null` skip sorting image files
+This is the format for the directory structure and naming scheme for image files. Set to `null` to skip sorting image files
-Unique Path Flags:
+In addition to the common sorting format fields, this option supports:
> `height`: vertical pixel count
>
@@ -100,13 +101,13 @@ Unique Path Flags:
| ----------------------- | --------------------------------------------- |
| `NonEmptyStr` or `null` | `{sort_dir}/{base_dir}/Video/{filename}{ext}` |
-This is the format for the directory structure and naming scheme for video files. Set to `null` skip sorting video files
+This is the format for the directory structure and naming scheme for video files. Set to `null` to skip sorting video files
-Unique Path Flags:
+In addition to the common sorting format fields, this option supports:
> `codec`: ex. h264
>
-> `duration`: video runtime
+> `duration`: video total runtime
>
> `fps`: ex. 24
>
@@ -124,7 +125,7 @@ Unique Path Flags:
| ----------------------- | --------------------------------------------- |
| `NonEmptyStr` or `null` | `{sort_dir}/{base_dir}/Other/{filename}{ext}` |
-This is the format for the directory structure and naming scheme for other files. Set to `null` skip sorting other files
+This is the format for the directory structure and naming scheme for other files. Set to `null` to skip sorting other files
## Group URLs
@@ -132,7 +133,7 @@ It is possible to treat a list of URLs as a group, allowing them to be downloade
To define a group, put a title above the URLs you want to be in the group, using the format: `--- <group_name>` or `=== <group_name>`.
-To define the end of a group, insert an group with no name. (`---` or `===`)
+To define the end of a group, add a new group with no name. (`---` or `===`)
Here is an example URL file with two groups:
diff --git a/docs/reference/configuration-options/special_setting_types.md b/docs/reference/configuration-options/special_setting_types.md
index b0489f782..a4ec0eaa4 100644
--- a/docs/reference/configuration-options/special_setting_types.md
+++ b/docs/reference/configuration-options/special_setting_types.md
@@ -45,7 +45,7 @@ A special kind of `bool`. Within a config file, it can have a `true` or `false`
## `ByteSize`
-A special kind of `int` that accepts suffixes like `GB`, `MiB` and `KB`
+A special kind of `int` that also accepts suffixes like `GB`, `MiB` and `KB` to specify valid values
In conformance with IEC 80000-13 Standard, `1KB` means `1000 bytes`, and `1KiB` means `1024 bytes`. In general, including a middle 'i' will cause the unit to be interpreted as a power of 2, rather than a power of 10.
diff --git a/docs/reference/supported-websites.md b/docs/reference/supported-websites.md
index 265d7b18d..dea93d654 100644
--- a/docs/reference/supported-websites.md
+++ b/docs/reference/supported-websites.md
@@ -17,7 +17,7 @@ Example: `https://cyberfile.me/folder/xUGg?password=1234`
| ------------------------------------------------------- |
| GoFile |
| Cyberfile |
-| Chevereto Sites (`JPG5`, `ImagePond.net` or `img.kiwi`) |
+| Chevereto Sites (`JPG5`, `ImagePond.net`) |
| Iceyfile.com |
## Additional Content Hosts with Real-Debrid
diff --git a/docs/transition-to-v6.md b/docs/transition-to-v6.md
index b89a2b6fb..adc252727 100644
--- a/docs/transition-to-v6.md
+++ b/docs/transition-to-v6.md
@@ -1,5 +1,5 @@
---
-description: This is the walk through for transitioning from V4 or V5 to V6
+description: This is the walk through for transitioning from V4, V5 or V6 to v7
icon: arrow-up-to-bracket
layout:
title:
@@ -14,10 +14,10 @@ layout:
visible: true
---
-# Transition from V4 or V5 to V6
+# Transition from V4, V5 or V6 to V7
{% hint style="danger" %}
-V6 introduces some breaking changes: Using a more strict config validation logic, replacing `md5` with `xxh128` as the default hashing algorithm, using a new database schema, among others.
+V6 introduced some breaking changes: Using a more strict config validation logic, replacing `md5` with `xxh128` as the default hashing algorithm, using a new database schema, among others.
It's recommended to do a manual backup of your current `AppData` folder. You won't be able to rollback to a previous version after the transfer is completed.
@@ -30,17 +30,17 @@ Even after a successful configuration migration, the program may not start if so
You can use the [Config Options page](reference/configuration-options/README.md) as reference for valid config values
{% endhint %}
-## Import V5 config to V6
+## Import V5 or V6 config into V7
Good news! If you have a v5 config, CDL can import it. The program will automatically detect the config in the current folder and handle the migration process for you.
-All you have to do is replace the v5 start script (for Windows, that's the `.bat` file) with the v6 start script. If CDL does not update automatically to v6, delete the `.venv` folder inside CDL's folder and run the start script again.
+All you have to do is replace your old start script (for Windows, that's the `.bat` file) with the v7 start script. If CDL does not update automatically to v7, delete the `.venv` folder inside CDL's folder and run the start script again.
Please note that after the migration is complete, you may still need to manually adjust some values in the config.
-If the migration fails, the database will be rolled back to its previous state. An automatic backup is also made before the migration begins. However, keep in mind that the backup only includes the database, not the config files.
+If you are migrating from v5 and the migration fails, the database will be rolled back to its previous state. An automatic backup is also made before the migration begins. However, keep in mind that the backup only includes the database, not the config files.
-## Import V4 config to V6
+## Import V4 config
{% hint style="warning" %}
V4 import is no longer supported. Last version with v4 support was `6.5.0`. To upgrade to the latest version, you need to perform a sequential upgrade. First, update CDL to `6.5.0`, import your v4 files, and then update to the latest version.
@@ -62,7 +62,7 @@ The config files will be located in the folder that you were previously running
{% hint style="info" %}
If you weren't using the config previously, you don't need to import it.
-However, if you were primarily using CLI Arguments with V4, some of the arguments will need to swapped into config options.
+However, if you were primarily using CLI Arguments with V4, some of the arguments has been replaced.
{% endhint %}
If you don't end up using the import feature, make sure you also change the default config in the program if that's something you want to do.