Bulk asynchronous downloader for multiple file hosts
Python asynciobunkrcyberdropdownloaderdropboxfilestergofilegoogle-drivemediafiremegamega-nzonedrivepatreonpawchivepinterestpixeldraintransfer-ittwitchtwittervk
docs: changelog and wiki updates for v8.1.0 release (#1346)
NTFSvolume9a6a09ff8d48
diff --git a/CHANGELOG.md b/CHANGELOG.mdindex ed6a1f3b4..6788fe39f 100644--- a/CHANGELOG.md+++ b/CHANGELOG.md@@ -22,6 +22,28 @@ All notable changes to this project will be documented here. For more details, vThe format is based on [Keep a Changelog](https://keepachangelog.com/en/1.1.0/),and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0.html).+## [8.1.0] - 2025-09-27++### Added++- imglike support+- Support for channel photos (xVideos)++### Changed++- Remove SimpCity disclaimer on first time startup+- Tiktok now has a hardcoded rate limit of 0.5 req/s++### Fixed++- `str object cannot be interpreted as an integer` error when using any retry option+- Date parsing (pixeldrain)+- Date parsing (bunkr)+- Channel's pagination (xVideos)+- Http errors being reported as `Not Implemented` (Pixeldrain, MediaFire, GoFile)+- Http errors causing CDL to crash while downloading galleries (xVideos)+- CDL getting stuck after 100 TCP connections+## [8.0.0] - 2025-09-23### Added@@ -41,7 +63,7 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0- Support for `pd.cybar.xyz` URLs to bypass pixeldrain daily limit- The `--separate-post` and `--separate-post-format` options now work for tiktok posts- CDL can now download private TikTok posts when using cookies-- New config option to download tiktok videos in original quality: <https://script-ware.gitbook.io/cyberdrop-dl/reference/configuration-options/settings/download_options#download-tiktok-src-quality-videos>+- New config option to download tiktok videos in original quality: <https://script-ware.gitbook.io/cyberdrop-dl/reference/cli-arguments#download-tiktok-src-quality-videos>- New `--max-thread-folder-depth` config option: <https://script-ware.gitbook.io/cyberdrop-dl/reference/configuration-options/settings/download_options#maximum_thread_folder_depth>- New `--ignore-coomer-post-content` config option: <https://script-ware.gitbook.io/cyberdrop-dl/reference/configuration-options/settings/ignore_options#ignore_coomer_post_content>- The `--ignore-coomer-ads` config option now affects all Kemono based crawlers (Coomer, Kemono & Nekohouse)diff --git a/cyberdrop_dl/crawlers/_chevereto.py b/cyberdrop_dl/crawlers/_chevereto.pyindex e7dc74cdf..0be522f85 100644--- a/cyberdrop_dl/crawlers/_chevereto.py+++ b/cyberdrop_dl/crawlers/_chevereto.py@@ -37,6 +37,7 @@ class CheveretoCrawler(Crawler, is_generic=True):"/album/<id>","/album/<name>.<id>",),+ "Category": "/category/<name>","Image": ("/img/<id>","/img/<name>.<id>",@@ -97,18 +98,26 @@ class CheveretoCrawler(Crawler, is_generic=True):scrape_item.setup_as_profile(title)self._process_page(scrape_item, soup)+ async def _get_final_album_url(self, url: AbsoluteHttpURL) -> AbsoluteHttpURL:+ if "category" in url.parts:+ return url++ # We need the full URL (aka "/<name>.<id>") to fetch sub albums+ if "." not in url.name:+ url = await self._get_redirect_url(url)++ # The first redirect may have only added a trailing slash, try again+ if not url.name and "." not in url.parts[-2]:+ url = await self._get_redirect_url(url)++ return url+@error_handling_wrapperasync def album(self, scrape_item: ScrapeItem, album_id: str) -> None:results = await self.get_album_results(album_id)title: str = ""- # We need the full URL (aka "/<name>.<id>") to fetch sub albums- if "." not in scrape_item.url.name:- scrape_item.url = await self._get_redirect_url(scrape_item.url)-- # The first redirect may have only added a trailing slash, try again- if not scrape_item.url.name and "." not in scrape_item.url.parts[-2]:- scrape_item.url = await self._get_redirect_url(scrape_item.url)+ scrape_item.url = await self._get_final_album_url(scrape_item.url)async for soup in self.web_pager(_sort_by_new(scrape_item.url), trim=False):if not title:@@ -116,7 +125,8 @@ class CheveretoCrawler(Crawler, is_generic=True):await self._unlock_password_protected_album(scrape_item)return await self.album(scrape_item, album_id)- title = open_graph.get_title(soup) or open_graph.get("description", soup)+ title = open_graph.get_title(soup) or open_graph.get("description", soup) or ""+ assert titletitle = self.create_title(title, album_id)scrape_item.setup_as_album(title, album_id=album_id)self._process_page(scrape_item, soup, results)diff --git a/docs/reference/cli-arguments.md b/docs/reference/cli-arguments.mdindex 3c088fbaf..8b0bfa168 100644--- a/docs/reference/cli-arguments.md+++ b/docs/reference/cli-arguments.md@@ -117,13 +117,19 @@ Download TikTok audios from posts and save them as separate files| ---------- | ------- | ------------ || `BoolFlag` | `False` | `store_true` |-Download TikTok videos in source quality.+By default, CDL will download the "optimized for streaming" version of tiktok videos. Setting this option to `True` will download videos in original (source) quality.++`_original` will be added as a prefix to their filename.{% hint style="warning" %}-Enabling this option will make downloads several orders of magnitude slower.+This will make video downloads several times slowerWhen it is set to `False` (the default) CDL can download 50 videos with a single request.When it is set to `True` , CDL needs to make at least 3 requests _per_ video to download them.++There's also a daily limit of the API CDL uses: 5000 requests per day++Setting this option to `True` will consume the daily limit faster{% endhint %}### `max-items-retry`@@ -266,9 +272,9 @@ media_duration_limits:--minimum-audio-duration MINIMUM_AUDIO_DURATIONfiles:- -d DOWNLOAD_FOLDER, --download-folder DOWNLOAD_FOLDER+ -d, --download-folder DOWNLOAD_FOLDER-j, --dump-json, --no-dump-json- -i INPUT_FILE, --input-file INPUT_FILE+ -i, --input-file INPUT_FILE--save-pages-html, --no-save-pages-htmlignore_options:diff --git a/docs/reference/supported-websites.md b/docs/reference/supported-websites.mdindex d5237ccca..1f9ff28ce 100644--- a/docs/reference/supported-websites.md+++ b/docs/reference/supported-websites.md@@ -30,7 +30,7 @@ Supported domains via Real-Debrid include `mega.nz`, `rapidgator`, `google drive## Supported sites-List of sites supported by cyberdrop-dl-patched as of version 8.0.0+List of sites supported by cyberdrop-dl-patched as of version 8.1.0| Site | Primary URL | Supported Domains | Supported Paths ||---------------------------|---------------------------------------------|----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|@@ -49,7 +49,7 @@ List of sites supported by cyberdrop-dl-patched as of version 8.0.0| Camwhores.tv | https://www.camwhores.tv | camwhores.tv | Categories: <br> - `/categories/...`<br>Members: <br> - `/members/<member_id>`<br>Search: <br> - `/search/...`<br>Tags: <br> - `/tags/...`<br>Videos: <br> - `/videos/...`<br> || Catbox | https://catbox.moe | files.catbox.moe<br>litter.catbox.moe | Direct Links<br> || CelebForum | https://celebforum.to | celebforum.* | Attachments: <br> - `/(attachments\|data\|uploads)/...`<br>Threads: <br> - `/(thread\|topic\|tema\|threads\|topics\|temas)/<thread_name_and_id>`<br> - `/goto/<post_id>`<br> - `/posts/<post_id>`<br><br><br>**NOTES**<br> - base crawler: Xenforo<br> |-| Chevereto | ::GENERIC CRAWLER:: | | Album: <br> - `/a/<id>`<br> - `/a/<name>.<id>`<br> - `/album/<id>`<br> - `/album/<name>.<id>`<br>Direct Links<br>Image: <br> - `/image/<id>`<br> - `/image/<name>.<id>`<br> - `/img/<id>`<br> - `/img/<name>.<id>`<br>Profiles: <br> - `/<user_name>`<br>Video: <br> - `/video/<id>`<br> - `/video/<name>.<id>`<br> - `/videos/<id>`<br> - `/videos/<name>.<id>`<br> |+| Chevereto | ::GENERIC CRAWLER:: | | Album: <br> - `/a/<id>`<br> - `/a/<name>.<id>`<br> - `/album/<id>`<br> - `/album/<name>.<id>`<br>Category: <br> - `/category/<name>`<br>Direct Links<br>Image: <br> - `/image/<id>`<br> - `/image/<name>.<id>`<br> - `/img/<id>`<br> - `/img/<name>.<id>`<br>Profiles: <br> - `/<user_name>`<br>Video: <br> - `/video/<id>`<br> - `/video/<name>.<id>`<br> - `/videos/<id>`<br> - `/videos/<name>.<id>`<br> || CloudflareStream | https://cloudflarestream.com | cloudflarestream.com<br>videodelivery.net | Public Video: <br> - `/<video_uid>`<br> - `/<video_uid>/iframe`<br> - `/<video_uid>/watch`<br> - `/embed/___.js?video=<video_uid>`<br>Restricted Access Video: <br> - `/<jwt_access_token>`<br> - `/<jwt_access_token>/iframe`<br> - `/<jwt_access_token>/watch`<br> - `/embed/___.js?video=<jwt_access_token>`<br> || Coomer | https://coomer.st | coomer.party<br>coomer.st<br>coomer.su | Direct links: <br> - `/data/...`<br> - `/thumbnail/...`<br>Favorites: <br> - `/favorites/<user_id>`<br>Individual Post: <br> - `/<service>/user/<user_id>/post/<post_id>`<br>Model: <br> - `/<service>/user/<user_id>`<br>Search: <br> - `/search?q=...`<br> || Cyberdrop | https://cyberdrop.me | cyberdrop.* | Album: <br> - `/a/...`<br>Direct Links<br>File: <br> - `/f/...`<br> |@@ -78,14 +78,15 @@ List of sites supported by cyberdrop-dl-patched as of version 8.0.0| HotPic | https://hotpic.cc | 2385290.xyz<br>hotpic.* | Album: <br> - `/album/...`<br>Image: <br> - `/i/...`<br> || Iceyfile | https://iceyfile.com | iceyfile.* | Files: <br> - `/<file_id>`<br> - `/<file_id>/<file_name>`<br>Public Folders: <br> - `/folder/<folder_id>`<br> - `/folder/<folder_id>/<folder_name>`<br>Shared folders: <br> - `/shared/<share_key>`<br> || ImageBam | https://www.imagebam.com | imagebam.* | Gallery: <br> - `/gallery/...`<br>Gallery or Image: <br> - `/view/...`<br>Image: <br> - `/image/...`<br>Thumbnails: <br> - `thumbs<x>.imagebam.com/...`<br> |-| ImagePond | https://imagepond.net | imagepond.net | Album: <br> - `/a/<id>`<br> - `/a/<name>.<id>`<br> - `/album/<id>`<br> - `/album/<name>.<id>`<br>Direct Links<br>Image: <br> - `/image/<id>`<br> - `/image/<name>.<id>`<br> - `/img/<id>`<br> - `/img/<name>.<id>`<br>Profiles: <br> - `/<user_name>`<br>Video: <br> - `/video/<id>`<br> - `/video/<name>.<id>`<br> - `/videos/<id>`<br> - `/videos/<name>.<id>`<br> |+| ImagePond | https://imagepond.net | imagepond.net | Album: <br> - `/a/<id>`<br> - `/a/<name>.<id>`<br> - `/album/<id>`<br> - `/album/<name>.<id>`<br>Category: <br> - `/category/<name>`<br>Direct Links<br>Image: <br> - `/image/<id>`<br> - `/image/<name>.<id>`<br> - `/img/<id>`<br> - `/img/<name>.<id>`<br>Profiles: <br> - `/<user_name>`<br>Video: <br> - `/video/<id>`<br> - `/video/<name>.<id>`<br> - `/videos/<id>`<br> - `/videos/<name>.<id>`<br> || ImgBB | https://ibb.co | ibb.co<br>imgbb.co | Album: <br> - `/album/...`<br>Image: <br> - `/...`<br> || ImgBox | https://imgbox.com | imgbox.* | Album: <br> - `/g/...`<br>Direct Links<br>Image: <br> - `/...`<br> |+| ImgLike | https://imglike.com | imglike.com | Album: <br> - `/a/<id>`<br> - `/a/<name>.<id>`<br> - `/album/<id>`<br> - `/album/<name>.<id>`<br>Category: <br> - `/category/<name>`<br>Direct Links<br>Image: <br> - `/image/<id>`<br> - `/image/<name>.<id>`<br> - `/img/<id>`<br> - `/img/<name>.<id>`<br>Profiles: <br> - `/<user_name>`<br>Video: <br> - `/video/<id>`<br> - `/video/<name>.<id>`<br> - `/videos/<id>`<br> - `/videos/<name>.<id>`<br> || Imgur | https://imgur.com | imgur.* | Album: <br> - `/a/...`<br>Direct Links<br>Gallery: <br> - `/gallery/...`<br>Image: <br> - `/...`<br> || Imx.to | https://imx.to | imx.to | Image: <br> - `/i/...`<br>Thumbnail: <br> - `/t/...`<br> || IncestFlix | https://www.incestflix.com | incestflix.* | Tag: <br> - `/tag/...`<br>Video: <br> - `/watch/...`<br> || InfluencerBitches | https://influencerbitches.com | influencerbitches.* | Model: <br> - `/model/...`<br> |-| JPG5 | https://jpg6.su | host.church<br>jpeg.pet<br>jpg.church<br>jpg.fish<br>jpg.fishing<br>jpg.homes<br>jpg.pet<br>jpg1.su<br>jpg2.su<br>jpg3.su<br>jpg4.su<br>jpg5.su<br>jpg6.su<br>selti-delivery.ru | Album: <br> - `/a/<id>`<br> - `/a/<name>.<id>`<br> - `/album/<id>`<br> - `/album/<name>.<id>`<br>Direct Links<br>Image: <br> - `/image/<id>`<br> - `/image/<name>.<id>`<br> - `/img/<id>`<br> - `/img/<name>.<id>`<br>Profiles: <br> - `/<user_name>`<br> |+| JPG5 | https://jpg6.su | host.church<br>jpeg.pet<br>jpg.church<br>jpg.fish<br>jpg.fishing<br>jpg.homes<br>jpg.pet<br>jpg1.su<br>jpg2.su<br>jpg3.su<br>jpg4.su<br>jpg5.su<br>jpg6.su<br>selti-delivery.ru | Album: <br> - `/a/<id>`<br> - `/a/<name>.<id>`<br> - `/album/<id>`<br> - `/album/<name>.<id>`<br>Category: <br> - `/category/<name>`<br>Direct Links<br>Image: <br> - `/image/<id>`<br> - `/image/<name>.<id>`<br> - `/img/<id>`<br> - `/img/<name>.<id>`<br>Profiles: <br> - `/<user_name>`<br> || Kemono | https://kemono.cr | kemono.cr<br>kemono.party<br>kemono.su | Direct links: <br> - `/data/...`<br> - `/thumbnail/...`<br>Discord Server: <br> - `/discord/<server_id>`<br>Discord Server Channel: <br> - `/discord/server/<server_id>/<channel_id>#...`<br>Favorites: <br> - `/favorites/<user_id>`<br>Individual Post: <br> - `/<service>/user/<user_id>/post/<post_id>`<br>Model: <br> - `/<service>/user/<user_id>`<br>Search: <br> - `/search?q=...`<br> || LeakedModels | https://leakedmodels.com/forum | leakedmodels.* | Attachments: <br> - `/(attachments\|data\|uploads)/...`<br>Threads: <br> - `/(thread\|topic\|tema\|threads\|topics\|temas)/<thread_name_and_id>`<br> - `/goto/<post_id>`<br> - `/posts/<post_id>`<br><br><br>**NOTES**<br> - base crawler: Xenforo<br> || LeakedZone | https://leakedzone.com | leakedzone.* | Model: <br> - `/<model_id>`<br>Video: <br> - `/<model_id>/video/<video_id>`<br> |