Skip to content

Bulk asynchronous downloader for multiple file hosts

docs: update cli overview

NTFSvolumeb0c6b4afbdbc
diff --git a/docs/reference/cli-arguments.md b/docs/reference/cli-arguments.md
index 2053e5945..35dca4e98 100644
--- a/docs/reference/cli-arguments.md
+++ b/docs/reference/cli-arguments.md
@@ -315,7 +315,7 @@ sorting:
--sorted-video SORTED_VIDEO
general:
- --allow-insecure-connections, --no-allow-insecure-connections
+ --ssl-context SSL_CONTEXT
--disable-crawlers [DISABLE_CRAWLERS ...]
--enable-generic-crawler, --no-enable-generic-crawler
--flaresolverr FLARESOLVERR
@@ -345,4 +345,9 @@ ui_options:
--scraping-item-limit SCRAPING_ITEM_LIMIT
--vi-mode, --no-vi-mode
+generic_crawlers_instances:
+ --wordpress-media [WORDPRESS_MEDIA ...]
+ --wordpress-html [WORDPRESS_HTML ...]
+ --discourse [DISCOURSE ...]
+
```
diff --git a/docs/reference/supported-websites.md b/docs/reference/supported-websites.md
index dea93d654..0114779f0 100644
--- a/docs/reference/supported-websites.md
+++ b/docs/reference/supported-websites.md
@@ -28,34 +28,35 @@ Supported domains via Real-Debrid include `mega.nz`, `rapidgator`, `google drive
## Supported sites
-List of sites supported by cyberdrop-dl-patched as of version 6.10.1
+List of sites supported by cyberdrop-dl-patched as of version 7.0.0
| Site | Primary URL | Supported Domains | Supported Paths |
|---------------------------|---------------------------------------------|-----------------------------------------------------------------------------------------------------------------------------------------------------------------------|--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|
| 4chan | https://boards.4chan.org | 4chan | Board: <br> - `/<board>`<br>Thread: <br> - `/thread`<br> |
| 8Muses | https://comics.8muses.com | 8muses | Album: <br> - `/comics/album/...`<br> |
| AcidImg | https://acidimg.cc | acidimg.cc | Image: <br> - `/i/...`<br>Thumbnail: <br> - `/upload/...`<br> |
-| AllPornComix | https://forum.allporncomix.com | allporncomix | Attachments: <br> - `/(attachments\|data\|uploads)/...`<br>Threads: <br> - `/(thread\|threads\|topic\|topics\|tema\|temas)/<thread_name_and_id>`<br> - `/goto/<post_id>`<br> - `/posts/<post_id>`<br><br><br>**NOTES**<br> - base crawler: Xenforo<br> |
+| AllPornComix | https://forum.allporncomix.com | allporncomix | Attachments: <br> - `/(attachments\|data\|uploads)/...`<br>Threads: <br> - `/(thread\|topic\|tema\|threads\|topics\|temas)/<thread_name_and_id>`<br> - `/goto/<post_id>`<br> - `/posts/<post_id>`<br><br><br>**NOTES**<br> - base crawler: Xenforo<br> |
| ArchiveBate | https://www.archivebate.store | archivebate | Video: <br> - `/watch/<video_id>`<br> |
-| Bellazon | https://www.bellazon.com/main/ | bellazon | Attachments: <br> - `/(attachments\|data\|uploads)/...`<br>Threads: <br> - `/(thread\|threads\|topic\|topics\|tema\|temas)/<thread_name_and_id>`<br> - `/goto/<post_id>`<br> - `/posts/<post_id>`<br><br><br>**NOTES**<br> - base crawler: Xenforo<br> |
+| Beeg.com | https://beeg.com/ | beeg.com | Video: <br> - `/<video_id>`<br> - `/video/<video_id>`<br> |
+| Bellazon | https://www.bellazon.com/main/ | bellazon | Attachments: <br> - `/(attachments\|data\|uploads)/...`<br>Threads: <br> - `/(thread\|topic\|tema\|threads\|topics\|temas)/<thread_name_and_id>`<br> - `/goto/<post_id>`<br> - `/posts/<post_id>`<br><br><br>**NOTES**<br> - base crawler: Invision<br> |
| BestPrettyGirl | https://bestprettygirl.com/ | bestprettygirl.com | All Posts: <br> - `/posts/`<br>Category: <br> - `/category/<category_slug>`<br>Date Range: <br> - `...?after=<date>`<br> - `...?before=<date&after=<date>`<br> - `...?before=<date>`<br>Post: <br> - `/<post_slug>/`<br>Tag: <br> - `/tag/<tag_slug>`<br><br><br>**NOTES**<br> - <br><br> For `Date Range`, <date> must be a valid iso 8601 date, ex: `2022-12-06`.<br><br> `Date Range` can be combined with `Category`, `Tag` and `All Posts`.<br> ex: To only download categories from a date range: ,<br> `/category/<category_slug>?before=<date>`<br> |
-| Box | https://www.box.com | app.box.com | File or Folder: <br> - `app.box.com/s?sh=<share_code>`<br> |
+| Box | https://www.box.com | app.box.com | Embeded File or Folder: <br> - `app.box.com/embed/s?sh=<share_code>`<br> - `app.box.com/embed_widget/s?sh=<share_code>`<br>File or Folder: <br> - `app.box.com/s?sh=<share_code>`<br> |
| Bunkrr | https://bunkr.site | bunkr<br>bunkrr | Album: <br> - `/a/...`<br>Direct Links<br>File: <br> - `/f/...`<br>Video: <br> - `/v/...`<br> |
| BuzzHeavier | https://buzzheavier.com | buzzheavier.com | Direct Links<br> |
| Camwhores.tv | https://www.camwhores.tv | camwhores.tv | Albums: <br> - `/albums/<album_name>`<br>Categories: <br> - `/categories/...`<br>Image: <br> - `/albums/<album_name>/<image_name>`<br>Members: <br> - `/members/<member_id>`<br>Search: <br> - `/search/?q=...`<br>Tags: <br> - `/tags/...`<br>Videos: <br> - `/videos/...`<br> |
| Catbox | https://catbox.moe | files.catbox.moe<br>litter.catbox.moe | Direct Links<br> |
-| CelebForum | https://celebforum.to | celebforum | Attachments: <br> - `/(attachments\|data\|uploads)/...`<br>Threads: <br> - `/(thread\|threads\|topic\|topics\|tema\|temas)/<thread_name_and_id>`<br> - `/goto/<post_id>`<br> - `/posts/<post_id>`<br><br><br>**NOTES**<br> - base crawler: Xenforo<br> |
+| CelebForum | https://celebforum.to | celebforum | Attachments: <br> - `/(attachments\|data\|uploads)/...`<br>Threads: <br> - `/(thread\|topic\|tema\|threads\|topics\|temas)/<thread_name_and_id>`<br> - `/goto/<post_id>`<br> - `/posts/<post_id>`<br><br><br>**NOTES**<br> - base crawler: Xenforo<br> |
| Coomer | https://coomer.su | coomer | Direct Links<br>Favorites: <br> - `/favorites/<user>`<br>Individual Post: <br> - `<service>/<user>/post/...`<br>Model: <br> - `/<service>/user/<user>`<br>Search: <br> - `/search?...`<br> |
| Cyberdrop | https://cyberdrop.me/ | cyberdrop | Album: <br> - `/a/...`<br>Direct Links<br>File: <br> - `/f/...`<br> |
| Cyberfile | https://cyberfile.me/ | cyberfile | Files: <br> - `/...`<br>Folder: <br> - `/folder/...`<br>Shared folder: <br> - `/shared/...`<br> |
| DirtyShip | https://dirtyship.com | dirtyship | Categorie: <br> - `/category/...`<br>Gallery: <br> - `/gallery/...`<br>Photo: <br> - `/gallery/.../...`<br>Tag: <br> - `/tag/...`<br>Video: <br> - `/<video_name>`<br> |
| Discourse | ::GENERIC CRAWLER:: | | Attachments: <br> - `/uploads/...`<br>Topic: <br> - `/t/<topic_name>/<topic_id>`<br> - `/t/<topic_name>/<topic_id>/<post_number>`<br><br><br>**NOTES**<br> - If the URL includes <post_number>, posts with a number lower that it won't be scraped<br> |
| DoodStream | https://doodstream.com/ | all3do.com<br>do7go.com<br>dood.re<br>dood.yt<br>doodcdn<br>doodstream<br>doodstream.co<br>vidply.com | Video: <br> - `/e/<video_id>`<br> |
-| Dropbox | https://dropbox.com/ | dropbox | File: <br> - `/s/...`<br> - `/scl/fi/...`<br>Folder: <br> - `/scl/fo/...`<br> - `/sh/...`<br><br><br>**NOTES**<br> - Folders will be downloaded as a zip file. See: [--download-dropbox-folders-as-zip](https://script-ware.gitbook.io/cyberdrop-dl/reference/cli-arguments#download-dropbox-folders-as-zip)<br> |
+| Dropbox | https://www.dropbox.com/ | dropbox | File: <br> - `/s/...`<br> - `/scl/fi/<file_id>`<br> - `/scl/fo/<token1>/<token2>?preview=<filename>`<br>Folder: <br> - `/scl/fo/<token1>/<token2>`<br> - `/sh/...`<br><br><br>**NOTES**<br> - Folders will be downloaded as a zip file. See: [--download-dropbox-folders-as-zip](https://script-ware.gitbook.io/cyberdrop-dl/reference/cli-arguments#download-dropbox-folders-as-zip)<br> |
| E-Hentai | https://e-hentai.org/ | e-hentai | Album: <br> - `/g/...`<br>File: <br> - `/s/...`<br> |
| E621 | https://e621.net | e621.net | Pools: <br> - `/pools/...`<br>Post: <br> - `/posts/...`<br>Tags: <br> - `/posts?tags=...`<br> |
| Erome | https://www.erome.com | erome | Album: <br> - `/a/...`<br>Profile: <br> - `/...`<br> |
-| F95Zone | https://f95zone.to | f95zone | Attachments: <br> - `/(attachments\|data\|uploads)/...`<br>Threads: <br> - `/(thread\|threads\|topic\|topics\|tema\|temas)/<thread_name_and_id>`<br> - `/goto/<post_id>`<br> - `/posts/<post_id>`<br><br><br>**NOTES**<br> - base crawler: Xenforo<br> |
+| F95Zone | https://f95zone.to | f95zone | Attachments: <br> - `/(attachments\|data\|uploads)/...`<br>Threads: <br> - `/(thread\|topic\|tema\|threads\|topics\|temas)/<thread_name_and_id>`<br> - `/goto/<post_id>`<br> - `/posts/<post_id>`<br><br><br>**NOTES**<br> - base crawler: Xenforo<br> |
| Fapello | https://fapello.su/ | fapello | Individual Post: <br> - `/.../...`<br>Model: <br> - `/...`<br> |
| Fileditch | https://fileditchfiles.me/ | fileditch | Direct Links<br> |
| FilesVC | https://files.vc | files.vc | Direct Links<br> |
@@ -76,7 +77,7 @@ List of sites supported by cyberdrop-dl-patched as of version 6.10.1
| InfluencerBitches | https://influencerbitches.com | influencerbitches | Model: <br> - `/model/...`<br> |
| JPG5 | https://jpg5.su | host.church<br>jpeg.pet<br>jpg.church<br>jpg.fish<br>jpg.fishing<br>jpg.homes<br>jpg.pet<br>jpg1.su<br>jpg2.su<br>jpg3.su<br>jpg4.su<br>jpg5.su | Album: <br> - `/a/...`<br>Direct Links<br>Image: <br> - `/img/...`<br>Profile: <br> - `/<user_name>`<br> |
| Kemono | https://kemono.su | kemono | Direct Links<br>Discord Server: <br> - `/discord/<server_id>`<br>Discord Server Channel: <br> - `/discord/server/...#...`<br>Favorites: <br> - `/favorites/<user>`<br>Individual Post: <br> - `<service>/<user>/post/...`<br>Model: <br> - `/<service>/user/<user>`<br>Search: <br> - `/search?...`<br> |
-| LeakedModels | https://leakedmodels.com/forum/ | leakedmodels | Attachments: <br> - `/(attachments\|data\|uploads)/...`<br>Threads: <br> - `/(thread\|threads\|topic\|topics\|tema\|temas)/<thread_name_and_id>`<br> - `/goto/<post_id>`<br> - `/posts/<post_id>`<br><br><br>**NOTES**<br> - base crawler: Xenforo<br> |
+| LeakedModels | https://leakedmodels.com/forum/ | leakedmodels | Attachments: <br> - `/(attachments\|data\|uploads)/...`<br>Threads: <br> - `/(thread\|topic\|tema\|threads\|topics\|temas)/<thread_name_and_id>`<br> - `/goto/<post_id>`<br> - `/posts/<post_id>`<br><br><br>**NOTES**<br> - base crawler: Xenforo<br> |
| Luscious | https://members.luscious.net | luscious | Album: <br> - `/albums/...`<br> |
| Mediafire | https://www.mediafire.com/ | mediafire | File: <br> - `/file/...`<br>Folder: <br> - `/folder/...`<br> |
| MissAV | https://missav.ws | missav | Video: <br> - `/...`<br> |
@@ -84,7 +85,7 @@ List of sites supported by cyberdrop-dl-patched as of version 6.10.1
| Motherless | https://motherless.com | motherless | Group: <br> - `/g/<group_name>`<br> - `/gi/<image>`<br> - `/gv/<video>`<br>Image: <br> - `/...`<br>User: <br> - `/f/...`<br> - `/u/...`<br>Video: <br> - `pending`<br><br><br>**NOTES**<br> - Galleries are NOT supported<br> |
| Nekohouse | https://nekohouse.su | nekohouse | Direct Links<br>Individual Post: <br> - `/<service>/<user>/post/...`<br>Model: <br> - `/<service>/user/<user>`<br> |
| NoodleMagazine | https://noodlemagazine.com | noodlemagazine | Search: <br> - `/video/<search_query`<br>Video: <br> - `/watch/<video_id>`<br> |
-| NudoStar | https://nudostar.com/forum/ | nudostar | Attachments: <br> - `/(attachments\|data\|uploads)/...`<br>Threads: <br> - `/(thread\|threads\|topic\|topics\|tema\|temas)/<thread_name_and_id>`<br> - `/goto/<post_id>`<br> - `/posts/<post_id>`<br><br><br>**NOTES**<br> - base crawler: Xenforo<br> |
+| NudoStar | https://nudostar.com/forum/ | nudostar | Attachments: <br> - `/(attachments\|data\|uploads)/...`<br>Threads: <br> - `/(thread\|topic\|tema\|threads\|topics\|temas)/<thread_name_and_id>`<br> - `/goto/<post_id>`<br> - `/posts/<post_id>`<br><br><br>**NOTES**<br> - base crawler: Xenforo<br> |
| NudoStarTV | https://nudostar.tv/ | nudostar.tv | Model: <br> - `/models/...`<br> |
| OmegaScans | https://omegascans.org | omegascans | Chapter: <br> - `/series/.../...`<br>Direct Links<br>Series: <br> - `/series/...`<br> |
| OneDrive | https://onedrive.com/ | 1drv.ms<br>onedrive.live.com | Access Link: <br> - `https://onedrive.live.com/?authkey=<KEY>&id=<ID>&cid=<CID>`<br>Share Link (anyone can access): <br> - `https://1drv.ms/b/<KEY>`<br> - `https://1drv.ms/f/<KEY>`<br> - `https://1drv.ms/t/<KEY>`<br> - `https://1drv.ms/u/<KEY>`<br> |
@@ -109,21 +110,22 @@ List of sites supported by cyberdrop-dl-patched as of version 6.10.1
| SendNow | https://send.now/ | send.now | Direct Links<br> |
| SendVid | https://sendvid.com/ | sendvid | Direct Links<br>Embeds: <br> - `/embed/...`<br>Videos: <br> - `/...`<br> |
| Sex.com | https://sex.com | sex | Shorts Profiles: <br> - `/shorts/<profile>`<br> |
-| SocialMediaGirls | https://forums.socialmediagirls.com | socialmediagirls | Attachments: <br> - `/(attachments\|data\|uploads)/...`<br>Threads: <br> - `/(thread\|threads\|topic\|topics\|tema\|temas)/<thread_name_and_id>`<br> - `/goto/<post_id>`<br> - `/posts/<post_id>`<br><br><br>**NOTES**<br> - base crawler: Xenforo<br> |
+| SocialMediaGirls | https://forums.socialmediagirls.com | socialmediagirls | Attachments: <br> - `/(attachments\|data\|uploads)/...`<br>Threads: <br> - `/(thread\|topic\|tema\|threads\|topics\|temas)/<thread_name_and_id>`<br> - `/goto/<post_id>`<br> - `/posts/<post_id>`<br><br><br>**NOTES**<br> - base crawler: Xenforo<br> |
| SpankBang | https://spankbang.com/ | spankbang | Playlist: <br> - `/playlist/<playlist-id>`<br>Video: <br> - `/embed/<video_id>`<br> - `/play/<video_id>`<br> - `/playlist/<playlist-id>-<video_id>`<br> - `/video/<video_id>`<br> |
| Streamable | https://streamable.com | streamable | Video: <br> - `/...`<br> |
| TWPornStars | https://www.twpornstars.com | indiantw.com<br>twanal.com<br>twgaymuscle.com<br>twgays.com<br>twlesbian.com<br>twmilf.com<br>twonfans.com<br>twpornstars.com<br>twteens.com<br>twtiktoks.com | Photo: <br> - `/...`<br> |
| ThisVid | https://thisvid.com | thisvid | Albums: <br> - `/albums/<album_name>`<br>Categories: <br> - `/categories/...`<br>Image: <br> - `/albums/<album_name>/<image_name>`<br>Members: <br> - `/members/<member_id>`<br>Search: <br> - `/search/?q=...`<br>Tags: <br> - `/tags/...`<br>Videos: <br> - `/videos/...`<br> |
| TikTok | https://tiktok.com/ | tiktok | Photo: <br> - `/@<user>/photo/<photo_id>`<br>User: <br> - `/@<user>`<br>Video: <br> - `/@<user>/video/<video_id>`<br> |
-| TitsInTops | https://titsintops.com/phpBB2 | titsintops | Attachments: <br> - `/(attachments\|data\|uploads)/...`<br>Threads: <br> - `/(thread\|threads\|topic\|topics\|tema\|temas)/<thread_name_and_id>`<br> - `/goto/<post_id>`<br> - `/posts/<post_id>`<br><br><br>**NOTES**<br> - base crawler: Xenforo<br> |
+| TitsInTops | https://titsintops.com/phpBB2 | titsintops | Attachments: <br> - `/(attachments\|data\|uploads)/...`<br>Threads: <br> - `/(thread\|topic\|tema\|threads\|topics\|temas)/<thread_name_and_id>`<br> - `/goto/<post_id>`<br> - `/posts/<post_id>`<br><br><br>**NOTES**<br> - base crawler: Xenforo<br> |
| Tokyomotion | https://www.tokyomotion.net | tokyomotion | Albums: <br> - `/album/<album_id>`<br> - `/user/<user>/albums/`<br>Photo: <br> - `/photo/<photo_id>`<br> - `/user/<user>/favorite/photos`<br>Playlist: <br> - `/user/<user>/favorite/videos`<br>Profiles: <br> - `/user/<user>`<br>Search Results: <br> - `/search?...`<br>Video: <br> - `/video/<video_id>`<br> |
| Toonily | https://toonily.com | toonily | Chapter: <br> - `/webtoon/.../...`<br>Direct Links<br>Webtoon: <br> - `/webtoon/...`<br> |
| TwitterImages | https://twimg.com/ | twimg | Photo: <br> - `/...`<br> |
+| ViperGirls | https://vipergirls.to | viper.click<br>vipergirls.to | Threads: <br> - `/goto/<post_id>`<br> - `/posts/<post_id>`<br> - `/threads/<thread_name>`<br> |
| Vipr.im | https://vipr.im | vipr.im | Image: <br> - `/...`<br>Thumbnail: <br> - `/th/...`<br> |
| WeTransfer | https://wetransfer.com/ | we.tl<br>wetransfer.com | Direct links: <br> - `download.wetransfer.com/...`<br>Public link: <br> - `wetransfer.com/<file_id>/<security_hash>`<br>Share Link: <br> - `wetransfer.com/<file_id>/<recipient_id>/<security_hash>`<br>Short Link: <br> - `we.tl/<id>`<br> |
| WordPressHTML | ::GENERIC CRAWLER:: | | All Posts: <br> - `/posts/`<br>Category: <br> - `/category/<category_slug>`<br>Date Range: <br> - `...?after=<date>`<br> - `...?before=<date&after=<date>`<br> - `...?before=<date>`<br>Post: <br> - `/<post_slug>/`<br>Tag: <br> - `/tag/<tag_slug>`<br><br><br>**NOTES**<br> - <br><br> For `Date Range`, <date> must be a valid iso 8601 date, ex: `2022-12-06`.<br><br> `Date Range` can be combined with `Category`, `Tag` and `All Posts`.<br> ex: To only download categories from a date range: ,<br> `/category/<category_slug>?before=<date>`<br> |
| WordPressMedia | ::GENERIC CRAWLER:: | | All Posts: <br> - `/posts/`<br>Category: <br> - `/category/<category_slug>`<br>Date Range: <br> - `...?after=<date>`<br> - `...?before=<date&after=<date>`<br> - `...?before=<date>`<br>Post: <br> - `/<post_slug>/`<br>Tag: <br> - `/tag/<tag_slug>`<br><br><br>**NOTES**<br> - <br><br> For `Date Range`, <date> must be a valid iso 8601 date, ex: `2022-12-06`.<br><br> `Date Range` can be combined with `Category`, `Tag` and `All Posts`.<br> ex: To only download categories from a date range: ,<br> `/category/<category_slug>?before=<date>`<br> |
-| XBunker | https://xbunker.nu/ | xbunker | Attachments: <br> - `/(attachments\|data\|uploads)/...`<br>Threads: <br> - `/(thread\|threads\|topic\|topics\|tema\|temas)/<thread_name_and_id>`<br> - `/goto/<post_id>`<br> - `/posts/<post_id>`<br><br><br>**NOTES**<br> - base crawler: Xenforo<br> |
+| XBunker | https://xbunker.nu/ | xbunker | Attachments: <br> - `/(attachments\|data\|uploads)/...`<br>Threads: <br> - `/(thread\|topic\|tema\|threads\|topics\|temas)/<thread_name_and_id>`<br> - `/goto/<post_id>`<br> - `/posts/<post_id>`<br><br><br>**NOTES**<br> - base crawler: Xenforo<br> |
| XBunkr | https://xbunkr.com | xbunkr | Albums: <br> - `/a/...`<br>Direct Links<br> |
| XXXBunker | https://xxxbunker.com | xxxbunker | Search: <br> - `/search/...`<br>Video: <br> - `/<video_id>`<br> |
| YandexDisk | https://disk.yandex.com.tr/ | disk.yandex<br>yadi.sk | Files: <br> - `/d/<folder_id>/<file_name>`<br>Folder: <br> - `/d/<folder_id>`<br><br><br>**NOTES**<br> - Does NOT support nested folders<br> |