Bulk asynchronous downloader for multiple file hosts
Python asynciobunkrcyberdropdownloaderdropboxfilestergofilegoogle-drivemediafiremegamega-nzonedrivepatreonpawchivepinterestpixeldraintransfer-ittwitchtwittervk
docs: update cli overview
NTFSvolumeb0c6b4afbdbc
diff --git a/docs/reference/cli-arguments.md b/docs/reference/cli-arguments.mdindex 2053e5945..35dca4e98 100644--- a/docs/reference/cli-arguments.md+++ b/docs/reference/cli-arguments.md@@ -315,7 +315,7 @@ sorting:--sorted-video SORTED_VIDEOgeneral:- --allow-insecure-connections, --no-allow-insecure-connections+ --ssl-context SSL_CONTEXT--disable-crawlers [DISABLE_CRAWLERS ...]--enable-generic-crawler, --no-enable-generic-crawler--flaresolverr FLARESOLVERR@@ -345,4 +345,9 @@ ui_options:--scraping-item-limit SCRAPING_ITEM_LIMIT--vi-mode, --no-vi-mode+generic_crawlers_instances:+ --wordpress-media [WORDPRESS_MEDIA ...]+ --wordpress-html [WORDPRESS_HTML ...]+ --discourse [DISCOURSE ...]+```diff --git a/docs/reference/supported-websites.md b/docs/reference/supported-websites.mdindex dea93d654..0114779f0 100644--- a/docs/reference/supported-websites.md+++ b/docs/reference/supported-websites.md@@ -28,34 +28,35 @@ Supported domains via Real-Debrid include `mega.nz`, `rapidgator`, `google drive## Supported sites-List of sites supported by cyberdrop-dl-patched as of version 6.10.1+List of sites supported by cyberdrop-dl-patched as of version 7.0.0| Site | Primary URL | Supported Domains | Supported Paths ||---------------------------|---------------------------------------------|-----------------------------------------------------------------------------------------------------------------------------------------------------------------------|--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|| 4chan | https://boards.4chan.org | 4chan | Board: <br> - `/<board>`<br>Thread: <br> - `/thread`<br> || 8Muses | https://comics.8muses.com | 8muses | Album: <br> - `/comics/album/...`<br> || AcidImg | https://acidimg.cc | acidimg.cc | Image: <br> - `/i/...`<br>Thumbnail: <br> - `/upload/...`<br> |-| AllPornComix | https://forum.allporncomix.com | allporncomix | Attachments: <br> - `/(attachments\|data\|uploads)/...`<br>Threads: <br> - `/(thread\|threads\|topic\|topics\|tema\|temas)/<thread_name_and_id>`<br> - `/goto/<post_id>`<br> - `/posts/<post_id>`<br><br><br>**NOTES**<br> - base crawler: Xenforo<br> |+| AllPornComix | https://forum.allporncomix.com | allporncomix | Attachments: <br> - `/(attachments\|data\|uploads)/...`<br>Threads: <br> - `/(thread\|topic\|tema\|threads\|topics\|temas)/<thread_name_and_id>`<br> - `/goto/<post_id>`<br> - `/posts/<post_id>`<br><br><br>**NOTES**<br> - base crawler: Xenforo<br> || ArchiveBate | https://www.archivebate.store | archivebate | Video: <br> - `/watch/<video_id>`<br> |-| Bellazon | https://www.bellazon.com/main/ | bellazon | Attachments: <br> - `/(attachments\|data\|uploads)/...`<br>Threads: <br> - `/(thread\|threads\|topic\|topics\|tema\|temas)/<thread_name_and_id>`<br> - `/goto/<post_id>`<br> - `/posts/<post_id>`<br><br><br>**NOTES**<br> - base crawler: Xenforo<br> |+| Beeg.com | https://beeg.com/ | beeg.com | Video: <br> - `/<video_id>`<br> - `/video/<video_id>`<br> |+| Bellazon | https://www.bellazon.com/main/ | bellazon | Attachments: <br> - `/(attachments\|data\|uploads)/...`<br>Threads: <br> - `/(thread\|topic\|tema\|threads\|topics\|temas)/<thread_name_and_id>`<br> - `/goto/<post_id>`<br> - `/posts/<post_id>`<br><br><br>**NOTES**<br> - base crawler: Invision<br> || BestPrettyGirl | https://bestprettygirl.com/ | bestprettygirl.com | All Posts: <br> - `/posts/`<br>Category: <br> - `/category/<category_slug>`<br>Date Range: <br> - `...?after=<date>`<br> - `...?before=<date&after=<date>`<br> - `...?before=<date>`<br>Post: <br> - `/<post_slug>/`<br>Tag: <br> - `/tag/<tag_slug>`<br><br><br>**NOTES**<br> - <br><br> For `Date Range`, <date> must be a valid iso 8601 date, ex: `2022-12-06`.<br><br> `Date Range` can be combined with `Category`, `Tag` and `All Posts`.<br> ex: To only download categories from a date range: ,<br> `/category/<category_slug>?before=<date>`<br> |-| Box | https://www.box.com | app.box.com | File or Folder: <br> - `app.box.com/s?sh=<share_code>`<br> |+| Box | https://www.box.com | app.box.com | Embeded File or Folder: <br> - `app.box.com/embed/s?sh=<share_code>`<br> - `app.box.com/embed_widget/s?sh=<share_code>`<br>File or Folder: <br> - `app.box.com/s?sh=<share_code>`<br> || Bunkrr | https://bunkr.site | bunkr<br>bunkrr | Album: <br> - `/a/...`<br>Direct Links<br>File: <br> - `/f/...`<br>Video: <br> - `/v/...`<br> || BuzzHeavier | https://buzzheavier.com | buzzheavier.com | Direct Links<br> || Camwhores.tv | https://www.camwhores.tv | camwhores.tv | Albums: <br> - `/albums/<album_name>`<br>Categories: <br> - `/categories/...`<br>Image: <br> - `/albums/<album_name>/<image_name>`<br>Members: <br> - `/members/<member_id>`<br>Search: <br> - `/search/?q=...`<br>Tags: <br> - `/tags/...`<br>Videos: <br> - `/videos/...`<br> || Catbox | https://catbox.moe | files.catbox.moe<br>litter.catbox.moe | Direct Links<br> |-| CelebForum | https://celebforum.to | celebforum | Attachments: <br> - `/(attachments\|data\|uploads)/...`<br>Threads: <br> - `/(thread\|threads\|topic\|topics\|tema\|temas)/<thread_name_and_id>`<br> - `/goto/<post_id>`<br> - `/posts/<post_id>`<br><br><br>**NOTES**<br> - base crawler: Xenforo<br> |+| CelebForum | https://celebforum.to | celebforum | Attachments: <br> - `/(attachments\|data\|uploads)/...`<br>Threads: <br> - `/(thread\|topic\|tema\|threads\|topics\|temas)/<thread_name_and_id>`<br> - `/goto/<post_id>`<br> - `/posts/<post_id>`<br><br><br>**NOTES**<br> - base crawler: Xenforo<br> || Coomer | https://coomer.su | coomer | Direct Links<br>Favorites: <br> - `/favorites/<user>`<br>Individual Post: <br> - `<service>/<user>/post/...`<br>Model: <br> - `/<service>/user/<user>`<br>Search: <br> - `/search?...`<br> || Cyberdrop | https://cyberdrop.me/ | cyberdrop | Album: <br> - `/a/...`<br>Direct Links<br>File: <br> - `/f/...`<br> || Cyberfile | https://cyberfile.me/ | cyberfile | Files: <br> - `/...`<br>Folder: <br> - `/folder/...`<br>Shared folder: <br> - `/shared/...`<br> || DirtyShip | https://dirtyship.com | dirtyship | Categorie: <br> - `/category/...`<br>Gallery: <br> - `/gallery/...`<br>Photo: <br> - `/gallery/.../...`<br>Tag: <br> - `/tag/...`<br>Video: <br> - `/<video_name>`<br> || Discourse | ::GENERIC CRAWLER:: | | Attachments: <br> - `/uploads/...`<br>Topic: <br> - `/t/<topic_name>/<topic_id>`<br> - `/t/<topic_name>/<topic_id>/<post_number>`<br><br><br>**NOTES**<br> - If the URL includes <post_number>, posts with a number lower that it won't be scraped<br> || DoodStream | https://doodstream.com/ | all3do.com<br>do7go.com<br>dood.re<br>dood.yt<br>doodcdn<br>doodstream<br>doodstream.co<br>vidply.com | Video: <br> - `/e/<video_id>`<br> |-| Dropbox | https://dropbox.com/ | dropbox | File: <br> - `/s/...`<br> - `/scl/fi/...`<br>Folder: <br> - `/scl/fo/...`<br> - `/sh/...`<br><br><br>**NOTES**<br> - Folders will be downloaded as a zip file. See: [--download-dropbox-folders-as-zip](https://script-ware.gitbook.io/cyberdrop-dl/reference/cli-arguments#download-dropbox-folders-as-zip)<br> |+| Dropbox | https://www.dropbox.com/ | dropbox | File: <br> - `/s/...`<br> - `/scl/fi/<file_id>`<br> - `/scl/fo/<token1>/<token2>?preview=<filename>`<br>Folder: <br> - `/scl/fo/<token1>/<token2>`<br> - `/sh/...`<br><br><br>**NOTES**<br> - Folders will be downloaded as a zip file. See: [--download-dropbox-folders-as-zip](https://script-ware.gitbook.io/cyberdrop-dl/reference/cli-arguments#download-dropbox-folders-as-zip)<br> || E-Hentai | https://e-hentai.org/ | e-hentai | Album: <br> - `/g/...`<br>File: <br> - `/s/...`<br> || E621 | https://e621.net | e621.net | Pools: <br> - `/pools/...`<br>Post: <br> - `/posts/...`<br>Tags: <br> - `/posts?tags=...`<br> || Erome | https://www.erome.com | erome | Album: <br> - `/a/...`<br>Profile: <br> - `/...`<br> |-| F95Zone | https://f95zone.to | f95zone | Attachments: <br> - `/(attachments\|data\|uploads)/...`<br>Threads: <br> - `/(thread\|threads\|topic\|topics\|tema\|temas)/<thread_name_and_id>`<br> - `/goto/<post_id>`<br> - `/posts/<post_id>`<br><br><br>**NOTES**<br> - base crawler: Xenforo<br> |+| F95Zone | https://f95zone.to | f95zone | Attachments: <br> - `/(attachments\|data\|uploads)/...`<br>Threads: <br> - `/(thread\|topic\|tema\|threads\|topics\|temas)/<thread_name_and_id>`<br> - `/goto/<post_id>`<br> - `/posts/<post_id>`<br><br><br>**NOTES**<br> - base crawler: Xenforo<br> || Fapello | https://fapello.su/ | fapello | Individual Post: <br> - `/.../...`<br>Model: <br> - `/...`<br> || Fileditch | https://fileditchfiles.me/ | fileditch | Direct Links<br> || FilesVC | https://files.vc | files.vc | Direct Links<br> |@@ -76,7 +77,7 @@ List of sites supported by cyberdrop-dl-patched as of version 6.10.1| InfluencerBitches | https://influencerbitches.com | influencerbitches | Model: <br> - `/model/...`<br> || JPG5 | https://jpg5.su | host.church<br>jpeg.pet<br>jpg.church<br>jpg.fish<br>jpg.fishing<br>jpg.homes<br>jpg.pet<br>jpg1.su<br>jpg2.su<br>jpg3.su<br>jpg4.su<br>jpg5.su | Album: <br> - `/a/...`<br>Direct Links<br>Image: <br> - `/img/...`<br>Profile: <br> - `/<user_name>`<br> || Kemono | https://kemono.su | kemono | Direct Links<br>Discord Server: <br> - `/discord/<server_id>`<br>Discord Server Channel: <br> - `/discord/server/...#...`<br>Favorites: <br> - `/favorites/<user>`<br>Individual Post: <br> - `<service>/<user>/post/...`<br>Model: <br> - `/<service>/user/<user>`<br>Search: <br> - `/search?...`<br> |-| LeakedModels | https://leakedmodels.com/forum/ | leakedmodels | Attachments: <br> - `/(attachments\|data\|uploads)/...`<br>Threads: <br> - `/(thread\|threads\|topic\|topics\|tema\|temas)/<thread_name_and_id>`<br> - `/goto/<post_id>`<br> - `/posts/<post_id>`<br><br><br>**NOTES**<br> - base crawler: Xenforo<br> |+| LeakedModels | https://leakedmodels.com/forum/ | leakedmodels | Attachments: <br> - `/(attachments\|data\|uploads)/...`<br>Threads: <br> - `/(thread\|topic\|tema\|threads\|topics\|temas)/<thread_name_and_id>`<br> - `/goto/<post_id>`<br> - `/posts/<post_id>`<br><br><br>**NOTES**<br> - base crawler: Xenforo<br> || Luscious | https://members.luscious.net | luscious | Album: <br> - `/albums/...`<br> || Mediafire | https://www.mediafire.com/ | mediafire | File: <br> - `/file/...`<br>Folder: <br> - `/folder/...`<br> || MissAV | https://missav.ws | missav | Video: <br> - `/...`<br> |@@ -84,7 +85,7 @@ List of sites supported by cyberdrop-dl-patched as of version 6.10.1| Motherless | https://motherless.com | motherless | Group: <br> - `/g/<group_name>`<br> - `/gi/<image>`<br> - `/gv/<video>`<br>Image: <br> - `/...`<br>User: <br> - `/f/...`<br> - `/u/...`<br>Video: <br> - `pending`<br><br><br>**NOTES**<br> - Galleries are NOT supported<br> || Nekohouse | https://nekohouse.su | nekohouse | Direct Links<br>Individual Post: <br> - `/<service>/<user>/post/...`<br>Model: <br> - `/<service>/user/<user>`<br> || NoodleMagazine | https://noodlemagazine.com | noodlemagazine | Search: <br> - `/video/<search_query`<br>Video: <br> - `/watch/<video_id>`<br> |-| NudoStar | https://nudostar.com/forum/ | nudostar | Attachments: <br> - `/(attachments\|data\|uploads)/...`<br>Threads: <br> - `/(thread\|threads\|topic\|topics\|tema\|temas)/<thread_name_and_id>`<br> - `/goto/<post_id>`<br> - `/posts/<post_id>`<br><br><br>**NOTES**<br> - base crawler: Xenforo<br> |+| NudoStar | https://nudostar.com/forum/ | nudostar | Attachments: <br> - `/(attachments\|data\|uploads)/...`<br>Threads: <br> - `/(thread\|topic\|tema\|threads\|topics\|temas)/<thread_name_and_id>`<br> - `/goto/<post_id>`<br> - `/posts/<post_id>`<br><br><br>**NOTES**<br> - base crawler: Xenforo<br> || NudoStarTV | https://nudostar.tv/ | nudostar.tv | Model: <br> - `/models/...`<br> || OmegaScans | https://omegascans.org | omegascans | Chapter: <br> - `/series/.../...`<br>Direct Links<br>Series: <br> - `/series/...`<br> || OneDrive | https://onedrive.com/ | 1drv.ms<br>onedrive.live.com | Access Link: <br> - `https://onedrive.live.com/?authkey=<KEY>&id=<ID>&cid=<CID>`<br>Share Link (anyone can access): <br> - `https://1drv.ms/b/<KEY>`<br> - `https://1drv.ms/f/<KEY>`<br> - `https://1drv.ms/t/<KEY>`<br> - `https://1drv.ms/u/<KEY>`<br> |@@ -109,21 +110,22 @@ List of sites supported by cyberdrop-dl-patched as of version 6.10.1| SendNow | https://send.now/ | send.now | Direct Links<br> || SendVid | https://sendvid.com/ | sendvid | Direct Links<br>Embeds: <br> - `/embed/...`<br>Videos: <br> - `/...`<br> || Sex.com | https://sex.com | sex | Shorts Profiles: <br> - `/shorts/<profile>`<br> |-| SocialMediaGirls | https://forums.socialmediagirls.com | socialmediagirls | Attachments: <br> - `/(attachments\|data\|uploads)/...`<br>Threads: <br> - `/(thread\|threads\|topic\|topics\|tema\|temas)/<thread_name_and_id>`<br> - `/goto/<post_id>`<br> - `/posts/<post_id>`<br><br><br>**NOTES**<br> - base crawler: Xenforo<br> |+| SocialMediaGirls | https://forums.socialmediagirls.com | socialmediagirls | Attachments: <br> - `/(attachments\|data\|uploads)/...`<br>Threads: <br> - `/(thread\|topic\|tema\|threads\|topics\|temas)/<thread_name_and_id>`<br> - `/goto/<post_id>`<br> - `/posts/<post_id>`<br><br><br>**NOTES**<br> - base crawler: Xenforo<br> || SpankBang | https://spankbang.com/ | spankbang | Playlist: <br> - `/playlist/<playlist-id>`<br>Video: <br> - `/embed/<video_id>`<br> - `/play/<video_id>`<br> - `/playlist/<playlist-id>-<video_id>`<br> - `/video/<video_id>`<br> || Streamable | https://streamable.com | streamable | Video: <br> - `/...`<br> || TWPornStars | https://www.twpornstars.com | indiantw.com<br>twanal.com<br>twgaymuscle.com<br>twgays.com<br>twlesbian.com<br>twmilf.com<br>twonfans.com<br>twpornstars.com<br>twteens.com<br>twtiktoks.com | Photo: <br> - `/...`<br> || ThisVid | https://thisvid.com | thisvid | Albums: <br> - `/albums/<album_name>`<br>Categories: <br> - `/categories/...`<br>Image: <br> - `/albums/<album_name>/<image_name>`<br>Members: <br> - `/members/<member_id>`<br>Search: <br> - `/search/?q=...`<br>Tags: <br> - `/tags/...`<br>Videos: <br> - `/videos/...`<br> || TikTok | https://tiktok.com/ | tiktok | Photo: <br> - `/@<user>/photo/<photo_id>`<br>User: <br> - `/@<user>`<br>Video: <br> - `/@<user>/video/<video_id>`<br> |-| TitsInTops | https://titsintops.com/phpBB2 | titsintops | Attachments: <br> - `/(attachments\|data\|uploads)/...`<br>Threads: <br> - `/(thread\|threads\|topic\|topics\|tema\|temas)/<thread_name_and_id>`<br> - `/goto/<post_id>`<br> - `/posts/<post_id>`<br><br><br>**NOTES**<br> - base crawler: Xenforo<br> |+| TitsInTops | https://titsintops.com/phpBB2 | titsintops | Attachments: <br> - `/(attachments\|data\|uploads)/...`<br>Threads: <br> - `/(thread\|topic\|tema\|threads\|topics\|temas)/<thread_name_and_id>`<br> - `/goto/<post_id>`<br> - `/posts/<post_id>`<br><br><br>**NOTES**<br> - base crawler: Xenforo<br> || Tokyomotion | https://www.tokyomotion.net | tokyomotion | Albums: <br> - `/album/<album_id>`<br> - `/user/<user>/albums/`<br>Photo: <br> - `/photo/<photo_id>`<br> - `/user/<user>/favorite/photos`<br>Playlist: <br> - `/user/<user>/favorite/videos`<br>Profiles: <br> - `/user/<user>`<br>Search Results: <br> - `/search?...`<br>Video: <br> - `/video/<video_id>`<br> || Toonily | https://toonily.com | toonily | Chapter: <br> - `/webtoon/.../...`<br>Direct Links<br>Webtoon: <br> - `/webtoon/...`<br> || TwitterImages | https://twimg.com/ | twimg | Photo: <br> - `/...`<br> |+| ViperGirls | https://vipergirls.to | viper.click<br>vipergirls.to | Threads: <br> - `/goto/<post_id>`<br> - `/posts/<post_id>`<br> - `/threads/<thread_name>`<br> || Vipr.im | https://vipr.im | vipr.im | Image: <br> - `/...`<br>Thumbnail: <br> - `/th/...`<br> || WeTransfer | https://wetransfer.com/ | we.tl<br>wetransfer.com | Direct links: <br> - `download.wetransfer.com/...`<br>Public link: <br> - `wetransfer.com/<file_id>/<security_hash>`<br>Share Link: <br> - `wetransfer.com/<file_id>/<recipient_id>/<security_hash>`<br>Short Link: <br> - `we.tl/<id>`<br> || WordPressHTML | ::GENERIC CRAWLER:: | | All Posts: <br> - `/posts/`<br>Category: <br> - `/category/<category_slug>`<br>Date Range: <br> - `...?after=<date>`<br> - `...?before=<date&after=<date>`<br> - `...?before=<date>`<br>Post: <br> - `/<post_slug>/`<br>Tag: <br> - `/tag/<tag_slug>`<br><br><br>**NOTES**<br> - <br><br> For `Date Range`, <date> must be a valid iso 8601 date, ex: `2022-12-06`.<br><br> `Date Range` can be combined with `Category`, `Tag` and `All Posts`.<br> ex: To only download categories from a date range: ,<br> `/category/<category_slug>?before=<date>`<br> || WordPressMedia | ::GENERIC CRAWLER:: | | All Posts: <br> - `/posts/`<br>Category: <br> - `/category/<category_slug>`<br>Date Range: <br> - `...?after=<date>`<br> - `...?before=<date&after=<date>`<br> - `...?before=<date>`<br>Post: <br> - `/<post_slug>/`<br>Tag: <br> - `/tag/<tag_slug>`<br><br><br>**NOTES**<br> - <br><br> For `Date Range`, <date> must be a valid iso 8601 date, ex: `2022-12-06`.<br><br> `Date Range` can be combined with `Category`, `Tag` and `All Posts`.<br> ex: To only download categories from a date range: ,<br> `/category/<category_slug>?before=<date>`<br> |-| XBunker | https://xbunker.nu/ | xbunker | Attachments: <br> - `/(attachments\|data\|uploads)/...`<br>Threads: <br> - `/(thread\|threads\|topic\|topics\|tema\|temas)/<thread_name_and_id>`<br> - `/goto/<post_id>`<br> - `/posts/<post_id>`<br><br><br>**NOTES**<br> - base crawler: Xenforo<br> |+| XBunker | https://xbunker.nu/ | xbunker | Attachments: <br> - `/(attachments\|data\|uploads)/...`<br>Threads: <br> - `/(thread\|topic\|tema\|threads\|topics\|temas)/<thread_name_and_id>`<br> - `/goto/<post_id>`<br> - `/posts/<post_id>`<br><br><br>**NOTES**<br> - base crawler: Xenforo<br> || XBunkr | https://xbunkr.com | xbunkr | Albums: <br> - `/a/...`<br>Direct Links<br> || XXXBunker | https://xxxbunker.com | xxxbunker | Search: <br> - `/search/...`<br>Video: <br> - `/<video_id>`<br> || YandexDisk | https://disk.yandex.com.tr/ | disk.yandex<br>yadi.sk | Files: <br> - `/d/<folder_id>/<file_name>`<br>Folder: <br> - `/d/<folder_id>`<br><br><br>**NOTES**<br> - Does NOT support nested folders<br> |