Bulk asynchronous downloader for multiple file hosts
Python asynciobunkrcyberdropdownloaderdropboxfilestergofilegoogle-drivemediafiremegamega-nzonedrivepatreonpawchivepinterestpixeldraintransfer-ittwitchtwittervk
docs: update contributing
NTFSvolumea91d493dbcf3
diff --git a/CONTRIBUTING.md b/CONTRIBUTING.mdindex 5dcbe8c94..66c82c855 100644--- a/CONTRIBUTING.md+++ b/CONTRIBUTING.md@@ -6,66 +6,66 @@ Search for existing issues before creating a new one. If a bug already has issuewith extra information that might help reproduce or fix the problem. Duplicate issues will be closed with areference to the existing issue.-You _should_ [read the wiki](https://script-ware.gitbook.io/cyberdrop-dl/frequently-asked-questions) as it+You **SHOULD** [read the wiki](https://script-ware.gitbook.io/cyberdrop-dl/frequently-asked-questions) as itincludes solutions to some common problems.If your issue is a bug and hasn't been reported, open a new issue at <https://github.com/Cyberdrop-DL/cyberdrop-dl/issues/new/choose>.## Feature requests-You _may_ request a new feature by [opening an issue](#submitting-an-issue). If you would like to _implement_ a new feature,-you _should_ open an issue or a discussion with a proposal first and at least one use case. You _may_ submit a PR directly if your feature is small+You **MAY** request a new feature by [opening an issue](#submitting-an-issue). If you would like to implement a new feature,+you **SHOULD** open an issue or a discussion with a proposal first and at least one use case. You **MAY** submit a PR directly if your feature is smallwith a narrow scope.## Setting up the development environment`cyberdrop-dl` uses `uv` as a project management tool and requires python 3.12+. You can install it from <https://docs.astral.sh/uv/getting-started/installation/>.-You do not need to manually install install python as `uv` will automatically install a compatible python version (if required).+You do not need to manually install python as `uv` will automatically install a compatible python version (if required).Once you have `uv` installed, follow these steps:1. Clone the repo- ```sh+ ```powershellgit clone "https://github.com/Cyberdrop-DL/cyberdrop-dl"cd cyberdrop-dl```1. Install the project and its dependencies- ```sh+ ```powershelluv sync --all-extras```1. Install the pre-commit hooks:- ```sh+ ```powershelluv run prek install```1. Optionally, run the tests suite to make sure your dev environment is setup correctly-```sh- uv run pytest -v+```powershell+ uv run pytest```## Code Style### Standards-- Formatting:+#### FormattingThis project uses [ruff](https://docs.astral.sh/ruff) for formatting, linting and import sorting.We recommend [setting up your IDE](https://docs.astral.sh/ruff/editors/) to format and check with `ruff`, but you can always run`uv run ruff check --fix` then `uv run ruff format` in the root directory before submitting a pull request.-- Type checking:+#### Type checkingNot enforced but highly recommended. The project includes hardcoded config options for [`basedpyright`](https://docs.basedpyright.com/latest)-- Line width:+#### Line width120 character max@@ -85,60 +85,63 @@ all you need to do is `git add` those files again and retry your commit.## Implementing new changes> [!NOTE]-> `cyberdrop-dl` requires python 3.12+. You _must_ not use any syntax features not compatible with it+> `cyberdrop-dl` requires python 3.12+. You **MUST NOT** use any syntax features not compatible with it### Submitting a Pull Request (PR)> [!IMPORTANT]-> Before you start writing any code, you _should_ search the [repository](https://github.com/Cyberdrop-DL/cyberdrop-dl/pulls) for an open or closed PR+> Before you start writing any code, you **SHOULD** search the [repository](https://github.com/Cyberdrop-DL/cyberdrop-dl/pulls) for an open or closed PR> that relates to your submission. You don't want to duplicate effort.-- Checkout a _new_ branch before working on any changes, branching from `main`. You _must not_ work directly on `main`-- Implement your changes. Make sure to follow [code style conventions](#code-style). You _should_ add tests for any new functionality you introduce+- Checkout a **new** branch before working on any changes, branching from `main`. You **MUST NOT** work directly on `main`+- Implement your changes. Make sure to follow [code style conventions](#code-style). You **SHOULD** add tests for any new non-crawler specific functionality you introduce- Commit your changes using a descriptive commit message-- Fork the repository on GitHub, push your branch to your fork and open a pull request against the `main` branch of base repository.-- Make sure your pull request has a descriptive title and a short description. Sometimes the title of the PR is enough and no description is required-- You _must not_ use AI to generate your PR title/description. Write it with your own words-- Wait for a maintainer to review your PR and then address any comments they might have.+- Fork the repository on GitHub, push your branch to the fork and open a pull request against the `main` branch of base repository.+- Make sure your pull request has a descriptive title and a short description. Sometimes the title is enough and no description is required+- You **MUST NOT** use AI to generate your PR title/description. Write it with your own words+- Wait for a maintainer to review your PR and then address any comments they might haveIf everything is okay, your changes will be merged into the project.-### Adding support for a new site+## Adding support for a new site-There's no standard template for a new crawler, but there are a few conventions. You _should_ look at the code of another (hopefully similar) crawler+There's no standard template for a new crawler, but there are a few conventions. Some of these conventions are enforced by tests.++You **SHOULD** look at the code of another (hopefully similar) crawlerand based your crawler on that. A simple crawler to use as base is [CloudFlareStream](cyberdrop_dl/crawlers/cloudflare_stream.py).-- All crawlers _must_ inherit from the [base crawler](cyberdrop_dl/crawlers/crawler.py).-- All Crawlers _must_ have the word `Crawler` at the end of their class name-- All Crawlers _must_ override the `fetch` method. It's the primary method that decides what to do with an URL that matches with that crawler-- You _must_ not call any method that may fail (raise an exception) from within `fetch`-- All async method called from within fetch _must_ be public methods-- All public method _must_ be decorated with `error_handling_wrapper` to catch any unknown error-- You _should_ catch **expected** errors and re-raise them as a `ScrapeError` with an appropiate code and message-- You _should_ use valid HTTP codes for any `ScrapeError`. ex: raise `ScrapeError(410)` if you know a file was deleted-- You _may_ use HTTP code `422` (Unprocessable entity) for **expected** errors if no other HTTP code feels adequate-- You _may_ use a short string instead of an HTTP code to convey a better context for error (if required).- The string _must_ 20 characters or less as it will be shown on the `Scrape Errors` sections of the TUI-- You _should not_ write logic in a defensive way for **unexpected** errors. If you need to perform an operation that _may_ fail+- All crawlers **MUST** inherit from the [base crawler](cyberdrop_dl/crawlers/crawler.py).+- All crawlers **MUST** have the word `Crawler` at the end of their class name+- All crawlers **MUST** override the `fetch` method. It's the primary method that decides what to do with an URL that matches with that crawler+- You **MUST** not call any method that may fail (raise an exception) from within `fetch`+- All async method called from within fetch **MUST** be public methods+- All public method **MUST** be decorated with `error_handling_wrapper` to catch any unknown error+- You **SHOULD** catch **expected** errors and re-raise them as a `ScrapeError` with an appropiate code and message+- You **SHOULD** use valid HTTP codes for any `ScrapeError`. ex: raise `ScrapeError(410)` if you know a file was deleted+- You **MAY** use HTTP code `422` (Unprocessable entity) for **expected** errors if no other HTTP code feels adequate+- You **MAY** use a short string instead of an HTTP code to convey a better context for an error (if required).+ The string **SHOULD** be at most 20 characters long as it will be shown on the `Scrape Errors` sections of the TUI+- You **SHOULD NOT** write logic in a defensive way for **unexpected** errors. If you need to perform an operation that **MAY** failbut is not **expected** to fail (ex: a dictionary key lookup on an API response), do not try to catch the exception, let it bubble up.It will eventually be logged by the `error_handling_wrapper`-- You _should_ model site specific data into dataclasses, if possible-- You _must_ create a new task for any coroutine whose result is not needed to complete the current task-- You _must not_ create `MediaItem` objects manually. You _must_ use the `handle_file` method-- You _should_ look at all methods on the base crawler for possible helper functions. Their doctrings explain expected usage of each one.-- You _may_ add site specific CLI/config options at `cyberdrop_dl/config/crawlers.py`[cyberdrop_dl/config/crawlers.py] to use within the crawler.-- You _should_ not added unless absolutely required-- You _should_ add at least 1 test case for the crawler+- You **SHOULD** model site specific data into dataclasses, if possible+- You **MUST** create a new task for any coroutine whose result is not needed to complete the current task+- You **MUST NOT** create `MediaItem` objects manually. You **MUST** use the `handle_file` method+- You **SHOULD** look at all methods on the base crawler for possible helper functions. Their doctrings explain expected usage of each one.+- You **MAY** add site specific CLI/config options at [cyberdrop_dl/config/crawlers.py](cyberdrop_dl/config/crawlers.py) to use within the crawler.+- You **SHOULD NOT** add crawler specific CLI/options unless absolutely necessary+- You **SHOULD** add at least 1 test case for the crawler-#### Creating tests for a new site+## Creating tests for a new site1. Create a file with the name `URLs.txt` at the root of the repo-2. Paste all the URLs you want to test inside the file-3. Run [scripts/tools/make_test_cases.py](scripts/tools/make_test_cases.py). If successful, it will generate a file at [tests/crawlers/test_cases](tests/crawlers/test_cases)-4. Check that the scraped data matches the expected result-5. Rename the generated test file, from `test_case_<domain>.py` to just `<domain>.py`-6. Run the test again manually to confirm it it is reproducible:--```sh+1. Paste all the URLs you want to test inside the file+1. Run [scripts/tools/make_test_cases.py](scripts/tools/make_test_cases.py). If successful, it will generate a+ new file inside [tests/crawlers/test_cases](tests/crawlers/test_cases)+1. Check that the scraped data matches the expected result+1. Rename the generated test file, from `test_case_<domain>.py` to just `<domain>.py`+1. Run the test again manually to confirm it it is reproducible:++```powershelluv run pytest -x --test-crawlers <domain>```