🕷️ The BEST Web Scraping framework that handles everything from a single request to a full-scale crawl! Don't be shy, join here: https://discord.gg/EMgGbDceNQ and follow here for daily tips and tricks: https://x.com/Scrapling_dev
Python aiai-scrapingautomationcrawlercrawlingcrawling-pythondatadata-extractionmcpmcp-serverplaywrightpythonscrapingselectorsstealthweb-scraperweb-scrapingweb-scraping-pythonwebscrapingxpath
docs: Updating the contribution guide
Karim shoair763363457fe9
diff --git a/.github/PULL_REQUEST_TEMPLATE.md b/.github/PULL_REQUEST_TEMPLATE.mdindex cd477fe..b8eac74 100644--- a/.github/PULL_REQUEST_TEMPLATE.md+++ b/.github/PULL_REQUEST_TEMPLATE.md@@ -5,10 +5,8 @@## Proposed change<!--- Describe the big picture of your changes here to communicate to the- maintainers why we should accept this pull request. If it fixes a bug- or resolves a feature request, be sure to link to that issue in the- additional information section.+ Describe the big picture of your changes here to communicate to the maintainers why we should accept this pull request.+ If it fixes a bug or resolves a feature request, be sure to link to that issue in the additional information section.-->@@ -34,12 +32,12 @@### Additional information<!--- Details are important, and help maintainers processing your PR.+ Details are important and help maintainers processing your PR.Please be sure to fill out additional details, if applicable.-->-- This PR fixes or closes issue: fixes #-- This PR is related to issue:+- This PR fixes or closes an issue: fixes #+- This PR is related to an issue: #- Link to documentation pull request: **### Checklist:diff --git a/CONTRIBUTING.md b/CONTRIBUTING.mdindex 0b1fe2b..740033b 100644--- a/CONTRIBUTING.md+++ b/CONTRIBUTING.md@@ -1,39 +1,106 @@# Contributing to Scrapling-Everybody is invited and welcome to contribute to Scrapling. Smaller changes have a better chance to get included in a timely manner. Adding unit tests for new features or test cases for bugs you've fixed help us to ensure that the Pull Request (PR) is fine.-There is a lot to do...-- If you are not a developer perhaps you would like to help with the [documentation](https://github.com/D4Vinci/Scrapling/tree/docs)?-- If you are a developer, most of the features I'm planning to add in the future are moved to [roadmap file](https://github.com/D4Vinci/Scrapling/blob/main/ROADMAP.md) so consider reading it.+Thank you for your interest in contributing to Scrapling!-Scrapling includes a comprehensive test suite which can be executed with pytest:-```bash-$ pytest-=============================== test session starts ===============================-platform darwin -- Python 3.12.7, pytest-8.3.3, pluggy-1.5.0-rootdir: /<some_where>/Scrapling-configfile: pytest.ini-plugins: cov-5.0.0, anyio-4.6.0-collected 16 items+Everybody is invited and welcome to contribute to Scrapling.-tests/test_parser_functions.py ................ [100%]+Minor changes have a better chance of being included promptly. Adding unit tests for new features or test cases for bugs you've fixed helps us ensure that the Pull Request (PR) is acceptable.++There are many ways to contribute to Scrapling. Here are some of them:++- Report bugs and request features using the [GitHub issues](https://github.com/D4Vinci/Scrapling/issues). Please follow the issue template to help us resolve your issue quickly.+- Blog about Scrapling. Tell the world how you’re using Scrapling. This will help newcomers with more examples and increase the Scrapling project's visibility.+- Join the [Discord community](https://discord.gg/EMgGbDceNQ) and share your ideas on how to improve Scrapling. We’re always open to suggestions.+- If you are not a developer, perhaps you would like to help with translating the [documentation](https://github.com/D4Vinci/Scrapling/tree/docs)?-=============================== 16 passed in 0.22s ================================-```-Also, consider setting the scrapling logging level to `debug` so it's easier to know what's happening in the background.-```python->>> import logging->>> logging.getLogger("scrapling").setLevel(logging.DEBUG)-```-### The process is straight-forward.+## Finding work- - Read [How to get faster PR reviews](https://github.com/kubernetes/community/blob/master/contributors/guide/pull-requests.md#best-practices-for-faster-reviews) by Kubernetes (but skip step 0 and 1)- - Fork Scrapling [git repository](https://github.com/D4Vinci/Scrapling).- - Make your changes.- - Ensure tests work.- - Create a Pull Request against the [**dev**](https://github.com/D4Vinci/Scrapling/tree/dev) branch of Scrapling.+If you have decided to make a contribution to Scrapling, but you do not know what to contribute, here are some ways to find pending work:-### Installing the latest changes from the dev branch+- Check out the [contribution](https://github.com/D4Vinci/Scrapling/contribute) GitHub page, which lists open issues tagged as good first issue. These issues provide a good starting point.+- There are also the [help wanted](https://github.com/D4Vinci/Scrapling/issues?q=is%3Aissue%20label%3A%22help%20wanted%22%20state%3Aopen) issues, but know that some may require familiarity with the Scrapling code base first. You can also target any other issue, provided it is not tagged as `invalid`, `wontfix`, or similar tags.+- If you enjoy writing automated tests, you can work on increasing our test coverage. Currently, the test coverage is around 90–92%.+- Join the [Discord community](https://discord.gg/EMgGbDceNQ) and ask questions in the `#help` channel.++## Coding style+Please follow these coding conventions as we do when writing code for Scrapling:+- We use [pre-commit](https://pre-commit.com/) to automatically address simple code issues before every commit, so please install it and run `pre-commit install` to set it up. This will install hooks to run [ruff](https://docs.astral.sh/ruff/), [bandit](https://github.com/PyCQA/bandit), and [vermin](https://github.com/netromdk/vermin) on every commit. We are currently using a workflow to automatically run these tools on every PR, so if your code doesn't pass these checks, the PR will be rejected.+- We use type hints for better code clarity and [pyright](https://github.com/microsoft/pyright) for static type checking, which depends on the type hints, of course.+- We use the conventional commit messages format as [here](https://gist.github.com/qoomon/5dfcdf8eec66a051ecd85625518cfd13#types), so for example, we use the following prefixes for commit messages:++ | Prefix | When to use it |+ |-------------|--------------------------|+ | `feat:` | New feature added |+ | `fix:` | Bug fix |+ | `docs:` | Documentation change/add |+ | `test:` | Tests |+ | `refactor:` | Code refactoring |+ | `chore:` | Maintenance tasks |++ Then include the details of the change in the body/description of the commit message.++ Example:+ ```+ feat: add `adaptive` for similar elements++ - Added find_similar() method+ - Implemented pattern matching+ - Added tests and documentation+ ```++> Please don’t put your name in the code you contribute; git provides enough metadata to identify the author of the code.++## Development+Setting the scrapling logging level to `debug` makes it easier to know what's happening in the background.+```python+import logging+logging.getLogger("scrapling").setLevel(logging.DEBUG)+```+Bonus: You can install the beta of the upcoming update from the dev branch as follows```commandlinepip3 install git+https://github.com/D4Vinci/Scrapling.git@dev```++## Building Documentation+Documentation is built using [MkDocs](https://www.mkdocs.org/). You can build it locally using the following commands:+```bash+pip install mkdocs-material+mkdocs serve # Local preview+mkdocs build # Build the static site+```++## Tests+Scrapling includes a comprehensive test suite that can be executed with pytest. However, first, you need to install all libraries and `pytest-plugins` listed in `tests/requirements.txt`. Then, running the tests will result in an output like this:+ ```bash+ $ pytest tests -n auto+ =============================== test session starts ===============================+ platform darwin -- Python 3.13.8, pytest-8.4.2, pluggy-1.6.0 -- /Users/<redacted>/.venv/bin/python3.13+ cachedir: .pytest_cache+ rootdir: /Users/<redacted>/scrapling+ configfile: pytest.ini+ plugins: asyncio-1.2.0, anyio-4.11.0, xdist-3.8.0, httpbin-2.1.0, cov-7.0.0+ asyncio: mode=Mode.AUTO, debug=False, asyncio_default_fixture_loop_scope=function, asyncio_default_test_loop_scope=function+ 10 workers [271 items]+ scheduling tests via LoadScheduling++ ...<shortened>...++ =============================== 271 passed in 52.68s ==============================+ ```+Hence, we used `-n auto` in the command above to run tests in threads to increase speed.++Bonus: You can also see the test coverage with the `pytest` plugin below+```bash+pytest --cov=scrapling tests/+```++## Making a Pull Request+To ensure that your PR gets accepted, please make sure that your PR is based on the latest changes from the dev branch and that it satisfies the following requirements:++- The PR should be made against the [**dev**](https://github.com/D4Vinci/Scrapling/tree/dev) branch of Scrapling. Any PR made against the main branch will be rejected.+- The code should be passing all available tests. We are using tox with GitHub's CI to run the current tests on all supported Python versions with every commit.+- The code should be passing all code quality checks we mentioned above. We are using GitHub's CI to enforce the code style checks performed by pre-commit. If you were using the pre-commit hooks we discussed above, you should not see any issues when committing your changes.+- Make your changes, keep the code clean with an explanation of any part that might be vague, and remember to create a separate virtual environment for this project.+- If you are adding a new feature, please add tests for it.+- If you are fixing a bug, please add code with the PR that reproduces the bug.\ No newline at end of filediff --git a/docs/contributing.md b/docs/contributing.mddeleted file mode 100644index 463ba85..0000000--- a/docs/contributing.md+++ /dev/null@@ -1,102 +0,0 @@-Thank you for your interest in contributing to Scrapling!--Everybody is invited and welcome to contribute to Scrapling.--Smaller changes have a better chance of getting included in a timely manner. Adding unit tests for new features or test cases for bugs you've fixed helps us to ensure that the Pull Request (PR) is acceptable.--There is a lot to do...--- If you are not a developer, you can help us improve the documentation.-- If you are a developer, most of the features I'm planning to add in the future are moved to [roadmap file](https://github.com/D4Vinci/Scrapling/blob/main/ROADMAP.md), so consider reading it.--## Running tests-Scrapling includes a comprehensive test suite that can be executed with pytest, but first, you need to install all libraries and `pytest-plugins` inside `tests/requirements.txt`. Then, running the tests will result in an output like this:- ```bash- $ pytest tests- =============================== test session starts ===============================- platform darwin -- Python 3.12.8, pytest-8.3.3, pluggy-1.5.0 -- /Users/<redacted>/.venv/bin/python3.12- cachedir: .pytest_cache- rootdir: /Users/<redacted>/scrapling- configfile: pytest.ini- plugins: cov-5.0.0, asyncio-0.25.0, base-url-2.1.0, httpbin-2.1.0, playwright-0.5.2, anyio-4.6.2.post1, xdist-3.6.1, typeguard-4.3.0- asyncio: mode=Mode.AUTO, asyncio_default_fixture_loop_scope=function- collected 83 items-- ...<shortened>...-- =============================== 83 passed in 157.52s (0:02:37) =====================- ```-Hence, you can add `-n auto` to the command above to run tests in threads to increase speed.--Bonus: You can also see the test coverage with the pytest plugin below-```bash-pytest --cov=scrapling tests/-```--## Installing the latest unstable version from the dev branch-```bash-pip3 install git+https://github.com/D4Vinci/Scrapling.git@dev-```--## Development-Setting the scrapling logging level to `debug` makes it easier to know what's happening in the background.- ```python- >>> import logging- >>> logging.getLogger("scrapling").setLevel(logging.DEBUG)- ```-### Code Style--We use:--1. Type hints for better code clarity-2. Flake8, bandit, isort, and other hooks through `pre-commit`. <br/>Please install the hooks before committing with:- ```bash- pip install pre-commit- pre-commit install- ```- It will run automatically on the code you push with each commit.-3. Conventional commit messages format. We use the below format for commit messages-- | Prefix | When to use it |- |-------------|--------------------------|- | `feat:` | New feature added |- | `fix:` | Bug fix |- | `docs:` | Documentation change/add |- | `test:` | Tests |- | `refactor:` | Code refactoring |- | `chore:` | Maintenance tasks |-- Example:- ```- feat: add `adaptive` for similar elements-- - Added find_similar() method- - Implemented pattern matching- - Added tests and documentation- ```--### Push changes to the library--Then, the process is straightforward.-- - Read [How to get faster PR reviews](https://github.com/kubernetes/community/blob/master/contributors/guide/pull-requests.md#best-practices-for-faster-reviews) by Kubernetes (but skip step 0 and 1)- - Fork Scrapling [Git repository](https://github.com/D4Vinci/Scrapling.git).- - Make your changes, and don't forget to create a separate virtual environment for this project.- - Ensure all tests are passing.- - Create a Pull Request against the [**dev**](https://github.com/D4Vinci/Scrapling/tree/dev) branch of Scrapling.--A bonus: if you have more than one version of Python installed, you can use tox to run tests on each version with:-```bash-pip install tox-tox-```--> Note: All tests are automatically run with each push on Github on all supported Python versions using tox, so ensure all tests pass, or your PR will not be accepted.---## Building Documentation-```bash-pip install mkdocs-material-mkdocs serve # Local preview-mkdocs build # Build the static site-```\ No newline at end of filediff --git a/mkdocs.yml b/mkdocs.ymlindex 35a1a62..ea2dd7a 100644--- a/mkdocs.yml+++ b/mkdocs.yml@@ -79,7 +79,7 @@ nav:- Writing your retrieval system: development/adaptive_storage_system.md- Using Scrapling's custom types: development/scrapling_custom_types.md- Support and Advertisement: donate.md- - Contributing: contributing.md+ - Contributing: 'https://github.com/D4Vinci/Scrapling/blob/main/CONTRIBUTING.md'- Changelog: 'https://github.com/D4Vinci/Scrapling/releases'markdown_extensions: