Skip to content

Scrapy, a fast high-level web crawling & scraping framework for Python.

DOC update "Contributing" docs:

Mikhail Korobovf716843a6682
diff --git a/docs/contributing.rst b/docs/contributing.rst
index f3732ab06..eb736bf30 100644
--- a/docs/contributing.rst
+++ b/docs/contributing.rst
@@ -19,12 +19,16 @@ There are many ways to contribute to Scrapy. Here are some of them:
the guidelines detailed in `Reporting bugs`_ below.
* Submit patches for new functionality and/or bug fixes. Please read
- `Writing patches`_ and `Submitting patches`_ below for details on how to
+ :ref:`writing-patches` and `Submitting patches`_ below for details on how to
write and submit a patch.
* Join the `Scrapy subreddit`_ and share your ideas on how to
improve Scrapy. We're always open to suggestions.
+* Answer Scrapy questions at
+ `Stack Overflow <https://stackoverflow.com/questions/tagged/scrapy>`__.
+
+
Reporting bugs
==============
@@ -40,9 +44,14 @@ guidelines when reporting a new bug.
* check the :ref:`FAQ <faq>` first to see if your issue is addressed in a
well-known question
+* if you have a general question about scrapy usage, please ask it at
+ `Stack Overflow <https://stackoverflow.com/questions/tagged/scrapy>`__
+ (use "scrapy" tag).
+
* check the `open issues`_ to see if it has already been reported. If it has,
- don't dismiss the report but check the ticket history and comments, you may
- find additional useful information to contribute.
+ don't dismiss the report, but check the ticket history and comments. If you
+ have additional useful information, please leave a comment, or consider
+ :ref:`sending a pull request <writing-patches>` with a fix.
* search the `scrapy-users`_ list and `Scrapy subreddit`_ to see if it has
been discussed there, or if you're not sure if what you're seeing is a bug.
@@ -54,12 +63,20 @@ guidelines when reporting a new bug.
it. See for example StackOverflow's guide on creating a
`Minimal, Complete, and Verifiable example`_ exhibiting the issue.
+* the most awesome way to provide a complete reproducible example is to
+ send a pull request which adds a failing test case to the
+ Scrapy testing suite (see :ref:`submitting-patches`).
+ This is helpful even if you don't have an intention to
+ fix the issue yourselves.
+
* include the output of ``scrapy version -v`` so developers working on your bug
know exactly which version and platform it occurred on, which is often very
helpful for reproducing it, or knowing if it was already fixed.
.. _Minimal, Complete, and Verifiable example: https://stackoverflow.com/help/mcve
+.. _writing-patches:
+
Writing patches
===============
@@ -83,6 +100,8 @@ Well-written patches should:
the documentation changes in the same patch. See `Documentation policies`_
below.
+.. _submitting-patches:
+
Submitting patches
==================
@@ -100,11 +119,22 @@ starting point is to send a pull request on GitHub. It can be simple enough to
illustrate your idea, and leave documentation/tests for later, after the idea
has been validated and proven useful. Alternatively, you can start a
conversation in the `Scrapy subreddit`_ to discuss your idea first.
+
+Sometimes there is an existing pull request for the problem you'd like to
+solve, which is stalled for some reason. Often the pull request is in a
+right direction, but changes are requested by Scrapy maintainers, and the
+original pull request author haven't had time to address them.
+In this case consider picking up this pull request: open
+a new pull request with all commits from the original pull request, as well as
+additional changes to address the raised issues. Doing so helps a lot; it is
+not considered rude as soon as original the author is acknowledged by keeping
+his/her commits.
+
When writing GitHub pull requests, try to keep titles short but descriptive.
E.g. For bug #411: "Scrapy hangs if an exception raises in start_requests"
prefer "Fix hanging when exception occurs in start_requests (#411)"
-instead of "Fix for #411".
-Complete titles make it easy to skim through the issue tracker.
+instead of "Fix for #411". Complete titles make it easy to skim through
+the issue tracker.
Finally, try to keep aesthetic changes (:pep:`8` compliance, unused imports
removal, etc) in separate commits than functional changes. This will make pull
@@ -121,21 +151,29 @@ Scrapy:
* It's OK to use lines longer than 80 chars if it improves the code
readability.
-* Don't put your name in the code you contribute. Our policy is to keep
- the contributor's name in the `AUTHORS`_ file distributed with Scrapy.
+* Don't put your name in the code you contribute; git provides enough
+ metadata to identify author of the code.
+ See https://help.github.com/articles/setting-your-username-in-git/ for
+ setup instructions.
Documentation policies
======================
* **Don't** use docstrings for documenting classes, or methods which are
- already documented in the official (sphinx) documentation. For example, the
- :meth:`ItemLoader.add_value` method should be documented in the sphinx
- documentation, not its docstring.
+ already documented in the official (sphinx) documentation. Alternatively,
+ **do** provide a docstring, but make sure sphinx documentation uses
+ autodoc_ extension to pull the docstring. For example, the
+ :meth:`ItemLoader.add_value` method should be either
+ documented only in the sphinx documentation (not it a docstring), or
+ it should have a docstring which is pulled to sphinx documentation using
+ autodoc_ extension.
* **Do** use docstrings for documenting functions not present in the official
(sphinx) documentation, such as functions from ``scrapy.utils`` package and
its sub-modules.
+.. _autodoc: http://www.sphinx-doc.org/en/stable/ext/autodoc.html
+
Tests
=====