Skip to content

Scrapy, a fast high-level web crawling & scraping framework for Python.

Add AGENTS.md with conventions for coding agents (#8157)

Adriancace2dad8fd8
diff --git a/AGENTS.md b/AGENTS.md
new file mode 100644
index 000000000..7105ce641
--- /dev/null
+++ b/AGENTS.md
@@ -0,0 +1,93 @@
+# Working on Scrapy
+
+Read the
+[contributing guide](https://docs.scrapy.org/en/master/contributing.md) first.
+This file adds what agents tend to get wrong.
+
+## Before you finish
+
+- Run `pre-commit run --all-files`, `tox -e mypy` and `tox -e pylint`, and
+ `tox -e docs` and `tox -e docs-tests` as well if you touched docs or
+ docstrings. CI runs these first and skips the test matrix when they fail.
+- Run the tests you touched: `tox -e py -- tests/test_foo.py`. Anything after
+ `tox -e py --` is passed to pytest.
+- Review your own diff. Remove whatever the change does not need: unused
+ parameters, dead code, leftover debugging, comments that restate the code.
+
+## Code
+
+- Make the smallest change that solves the issue. Do not add settings,
+ parameters or abstractions for needs nobody has yet.
+- Prefix new modules, classes, functions, constants and attributes with `_`
+ unless users must call them by name. Do not re-export or document private
+ APIs.
+- Write comments for a reader who has never seen the code any other way.
+ Explain what the lines do and why, never what they replaced or which bug
+ prompted them. Most lines need no comment.
+- Do not write module docstrings. Document classes and functions instead.
+- Log messages are f-strings: `logger.debug(f"Ignoring {request.url}")`.
+- Use `assert` for invariants and `if` for conditions that can actually
+ happen.
+- Make new boolean parameters keyword-only.
+
+## Tests
+
+- Cover every line you add, including error branches. Where a branch cannot be
+ exercised, mark it `# pragma: no cover`.
+- Test through public APIs. If behavior is only reachable through a private
+ function, reconsider whether it needs its own test.
+- Use `pytest.mark.parametrize` instead of looping over inputs inside one
+ test, so that one bad input fails on its own.
+- Build components with `build_from_crawler()` from `scrapy.utils.misc` rather
+ than calling their constructors.
+- Test spiders by crawling them against the mock server, not by calling their
+ private methods.
+- Use `example.com` or any `.example` domain, never a real one, even in URLs
+ that are never fetched.
+- New test modules must type-check under `tox -e mypy`.
+
+## Documentation
+
+- API reference lives in docstrings and is pulled in with autodoc. Keep
+ docstrings short and link to the narrative docs instead of repeating them.
+- Describe parameters in prose, naming them in italics like the standard
+ library does (*include_universal*). No `:param:` lists.
+- Put `versionadded` and `versionchanged` right after the first paragraph, and
+ always write `VERSION` as the version.
+- Do not document deprecated features, known bugs, or implementation details
+ such as the order in which things happen internally. Never use the
+ `deprecated` directive.
+- Do not edit `docs/news.rst`. Release notes are written at release time from
+ the merged pull requests.
+- Say what something is, not what it is not.
+- Link instead of duplicating. Put the details in one place; the other pages
+ link to it.
+- Every example line must show something the other lines do not.
+- Mark file names and paths with the `:file:` role.
+- Use sentence case for titles. An underline is exactly as long as its title;
+ verify with `awk '{print length, $0}'`, not by eye.
+- Give new anchors short names without the `topics-` prefix. Keep existing
+ anchors.
+- Wrap paragraphs at 79 characters.
+- `sphinx-scrapy` is pinned in three places that must match: `tox.ini`,
+ `docs/requirements.in` (then run `uv pip compile requirements.in -o
+ requirements.txt` in `docs/`) and the `rev` in `.pre-commit-config.yaml`.
+
+## Git and GitHub
+
+- One change per pull request. Keep cosmetic changes out of functional
+ commits.
+- A pull request description is one or two sentences on what and why, plus
+ `Closes #N` for every issue or pull request it resolves. The diff and the
+ tests already show what changed, so do not summarize them, add section
+ headers, or list the files touched.
+- Do not open an issue before a pull request. The pull request is enough.
+- Comments answer the question asked, in as few words as it takes.
+- A commit message is a title. Add a body only when the why does not fit in
+ it.
+- Once a pull request has been reviewed, only add commits on top. To catch up
+ with `master`, merge it in. Never rebase, amend or force-push: pull requests
+ are squash-merged, so rewriting history buys nothing and hides what changed
+ since the last review.
+- When a review asks for changes, update the existing pull request instead of
+ opening a new one.
diff --git a/CLAUDE.md b/CLAUDE.md
new file mode 100644
index 000000000..43c994c2d
--- /dev/null
+++ b/CLAUDE.md
@@ -0,0 +1 @@