Skip to content

Scrapy, a fast high-level web crawling & scraping framework for Python.

Commits on master

Warn or fail with exception on unknown policies
Paul Tremberth committed 9 years ago
6916dd6
Add test for Referer header on HTTP redirections
Paul Tremberth committed 9 years ago
8226e77
Update tests
Paul Tremberth committed 9 years ago
d2aa51c
Pass URLs around instead of Request/Responses
Paul Tremberth committed 9 years ago
04e4d08
Rename setting to REFERRER_POLICY (with 2 Rs)
Paul Tremberth committed 9 years ago
bc200d1
Add autoclass directives to document built-in policies
Paul Tremberth committed 9 years ago
537683f
Use table for referrer policy options
Paul Tremberth committed 9 years ago
3dc09ee
Edit text
Paul Tremberth committed 9 years ago
605935f
Reword warning on no-referrer-when-downgrade policy
Paul Tremberth committed 9 years ago
eb07285
Update docs for new "referrer_policy" Request.meta key
Paul Tremberth committed 9 years ago
03ff19d
Update docs
Paul Tremberth committed 9 years ago
e249abc
Update docs with "strict-..." policies
Paul Tremberth committed 9 years ago
c86f568
Fix tests
Paul Tremberth committed 9 years ago
b6c761d
Update StrictOriginPolicy
Paul Tremberth committed 9 years ago
ebcacd3
Update NoReferrerWhenDowngradePolicy
Paul Tremberth committed 9 years ago
deb8567
Fix implementation
Paul Tremberth committed 9 years ago
77aec5a
Update Referrer tests for "strict-" policies
Paul Tremberth committed 9 years ago
5cef67a
Add new "strict-" policies
Paul Tremberth committed 9 years ago
c808a97
0a0b60a
Rename helper function to strip_url() + add more tests
Paul Tremberth committed 10 years ago
8864d0e
Move URL credentials stripping to a helper function
Paul Tremberth committed 10 years ago
5dd7311
Update documentation about REFERER_POLICY setting
Paul Tremberth committed 10 years ago
c9c59db
Patch "Referer" header on HTTP redirects if necessary
Paul Tremberth committed 10 years ago
285d5bc
d3d4d66
Add test for custom referrer policy via settings
Paul Tremberth committed 10 years ago
e50e670
Change __init__ default "settings" arg handling
Paul Tremberth committed 10 years ago
ec8b4c1
Support case-insensitive policy names in settings
Paul Tremberth committed 10 years ago
0344f57
e72b6e3
Make default referrer policy customizable via settings
Paul Tremberth committed 10 years ago
842ce13
Refactor ReferrerPolicy methods
Paul Tremberth committed 10 years ago
f620577