Scrapy, a fast high-level web crawling & scraping framework for Python.
Commits on master
Add a schemes parameter to LinkExtractor, and extract data: and s3: links by default (#7994)
46e3fa1 Adrian committed 1 month ago
Bound the media pipeline result cache with MEDIA_CACHE_SIZE (#8193)
7ed2690 Adrian committed 1 month ago
Add JOBDIR_SYNC_EVERY to keep active.json current, written atomically (#8152)
db5bad1 Adrian committed 1 month ago
Ignore docstring @ lines that are not registered contracts (#8226)
36747fc Adrian committed 1 month ago
Build selectors from bytes only with parsel 1.12.0+, fixing UTF-16 (#8222)
9d1085d Adrian committed 1 month ago
Fail the version bump on version directives older than 3 years (#8219)
4aa119b Adrian committed 1 month ago
Support propagating a spider-triggered error as an exit code (#8204)
bb93093 Adrian committed 1 month ago
Add JsonValidationMiddleware to retry malformed JSON responses (#8215)
294cada Adrian committed 1 month ago
Let process_response() exceptions reach process_exception() (#8177)
7af73d4 Adrian committed 1 month ago
Upgrade to sphinx-scrapy 0.10.0 and sphinx-llm-friendly (#8214)
8da9459 Adrian committed 1 month ago
Skip only the misconfigured feed on a bad URI placeholder (#8205)
cd4f213 Adrian committed 1 month ago
Add IMAGES_PRESERVE_FORMAT to skip forced JPEG conversion (#8198)
4439b82 Adrian committed 1 month ago
Document HTTPCACHE_IGNORE_HTTP_CODES/RETRY_HTTP_CODES interaction (#8202)
8fab802 Adrian committed 1 month ago
CrawlSpider: extract links even when a rule's callback raises (#8200)
6651567 Adrian committed 1 month ago
Exit with code 1 when Spider.start() raises an exception (#8203)
afd71bf Adrian committed 1 month ago
Retry FTP downloads when the server drops the connection (#8206)
e3c48e2 Adrian committed 1 month ago
docs: recipe for deduplicating query params in request fingerprints (#8209)
968d425 Adrian committed 1 month ago
Avoid decoding response bodies for html/xml selectors when possible (#8195)
506555b Adrian committed 2 months ago
Never fail decoding headers in to_unicode_dict() and to_tuple_list() (#8192)
40dc142 Adrian committed 2 months ago