Skip to content

Scrapy, a fast high-level web crawling & scraping framework for Python.

Commits on master

check CONCURRENT_REQUESTS_PER_IP is not set
Vostretsov Nikita committed 7 years ago
8e8ce30
test ip concurrency incompatibility with DAPQ
Vostretsov Nikita committed 7 years ago
987c2ae
no need for this
Vostretsov Nikita committed 7 years ago
4163a7a
Needs to be installed within tox env
Daniel Graña committed 7 years ago
6ff2574
Fix boto import error under Jessie testing environment
Daniel Graña committed 7 years ago
7d3175a
use regular comments
Vostretsov Nikita committed 7 years ago
a46613a
common indentation for comment
Vostretsov Nikita committed 7 years ago
6af964c
less complex implementation
Vostretsov Nikita committed 7 years ago
484927b
use urlparse_cached where it is possible
Vostretsov Nikita committed 7 years ago
0e06b9a
Document the SCRAPY_PROJECT environment variable
Adrián Chaves committed 7 years ago
9c31480
afdb69e
First implementation handle exception use O(N) instead of O(NlogN) here we have request as struct additional check for meptiness small performance improvement do not consume another request test number of responses mark requests back to 3 slots test case raise exceptions in case of missed meta add marks to requests and work only with your own requests only disk queue should obtain signals separate functions for slot rasd/write use signlas without variable stop crawler get signals in correct place logic test for download-aware priority queue update comment for structure ensure text type transform slot name to path use implicit structure use unicode type implicitly use real crawler add signals more slot accounting simple implementation of pop small slot accounting code no need for custom len function ability to call super in py27 add slots generic tests for downloader aware queue dummy implementation of crawler aware priority queue move common logic to base class rename class pass crawler to pqclass constructor do not copy quelib.PriorityQueue code add comment about new class remove obsolete function modify behaviour of queuelib.PriorityQueue to dodge very complex priority better way to get name remove obsolete commentary check boundaries function for priority convertion with known limits correct import path move file do not switch on by deffault as ip concurrency not supported set scheduler slot in case of empty slot use constant single place for added urls single place for constants use as default queue correct format for error text test migration from old version with on disk queue in these tests we have only two inflection points - jobdir and priority_queue_cls we do not need separate mock spider, use usual one do not rely on order of dict elements, imply order of list test round robiness of priority queue add comments and requirements for our magick function remove debug logging put queues into slot as we fabricate priorities we do not need special types anymore fabricate priority for priority queue more versatile priorities Scheduler class is not inflection point wrap correct types check for emptinees before initialization tests for new priority queue correct default type for startprios use exact values put common settings to base class test priorities for disk scheduler test dequeue for disk scheduler test length for disk scheduler setUp/tearDown methods for on disk schedulers new methods remove excessive line base class to handle scheduler creation correct method names test priorities deque test close scheduler on test end enqueue some requests test template for scheduler use downloader slot I/O implementation for RoundRobin queue round-robin implementation without I/O and slot detection wrappers for every disk queue class
Vostretsov Nikita committed 7 years ago
821f5bb
Fix numeration
Marc Hernández committed 7 years ago
70aa5b1
Update test_http_request.py
Maram Sumanth committed 7 years ago
4c89e53
Update form.py
Maram Sumanth committed 7 years ago
282f24c
doc for creating custom cache storage backend.
Anubhav Patel committed 7 years ago
0443189
Rule.process_request - Renaming
Eugenio Lacuesta committed 7 years ago
92bbc52
Rename Request.kwargs to Request.cb_kwargs
Eugenio Lacuesta committed 7 years ago
6760bca
Count keyword argument checks
Eugenio Lacuesta committed 7 years ago
645e8d1
make tests work with new convert_image
Anubhav Patel committed 7 years ago
07487dd
PEP8 changes to test_spider.py
Eugenio Lacuesta committed 7 years ago
01ed605
Merge branch 'master' into feed-storage-s3-acl
Victor Torres committed 7 years ago
fda1d04
do not degrade JPEG files.
Anubhav Patel committed 7 years ago
25e616f
d346b8c
Merge pull request #3610 from Gallaecio/spidercls-check
Adrián Chaves committed 7 years ago
70a0f1f