A Smart, Automatic, Fast and Lightweight Web Scraper for Python
Python aiartificial-intelligenceautomationcrawlermachine-learningpythonscrapescraperscrapingweb-scrapingwebautomationwebscraping
update readme
alireza7c0ae92f153b
diff --git a/README.rst b/README.mdsimilarity index 65%rename from README.rstrename to README.mdindex 22fbfa8..bb78590 100644--- a/README.rst+++ b/README.md@@ -1,41 +1,36 @@-AutoScraper: A Smart Automatic Scraper for Python--------------------------------------------------+# AutoScraper: A Smart Automatic Scraper for Python+This project is made for automatic web scraping to make scraping easy.-It gets a url or html content of a web page and a list of sample data which we want to scrape from that page; Then it learns the scraping rules and returns the similar elements.-Then you can use this learnt object with new urls to get similar content of those new pages.+It gets an url or the html content of a web page and a list of sample data which we want to scrape from that page. It learns the scraping rules and returns the similar elements. Then you can use this learned object with new urls to get similar content of those new pages.++## Installation+-------------------------------------Installation-------------It's compatible with python3.Install from source:-.. code-block:: bash-$ python setup.py installInstall latest version from git repository using pip:-.. code-block:: bash-$ pip install git+https://github.com/alirezamika/autoscraper.git-How to use - Getting similar results+## How to use------------------------------------+### Getting similar resultsSay we want to fetch all related post titles in a stackoverflow page:--.. code:: python-from autoscraper import AutoScraperurl = 'https://stackoverflow.com/questions/2081586/web-scraping-with-python'- # We can add one or multiple candidates here+ # We can add one or multiple candidates here.+ # You can also put urls here to retrive urls.wanted_list = ["How to call an external command?"]scraper = AutoScraper()@@ -45,9 +40,6 @@ Say we want to fetch all related post titles in a stackoverflow page:Here's the output:--.. code:: python-['How do I merge two dictionaries in a single expression in Python (taking union of dictionaries)?','How to call an external command?',@@ -60,20 +52,14 @@ Here's the output:'Why is “1000000000000000 in range(1000000000000001)” so fast in Python 3?']-You can also put urls in the wanted list.-Now you can use the `scraper` object to get related topics to any stackoverflow post:--.. code:: python+Now you can use the `scraper` object to get related topics of any stackoverflow page:scraper.get_result_similar('https://stackoverflow.com/questions/606191/convert-bytes-to-a-string')-How to use - Getting exact results-------------------------------------Say we want to scrape stock live price from nasdaq:+### Getting exact results-.. code:: python+Say we want to scrape stock live prices from Nasdaq:from autoscraper import AutoScraper@@ -87,9 +73,7 @@ Say we want to scrape stock live price from nasdaq:result = scraper.build(url, wanted_list)print(result)-You can also pass any custom `requests` attributes. for example you may want to use proxies:--.. code:: python+You can also pass any custom `requests` module attribute. for example you may want to use proxies:proxies = {"http": 'http://127.0.0.1:8001',@@ -100,20 +84,15 @@ You can also pass any custom `requests` attributes. for example you may want toNow we can get the price of any nasdaq symbol:-.. code:: python-scraper.get_result_exact('https://www.google.com/finance?oq=msft')-How to use - Generating the scraper python code------------------------------------------------+### Generating the scraper python codeWe can generate a code for the built scraper to use it later:-.. code:: python-scraper.generate_python_code()It will print the generated code. There's a class named `GeneratedAutoScraper` which has the methods `get_result_similar` and-`get_result_exact` which you can use. (or use `get_result` method to get both)+`get_result_exact` which you can use. You can also use `get_result` method to get both.