Skip to content

A Smart, Automatic, Fast and Lightweight Web Scraper for Python

initial commit

alireza5bfef1553dfc
diff --git a/.gitignore b/.gitignore
new file mode 100644
index 0000000..d228f30
--- /dev/null
+++ b/.gitignore
@@ -0,0 +1,103 @@
+# Byte-compiled / optimized / DLL files
+__pycache__/
+*.py[cod]
+*$py.class
+.idea/
+.vscode/
+
+# C extensions
+*.so
+
+# Distribution / packaging
+.Python
+env/
+build/
+develop-eggs/
+dist/
+downloads/
+eggs/
+.eggs/
+lib/
+lib64/
+parts/
+sdist/
+var/
+wheels/
+*.egg-info/
+.installed.cfg
+*.egg
+
+# PyInstaller
+# Usually these files are written by a python script from a template
+# before PyInstaller builds the exe, so as to inject date/other infos into it.
+*.manifest
+*.spec
+
+# Installer logs
+pip-log.txt
+pip-delete-this-directory.txt
+
+# Unit test / coverage reports
+htmlcov/
+.tox/
+.coverage
+.coverage.*
+.cache
+nosetests.xml
+coverage.xml
+*.cover
+.hypothesis/
+
+# Translations
+*.mo
+*.pot
+
+# Django stuff:
+*.log
+local_settings.py
+
+# Flask stuff:
+instance/
+.webassets-cache
+
+# Scrapy stuff:
+.scrapy
+
+# Sphinx documentation
+docs/_build/
+
+# PyBuilder
+target/
+
+# Jupyter Notebook
+.ipynb_checkpoints
+
+# pyenv
+.python-version
+
+# celery beat schedule file
+celerybeat-schedule
+
+# SageMath parsed files
+*.sage.py
+
+# dotenv
+.env
+
+# virtualenv
+.venv
+venv/
+ENV/
+
+# Spyder project settings
+.spyderproject
+.spyproject
+
+# Rope project settings
+.ropeproject
+
+# mkdocs documentation
+/site
+
+# mypy
+.mypy_cache/
diff --git a/LICENSE.txt b/LICENSE.txt
new file mode 100644
index 0000000..e662c78
--- /dev/null
+++ b/LICENSE.txt
@@ -0,0 +1,5 @@
+Permission is hereby granted, free of charge, to any person obtaining a copy of this software and associated documentation files (the "Software"), to deal in the Software without restriction, including without limitation the rights to use, copy, modify, merge, publish, distribute, sublicense, and/or sell copies of the Software, and to permit persons to whom the Software is furnished to do so, subject to the following conditions:
+
+The above copyright notice and this permission notice shall be included in all copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE.
\ No newline at end of file
diff --git a/README.rst b/README.rst
new file mode 100644
index 0000000..a936f58
--- /dev/null
+++ b/README.rst
@@ -0,0 +1,115 @@
+AutoScraper: A Smart Automatic Scraper for Python
+-------------------------------------------------
+
+This project is made for automatic web scraping to make scraping easy.
+It gets a url or html content of a web page and a list of sample data which we want to scrape from that page; Then it learns the scraping rules and returns the similar elements.
+Then you can use this learnt object with new urls to get similar content of those new pages.
+
+Installation
+------------
+It's compatible with python3.
+
+Install from source:
+
+.. code-block:: bash
+
+ $ python setup.py install
+
+
+Install latest version from git repository using pip:
+
+.. code-block:: bash
+
+ $ pip install git+https://github.com/alirezamika/evostra.git
+
+
+How to use - Getting similar results
+------------------------------------
+
+Say we want to fetch all related post titles in a stackoverflow page:
+
+
+.. code:: python
+
+ from autoscraper import AutoScraper
+
+ url = 'https://stackoverflow.com/questions/2081586/web-scraping-with-python'
+
+ # We can add one or multiple candidates here
+ wanted_list = ["How to call an external command?"]
+
+ scraper = AutoScraper()
+ result = scraper.build(url, wanted_list)
+ print(result)
+
+Here's the output:
+
+.. code::
+ [
+ 'How do I merge two dictionaries in a single expression in Python (taking union of dictionaries)?',
+ 'How to call an external command?',
+ 'What are metaclasses in Python?',
+ 'Does Python have a ternary conditional operator?',
+ 'How do you remove duplicates from a list whilst preserving order?',
+ 'Convert bytes to a string',
+ 'How to get line count of a large file cheaply in Python?',
+ "Does Python have a string 'contains' substring method?",
+ 'Why is “1000000000000000 in range(1000000000000001)” so fast in Python 3?'
+ ]
+
+You can also put urls in the wanted list.
+Now you can use the `scraper` object to get related topics to any stackoverflow post:
+
+.. code:: python
+
+ scraper.get_result_similar('https://stackoverflow.com/questions/606191/convert-bytes-to-a-string')
+
+
+How to use - Getting exact results
+----------------------------------
+
+Say we want to scrape stock live price from nasdaq:
+
+.. code:: python
+
+ from autoscraper import AutoScraper
+
+ url = 'https://www.google.com/finance?oq=aapl'
+
+ wanted_list = ["124.81"]
+
+ scraper = AutoScraper()
+
+ # Here we can also pass html content via the html parameter instead of the url (html=html_content)
+ result = scraper.build(url, wanted_list)
+ print(result)
+
+You can also pass any custom `requests` attributes. for example you may want to use proxies:
+
+.. code:: python
+ proxies = {
+ "http": 'http://127.0.0.1:8001',
+ "https": 'https://127.0.0.1:8001',
+ }
+
+ result = scraper.build(url, wanted_list, request_args=dict(proxies=proxies))
+
+Now we can get the price of any nasdaq symbol:
+
+.. code:: python
+
+ scraper.get_result_exact('https://www.google.com/finance?oq=msft')
+
+
+How to use - Generating the scraper python code
+-----------------------------------------------
+
+We can generate a code for the built scraper to use it later:
+
+.. code:: python
+
+ scraper.generate_python_code()
+
+
+It will print the generated code. There's a class named `GeneratedAutoScraper` which has the methods `get_result_similar` and
+`get_result_exact` which you can use. (or use `get_result` method to get both)
diff --git a/setup.py b/setup.py
new file mode 100644
index 0000000..d642c89
--- /dev/null
+++ b/setup.py
@@ -0,0 +1,38 @@
+from setuptools import setup, find_packages
+from codecs import open
+from os import path
+
+here = path.abspath(path.dirname(__file__))
+
+with open(path.join(here, 'README.rst'), encoding='utf-8') as f:
+ long_description = f.read()
+
+setup(
+ name='autoscraper',
+
+ version='1.0.0',
+
+ description='Smart Automatic Scraper in Python',
+ long_description=long_description,
+
+ url='https://github.com/alirezamika/autoscraper',
+
+ author='Alireza Mika',
+ author_email='[email protected]',
+
+ license='MIT',
+
+ classifiers=[
+ 'Development Status :: 4 - Beta',
+ 'License :: OSI Approved :: MIT License',
+ 'Programming Language :: Python :: 3',
+ ],
+
+ keywords='scraping - scraper',
+
+ packages=find_packages(exclude=['contrib', 'docs', 'tests']),
+
+ python_requires='>=3.6',
+ install_requires=['requests', 'bs4'],
+
+)