A Smart, Automatic, Fast and Lightweight Web Scraper for Python
Python aiartificial-intelligenceautomationcrawlermachine-learningpythonscrapescraperscrapingweb-scrapingwebautomationwebscraping
add ability to save and load
alireza3fb5912a8e7d
diff --git a/README.md b/README.mdindex 2cfd087..1fabbc8 100644--- a/README.md+++ b/README.md@@ -92,9 +92,24 @@ Now we can get the price of any symbol:scraper.get_result_exact('https://finance.yahoo.com/quote/MSFT/')```+### Saving the model++We can now save the built model to use it later. To save:++```python+# Give it a file path+scraper.save('yahoo-finance')+```++And to load:++```python+scraper.load('yahoo-finance')+```+### Generating the scraper python code-We can generate a code for the built scraper to use it later:+We can also generate a stand-alone code for the learned scraper to use it anywhere:```pythonscraper.generate_python_code()diff --git a/autoscraper/auto_scraper.py b/autoscraper/auto_scraper.pyindex fa73966..f89a711 100644--- a/autoscraper/auto_scraper.py+++ b/autoscraper/auto_scraper.py@@ -1,3 +1,4 @@+import jsonimport osfrom urllib.parse import urljoinimport requests@@ -14,6 +15,14 @@ class AutoScraper(object):self.stack_list = stack_listself.url = url+ def save(self, file_path):+ with open(file_path, 'w') as f:+ f.write(json.dumps(self.stack_list))++ def load(self, file_path):+ with open(file_path, 'r') as f:+ self.stack_list = json.loads(f.read())+@staticmethoddef _get_soup(url=None, html=None, request_args=None):if html: