Python script, which empowers people with no programming background to generate robust leads on a mass scale. This repo will be compiled of various versatile techniques used in lead generation.
New version
Madi-S23c095ade733
diff --git a/LICENSE b/LICENSEindex 2b52355..5714e5d 100644--- a/LICENSE+++ b/LICENSE@@ -1,6 +1,6 @@MIT License-Copyright (c) 2022 Madi Shaiken+Copyright (c) 2024 Madi ShaikenPermission is hereby granted, free of charge, to any person obtaining a copyof this software and associated documentation files (the "Software"), to dealdiff --git a/README.md b/README.mdindex 6bc6767..007a45b 100644--- a/README.md+++ b/README.md@@ -1,2 +1,56 @@# Lead-Generation-Python script, which empowers people with no programming background to generate robust leads on a mass scale. This repo will be compiled of various versatile techniques used in lead generation implemented in python.++The updated version of my outdated dirty clumsy and having shockingly high amount of stars repository on Python - https://github.com/Madi-S/Lead-Generation++# Installation++```bash+pip install py-lead-generation+```++OR++```bash+git clone https://github.com/Madi-S/Lead-Generation+cd Lead-Generation+python run.py+```++# Quickstart++```python+import asyncio+from py_lead_generation import GoogleMapsEngine, YelpEngine+++async def main() -> None:+ q = input('Enter your search query: ').strip() or 'Barbershop'+ addr = input('Enter the location you would like to search in: ').strip() \+ or 'Paris'+ zoom = float(input('[Optional] Enter google maps zoom: ').strip() or 12)++ engine = GoogleMapsEngine(q, addr, zoom)+ await engine.run()+ engine.save_to_csv()++ engine = YelpEngine('Pizza', 'Mexico, Pampanga, Philippines')+ await engine.run()+ engine.save_to_csv('pizza_leads.csv')++if __name__ == '__main__':+ asyncio.run(main())+```++# Current functionality++ - Parse Google Maps+ - Parse Yelp+ - Export collected data to a CSV file++# Expectations of this project:++ - Parse Google Maps and Yelp for telephone number, email, address and other information by given keyword+ - Somehow parse search results in Google Search for the same information using regex or other algorithm+ - Export all parsed data to csv or excel+ - For parsed emails send a message, which will be prevented from going to spam+ - For parsed telephone numbers send an SMS, which will b prevented from going to spam as welldiff --git a/__pycache__/bufferization.cpython-38.pyc b/__pycache__/bufferization.cpython-38.pycdeleted file mode 100644index ce02117..0000000Binary files a/__pycache__/bufferization.cpython-38.pyc and /dev/null differdiff --git a/__pycache__/config.cpython-38.pyc b/__pycache__/config.cpython-38.pycdeleted file mode 100644index 35ba21c..0000000Binary files a/__pycache__/config.cpython-38.pyc and /dev/null differdiff --git a/__pycache__/driver_retry.cpython-38.pyc b/__pycache__/driver_retry.cpython-38.pycdeleted file mode 100644index cc0ec1e..0000000Binary files a/__pycache__/driver_retry.cpython-38.pyc and /dev/null differdiff --git a/__pycache__/locators.cpython-38.pyc b/__pycache__/locators.cpython-38.pycdeleted file mode 100644index be993d6..0000000Binary files a/__pycache__/locators.cpython-38.pyc and /dev/null differdiff --git a/__pycache__/logger_config.cpython-38.pyc b/__pycache__/logger_config.cpython-38.pycdeleted file mode 100644index 9e9cc61..0000000Binary files a/__pycache__/logger_config.cpython-38.pyc and /dev/null differdiff --git a/__pycache__/my_config.cpython-38.pyc b/__pycache__/my_config.cpython-38.pycdeleted file mode 100644index c05d077..0000000Binary files a/__pycache__/my_config.cpython-38.pyc and /dev/null differdiff --git a/__pycache__/webdriver.cpython-38.pyc b/__pycache__/webdriver.cpython-38.pycdeleted file mode 100644index 5c5781a..0000000Binary files a/__pycache__/webdriver.cpython-38.pyc and /dev/null differdiff --git a/app/__pycache__/app.cpython-38.pyc b/app/__pycache__/app.cpython-38.pycdeleted file mode 100644index dd73b8c..0000000Binary files a/app/__pycache__/app.cpython-38.pyc and /dev/null differdiff --git a/app/__pycache__/config.cpython-38.pyc b/app/__pycache__/config.cpython-38.pycdeleted file mode 100644index 37727e7..0000000Binary files a/app/__pycache__/config.cpython-38.pyc and /dev/null differdiff --git a/app/__pycache__/crypto.cpython-38.pyc b/app/__pycache__/crypto.cpython-38.pycdeleted file mode 100644index 497c7b0..0000000Binary files a/app/__pycache__/crypto.cpython-38.pyc and /dev/null differdiff --git a/app/__pycache__/forms.cpython-38.pyc b/app/__pycache__/forms.cpython-38.pycdeleted file mode 100644index 4c41f63..0000000Binary files a/app/__pycache__/forms.cpython-38.pyc and /dev/null differdiff --git a/app/__pycache__/models.cpython-38.pyc b/app/__pycache__/models.cpython-38.pycdeleted file mode 100644index dd28994..0000000Binary files a/app/__pycache__/models.cpython-38.pyc and /dev/null differdiff --git a/app/__pycache__/views.cpython-38.pyc b/app/__pycache__/views.cpython-38.pycdeleted file mode 100644index 690ca8c..0000000Binary files a/app/__pycache__/views.cpython-38.pyc and /dev/null differdiff --git a/app/db/lg.db b/app/db/lg.dbdeleted file mode 100644index 0dff396..0000000Binary files a/app/db/lg.db and /dev/null differdiff --git a/archived/LICENSE b/archived/LICENSEnew file mode 100644index 0000000..2b52355--- /dev/null+++ b/archived/LICENSE@@ -0,0 +1,21 @@+MIT License++Copyright (c) 2022 Madi Shaiken++Permission is hereby granted, free of charge, to any person obtaining a copy+of this software and associated documentation files (the "Software"), to deal+in the Software without restriction, including without limitation the rights+to use, copy, modify, merge, publish, distribute, sublicense, and/or sell+copies of the Software, and to permit persons to whom the Software is+furnished to do so, subject to the following conditions:++The above copyright notice and this permission notice shall be included in all+copies or substantial portions of the Software.++THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR+IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,+FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE+AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER+LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,+OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE+SOFTWARE.diff --git a/archived/README.md b/archived/README.mdnew file mode 100644index 0000000..6bc6767--- /dev/null+++ b/archived/README.md@@ -0,0 +1,2 @@+# Lead-Generation+Python script, which empowers people with no programming background to generate robust leads on a mass scale. This repo will be compiled of various versatile techniques used in lead generation implemented in python.diff --git a/app/app.py b/archived/app/app.pysimilarity index 99%rename from app/app.pyrename to archived/app/app.pyindex 2c704f5..d8534b3 100644--- a/app/app.py+++ b/archived/app/app.py@@ -13,4 +13,3 @@ app.config.from_object(Config)db = SQLAlchemy(app)migrate = Migrate(app, db)-diff --git a/app/config.py b/archived/app/config.pysimilarity index 88%rename from app/config.pyrename to archived/app/config.pyindex 068158d..a1d0ac2 100644--- a/app/config.py+++ b/archived/app/config.py@@ -6,11 +6,11 @@ class Config(object):DEBUG = TrueWTF_CSRF_SECRET_KEY = 'a random string'-+SECRET_KEY = '6554410059521c8b5426eba21e54a7a193aaf30e67645e8c24a2f6eb8779790711a4ca0b9d62ef43e79caddf1e5a123808a4ea370f69c59cd64003e1c9ac8659'SQLALCHEMY_TRACK_MODIFICATIONS = False-+if platform.startswith('w'):# For WindowsSQLALCHEMY_DATABASE_URI = r'sqlite:///C:\Users\khova\Desktop\Python\Code\Lead Generation\Lead-Generation\app\db\lg.db'@@ -19,12 +19,11 @@ class Config(object):# For UbuntuSQLALCHEMY_DATABASE_URI = 'mysql+mysqlconnector://root:1234@localhost/lg'-- SESSION_TYPE = 'redis' # redis, memcached, filesystem or mongodb+ SESSION_TYPE = 'redis' # redis, memcached, filesystem or mongodbPERMANENT_SESSION_LIFETIME = timedelta(days=90)RECAPTCHA_PUBLIC_KEY = '6LeENmoaAAAAAH1dMElRkLRGE2UiSP-CK831656U'RECAPTCHA_PRIVATE_KEY = '6LeENmoaAAAAANfgmGgA9Of_IPlJWtppMLMoMoV4'# RECAPTCHA_THEME = 'dark'# RECAPTCHA_TYPE = 'audio'- # RECAPTCHA_SIZE = 'compact'\ No newline at end of file+ # RECAPTCHA_SIZE = 'compact'diff --git a/app/crypto.py b/archived/app/crypto.pysimilarity index 100%rename from app/crypto.pyrename to archived/app/crypto.pydiff --git a/app/forms.py b/archived/app/forms.pysimilarity index 65%rename from app/forms.pyrename to archived/app/forms.pyindex 630c6fb..c96a10e 100644--- a/app/forms.py+++ b/archived/app/forms.py@@ -4,10 +4,14 @@ from wtforms.validators import DataRequired, Length, EqualToclass UserRegistrationForm(FlaskForm):- first_name = StringField('First Name', [DataRequired(), Length(4, 100, 'First name should consist of 4-200 characters')])- last_name = StringField('Last Name', [DataRequired(), Length(4, 100, 'Last name should consist of 4-200 characters')])- username = StringField('Username (4-30)', [DataRequired(), Length(4, 30, 'Username should consist of 4-30 characters')])- password = PasswordField('Password (6-40)', [DataRequired(), Length(6, 40, 'Passowrd should consist of 6-40 characters')])+ first_name = StringField('First Name', [DataRequired(), Length(+ 4, 100, 'First name should consist of 4-200 characters')])+ last_name = StringField('Last Name', [DataRequired(), Length(+ 4, 100, 'Last name should consist of 4-200 characters')])+ username = StringField('Username (4-30)', [DataRequired(), Length(+ 4, 30, 'Username should consist of 4-30 characters')])+ password = PasswordField(+ 'Password (6-40)', [DataRequired(), Length(6, 40, 'Passowrd should consist of 6-40 characters')])repeat_password = PasswordField('Repeat Password')accept_terms = BooleanField('I accept the TOS', [DataRequired()])recaptcha = RecaptchaField()@@ -19,11 +23,13 @@ class UserRegistrationForm(FlaskForm):class UserLoginForm(FlaskForm):- username = StringField('Username (4-30)', validators=[DataRequired(), Length(4, 30, 'Username should consist of 4-30 characters')])- password = PasswordField('Password (6-40)', validators=[DataRequired(), Length(6, 40, 'Passowrd should consist of 6-40 characters')])+ username = StringField('Username (4-30)', validators=[DataRequired(), Length(+ 4, 30, 'Username should consist of 4-30 characters')])+ password = PasswordField('Password (6-40)', validators=[DataRequired(), Length(+ 6, 40, 'Passowrd should consist of 6-40 characters')])recaptcha = RecaptchaField()submit = SubmitField('Sign In')-+@propertydef visible_fields(self):- return (self.username, self.password)\ No newline at end of file+ return (self.username, self.password)diff --git a/app/main.py b/archived/app/main.pysimilarity index 51%rename from app/main.pyrename to archived/app/main.pyindex eaf8bc9..5ad3ef6 100644--- a/app/main.py+++ b/archived/app/main.py@@ -2,5 +2,5 @@ from app import app, dbfrom views import *-if __name__ == '__main__':- app.run()\ No newline at end of file+if __name__ == '__main__':+ app.run()diff --git a/app/models.py b/archived/app/models.pysimilarity index 95%rename from app/models.pyrename to archived/app/models.pyindex 5068be6..4d20d0a 100644--- a/app/models.py+++ b/archived/app/models.py@@ -4,7 +4,6 @@ from sqlalchemy.sql import funcfrom crypto import encrypt-class ReprMixin:def __repr__(self):return f'<{self.__class__.__name__} object id: {self.id}, username: {self.username}, created at: {self.created}>'@@ -14,12 +13,12 @@ class SecretMixin:@classmethoddef create(cls, pwd, **kwargs):obj = cls(password=encrypt(pwd), **kwargs)-+db.session.add(obj)db.session.commit()print(f'Created: {obj}')-+return obj@classmethod@@ -28,11 +27,10 @@ class SecretMixin:print(f'User {username} found: {obj}')if obj and obj.password == encrypt(pwd):return obj-+@classmethoddef update_pwd(cls, id, pwd):- pass-+ passclass User(ReprMixin, SecretMixin, db.Model):diff --git a/app/static/css/fonts.css b/archived/app/static/css/fonts.csssimilarity index 100%rename from app/static/css/fonts.cssrename to archived/app/static/css/fonts.cssdiff --git a/app/templates/base.html b/archived/app/templates/base.htmlsimilarity index 100%rename from app/templates/base.htmlrename to archived/app/templates/base.htmldiff --git a/app/templates/error.html b/archived/app/templates/error.htmlsimilarity index 100%rename from app/templates/error.htmlrename to archived/app/templates/error.htmldiff --git a/app/templates/index.html b/archived/app/templates/index.htmlsimilarity index 100%rename from app/templates/index.htmlrename to archived/app/templates/index.htmldiff --git a/app/templates/login.html b/archived/app/templates/login.htmlsimilarity index 100%rename from app/templates/login.htmlrename to archived/app/templates/login.htmldiff --git a/app/templates/register.html b/archived/app/templates/register.htmlsimilarity index 100%rename from app/templates/register.htmlrename to archived/app/templates/register.htmldiff --git a/app/templates/subscriptions.html b/archived/app/templates/subscriptions.htmlsimilarity index 100%rename from app/templates/subscriptions.htmlrename to archived/app/templates/subscriptions.htmldiff --git a/app/views.py b/archived/app/views.pysimilarity index 98%rename from app/views.pyrename to archived/app/views.pyindex 1c46cb7..60e8fe8 100644--- a/app/views.py+++ b/archived/app/views.py@@ -5,6 +5,7 @@ from models import Userfrom functools import wraps+bad_user_agents = ['request','robot',@@ -88,7 +89,7 @@ def login():if request.method == 'GET':return render_template('login.html', form=form)- if form.validate_on_submit():+ if form.validate_on_submit():user = User.validate_user(form.username.data, form.password.data)if user:session['user_id'] = user.iddiff --git a/bufferization.py b/archived/bufferization.pysimilarity index 91%rename from bufferization.pyrename to archived/bufferization.pyindex a18cf83..88079db 100644--- a/bufferization.py+++ b/archived/bufferization.py@@ -2,13 +2,14 @@ import csvfrom logger_config import get_logger+logger = get_logger('buffer')class Buffer:lower_limit = 5upper_limit = 200- fn = ['Title','Address','WebSite','PhoneNumber']+ fn = ['Title', 'Address', 'WebSite', 'PhoneNumber']def __init__(self, filename: str = 'leads.csv', buffer_size: int = 5):if not filename.endswith('.csv'):@@ -34,9 +35,9 @@ class Buffer:self._data.clear()def store(self, info: dict):- if info.get('Title') and not info in self._data:+ if info.get('Title') and not info in self._data:self._data.append(info)if len(self._data) >= self._buffer_size:self.dump()else:- logger.warning('Got duplicate or empty dictionary %s', info)\ No newline at end of file+ logger.warning('Got duplicate or empty dictionary %s', info)diff --git a/emails/config.py b/archived/emails/config.pysimilarity index 100%rename from emails/config.pyrename to archived/emails/config.pydiff --git a/emails/extract_emails.py b/archived/emails/extract_emails.pysimilarity index 100%rename from emails/extract_emails.pyrename to archived/emails/extract_emails.pydiff --git a/emails/hunterio.py b/archived/emails/hunterio.pysimilarity index 100%rename from emails/hunterio.pyrename to archived/emails/hunterio.pydiff --git a/emails/locators.py b/archived/emails/locators.pysimilarity index 100%rename from emails/locators.pyrename to archived/emails/locators.pydiff --git a/google-maps/extractor.py b/archived/google-maps/extractor.pysimilarity index 97%rename from google-maps/extractor.pyrename to archived/google-maps/extractor.pyindex 95bf589..8712218 100644--- a/google-maps/extractor.py+++ b/archived/google-maps/extractor.py@@ -1,5 +1,7 @@import reimport asyncio+from webdriver import Webdriver+from logger_config import get_loggerfrom locators import *from my_config import *@@ -12,8 +14,6 @@ from pyppeteer.errors import PageErrorimport syssys.path.append('..')-from logger_config import get_logger-from webdriver import Webdriverlogger = get_logger('webdriver.google_maps')@@ -30,27 +30,26 @@ class GoogleMaps(Webdriver)::return: returns nothing'''-if not '_page' in self.__dict__:raise ValueError('Initialize the browser before searching by `await *.init_broswser()`')-+if location and keyword and not url:logger.debug('Working based on location `%s` and keyword `%s`', location, keyword)self._location = locationself._keyword = keywordawait self._locate()-+elif not (location and keyword) and url:await self._jump(url, search_xpath)logger.debug('Working based on given URL `%s`', url)-+else:await self._shut_browser()raise ValueError(f'Specify location and keyword. Otherwise, specify URL only. Got location `{location}`, keyword `{keyword}` and URL `{url}`')-+await self._scrape()await self._shut_browser()@@ -117,7 +116,7 @@ class GoogleMaps(Webdriver):back = (await self._page.xpath(back_xpath))[0]await self._do_retry(back.click, result_xpath)except:- logger.warning('Error occured',exc_info=True)+ logger.warning('Error occured', exc_info=True)next_button = (await self._page.xpath(next_xpath))[0]if not next_button:diff --git a/google-maps/locators.py b/archived/google-maps/locators.pysimilarity index 79%rename from google-maps/locators.pyrename to archived/google-maps/locators.pyindex fde8e0e..816a8d3 100644--- a/google-maps/locators.py+++ b/archived/google-maps/locators.py@@ -1,8 +1,6 @@button_xpath = '//*[@class="iRxY3GoUYUY__button gm2-hairline-border section-action-chip-button"]'search_xpath = '//*[@role="gridcell"]'next_xpath = '//*[@id="n7lv7yjyC35__section-pagination-button-next"]'-# check_xpath = '//*[@src="//www.gstatic.com/images/icons/material/system_gm/2x/share_gm_blue_18dp.png"]'place_xpath = '//*[@class="section-hero-header-title-title GLOBAL__gm2-headline-5"]'result_xpath = '//*[@class="section-result-content"]'back_xpath = '//*[@class="section-back-to-list-button blue-link noprint"]'-diff --git a/google-maps/my_config.py b/archived/google-maps/my_config.pysimilarity index 100%rename from google-maps/my_config.pyrename to archived/google-maps/my_config.pydiff --git a/linkedin/extractor.py b/archived/linkedin/extractor.pysimilarity index 79%rename from linkedin/extractor.pyrename to archived/linkedin/extractor.pyindex ddfa9fd..9089dbd 100644--- a/linkedin/extractor.py+++ b/archived/linkedin/extractor.py@@ -1,5 +1,6 @@-import reimport asyncio+from webdriver import Webdriver+from logger_config import get_loggerfrom locators import *from my_config import *@@ -10,12 +11,9 @@ from random import uniform, randintfrom bs4 import BeautifulSoupfrom fake_useragent import UserAgent-from pyppeteer.errors import PageErrorimport syssys.path.append('..')-from logger_config import get_logger-from webdriver import Webdriverlogger = get_logger('webdriver.linkedin')@@ -25,16 +23,17 @@ ua = UserAgent()class LinkedIn(Webdriver):-def __init__(self, email=None, pwd=None):if not (email and pwd):try:self.email = environ.get('linkedin_mail')self.pwd = environ.get('linkedin_pwd')except:- logger.fatal('Cannot get email and password to login in LinkedIn account', exc_info=True)+ logger.fatal(+ 'Cannot get email and password to login in LinkedIn account', exc_info=True)else:- logger.warning('Specify email and pwd as local variables `linkedin_mail` and `linkedin_pwd` for your privacy')+ logger.warning(+ 'Specify email and pwd as local variables `linkedin_mail` and `linkedin_pwd` for your privacy')self.email = emailself.pwd = pwd@@ -42,13 +41,14 @@ class LinkedIn(Webdriver):await self._page.goto(linkedin_link)sleep(uniform(1.7, 3.4))await asyncio.wait([- self._page.click(login_locator, {'button':'left','delay': uniform(50, 100)}),+ self._page.click(+ login_locator, {'button': 'left', 'delay': uniform(50, 100)}),self._page.waitForNavigation()])sleep(uniform(1.7, 3.4))await self.type_(self.email)-+await self._page.click(pwd_locator)sleep(uniform(0.3, 0.9))await self.type_(self.pwd)@@ -62,7 +62,7 @@ class LinkedIn(Webdriver):async def search_title(self, title):# await self._page.click(search_locator)# sleep(uniform(2, 3))-+# await self.type_(title)# sleep(uniform(0.5, 1))@@ -90,7 +90,8 @@ class LinkedIn(Webdriver):soup = BeautifulSoup(html, 'html.parser')position = soup.find(string=head_regex)- name = position.parent.parent.parent.find(attrs={'aria-hidden':True}).text+ name = position.parent.parent.parent.find(+ attrs={'aria-hidden': True}).textreturn (position, name)@@ -98,9 +99,9 @@ class LinkedIn(Webdriver):passasync def type_(self, string):- for letter in string:- self._page.keyboard.type(letter)- sleep(uniform(0.1, 0.3))+ for letter in string:+ self._page.keyboard.type(letter)+ sleep(uniform(0.1, 0.3))async def behave_like_human(self):- pass\ No newline at end of file+ passdiff --git a/linkedin/locators.py b/archived/linkedin/locators.pysimilarity index 85%rename from linkedin/locators.pyrename to archived/linkedin/locators.pyindex d4f990f..274086f 100644--- a/linkedin/locators.py+++ b/archived/linkedin/locators.py@@ -4,4 +4,4 @@ show_pwd_locator = 'span#password-visibility-toggle'search_locator = 'input.search-global-typeahead__input.always-show-placeholder'companies_locator = '.entity-result__title-text a'-select_bar_xpath = '//*[@d="M8 11L3 6h10z"]'\ No newline at end of file+select_bar_xpath = '//*[@d="M8 11L3 6h10z"]'diff --git a/linkedin/my_config.py b/archived/linkedin/my_config.pysimilarity index 53%rename from linkedin/my_config.pyrename to archived/linkedin/my_config.pyindex f5d57fe..6427b94 100644--- a/linkedin/my_config.py+++ b/archived/linkedin/my_config.py@@ -1,6 +1,8 @@import re+__all__ = ['linkedin_link', 'head_regex']linkedin_link = 'https://www.linkedin.com/home'-head_regex = re.compile(r'(ceo|owner|manager|chief|manager)', flags=re.IGNORECASE)\ No newline at end of file+head_regex = re.compile(+ r'(ceo|owner|manager|chief|manager)', flags=re.IGNORECASE)diff --git a/linkedin/test_regex.py b/archived/linkedin/test_regex.pysimilarity index 52%rename from linkedin/test_regex.pyrename to archived/linkedin/test_regex.pyindex 396cdbe..5d74e1f 100644--- a/linkedin/test_regex.py+++ b/archived/linkedin/test_regex.py@@ -1,12 +1,14 @@-import re+import refrom bs4 import BeautifulSoup-head_regex = re.compile(r'(ceo|owner|manager|chief|manager)', flags=re.IGNORECASE)++head_regex = re.compile(+ r'(ceo|owner|manager|chief|manager)', flags=re.IGNORECASE)html = open('test.html', encoding='utf-8').read()soup = BeautifulSoup(html, 'html.parser')head_man = soup.find(string=head_regex)print(head_man)-name = head_man.parent.parent.parent.find(attrs={'aria-hidden':True}).text+name = head_man.parent.parent.parent.find(attrs={'aria-hidden': True}).textprint(name)diff --git a/logger_config.py b/archived/logger_config.pysimilarity index 99%rename from logger_config.pyrename to archived/logger_config.pyindex 4ceac7f..c8fcdb9 100644--- a/logger_config.py+++ b/archived/logger_config.py@@ -8,7 +8,7 @@ def get_logger(name: str)::param name: Name of the logger:return: returns full-fledged logger'''-+if isinstance(name, str):f_formatter = logging.Formatter(diff --git a/webdriver.py b/archived/webdriver.pysimilarity index 94%rename from webdriver.pyrename to archived/webdriver.pyindex 7572586..0aa81bf 100644--- a/webdriver.py+++ b/archived/webdriver.py@@ -14,12 +14,11 @@ from bufferization import Bufferlogger = get_logger('webdriver')-class Webdriver:_ua = UserAgent()_buf = Buffer()_monitor = get_monitors()[0]- _viewport = {'width':_monitor.width, 'height': _monitor.height,}+ _viewport = {'width': _monitor.width, 'height': _monitor.height, }async def init_browser(self, hidden=False, language='en-gb'):'''@@ -55,7 +54,8 @@ class Webdriver:async def _do_retry(self, operation, xpath, dest=None, retries=0):if retries == 10:- raise SystemError('Max 10 retries exceeded when clicking the place')+ raise SystemError(+ 'Max 10 retries exceeded when clicking the place')try:if dest:await operation(dest)@@ -68,7 +68,7 @@ class Webdriver:await self._do_retry(operation, xpath, dest, retries + 1)except:logger.error('Exception found when retrying', exc_info=True)-+async def _jump(self, url, xpath):try:await self._page.goto(url)@@ -78,5 +78,3 @@ class Webdriver:logger.warning('Webdriver got invalid URL for lead generation')await self._shut_browser()raise ValueError('Got invalid URL for google maps')--diff --git a/yellow-pages/extractor.py b/archived/yellow-pages/extractor.pysimilarity index 78%rename from yellow-pages/extractor.pyrename to archived/yellow-pages/extractor.pyindex ec532f8..ac18f8c 100644--- a/yellow-pages/extractor.py+++ b/archived/yellow-pages/extractor.py@@ -1,6 +1,8 @@import json-import requestsimport asyncio+import requests+from logger_config import get_logger+from webdriver import Webdriverfrom pyppeteer.errors import TimeoutError@@ -14,15 +16,14 @@ from my_config import *import syssys.path.append('..')-from webdriver import Webdriver-from logger_config import get_loggerlogger = get_logger('webdriver.yelp')ua = UserAgent()-_found = lambda loc, desc: not f'No Results for {desc}' in requests.get(yelp.format(desc, loc), headers={'User-Agent': ua.random}).text+def _found(loc, desc): return not f'No Results for {desc}' in requests.get(+ yelp.format(desc, loc), headers={'User-Agent': ua.random}).textclass Yelp(Webdriver):@@ -37,11 +38,13 @@ class Yelp(Webdriver)::return: returns nothing'''if not '_page' in self.__dict__:- raise ValueError('Initialize the browser before searching by `await *.init_broswser()`')+ raise ValueError(+ 'Initialize the browser before searching by `await *.init_broswser()`')if loc and desc and not url:if _found(loc, desc):- logger.debug('Working based on location `%s` and keyword `%s`', loc, desc)+ logger.debug(+ 'Working based on location `%s` and keyword `%s`', loc, desc)self._data = [desc, loc]await self._locate()else:@@ -53,7 +56,8 @@ class Yelp(Webdriver):else:await self._shut_browser()- raise ValueError('Specify only location and description or only URL for yelp page')+ raise ValueError(+ 'Specify only location and description or only URL for yelp page')await self._scrape()await self._shut_browser()@@ -77,9 +81,11 @@ class Yelp(Webdriver):soup = BeautifulSoup(html, 'html.parser')try:- raw_data = json.loads(soup.find('script', attrs=json_attrs).string)+ raw_data = json.loads(+ soup.find('script', attrs=json_attrs).string)except AttributeError:- logger.warning('Could not locate `raw_data` skipping current page')+ logger.warning(+ 'Could not locate `raw_data` skipping current page')returntry:@@ -89,11 +95,11 @@ class Yelp(Webdriver):try:data = {- 'WebSite': website,- 'Title': raw_data['name'],- 'PhoneNumber': raw_data['telephone'],- 'Address': ', '.join(list(raw_data['address'].values())),- }+ 'WebSite': website,+ 'Title': raw_data['name'],+ 'PhoneNumber': raw_data['telephone'],+ 'Address': ', '.join(list(raw_data['address'].values())),+ }logger.debug('Extracted: %s', list(data.values()))self._buf.store(data)except Exception:@@ -120,7 +126,7 @@ class Yelp(Webdriver):parse(await self._page.content())await self._page.goBack()- sleep(uniform(1,2))+ sleep(uniform(1, 2))await self._page.waitForXPath(results_xpath, {'visible': True})next_button = await self._page.xpath(next_xpath)diff --git a/yellow-pages/locators.py b/archived/yellow-pages/locators.pysimilarity index 100%rename from yellow-pages/locators.pyrename to archived/yellow-pages/locators.pydiff --git a/yellow-pages/my_config.py b/archived/yellow-pages/my_config.pysimilarity index 95%rename from yellow-pages/my_config.pyrename to archived/yellow-pages/my_config.pyindex 7e54410..b28419a 100644--- a/yellow-pages/my_config.py+++ b/archived/yellow-pages/my_config.py@@ -1 +1 @@-yelp = 'https://www.yelp.com/search?find_desc={}&find_loc={}&ns=1'\ No newline at end of file+yelp = 'https://www.yelp.com/search?find_desc={}&find_loc={}&ns=1'diff --git a/google-maps/__pycache__/config.cpython-38.pyc b/google-maps/__pycache__/config.cpython-38.pycdeleted file mode 100644index e190f10..0000000Binary files a/google-maps/__pycache__/config.cpython-38.pyc and /dev/null differdiff --git a/google-maps/__pycache__/extractor.cpython-38.pyc b/google-maps/__pycache__/extractor.cpython-38.pycdeleted file mode 100644index 7cdfbca..0000000Binary files a/google-maps/__pycache__/extractor.cpython-38.pyc and /dev/null differdiff --git a/google-maps/__pycache__/locators.cpython-38.pyc b/google-maps/__pycache__/locators.cpython-38.pycdeleted file mode 100644index eb8b3da..0000000Binary files a/google-maps/__pycache__/locators.cpython-38.pyc and /dev/null differdiff --git a/google-maps/__pycache__/my_config.cpython-38.pyc b/google-maps/__pycache__/my_config.cpython-38.pycdeleted file mode 100644index 5ab18e2..0000000Binary files a/google-maps/__pycache__/my_config.cpython-38.pyc and /dev/null differdiff --git a/google-maps/leads.csv b/google-maps/leads.csvdeleted file mode 100644index 6828319..0000000--- a/google-maps/leads.csv+++ /dev/null@@ -1,286 +0,0 @@-Title,Address,WebSite,PhoneNumber-Kazzhol Hotel Astana,"Balqantau St 213, Nur-Sultan 010000",hotelkazzhol.kz,8 (7172) 40 52 52-Eco Apart Hotel Astana,"Qabanbay Batyr Avenue h. 58B pod.3 ap.50, Nur-Sultan 020000",ecoaparthotel.kz,8 (708) 311 6378-Hampton by Hilton Astana Triumphal Arch,"Prospekt Mangilik Yel. 43A, Nur-Sultan 010000",hilton.com,8 (7172) 70 35 00-Diplomat Hotel & Business Center,"Dinmukhamed Qonayev St 29/1, Nur-Sultan 010000",diplomathotel.kz,8 (7172) 55 00 01-ibis Nur-Sultan,"Tauelsizdik Ave 38, Nur-Sultan 010000",all.accor.com,8 (7172) 70 82 00-King Hotel Astana,"Shokan Valikhanov Street 7, Nur-Sultan 010000",kinghotelastana.kz,8 (7172) 70 57 05-Astana Best Hostel,"Zhk Infiniti 2, Qabanbay Batyr Ave 42, Nur-Sultan 010000",-,8 (7172) 28 68 29-Park Avenue,"Prospekt Mangilik Yel. 37/1, Nur-Sultan 010000",hotelparkavenue.kz,8 (701) 052 2266-Alanda Hotel,"Tauelsizdik Ave 33, Nur-Sultan 020000",alanda-hotel.kz,8 (7172) 72 93 09-MyLINE relax complex,"Bayimbet Maylin St 37 А, Nur-Sultan 010000",-,--Caps Lock Future Hotel,"Dinmukhamed Qonayev St 29, Nur-Sultan 020000",capslock.kz,8 (771) 777 8111-Hostel S size,"Ул. Бухар Жырау 36, Nur-Sultan 020000",-,8 (775) 103 3337-Antalya Hotel,"Balqantau St 94, Nur-Sultan 020000",antalyahotel.kz,8 (705) 555 3772-San Marino,"Zhumabek Tashenov St 17, Nur-Sultan 010000",-,8 (7172) 41 92 92-Come INN Hotel,"Арганаты 13, Nur-Sultan 010000",comeinnhotel.com,8 (7172) 57 93 26-HELLO INN | Ex-Hostelland Astana,"Zhumabek Tashenov St 21/1, Nur-Sultan 010000",helloinn.kz,8 (778) 999 0932-Manhattan Astana Hotel,"Tauelsizdik Ave 5, Nur-Sultan 020000",manhattanhotel.kz,8 (7172) 36 15 36-Kapsula Hotel Astana,"Qabanbay Batyr Ave 7/3, Nur-Sultan 020000",kapsula.kz,8 (7172) 50 74 04-Jelsomino Boutique Hotel,"Amangeldi Imanov Street 26, Nur-Sultan 010000",jelsominohotel.kz,8 (701) 100 0201-G Empire,"Abay Ave 63, Nur-Sultan 010000",gempire.kz,8 (7172) 40 55 01-Red Village Hotel,"ул. Мустафина 1/1, Nur-Sultan 010009",redvillagehotel.kz,8 (7172) 57 08 08-MAXX inn,"46/1, Qabanbay Batyr Ave, Nur-Sultan 010000",-,8 (701) 214 0106-Alanda Hotel,"Tauelsizdik Ave 33, Nur-Sultan 020000",alanda-hotel.kz,8 (7172) 72 93 09-Hostel Forum,"Qabanbay Batyr Ave 9, Nur-Sultan 010000",hostelforum.business.site,8 (778) 950 8631-Inzhu,"Aleksandr Barayev St 9, Nur-Sultan 010000",-,8 (7172) 30 23 31-Astana Plaza,"Zhengis Ave 27А, Nur-Sultan 020000",-,8 (7172) 44 48 61-Makenty house hotel,"Qabanbay Batyr Ave 40, Nur-Sultan 010000",-,8 (778) 135 5032-Alanda Hotel,"Tauelsizdik Ave 33, Nur-Sultan 020000",alanda-hotel.kz,8 (7172) 72 93 09-Bozok,"Abay Ave, Nur-Sultan 010000",-,--Sultan Beybars,"Abylay Khan Ave 16, Nur-Sultan 010000",sultanbeibarys.com,8 (7172) 56 21 80-NCityHotel,"Sarayshyq St 40, Nur-Sultan 020000",-,--San Marino,"Zhumabek Tashenov St 17, Nur-Sultan 010000",-,8 (7172) 41 92 92-Sky House Astana,"Dostyq St 5, Nur-Sultan 020000",-,--miniГостиница,"Qabanbay Batyr Ave 29, Nur-Sultan 020000",-,--Rixos President Astana,"Kunayeva 7, Nur-Sultan 010000",all.accor.com,8 (7172) 24 50 50-Kapsula Hotel Astana,"Qabanbay Batyr Ave 7/3, Nur-Sultan 020000",kapsula.kz,8 (7172) 50 74 04-Jelsomino Boutique Hotel,"Amangeldi Imanov Street 26, Nur-Sultan 010000",jelsominohotel.kz,8 (701) 100 0201-Park Inn by Radisson Astana Hotel,"Saryarka Avenue 8, Nur-Sultan 010000",radissonhotels.com,8 (7172) 67 00 00-Wyndham Garden Astana,"Хусейн бен Талал 25, Nur-Sultan 010000",wyndhamgardenastana.com,8 (775) 030 4444-G Empire,"Abay Ave 63, Nur-Sultan 010000",gempire.kz,8 (7172) 40 55 01-Orynbor,"Prospekt Mangilik Yel. 50, Nur-Sultan 010000",-,8 (707) 101 7457-Royal Park Hotel & Spa,"Kerey and Zhanibek Khans St 18, Nur-Sultan 010000",-,8 (7172) 42 22 01-Отель The One - Гостиница в Астане - Нур-султане,"Qadyrghali Zhalayyri St, Nur-Sultan 020000",theone-astana.kz,8 (771) 222 2400-Gostinitsa Atlas,"Vol'fgan Gete St 12/1, Nur-Sultan 010000",-,8 (778) 699 7626-Manhattan Astana Hotel,"Tauelsizdik Ave 5, Nur-Sultan 020000",manhattanhotel.kz,8 (7172) 36 15 36-ibis Nur-Sultan,"Tauelsizdik Ave 38, Nur-Sultan 010000",all.accor.com,8 (7172) 70 82 00-Manhattan Astana Hotel,"Tauelsizdik Ave 5, Nur-Sultan 020000",manhattanhotel.kz,8 (7172) 36 15 36-East Astana Hotel,"Auezova street 12, Nur-Sultan 010000",easthotelastana.kz,8 (775) 333 0055-Hotel Inju Hills,"Mukhtar Auezov Street 50/1, Nur-Sultan 010000",-,8 (771) 161 2290-Sheraton Nur-Sultan Hotel,"Syghanaq St 16/1, Nur-Sultan 010000",marriott.com,8 (7172) 70 05 55-Residenz,"Petrov St 21/1, Nur-Sultan 010000",-,8 (7172) 34 22 49-Hotel Inju on Baraeva 9,"Aleksandr Barayev St 9, Nur-Sultan 001000",-,8 (776) 230 2335-Prestige,"Zheltoqsan St 1, Nur-Sultan 010000",-,8 (7172) 32 51 81-"The Ritz-Carlton, Astana","Dostyk Street 16, Nur-Sultan 010016",ritzcarlton.com,8 (7172) 73 40 00-ЖК The Arc Residence,"ул. Мәңгілік Ел, Nur-Sultan 020000",thearc.kz,8 (777) 722 1010-Hostel Nomad4x4,"Нуртаса Ондасынова, Чубары микрорайон 52А, Nur-Sultan 010001",hostel-nomad4x4.business.site,8 (708) 865 1940-Elite Hostel,"Shokan Valikhanov Street 9/2, Nur-Sultan 010000",-,8 (775) 653 0013-Goldman Empire,"Сатбаев көшесі 13/1, Nur-Sultan 010000",goldmanempire.kz,8 (7172) 68 20 20-"Гостевой дом ""Altyn hotel#","Tsentr Mezhdunarodnoy Torgovli, Dostyq St 1, Nur-Sultan 020000",-,8 (702) 250 6000-"Mini Hotel ""Prospect""","Syghanaq St 9, Nur-Sultan 020000",-,8 (701) 301 4074-The St. Regis Astana,"Qabanbay Batyr Ave 1, Nur-Sultan 010000",marriott.com,8 (7172) 79 08 88-Zhassybi Hotel,"Shokan Valikhanov Street 26, Nur-Sultan 010000",-,--Altyn-Adam Hotel,"Säken Seyfullin Street 38, Nur-Sultan 010000",-,--Lion,"Maskeu St 37/2, Nur-Sultan 010000",-,8 (7172) 39 46 20-VIP House Apartments,"Dostyq St 5/1, Nur-Sultan 010000",-,8 (700) 375 5774-Kazzhol Hotel Astana,"Balqantau St 213, Nur-Sultan 010000",hotelkazzhol.kz,8 (7172) 40 52 52-Astina Inn Hotel,"Burabay St, Nur-Sultan 010000",astina.kz,8 (747) 909 9918-Pulsar Mini-Hotel,"Tauelsizdik Ave 49/3, Nur-Sultan 010011",-,8 (771) 881 1100-Elite Apart-Hotel,"Raqymjan Qoshqarbayev Avenue 10\1, Nur-Sultan 010017",eliteapart.kz,8 (7172) 76 97 33-Mini-hotel Ulpan,"Khalel Dosmukhameduly St 11, Nur-Sultan 010000",hotelulpan.com,8 (7172) 24 03 44-Отель Union Plaza,"Almaty St 1/1, Nur-Sultan 020000",-,8 (776) 425 2226-Skyversal гостиница,"Sarayshyq St 40, Nur-Sultan 010000",-,8 (702) 136 2499-"Radisson Hotel, Astana","Saryarka Avenue 4, Nur-Sultan 010000",radissonhotels.com,8 (7172) 67 07 77-QonaQ hotel,"пр. Мәңгілік Ел дом 50 НП 19, Отель/кафе ""QonaQ"", Nur-Sultan 010000",-,8 (771) 099 0555-Mini Hotel Bereket Dip,"Sarayshyq St 34, Nur-Sultan 010000",-,8 (707) 777 2310-Astina Inn Hotel,"Burabay St, Nur-Sultan 010000",astina.kz,8 (747) 909 9918-Tengri Hotel,"Bayimbet Maylin St 1A, Nur-Sultan 020000",tengrihotel.kz,8 (7172) 41 38 38-Seven Inn Boutique Hotel,"36th St, Nur-Sultan 010000",-,8 (7172) 65 00 01-ARENA Hotel,"ЖК""Sezim Qala, Turan Ave 55, Nur-Sultan 010000",-,--Gokai Hotel,"Bauyrzhan Momyshuly Ave 24, Nur-Sultan 010009",gokai-hotel.business.site,8 (705) 134 5533-The Veil,"Q. Zhalayyri street, Nur-Sultan 010000",-,8 (701) 068 8800-Torgay,"Qanysh Satbaev St 15, Nur-Sultan 020000",torgai.kz,8 (7172) 97 82 76-Shaghyn,"Sauran Street 2, Nur-Sultan 020000",-,8 (701) 272 8773-Nomad Hotel,"Kuyshi Dina St 25/1, Nur-Sultan 010000",-,--Гостиница QAINAR HOTEL,"Aqkent Ln дом 5А, Nur-Sultan 020000",-,8 (701) 543 0893-Orion,"Alexander Pushkin St 3/2, Nur-Sultan 010000",orionhotel.kz,8 (701) 223 3636-Astana Central Hotel,"м-н 2, 010000, Ayagoz St, Nur-Sultan",astanacentralhotel.kz,8 (7172) 99 99 22-Alash Hotel,"Блок А, Bauyrzhan Momyshuly Ave 12, Nur-Sultan",-,8 (7172) 99 84 15-"Апарт-отель ""MONACO"" Астана","Charles de Gaulle St 1, Nur-Sultan 010000",monacohotel.kz,8 (7172) 22 22 82-Hostel Nomad4x4,"Нуртаса Ондасынова, Чубары микрорайон 52А, Nur-Sultan 010001",hostel-nomad4x4.business.site,8 (708) 865 1940-Capital Hostel,"Novyy Mir, Dinmukhamed Qonayev St 35, Nur-Sultan 020000",-,8 (747) 295 4988-Hotel Akkaiyn,"Zhayyq St 30, Nur-Sultan 020000",charleston.kz,8 (701) 888 1208-Porta Fortuna,"Sembinov St, Nur-Sultan 010000",-,--Versal Hotel,"Sarayshyq St 40, Nur-Sultan 010000",versal-hotel.hoteleastana.com,8 (7172) 56 94 43-Hotel AyKun,"Kerey and Zhanibek Khans St 12 Б, Nur-Sultan 010016",aykun.kz,8 (775) 121 1152-Pulsar Mini-Hotel,"Tauelsizdik Ave 49/3, Nur-Sultan 010011",-,8 (771) 881 1100-Interia,"Abylay Khan Ave 37/1, Nur-Sultan 010000",-,--Traditional,"Turan Ave 33, Nur-Sultan 010000",traditional.kz,8 (7172) 40 21 81-Hostel Americana,"Baltalı 2, Nur-Sultan 010000",-,8 (775) 828 7750-Eco Apart Hotel Astana,"Qabanbay Batyr Avenue h. 58B pod.3 ap.50, Nur-Sultan 020000",ecoaparthotel.kz,8 (708) 311 6378-"Апарт-отель ""MONACO"" Астана","Charles de Gaulle St 1, Nur-Sultan 010000",monacohotel.kz,8 (7172) 22 22 82-O Azamat,"Grigorii Potanin St 9, Nur-Sultan 020000",oazamat-hotel.kz,8 (7172) 57 80 80-Minigostinitsa Bereket,"Zhk Zhagalau, Taras Shevchenko St 10, Nur-Sultan 020000",-,8 (7172) 40 40 66-Hostel Forum,"Qabanbay Batyr Ave 9, Nur-Sultan 010000",hostelforum.business.site,8 (778) 950 8631-Hotel Aner,"проспект М.Жумабаева 25/4 25/4, Nur-Sultan 010000",hotelaner.tilda.ws,8 (775) 000 7081-"Mini hotel ""SULTAN""","Sarayshyq St 34, Nur-Sultan 010000",-,8 (771) 123 4145-Rush Hotel,"Zhenis Avenue 53/1, Nur-Sultan 010000",rush-hotel.kz,8 (7172) 38 14 51-Business Hotel,"Amangeldi Imanov Street 12, Nur-Sultan 020000",-,8 (7172) 53 73 26-Saryarka Hotel,"Sembinov St 36, Nur-Sultan 010000",-,--Botakoz,"Syghanaq St 10, Nur-Sultan 020000",-,8 (7172) 51 04 92-Hotel Akkaiyn,"Zhayyq St 30, Nur-Sultan 020000",charleston.kz,8 (701) 888 1208-Elite Apart-Hotel,"Raqymjan Qoshqarbayev Avenue 10\1, Nur-Sultan 010017",eliteapart.kz,8 (7172) 76 97 33-Comfort Hotel Astana,"Cosmonauts St 60, Nur-Sultan 020000",comforthotel.kz,8 (7172) 24 44 44-Brown Hotel,"Malik Ghabdullin St 20/1, Nur-Sultan 010000",-,8 (7172) 48 55 77-Aybat,"Ulitsa Karasakal Yerimbet 65, Nur-Sultan 020000",jumbaktas.kz,8 (7172) 57 87 77-JJ Hotel,"Qabanbay Batyr Ave 43, Nur-Sultan 010000",-,8 (776) 001 3330-Boutique Hotel Tomiris,"Kurgaldzhinskoye Highway 9, Nur-Sultan 010000",-,8 (707) 555 5950-Beijing Palace Soluxe Hotel Astana,"Syghanaq St 27, Nur-Sultan 010000",soluxe-astana.kz,8 (7172) 70 15 15-Hotel Kausar,"Bogenbay Batyr Ave 6/2, Nur-Sultan 010000",hotel-kausar.kz,8 (778) 808 1616-Aisha bibi hotel and apart,"Aysha Bibi St 38, Nur-Sultan 020000",aisha-bibi.com,8 (771) 899 4205-Grace Point Hotel,"Syghanaq St 2, Nur-Sultan 010000",gracepoint.kz,8 (7172) 29 98 80-"Гостиница ""Bed and Breakfast""","Sarayshyq St 38, ВП-6, Nur-Sultan 010017",hotelbb.kz,8 (702) 387 3534-Triumph Astana Hostel,"Проспект Кабанбай батыра 11, 6 секция , 18 этаж , 386 квартира, Nur-Sultan 010000",astana-hostel.kz,--Astana Transit Hotel,"Астананың Халықаралық әуежайы, Qabanbay Batyr Ave 119, Nur-Sultan",astanaairport.com,8 (7172) 77 71 69-Hotel Grand Park Esil,"Beibitshilik Street 8, Nur-Sultan 020000",grandparkesil.kz,8 (7172) 59 19 01-Kel Inn,"Shakarim Kudayberdiuly Avenue 24/193, Nur-Sultan 020000",-,8 (777) 176 5566-Zhastar,"улица А. Бараева 10/3, Nur-Sultan 010000",-,8 (775) 202 8688-Арыстан,"Арқайым көшесі 11, Nur-Sultan 010000",instagram.com,8 (702) 181 7229-Altyn Orda,"Azerbaijan Mambetov St 16, Nur-Sultan 010000",-,8 (7172) 68 06 80-Гостиница B&B Akbulak2,"Sargul St дом 7, Nur-Sultan 010010",-,8 (747) 941 5887-Merkuriy I K,"Tsiolkovskiy Str., Nur-Sultan 010000",-,8 (7172) 37 43 96-Sunkar,"Sembinov Str. 15, Nur-Sultan 010000",hotel-sunkar.kz,8 (7172) 56 96 70-ЖК The Arc Residence,"ул. Мәңгілік Ел, Nur-Sultan 020000",thearc.kz,8 (777) 722 1010-Premier Inn Astana,"Balqantau St 187, Nur-Sultan 010000",premierinn.kz,8 (771) 333 3336-Zhan Villa,"Balqantau St 124, Nur-Sultan 010000",-,8 (7172) 22 20 33-Royal M Hotel,"Alash Hwy 7, Nur-Sultan 010000",-,8 (7172) 24 84 79-Mokka Hostel,"Magzhan Zhumabaev Avenue 17, Nur-Sultan 010000",-,8 (708) 695 0025-Керей,"Yelubay Taybekov St, Nur-Sultan 020000",-,8 (7172) 24 29 54-Sophia,"Abylay Khan Ave 4/4, Nur-Sultan 010000",-,8 (7172) 35 30 17-Bayterek,"Sarayshyq St 38, Nur-Sultan 010017",baiterekpremium.com,8 (7172) 57 43 21-Holiday Inn Express Astana - Turan,"Intersection of Turan and 24 Street, Nur-Sultan 020000",ihg.com,8 (7172) 31 29 14-Absolute,"Beibitshilik Street 1, Nur-Sultan 010000",absolutehotel.kz,8 (7172) 31 47 42-Shanyrak on Abaya,"Abay Ave 98, Nur-Sultan 000902",-,8 (7172) 27 43 50-"Hotel ""Duman""","Коргальджинское шоссе, 2А, Nur-Sultan",hoteldumanastana.com,8 (7172) 79 15 00-Astana Park Hotel,"Turan Ave 2, Nur-Sultan 020000",-,8 (7172) 55 63 33-Otel' A1,"55, Zhankent St, Nur-Sultan 020000",-,--Astana Zhamilya,"Tsentr Mezhdunarodnoy Torgovli, Dostyq St 1, Nur-Sultan 020000",apartamenty.kz,8 (7172) 52 43 53-Hostel Astana,"д., Kenesary St 42, Kentau",-,8 (702) 212 8282-Apartments on Kerey and Zhanibek Khans 22,"Kerey and Zhanibek Khans St 22, Nur-Sultan 010000",altynecopark.com,8 (778) 999 3805-Oasis Inn,"Bauyrzhan Momyshuly Ave 12a, Nur-Sultan 010000",oasisinn.kz,8 (778) 128 4955-Lyubimyy,"Balqantau St 77, Nur-Sultan 020000",-,8 (7172) 53 99 39-Urban Hostel,"Magzhan Zhumabaev Avenue, Nur-Sultan 010000",-,8 (708) 999 9191-Luxe,"Vol'fgan Gete St 8, Nur-Sultan 010000",-,8 (775) 387 0115-Health Club & SPA Grand Prix,"Светлый, Sarayshyq Street 1, Nur-Sultan 010000",grandprixclub.kz,8 (707) 177 7222-ЖК The Arc Residence,"ул. Мәңгілік Ел, Nur-Sultan 020000",thearc.kz,8 (777) 722 1010-Premier Inn Astana,"Balqantau St 187, Nur-Sultan 010000",premierinn.kz,8 (771) 333 3336-Gostinitsa Zhasybay,"Shokan Valikhanov Street 26, Nur-Sultan 020000",-,8 (7172) 21 79 79-Alau,"Qabanbay Batyr Ave 47, Nur-Sultan 020000",-,8 (7172) 70 67 18-Gk Ak Bulak,"Asqar Toqpanov Street 29, Nur-Sultan 020000",-,--Аксу,"Yegemen Qazaqstan Gazeti St 9/3, Nur-Sultan 020000",aksu-astana.kz,8 (701) 739 1326-Hostel infiniti,"ukl Kabanbay batyra .42 8etazh ap. 460, Nur-Sultan 010000",-,8 (700) 700 2770-Elysian hotel,"2 Есильсеий, Yengbekshiler Street, Nur-Sultan 020000",elysian-hotel.business.site,8 (700) 269 8055-Home Hotel Astana,"Manas 9A, Nur-Sultan 010000",-,8 (7172) 54 91 90-Azat Hotel,"Bayimbet Maylin St 35, Nur-Sultan 010000",hotel-13016.business.site,8 (702) 902 4442-"Гостинично-ресторанный комплекс ""Турист""","Respublika Ave 33, Nur-Sultan 010000",-,8 (7172) 65 02 00-Myrza,"Syghanaq St 21/1, Nur-Sultan 020000",-,8 (7172) 50 40 06-Qaton Qaragay,"18 Куйшы дина, Nur-Sultan 010009",-,8 (7172) 40 06 33-Hotel BM,"Dostyq St 5, Nur-Sultan 020000",-,8 (7172) 64 24 70-Алтын Дала,"Bigeldinov St 8/1, Almaty 020000",-,8 (7172) 32 33 11-Belon Life Hotel,"Abylay Khan Ave 24/1, Nur-Sultan 010000",belon.kz,--Azamat Hotel,"Shakarim Kudayberdiuly Avenue 7, Nur-Sultan 010000",azamat-hotel.business.site,8 (7172) 34 21 14-Отель Реал,"Respublika Ave 54, Nur-Sultan 010000",-,8 (778) 153 3505-YURT Hostel Astana,"Shamshi Qaldayaqov St 15/1, Nur-Sultan 020000",-,8 (708) 354 0059-Nurbi,"Гостиница ""Нурби, Kuyshi Dina St 18/1, Nur-Sultan 020000",-,8 (7172) 40 00 00-"""H8"" - hostel","zhk ""Montblanc De Lux"" blok G, Kenesary St 8, Nur-Sultan 010000",-,8 (771) 800 1001-"Condo ""Lazurny Kvartal""","Sarayshyq St 5, Nur-Sultan 020000",-,8 (700) 222 0784-O Azamat,"Grigorii Potanin St 9, Nur-Sultan 020000",oazamat-hotel.kz,8 (7172) 57 80 80-Business Hotel,"Amangeldi Imanov Street 12, Nur-Sultan 020000",-,8 (7172) 53 73 26-Egorkino,-,egorkino.kz,8 (707) 230 2222-Jasmine Hotel,"Sarayshyq St 36, Nur-Sultan 020000",-,8 (747) 197 8188-THE SQUARE,"Turan Ave 46/1, Nur-Sultan 020000",shaheeninvest.kz,8 (747) 885 8250-Gostinitsa Astana Art Otel',"Zheltoqsan St 42, Nur-Sultan 020000",-,8 (7172) 38 34 20-KEMER LUX,"Baktyoraz Beysekbayev St 25/1а, Nur-Sultan 010000",-,8 (778) 999 8555-Maminn Hostel,"Seiitkali Mendeshev St, Nur-Sultan 020000",facebook.com,8 (7172) 25 61 63-Agidel',"Vol'fgan Gete St 1Б, Nur-Sultan 010000",-,8 (7172) 93 35 51-Belon-Lux Hotel,"Dostyq St 13, Nur-Sultan 010016",belon.kz,8 (7172) 79 54 44-Otyrar Hotel,"Otyrar St 10, Nur-Sultan 010000",otyrar-hotel.kz,8 (776) 111 6501-Hotel 7 palat,"Almaty street 13 , ЖК Отандостар.Левое крыло, Nur-Sultan 020000",-,8 (7172) 51 57 59-Zhibek Zholy Hotel,"Zhanibek Tarkhan St 2, Nur-Sultan 010000",-,--Nochleg,"Sarayshyq St 38, Nur-Sultan 010000",-,8 (7172) 50 88 20-Madi,"Temirkazyk St 7, Nur-Sultan 020000",-,8 (701) 726 3766-Bayan Sulu,"Petrov St 16/4, Nur-Sultan 020000",bayansulu.kz,8 (7172) 36 60 79-"Hotel ""Daniyar""","Tauelsizdik Ave, Nur-Sultan 020000",-,8 (701) 416 6005-Korsar Hotel,"B.Momyshuly Str 13, Nur-Sultan 010000",-,8 (7172) 77 60 50-Akzhayyk,"Iliyas Yesenberlin St 7, Nur-Sultan 010000",akzhaik-hotel.kz,8 (7172) 38 00 91-Tamgaly,"Kuyshi Dina St 25, Nur-Sultan 010000",-,--Hotel Tarlan,"Tупик Льва Толстого 5, гостиница Тарлан, Nur-Sultan 010000",-,8 (701) 101 4442-Жулдыз,"Sembinov St 16, Nur-Sultan 010000",-,8 (7172) 20 09 99-Goldman Empire,"Сатбаев көшесі 13/1, Nur-Sultan 010000",goldmanempire.kz,8 (7172) 68 20 20-Kapsula Hotel Astana,"Qabanbay Batyr Ave 7/3, Nur-Sultan 020000",kapsula.kz,8 (7172) 50 74 04-Magas Hotel,"Затаевича ул., Nur-Sultan 010000",magas-hotel.kz,8 (7172) 38 20 21-Kaisar hotel,"Zhumabek Tashenov St 7/3, Nur-Sultan 020000",-,8 (7172) 22 40 34-Apple Hostel,"Улица Генерала Сабыра Ракымова 22, Nur-Sultan 010000",-,8 (707) 847 3836-Club Hostel,"Office 16, Infinity 2 complex, Qabanbay Batyr Ave 42, Nur-Sultan 010017",-,8 (7172) 65 01 46-Hotel Nursat,"Zhengis Ave 63, Nur-Sultan 010000",-,--Uyut,"улица Cығанақ 18/1, Nur-Sultan 010000",-,8 (701) 383 1149-Hotel Milan,"г.Астана, Акбулак 2, д 12, Daraboz St 12, Nur-Sultan",mlife1.wix.com,8 (7172) 34 23 61-Alatau Hotel,"Zhaidarman Ln 3, Nur-Sultan 020000",-,8 (701) 111 3769-River house,"Azerbaijan Mambetov St 4, Nur-Sultan 010000",-,8 (747) 500 0129-Olimp Hostel,"Kerey and Zhanibek Khans St 22, Nur-Sultan 020000",-,8 (775) 000 6901-Hostel Millenium Park,"Jumeken Najimedenov Street 17, Nur-Sultan 010000",-,8 (702) 406 2895-Hotel Karat Camp,"Tengiz Ln, Tengiz 060000",-,8 (701) 530 0418-Agat Hotel Astana,"020000, Nur-Sultan",agathotel.kz,8 (7172) 52 13 44-Inn Sofia,"Prospect Abyla Khan 4/4, Nur-Sultan 010000",-,--Sky Lux Hotel Astana,"Zhengis Ave 43, Nur-Sultan 010000",-,--Ырысты,"Zh. Aymauytov St, Nur-Sultan 010000",-,8 (775) 357 2428-"""Ақ сұнқар"" қонақ үйі","Abay Ave 29, Nur-Sultan 010000",-,8 (7172) 32 73 40-Europe Palas,"Abay Ave 29, Nur-Sultan 010000",-,--Merdan Hall,"Bauyrzhan Momyshuly Ave 31a, Nur-Sultan 010000",-,8 (775) 909 6818-Ulytau,"Iliyas Yesenberlin St 26, Nur-Sultan 010000",-,8 (7172) 38 58 40-Kapsula Hotel Astana,"Qabanbay Batyr Ave 7/3, Nur-Sultan 020000",kapsula.kz,8 (7172) 50 74 04-Royal Park Hotel & Spa,"Kerey and Zhanibek Khans St 18, Nur-Sultan 010000",-,8 (7172) 42 22 01-Vintage,"Bayimbet Maylin St 16/10, Nur-Sultan 010000",-,8 (7172) 43 65 17-Hostel Dostyk 10,"ЖК Алматы, Dostyq St 10, Nur-Sultan 010000",-,8 (775) 276 5400-Mini Hotel Alikhan,"Dostyq St 1, Nur-Sultan 020000",-,--Adam I Yeva,"Qayrat Rysqulbekov St 17, Nur-Sultan 010000",inn-248.business.site,8 (778) 066 2706-Khostel-Panda,"Imanbayeva St 8, Nur-Sultan 010000",hostel-panda.kz,8 (777) 730 1964-Efendi,"Qabanbay Batyr Ave 8, Nur-Sultan 020000",-,8 (7172) 24 06 73-Arlan,"Mugalzhar Street 80, Nur-Sultan 020000",-,8 (771) 222 0888-Hostel RADA,"Respublika Ave 52/2, Nur-Sultan 010000",-,8 (7172) 73 86 86-Economy Hotel Zhyger,"Respublika Ave 59, Nur-Sultan 010000",-,8 (7172) 39 58 79-Hotel Tourist,"Republic Prospect 33, Nur-Sultan 010000",-,8 (777) 870 5001-Comfort Apartments 2,"020000, Nur-Sultan",-,8 (701) 958 2655-Hostel Baiterek Astana,"Syghanaq St 23, Nur-Sultan 010000",-,--"Hotel ""Akku""","Turar Ryskulov St 22, Nur-Sultan 010000",-,--Apartments of Astana,"улица Сауран 5G, Nur-Sultan",-,8 (775) 909 0800-Zolotoy Fazan,"Рыскулова 35, Koyandy 020000",fazan.kz,8 (777) 077 7050-Uyutnaya,"Seifullin Street 52/2(180), Nur-Sultan 010000",-,8 (778) 535 4824-Izumrud,"Nurtas Ongdasynov Street 46, Nur-Sultan 020000",-,8 (7172) 24 17 95-The Barrel Hostel,"Imanbayeva St, Nur-Sultan 020000",-,8 (747) 510 6310-WELKOM HOSTEL,"Baurzhan Momyshuly Street 2В 9этаж, Nur-Sultan 010010",-,8 (708) 819 0727-Hotel Shah - Гостиница сауна Шах,"Кайрат Рыскулбеков 11, Qayrat Rysqulbekov St 11, Nur-Sultan 010000",hotelinastana.wixsite.com,8 (778) 139 0188--,-,-,--Samruk Hostel,"st. Bokeyhana. 35, Nur-Sultan 010000",-,8 (701) 700 0969-Akzhol,"Bayimbet Maylin St 16/12, Nur-Sultan 020000",-,8 (7172) 49 09 09-HomeStayTursyn Astana Apartments,"Koshkarbayev, 40, Nur-Sultan 020000",-,8 (705) 170 5555-Mommy,"38 Sarayshyq Street 7 подъезд, 2 этаж, квартира 108, Nur-Sultan 010000",-,8 (708) 972 6234-hotel Summer garden,Nur-Sultan 020000,-,8 (7172) 23 15 27-Inn Dostar,"Qayrat Rysqulbekov St 16, Nur-Sultan 020000",-,8 (771) 833 3778-Akku,"Бухарбай батыра 22, Nur-Sultan 010000",-,8 (7172) 75 27 00-Saltanat,"Ulitsa Seyfullin 410, Nur-Sultan 050050",-,--Zeyd,"Agybay Batyr St 99, Nur-Sultan 020000",zeid.kz,8 (7172) 21 28 05-Hostel Mayak on Lepsy 46,"Астана, Казахстан Лепсы 46 ВП-1, Nur-Sultan 100000",hostelmayak.simplesite.com,8 (777) 001 9845-Hostel on Sauran,"Sauran Street 9, Nur-Sultan 010000",-,8 (701) 997 7129-Belon-Land Hotel,"Bogenbay Batyr Ave 45, Nur-Sultan 010000",belon.kz,8 (7172) 32 83 08-Dastan Hotel,"38, Shakarim Kudayberdiuly Avenue Здание, Nur-Sultan 010000",-,8 (701) 214 0075-Complex o'Rosso,"проезд Абылайхана ВИП городок, 19, Nur-Sultan 010000",-,8 (701) 879 2361-Dastan,"Qayrat Rysqulbekov St 16 корпус, Nur-Sultan 020000",-,--AliStar,"Zhk Meruyert, Syghanaq St 21/1, Nur-Sultan 020000",krisha.kz,8 (701) 999 6864-Topaz,"Zheltoqsan St 6, Nur-Sultan 010000",-,8 (708) 667 5063-Hostel North Astana,"Dostyq St 5, Nur-Sultan 010000",-,8 (702) 548 8798-Capital Apartments Astana,"Bayanauyl St 1, Nur-Sultan 020000",-,8 (7172) 96 82 89-Hostel A99,"99, Aynaköl St, Nur-Sultan 010000",-,--Residenz,"Petrov St 21/1, Nur-Sultan 010000",-,8 (7172) 34 22 49-"Economy Hotel ""Congress Hall""","Kenesary St 30, Nur-Sultan 010000",-,--Imanova1,"1, Иманова",-,--Hostel Khan Tengri,"Prospect Saryarka 18 9 etzh 76 ap, Nur-Sultan 010000",-,8 (775) 204 6055-Zhety Kazyna,"Bogenbay Batyr Ave 18, Nur-Sultan 010000",astana.restoran.kz,8 (7172) 23 07 72-K40,"Qabanbay Batyr Ave 40, Nur-Sultan 020000",-,8 (701) 771 9999-AG.KZ,"Asankayghy St 41, Nur-Sultan 010000",-,8 (702) 323 8916-Hostel on Dostyk 2,"Dostyq St 2, Nur-Sultan 010016",-,--Hostel on Seifullina 4/2,"Astana Seifullina 4/2, Nur-Sultan 020000",-,8 (775) 209 5400-"Khostel As, Zhk Infiniti","Qabanbay Batyr Ave 40, Nur-Sultan 020000",-,8 (776) 917 0404-Shama,"Mustafin St 12, Nur-Sultan 010000",-,8 (701) 257 4888-Hotel Uyut,"Ulitsa 187 16, Nur-Sultan 010000",-,8 (7172) 47 30 32-Residenz Hotel,"Zhumabek Tashenov Street 25, Nur-Sultan 020000",-,--ХОСТЕЛ В ЦЕНТРЕ АСТАНЫ. АКЦИЯ!,"Zheltoqsan St 48/1, Nur-Sultan 010000",-,8 (700) 199 1007-Hostel Edem Rooms,"Keensary 52, Nur-Sultan 010000",-,8 (771) 888 2223-Espresso Bar Expo Boulevard,"Qabanbay Batyr Ave 58Б, Nur-Sultan 020000",instagram.com,8 (778) 454 1605-"Gostinitsa ""Berlin """,Nur-Sultan 020000,-,--Female Hostel,"Syghanaq St 18, Nur-Sultan 020000",-,8 (708) 414 4292-Orion,"Alexander Pushkin St 3/2, Nur-Sultan 010000",orionhotel.kz,8 (701) 223 3636-Park Inn by Radisson Astana Hotel,"Saryarka Avenue 8, Nur-Sultan 010000",radissonhotels.com,8 (7172) 67 00 00-Сауна Charleston,"Zhayyq St 30, Nur-Sultan 010000",charleston.kz,8 (701) 888 1208-"Сауна Ардана, VIP Сауна в Астане (Нур-Султан)","Tulkibas St 49, Nur-Sultan 010000",-,8 (778) 260 7273diff --git a/google-maps/pylg.log b/google-maps/pylg.logdeleted file mode 100644index a303cf5..0000000--- a/google-maps/pylg.log+++ /dev/null@@ -1,259 +0,0 @@-2020-12-26 09:03:01,994 - WARNING | webdriver.py/_do_retry/67 | Cannot locate xpath retry #1-2020-12-26 09:04:41,661 - WARNING | webdriver.py/_do_retry/67 | Cannot locate xpath retry #1-2020-12-26 09:06:27,528 - WARNING | webdriver.py/_do_retry/67 | Cannot locate xpath retry #1-2020-12-26 09:08:11,889 - WARNING | webdriver.py/_do_retry/67 | Cannot locate xpath retry #1-2020-12-26 09:09:56,557 - WARNING | webdriver.py/_do_retry/67 | Cannot locate xpath retry #1-2020-12-26 09:11:08,657 - WARNING | webdriver.py/_do_retry/67 | Cannot locate xpath retry #1-2020-12-26 09:39:24,550 - ERROR | webdriver.py/_do_retry/70 | Exception found when retrying-Traceback (most recent call last):- File "C:\Users\khova\AppData\Local\Programs\Python\Python38-32\lib\asyncio\runners.py", line 43, in run- return loop.run_until_complete(main)- File "C:\Users\khova\AppData\Local\Programs\Python\Python38-32\lib\asyncio\base_events.py", line 603, in run_until_complete- self.run_forever()- File "C:\Users\khova\AppData\Local\Programs\Python\Python38-32\lib\asyncio\windows_events.py", line 316, in run_forever- super().run_forever()- File "C:\Users\khova\AppData\Local\Programs\Python\Python38-32\lib\asyncio\base_events.py", line 570, in run_forever- self._run_once()- File "C:\Users\khova\AppData\Local\Programs\Python\Python38-32\lib\asyncio\base_events.py", line 1823, in _run_once- event_list = self._selector.select(timeout)- File "C:\Users\khova\AppData\Local\Programs\Python\Python38-32\lib\asyncio\windows_events.py", line 430, in select- self._poll(timeout)- File "C:\Users\khova\AppData\Local\Programs\Python\Python38-32\lib\asyncio\windows_events.py", line 779, in _poll- status = _overlapped.GetQueuedCompletionStatus(self._iocp, ms)- File "C:\Users\khova\AppData\Local\Programs\Python\Python38-32\lib\site-packages\pyppeteer\launcher.py", line 151, in _close_process- self._loop.run_until_complete(self.killChrome())- File "C:\Users\khova\AppData\Local\Programs\Python\Python38-32\lib\asyncio\base_events.py", line 592, in run_until_complete- self._check_running()- File "C:\Users\khova\AppData\Local\Programs\Python\Python38-32\lib\asyncio\base_events.py", line 552, in _check_running- raise RuntimeError('This event loop is already running')-RuntimeError: This event loop is already running--During handling of the above exception, another exception occurred:--Traceback (most recent call last):- File "..\webdriver.py", line 63, in _do_retry- await operation()- File "C:\Users\khova\AppData\Local\Programs\Python\Python38-32\lib\site-packages\pyppeteer\element_handle.py", line 177, in click- await self._scrollIntoViewIfNeeded()- File "C:\Users\khova\AppData\Local\Programs\Python\Python38-32\lib\site-packages\pyppeteer\element_handle.py", line 68, in _scrollIntoViewIfNeeded- error = await self.executionContext.evaluate('''- File "C:\Users\khova\AppData\Local\Programs\Python\Python38-32\lib\site-packages\pyppeteer\execution_context.py", line 53, in evaluate- handle = await self.evaluateHandle(- File "C:\Users\khova\AppData\Local\Programs\Python\Python38-32\lib\site-packages\pyppeteer\execution_context.py", line 99, in evaluateHandle- _obj = await self._client.send('Runtime.callFunctionOn', {-asyncio.exceptions.CancelledError-2021-02-26 17:15:57,000 - WARNING | extractor.py/_scrape/120 | Error occured-Traceback (most recent call last):- File "C:\Users\khova\AppData\Local\Programs\Python\Python38-32\lib\asyncio\runners.py", line 43, in run- return loop.run_until_complete(main)- File "C:\Users\khova\AppData\Local\Programs\Python\Python38-32\lib\asyncio\base_events.py", line 603, in run_until_complete- self.run_forever()- File "C:\Users\khova\AppData\Local\Programs\Python\Python38-32\lib\asyncio\windows_events.py", line 316, in run_forever- super().run_forever()- File "C:\Users\khova\AppData\Local\Programs\Python\Python38-32\lib\asyncio\base_events.py", line 570, in run_forever- self._run_once()- File "C:\Users\khova\AppData\Local\Programs\Python\Python38-32\lib\asyncio\base_events.py", line 1823, in _run_once- event_list = self._selector.select(timeout)- File "C:\Users\khova\AppData\Local\Programs\Python\Python38-32\lib\asyncio\windows_events.py", line 430, in select- self._poll(timeout)- File "C:\Users\khova\AppData\Local\Programs\Python\Python38-32\lib\asyncio\windows_events.py", line 779, in _poll- status = _overlapped.GetQueuedCompletionStatus(self._iocp, ms)- File "C:\Users\khova\AppData\Local\Programs\Python\Python38-32\lib\site-packages\pyppeteer\launcher.py", line 151, in _close_process- self._loop.run_until_complete(self.killChrome())- File "C:\Users\khova\AppData\Local\Programs\Python\Python38-32\lib\asyncio\base_events.py", line 592, in run_until_complete- self._check_running()- File "C:\Users\khova\AppData\Local\Programs\Python\Python38-32\lib\asyncio\base_events.py", line 552, in _check_running- raise RuntimeError('This event loop is already running')-RuntimeError: This event loop is already running--During handling of the above exception, another exception occurred:--Traceback (most recent call last):- File "extractor.py", line 114, in _scrape- data = extract(await self._page.content())- File "C:\Users\khova\AppData\Local\Programs\Python\Python38-32\lib\site-packages\pyppeteer\page.py", line 803, in content- return await frame.content()- File "C:\Users\khova\AppData\Local\Programs\Python\Python38-32\lib\site-packages\pyppeteer\frame_manager.py", line 384, in content- return await self.evaluate('''- File "C:\Users\khova\AppData\Local\Programs\Python\Python38-32\lib\site-packages\pyppeteer\frame_manager.py", line 308, in evaluate- return await context.evaluate(- File "C:\Users\khova\AppData\Local\Programs\Python\Python38-32\lib\site-packages\pyppeteer\execution_context.py", line 53, in evaluate- handle = await self.evaluateHandle(- File "C:\Users\khova\AppData\Local\Programs\Python\Python38-32\lib\site-packages\pyppeteer\execution_context.py", line 99, in evaluateHandle- _obj = await self._client.send('Runtime.callFunctionOn', {-asyncio.exceptions.CancelledError-2021-02-26 17:18:00,115 - WARNING | webdriver.py/_do_retry/67 | Cannot locate xpath retry #1-2021-02-26 17:19:42,183 - WARNING | webdriver.py/_do_retry/67 | Cannot locate xpath retry #1-2021-02-26 17:21:25,899 - WARNING | webdriver.py/_do_retry/67 | Cannot locate xpath retry #1-2021-02-26 17:23:08,410 - WARNING | webdriver.py/_do_retry/67 | Cannot locate xpath retry #1-2021-02-26 17:24:53,350 - WARNING | webdriver.py/_do_retry/67 | Cannot locate xpath retry #1-2021-02-26 17:26:40,303 - WARNING | webdriver.py/_do_retry/67 | Cannot locate xpath retry #1-2021-02-26 17:28:27,005 - WARNING | webdriver.py/_do_retry/67 | Cannot locate xpath retry #1-2021-02-26 17:30:13,206 - WARNING | webdriver.py/_do_retry/67 | Cannot locate xpath retry #1-2021-02-26 17:31:58,209 - WARNING | webdriver.py/_do_retry/67 | Cannot locate xpath retry #1-2021-02-26 17:33:46,760 - WARNING | webdriver.py/_do_retry/67 | Cannot locate xpath retry #1-2021-02-26 17:35:35,841 - WARNING | webdriver.py/_do_retry/67 | Cannot locate xpath retry #1-2021-02-26 17:35:35,845 - ERROR | webdriver.py/_do_retry/70 | Exception found when retrying-Traceback (most recent call last):- File "..\webdriver.py", line 65, in _do_retry- await self._page.waitForXPath(xpath, {'visible': True})- File "C:\Users\khova\AppData\Local\Programs\Python\Python38-32\lib\site-packages\pyppeteer\frame_manager.py", line 855, in __await__- raise result-pyppeteer.errors.TimeoutError: Waiting for XPath "//*[@class="section-hero-header-title-title GLOBAL__gm2-headline-5"]" failed: timeout 30000ms exceeds.--During handling of the above exception, another exception occurred:--Traceback (most recent call last):- File "..\webdriver.py", line 63, in _do_retry- await operation()- File "C:\Users\khova\AppData\Local\Programs\Python\Python38-32\lib\site-packages\pyppeteer\element_handle.py", line 177, in click- await self._scrollIntoViewIfNeeded()- File "C:\Users\khova\AppData\Local\Programs\Python\Python38-32\lib\site-packages\pyppeteer\element_handle.py", line 99, in _scrollIntoViewIfNeeded- raise ElementHandleError(error)-pyppeteer.errors.ElementHandleError: Node is detached from document-2021-02-26 17:35:36,220 - WARNING | extractor.py/_scrape/120 | Error occured-Traceback (most recent call last):- File "extractor.py", line 117, in _scrape- back = (await self._page.xpath(back_xpath))[0]-IndexError: list index out of range-2021-02-26 17:36:45,623 - WARNING | extractor.py/_scrape/120 | Error occured-Traceback (most recent call last):- File "extractor.py", line 111, in _scrape- place = (await self._page.xpath(result_xpath))[i]-IndexError: list index out of range-2021-02-26 17:36:45,632 - WARNING | extractor.py/_scrape/120 | Error occured-Traceback (most recent call last):- File "extractor.py", line 111, in _scrape- place = (await self._page.xpath(result_xpath))[i]-IndexError: list index out of range-2021-02-26 17:37:17,806 - WARNING | webdriver.py/_do_retry/67 | Cannot locate xpath retry #1-2021-02-26 17:38:59,952 - WARNING | webdriver.py/_do_retry/67 | Cannot locate xpath retry #1-2021-02-26 17:39:16,978 - WARNING | extractor.py/_scrape/120 | Error occured-Traceback (most recent call last):- File "extractor.py", line 111, in _scrape- place = (await self._page.xpath(result_xpath))[i]-IndexError: list index out of range-2021-02-26 17:39:16,983 - WARNING | extractor.py/_scrape/120 | Error occured-Traceback (most recent call last):- File "extractor.py", line 111, in _scrape- place = (await self._page.xpath(result_xpath))[i]-IndexError: list index out of range-2021-02-26 17:39:16,989 - WARNING | extractor.py/_scrape/120 | Error occured-Traceback (most recent call last):- File "extractor.py", line 111, in _scrape- place = (await self._page.xpath(result_xpath))[i]-IndexError: list index out of range-2021-02-26 17:39:16,995 - WARNING | extractor.py/_scrape/120 | Error occured-Traceback (most recent call last):- File "extractor.py", line 111, in _scrape- place = (await self._page.xpath(result_xpath))[i]-IndexError: list index out of range-2021-02-26 17:39:17,000 - WARNING | extractor.py/_scrape/120 | Error occured-Traceback (most recent call last):- File "extractor.py", line 111, in _scrape- place = (await self._page.xpath(result_xpath))[i]-IndexError: list index out of range-2021-02-26 17:39:17,007 - WARNING | extractor.py/_scrape/120 | Error occured-Traceback (most recent call last):- File "extractor.py", line 111, in _scrape- place = (await self._page.xpath(result_xpath))[i]-IndexError: list index out of range-2021-02-26 17:39:17,014 - WARNING | extractor.py/_scrape/120 | Error occured-Traceback (most recent call last):- File "extractor.py", line 111, in _scrape- place = (await self._page.xpath(result_xpath))[i]-IndexError: list index out of range-2021-02-26 17:39:17,020 - WARNING | extractor.py/_scrape/120 | Error occured-Traceback (most recent call last):- File "extractor.py", line 111, in _scrape- place = (await self._page.xpath(result_xpath))[i]-IndexError: list index out of range-2021-02-26 17:39:17,025 - WARNING | extractor.py/_scrape/120 | Error occured-Traceback (most recent call last):- File "extractor.py", line 111, in _scrape- place = (await self._page.xpath(result_xpath))[i]-IndexError: list index out of range-2021-02-26 17:39:17,031 - WARNING | extractor.py/_scrape/120 | Error occured-Traceback (most recent call last):- File "extractor.py", line 111, in _scrape- place = (await self._page.xpath(result_xpath))[i]-IndexError: list index out of range-2021-02-26 17:39:17,036 - WARNING | extractor.py/_scrape/120 | Error occured-Traceback (most recent call last):- File "extractor.py", line 111, in _scrape- place = (await self._page.xpath(result_xpath))[i]-IndexError: list index out of range-2021-02-26 17:39:17,041 - WARNING | extractor.py/_scrape/120 | Error occured-Traceback (most recent call last):- File "extractor.py", line 111, in _scrape- place = (await self._page.xpath(result_xpath))[i]-IndexError: list index out of range-2021-02-26 17:39:17,047 - WARNING | extractor.py/_scrape/120 | Error occured-Traceback (most recent call last):- File "extractor.py", line 111, in _scrape- place = (await self._page.xpath(result_xpath))[i]-IndexError: list index out of range-2021-02-26 17:39:17,052 - WARNING | extractor.py/_scrape/120 | Error occured-Traceback (most recent call last):- File "extractor.py", line 111, in _scrape- place = (await self._page.xpath(result_xpath))[i]-IndexError: list index out of range-2021-02-26 17:39:17,056 - WARNING | extractor.py/_scrape/120 | Error occured-Traceback (most recent call last):- File "extractor.py", line 111, in _scrape- place = (await self._page.xpath(result_xpath))[i]-IndexError: list index out of range-2021-02-26 17:39:17,062 - WARNING | extractor.py/_scrape/120 | Error occured-Traceback (most recent call last):- File "extractor.py", line 111, in _scrape- place = (await self._page.xpath(result_xpath))[i]-IndexError: list index out of range-2021-02-26 17:39:17,067 - WARNING | extractor.py/_scrape/120 | Error occured-Traceback (most recent call last):- File "extractor.py", line 111, in _scrape- place = (await self._page.xpath(result_xpath))[i]-IndexError: list index out of range-2021-02-26 17:39:49,279 - WARNING | webdriver.py/_do_retry/67 | Cannot locate xpath retry #1-2021-02-26 17:39:51,929 - ERROR | webdriver.py/_do_retry/70 | Exception found when retrying-Traceback (most recent call last):- File "..\webdriver.py", line 65, in _do_retry- await self._page.waitForXPath(xpath, {'visible': True})- File "C:\Users\khova\AppData\Local\Programs\Python\Python38-32\lib\site-packages\pyppeteer\frame_manager.py", line 855, in __await__- raise result-pyppeteer.errors.TimeoutError: Waiting for XPath "//*[@class="section-hero-header-title-title GLOBAL__gm2-headline-5"]" failed: timeout 30000ms exceeds.--During handling of the above exception, another exception occurred:--Traceback (most recent call last):- File "..\webdriver.py", line 63, in _do_retry- await operation()- File "C:\Users\khova\AppData\Local\Programs\Python\Python38-32\lib\site-packages\pyppeteer\element_handle.py", line 177, in click- await self._scrollIntoViewIfNeeded()- File "C:\Users\khova\AppData\Local\Programs\Python\Python38-32\lib\site-packages\pyppeteer\element_handle.py", line 99, in _scrollIntoViewIfNeeded- raise ElementHandleError(error)-pyppeteer.errors.ElementHandleError: Node is detached from document-2021-02-26 17:39:52,369 - WARNING | extractor.py/_scrape/120 | Error occured-Traceback (most recent call last):- File "extractor.py", line 117, in _scrape- back = (await self._page.xpath(back_xpath))[0]-IndexError: list index out of range-2021-02-26 17:39:52,382 - WARNING | extractor.py/_scrape/120 | Error occured-Traceback (most recent call last):- File "extractor.py", line 111, in _scrape- place = (await self._page.xpath(result_xpath))[i]-IndexError: list index out of range-2021-02-26 17:39:56,928 - WARNING | extractor.py/_scrape/120 | Error occured-Traceback (most recent call last):- File "extractor.py", line 111, in _scrape- place = (await self._page.xpath(result_xpath))[i]-IndexError: list index out of range-2021-02-26 17:39:56,987 - WARNING | extractor.py/_scrape/120 | Error occured-Traceback (most recent call last):- File "extractor.py", line 111, in _scrape- place = (await self._page.xpath(result_xpath))[i]-IndexError: list index out of range-2021-02-26 17:39:57,051 - WARNING | extractor.py/_scrape/120 | Error occured-Traceback (most recent call last):- File "extractor.py", line 111, in _scrape- place = (await self._page.xpath(result_xpath))[i]-IndexError: list index out of rangediff --git a/linkedin/__pycache__/locators.cpython-38.pyc b/linkedin/__pycache__/locators.cpython-38.pycdeleted file mode 100644index 602624d..0000000Binary files a/linkedin/__pycache__/locators.cpython-38.pyc and /dev/null differdiff --git a/linkedin/__pycache__/my_config.cpython-38.pyc b/linkedin/__pycache__/my_config.cpython-38.pycdeleted file mode 100644index 545b78b..0000000Binary files a/linkedin/__pycache__/my_config.cpython-38.pyc and /dev/null differdiff --git a/linkedin/leads.csv b/linkedin/leads.csvdeleted file mode 100644index 58ce79a..0000000--- a/linkedin/leads.csv+++ /dev/null@@ -1 +0,0 @@-Title,Address,WebSite,PhoneNumberdiff --git a/py_lead_generation/__init__.py b/py_lead_generation/__init__.pynew file mode 100644index 0000000..93d0df3--- /dev/null+++ b/py_lead_generation/__init__.py@@ -0,0 +1,2 @@+from py_lead_generation.src.yelp.engine import YelpEngine+from py_lead_generation.src.google_maps.engine import GoogleMapsEnginediff --git a/py_lead_generation/requirements.txt b/py_lead_generation/requirements.txtnew file mode 100644index 0000000..2788388--- /dev/null+++ b/py_lead_generation/requirements.txt@@ -0,0 +1,22 @@+beautifulsoup4==4.12.2+certifi==2023.11.17+charset-normalizer==3.3.2+colorama==0.4.6+geographiclib==2.0+geopy==2.4.1+greenlet==3.0.1+idna==3.6+iniconfig==2.0.0+packaging==23.2+playwright==1.40.0+pluggy==1.3.0+pyee==11.0.1+pytest==7.4.3+pytest-base-url==2.0.0+pytest-playwright==0.4.3+python-slugify==8.0.1+setuptools==69.0.3+soupsieve==2.5+text-unidecode==1.3+typing_extensions==4.9.0+urllib3==2.1.0\ No newline at end of filediff --git a/linkedin/pylg.log b/py_lead_generation/src/__init__.pysimilarity index 100%rename from linkedin/pylg.logrename to py_lead_generation/src/__init__.pydiff --git a/py_lead_generation/src/engines/__init__.py b/py_lead_generation/src/engines/__init__.pynew file mode 100644index 0000000..e69de29diff --git a/py_lead_generation/src/engines/abstract.py b/py_lead_generation/src/engines/abstract.pynew file mode 100644index 0000000..7c67cb9--- /dev/null+++ b/py_lead_generation/src/engines/abstract.py@@ -0,0 +1,35 @@+from abc import ABC+++class AbstractEngine(ABC):+ '''+ `AbstractEngine`++ Abstract class specifying main methods and attributes++ [CONSTANT] `BASE_URL` - base url for google maps website with substitute search query, coordinates and zoom++ [CONSTANT] `FIELD_NAMES` - field names of scraped data, extracts data entries to csv file based on these field names++ [EDITABLE] `FILENAME` - file name for exporting collected leads, must have .csv extension+ '''++ BASE_URL = ''+ FIELD_NAMES = []+ FILENAME = 'leads.csv'++ async def _get_search_results_urls(self, *args, **kwargs) -> list[str]:+ '''+ Retreiving search results URLs from the website (Yelp/Google Maps) method+ '''+ ...++ def _parse_data_with_soup(self, html: str) -> list[dict]:+ '''+ `html: str` - html representation of the page to parse++ Should be defined in child class++ Returns `list[dict]` typed parsed data - `[title, addr, phone, website]`+ '''+ ...diff --git a/py_lead_generation/src/engines/base.py b/py_lead_generation/src/engines/base.pynew file mode 100644index 0000000..6cdfff4--- /dev/null+++ b/py_lead_generation/src/engines/base.py@@ -0,0 +1,103 @@+import asyncio+from playwright.async_api import Playwright, async_playwright++from py_lead_generation.src.misc.writer import CsvWriter+from py_lead_generation.src.engines.playwright_config import PlaywrightEngineConfig+++class BaseEngine(PlaywrightEngineConfig):+ '''+ `BaseEngine`++ Base engine class expressing methods, which are shared between all engines, does not provide implementation of `AbstractEngine` methods+ '''++ async def run(self) -> None:+ '''+ Uses headles webdriver powered by Playwright++ Creates a web url based on initialized parameters++ Parses results by given query and url one by one++ Assigns collected results to `.entries`++ To save the results call `.save_to_csv()` method after+ '''+ async with async_playwright() as playwright:+ self.playwright: Playwright = playwright+ await self._setup_browser()+ await self._open_url_and_wait(self.url)+ urls: list[str] = await self._get_search_results_urls()+ self._entries: list[dict] = await self._get_search_results_entries(urls)+ await self.browser.close()++ def save_to_csv(self, filename: str = None) -> None:+ '''+ `filename: str = None` - optional parameter, by default uses `self.FILENAME`++ If file with such name already exists, it will not overwrite it but append newly found entries to existing ones++ If file with such name does not exist, it will create a new csv file with predetermined fieldnames+ '''+ if filename:+ self.FILENAME = filename++ if not self.FILENAME.endswith('.csv'):+ raise ValueError('Use .csv file extension')+ if not self._entries:+ raise NotImplementedError(+ 'Entries are empty, call .run() method first to save them'+ )+ csv_writer = CsvWriter(self.FILENAME, self.FIELD_NAMES)+ csv_writer.append(self._entries)++ @property+ def entries(self) -> list[dict]:+ '''+ Returns `list[dict]` typed entries once `.run()` method was called+ '''+ if not self._entries:+ raise NotImplementedError(+ 'Entries are empty, call .run() method first to create them'+ )+ return self._entries++ @entries.setter+ def entries(self, _) -> None:+ '''+ You cannot do that ;)++ Made for durability reasons+ '''+ raise ValueError('Cannot set value to data. This is not allowed')++ async def _open_url_and_wait(self, url: str, sleep_duration_s: int = 3) -> None:+ '''+ `url: str` - url to open+ `sleep_duration_s: int = 3` - amount of seconds to wait for the page to load before any operations on that page++ Navigates to initialized `self.url` and waits for `sleep_duration_s` seconds+ '''+ await self.page.goto(url)+ await asyncio.sleep(sleep_duration_s)++ async def _get_search_results_entries(self, urls: list[str]) -> list[dict]:+ '''+ `urls: list[str]` - list of urls for google maps entities to scrape for++ Goes over each url in the list and retreives data from opening page one by one++ Searches for data using exact image source values selectors, no other unique way to extract data using selectors is not found yet++ Returns `list[dict]` typed search result entries for each seperate given url from `urls`+ '''+ entries = []+ for url in urls:+ await self._open_url_and_wait(url, 1.5)+ html = await self.page.content()+ data = self._parse_data_with_soup(html)+ entry = dict(zip(self.FIELD_NAMES, data))+ entries.append(entry)++ return entriesdiff --git a/py_lead_generation/src/engines/playwright_config.py b/py_lead_generation/src/engines/playwright_config.pynew file mode 100644index 0000000..4db09f7--- /dev/null+++ b/py_lead_generation/src/engines/playwright_config.py@@ -0,0 +1,24 @@+from playwright.async_api import Browser, Page, BrowserType+++class PlaywrightEngineConfig:+ '''+ `PlaywrightEngineConfig`++ Playwright setup configuration parameters++ [EDITABLE] `BROWSER_PARAMS` - chromiun browser parameters configuration from playwright, additional parameters can be set if needed++ [EDITABLE] `PAGE_PARAMS` - browser page parameters configuration from playwright, additional parameters can be set if needed+ '''++ BROWSER_PARAMS = {'headless': False, 'proxy': None, 'slow_mo': 150}+ PAGE_PARAMS = {'java_script_enabled': True, 'bypass_csp': True}++ async def _setup_browser(self) -> None:+ '''+ Sets up the browser by initializing `playwright.chromiun` and `Browser` along with `Page` after+ '''+ chromium: BrowserType = self.playwright.chromium+ self.browser: Browser = await chromium.launch(**self.BROWSER_PARAMS)+ self.page: Page = await self.browser.new_page(**self.PAGE_PARAMS)diff --git a/py_lead_generation/src/google_maps/__init__.py b/py_lead_generation/src/google_maps/__init__.pynew file mode 100644index 0000000..e69de29diff --git a/py_lead_generation/src/google_maps/engine.py b/py_lead_generation/src/google_maps/engine.pynew file mode 100644index 0000000..5c20577--- /dev/null+++ b/py_lead_generation/src/google_maps/engine.py@@ -0,0 +1,137 @@+import time+import asyncio+from bs4 import BeautifulSoup++from py_lead_generation.src.engines.base import BaseEngine+from py_lead_generation.src.engines.abstract import AbstractEngine+from py_lead_generation.src.misc.utils import get_coords_by_location+++class GoogleMapsEngine(BaseEngine, AbstractEngine):+ '''+ `GoogleMapsEngine`++ [EDITABLE] `SCROLL_TIME_DURATION_S` - scroll time duration to view the search results, preferably should be not less than 150, for testing purposes can be decreased++ [EDITABLE] `SLEEP_PER_SCROLL_S` - amount of seconds to wait before each scroll of search results so that google maps does not output endless loading ~ aka simulate human-like activity, preferable should be not less than 5++ Usage:++ `engine = GoogleMapsEngine(*args, **kwargs)`++ `await engine.run()`++ `await engine.save_to_csv()`++ `print(engine.entries)`+ '''++ BASE_URL = 'https://www.google.com/maps/search/{query}/@{coords},{zoom}z/data=!3m1!4b1?entry=ttu'+ FIELD_NAMES = ['Title', 'Address', 'PhoneNumber', 'WebsiteURL']+ FILENAME = 'google_maps_leads.csv'++ SLEEP_PER_SCROLL_S = 5+ SCROLL_TIME_DURATION_S = 200++ def __init__(self, query: str, location: str, zoom: int | float = 12) -> None:+ '''+ `query: str` - what are you looking for? e.g., `gym`+ `location: str` - where are you looking for that query? e.g., `Astana`+ `zoom: int | float` - google maps zoom e.g., `8.75`++ Creates `GoogleMapsEngine` instance+ '''+ self._entries = []+ self.zoom = zoom+ self.query = query+ self.location = location+ self.coords = get_coords_by_location(self.location)+ self.search_query = f'{self.query}%20{self.location}'+ self.url = self.BASE_URL.format(+ query=self.search_query, coords=','.join(self.coords), zoom=self.zoom+ )++ async def _get_search_results_urls(self) -> list[str]:+ '''+ Goes through the search results for `GoogleMapsEngine.SCROLL_TIME_DURATION_S` seconds++ Waits for `GoogleMapsEngine.SLEEP_PER_SCROLL_S` seconds so that GoogleMaps will not output endless load, aka simulate human-like activity++ Or scrapes the results urls, once the are no more results+ '''+ async def hover_search_results() -> None:+ '''+ Hovers on leftbar, where search results are located++ Needed so that scroll function is used properly - not on the map but on the search results div+ '''+ leftbar = await self.page.query_selector('[role="main"]')+ await leftbar.hover()+ await asyncio.sleep(0.5)++ async def scroll_and_sleep(delta_y: int = 1000) -> None:+ '''+ `delta_y: int = 1000` pixel units to scroll down along y-axis++ Scrolls down by `delta_y` and waits for `GoogleMapsEngine.SCROLL_TIME_DURATION_S` seconds+ '''+ await self.page.mouse.wheel(0, delta_y)+ await asyncio.sleep(self.SLEEP_PER_SCROLL_S)++ async def end_locator_is_present() -> bool:+ '''+ Retunrs `end_locator` as boolean value, which correlates with the possible end of the search results++ Once `end_locator` is Truthy, aka ElemenetHandle, it means that there is nothing more to scroll+ '''+ end_locator = await self.page.query_selector('.m6QErb.tLjsW.eKbjU')+ return bool(end_locator)++ async def scrape_urls() -> list[str]:+ '''+ Should be called once page is being scrolled all the way down (`end_locator` found) or `GoogleMapsEngine.SCROLL_TIME_DURATION_S` duration limit is exceeded++ Returns `list[str]` typed scraped urls list using query_selector+ '''+ urls = []+ link_elements = await self.page.query_selector_all('a.hfpxzc')+ for link_element in link_elements:+ url = await link_element.get_attribute('href')+ urls.append(url)+ return urls++ await hover_search_results()+ start_scroll_time = time.time()++ while True:+ await scroll_and_sleep()+ finish_scroll_time = time.time()+ if (await end_locator_is_present()) or (finish_scroll_time - start_scroll_time > self.SCROLL_TIME_DURATION_S):+ break++ urls = await scrape_urls()+ return urls++ def _parse_data_with_soup(self, html: str) -> list[str]:+ '''+ `html: str` - html representation of the page to parse++ Returns `list[str]` typed parsed data - `[title, addr, phone, website]`+ '''+ soup = BeautifulSoup(html, 'html.parser')+ data = []++ title = soup.select_one('.DUwDvf.lfPIob').get_text()+ data.append(title)++ addr_img_src = '//www.gstatic.com/images/icons/material/system_gm/2x/place_gm_blue_24dp.png'+ phone_img_src = '//www.gstatic.com/images/icons/material/system_gm/2x/phone_gm_blue_24dp.png'+ website_img_src = '//www.gstatic.com/images/icons/material/system_gm/2x/public_gm_blue_24dp.png'++ for img_src in (addr_img_src, phone_img_src, website_img_src):+ img_element = soup.select_one(f'img[src="{img_src}"]')+ info = img_element.parent.parent.parent.select_one('.Io6YTe').get_text() \+ if img_element else '-'+ data.append(info)++ return datadiff --git a/py_lead_generation/src/google_search/README.md b/py_lead_generation/src/google_search/README.mdnew file mode 100644index 0000000..9b25634--- /dev/null+++ b/py_lead_generation/src/google_search/README.md@@ -0,0 +1,11 @@+empty for now+but in the foreseeable future+i will finish it+using playwright and regex+to scrape google search results urls+locate leads information on them+and export it to csv+most importantly here+is the regex+which is complex+bb for nowdiff --git a/py_lead_generation/src/google_search/__init__.py b/py_lead_generation/src/google_search/__init__.pynew file mode 100644index 0000000..e69de29diff --git a/py_lead_generation/src/misc/__init__.py b/py_lead_generation/src/misc/__init__.pynew file mode 100644index 0000000..e69de29diff --git a/py_lead_generation/src/misc/utils.py b/py_lead_generation/src/misc/utils.pynew file mode 100644index 0000000..5ac83f3--- /dev/null+++ b/py_lead_generation/src/misc/utils.py@@ -0,0 +1,16 @@+from geopy.geocoders import Nominatim+++geolocator = Nominatim(user_agent='google-leads')+++def get_coords_by_location(location: str) -> tuple[str]:+ '''+ `location: str` - specific location or city name++ Returns a tuple of the coordinates, two strings represented as float numbers - (latitude, longitude)+ '''+ loc = geolocator.geocode(location)+ coords = (loc.latitude, loc.longitude)+ coords = list(map(str, coords))+ return coordsdiff --git a/py_lead_generation/src/misc/writer.py b/py_lead_generation/src/misc/writer.pynew file mode 100644index 0000000..580d557--- /dev/null+++ b/py_lead_generation/src/misc/writer.py@@ -0,0 +1,41 @@+import csv+import os.path+++class CsvWriter:+ '''+ CSW Writer class to create and append data to a csv file with fieldnames+ '''++ def __init__(self, filename: str, fieldnames: list[str]) -> None:+ '''+ `filename: str` - file name or file path to create, e.g., `'leads.csv'`+ `filename: list[str]` - field names/headings, should match data dictionary keys, e.g., `['id', 'name', 'phone_number']`++ Creates a new file or skip if file with such name already exists+ '''++ self.filename = filename+ self.fieldnames = fieldnames++ if not os.path.exists(filename):+ self._init()++ def _init(self) -> None:+ '''+ Initializes csv file by writing a new one with fieldnames+ '''+ with open(self.filename, 'w', newline='') as f:+ writer = csv.DictWriter(f, fieldnames=self.fieldnames)+ writer.writeheader()++ def append(self, data: list[dict]) -> None:+ '''+ `data: list[dict]` - data to write, keys of this dictionary should match fieldnames++ Appends incoming `data` to the csv file+ '''+ with open(self.filename, 'a', encoding='utf-8', newline='') as f:+ writer = csv.DictWriter(f, fieldnames=self.fieldnames)+ for d in data:+ writer.writerow(d)diff --git a/py_lead_generation/src/yelp/__init__.py b/py_lead_generation/src/yelp/__init__.pynew file mode 100644index 0000000..e69de29diff --git a/py_lead_generation/src/yelp/engine.py b/py_lead_generation/src/yelp/engine.pynew file mode 100644index 0000000..d65e1db--- /dev/null+++ b/py_lead_generation/src/yelp/engine.py@@ -0,0 +1,90 @@+import time+import asyncio+from bs4 import BeautifulSoup++from py_lead_generation.src.engines.base import BaseEngine+from py_lead_generation.src.engines.abstract import AbstractEngine+++class YelpEngine(BaseEngine, AbstractEngine):+ '''+ `YelpEngine`++ Usage:++ `engine = YelpEngine(*args, **kwargs)`++ `await engine.run()`++ `await engine.save_to_csv()`++ `print(engine.entries)`+ '''++ BASE_URL = 'https://www.yelp.com/search?find_desc={query}&find_loc={location}%2C+Philippines&start=0'+ FIELD_NAMES = ['Title', 'Address', 'PhoneNumber', 'Tags']+ FILENAME = 'yelp_leads.csv'++ def __init__(self, query: str, location: str) -> None:+ '''+ '''+ self._entries = []+ self.query = query+ self.location = location+ self.url = self.BASE_URL.format(+ query=self.query, location=self.location+ )++ async def _get_search_results_urls(self) -> list[str]:+ '''+ '''+ urls = []+ host = 'https://www.yelp.com'+ while True:+ await asyncio.sleep(3)+ link_elements = await self.page.query_selector_all('.css-1hqkluu')+ for link_element in link_elements:+ url = await link_element.get_attribute('href')+ url = host + url+ urls.append(url)++ next_page_btn = await self.page.query_selector(+ '.next-link.navigation-button__09f24__m9qRz.css-ahgoya'+ )+ if not next_page_btn:+ break++ await next_page_btn.scroll_into_view_if_needed()+ await next_page_btn.click()++ return urls++ def _parse_data_with_soup(self, html: str) -> list[str]:+ '''+ `html: str` - html representation of the page to parse++ Returns `list[str]` typed parsed data - `[title, addr, phone, tags]`+ '''+ soup = BeautifulSoup(html, 'html.parser')+ data = []++ title_selector = '.css-1se8maq'+ addr_selector = '.css-qyp8bo'+ phone_selector = '.css-djo2w .css-1p9ibgf'+ tags_selector = '.css-1xfc281 span.css-1fdy0l5'++ for selector in (title_selector, addr_selector, phone_selector):+ element = soup.select_one(selector)+ info = element.get_text() if element else '-'+ data.append(info)++ tags = ''+ tag_elements = soup.select(tags_selector)+ for tag_element in tag_elements:+ el = tag_element.find('a')+ tag = el.get_text() if el else '-'+ tags += tag + ','+ tags = tags[:-1]+ data.append(tags)++ return datadiff --git a/run.py b/run.pynew file mode 100644index 0000000..304f2ba--- /dev/null+++ b/run.py@@ -0,0 +1,20 @@+import asyncio+from py_lead_generation import GoogleMapsEngine, YelpEngine+++async def main() -> None:+ q = input('Enter your search query: ').strip() or 'Barbershop'+ addr = input('Enter the location you would like to search in: ').strip() \+ or 'Paris'+ zoom = float(input('[Optional] Enter google maps zoom: ').strip() or 12)++ engine = GoogleMapsEngine(q, addr, zoom)+ await engine.run()+ engine.save_to_csv()++ engine = YelpEngine('Pizza', 'Mexico, Pampanga, Philippines')+ await engine.run()+ engine.save_to_csv('pizza_leads.csv')++if __name__ == '__main__':+ asyncio.run(main())diff --git a/setup.py b/setup.pynew file mode 100644index 0000000..ac8c77c--- /dev/null+++ b/setup.py@@ -0,0 +1,32 @@+from setuptools import setup, find_packages+++with open('README.md', 'r') as f:+ long_description = f.read()++VERSION = '0.3.0'++setup(+ name='py_lead_generation',+ version=VERSION,+ license = 'MIT',+ author='Madi-S (Madi Shaiken)',+ author_email='<[email protected]>',+ description='Lead generation scripts',+ long_description=long_description,+ long_description_content_type='text/markdown',+ url='https://github.com/Madi-S/Lead-Generation',+ packages=find_packages(),+ install_requires=['playwright', 'beautifulsoup4', 'geopy'],+ python_requires='>=3.10',+ keywords=['python', 'lead generation', 'web automation',+ 'playwright', 'google maps', 'yelp'],+ classifiers=[+ 'Development Status :: 1 - Planning',+ 'Intended Audience :: Developers',+ 'Programming Language :: Python :: 3.10',+ 'Operating System :: Unix',+ 'Operating System :: MacOS :: MacOS X',+ 'Operating System :: Microsoft :: Windows',+ ]+)diff --git a/yellow-pages/__pycache__/config.cpython-38.pyc b/yellow-pages/__pycache__/config.cpython-38.pycdeleted file mode 100644index d46a9a4..0000000Binary files a/yellow-pages/__pycache__/config.cpython-38.pyc and /dev/null differdiff --git a/yellow-pages/__pycache__/locators.cpython-38.pyc b/yellow-pages/__pycache__/locators.cpython-38.pycdeleted file mode 100644index f0f5da1..0000000Binary files a/yellow-pages/__pycache__/locators.cpython-38.pyc and /dev/null differdiff --git a/yellow-pages/__pycache__/my_config.cpython-38.pyc b/yellow-pages/__pycache__/my_config.cpython-38.pycdeleted file mode 100644index e61acb0..0000000Binary files a/yellow-pages/__pycache__/my_config.cpython-38.pyc and /dev/null differdiff --git a/yellow-pages/leads.csv b/yellow-pages/leads.csvdeleted file mode 100644index 2740b1b..0000000--- a/yellow-pages/leads.csv+++ /dev/null@@ -1,289 +0,0 @@-Title,Address,WebSite,PhoneNumber-Sohostel,"91 Dean Street, London, GB, XGL, W1D 3SY",Unclaimed,020 8821 5154-YHA London Central,"104 Bolsover Street, London, GB, XGL, W1W 5NU",yha.org.uk,0845 371 9154-The Hoxton - Shoreditch,"81-83 Great Eastern Street, London, GB, XGL, EC2A 3HU",Unclaimed,020 7550 1000-Wombat's City Hostel London,"7 Dock Street, London, GB, XGL, E1 8LL",wombats-hostels.com,020 7680 7600-Palmers Lodge,"Swiss Cottage-40 College Crescent, London, GB, XGL, NW3 5LB",palmerslodges.com,020 7483 8470-The Dictionary Hostel,"12-20 Kingsland Road, London, GB, XGL, E2 8DA",thedictionaryhostel.com,020 7613 2784-Woodville Guesthouse,"35 Woodville Road-Walthamstow, London, GB, XGL, E17 7ER",Unclaimed,07901 155500-Generator London,"37 Tavistock Place, London, GB, XGL, WC1H 9SE",generatorhostels.com,020 7388 7666-St Christophers Inn,"121 Borough High Street, London, GB, XGL, SE1 1NP",Unclaimed,020 7407 2392-Astor Victoria,"71 Belgrave Road, London, GB, XGL, SW1V 2BG",Unclaimed,020 7834 3077-Holland House,"53 Eccleston Square, London, GB, XGL, SW1V 1PG",Unclaimed,020 7834 9104-Bestplace,"41-42 Cosser Street, London, GB, XGL, SE1 7BU",bestplaceinns.com,020 7928 0720-Clink 78,"78 Kings Cross Road, London, GB, XGL, WC1X 9QG",clinkhostels.com,020 7183 9400-Queen Elizabeth Hostel,"58 Bagleys Lane-Hammersmith, London, GB, XGL, SW6 2BH",Unclaimed,0844 472 1800-The Walrus Bar and Hostel,"172 Westminster Bridge Road, London, GB, XGL, SE1 7RW",Menus | The Walrus Bar and Hostel http://thewalrusbarandhostel.co.uk/bar/assets/walrus_bh_menu.pdf,020 7928 4368-Arsenal Tavern Backpackers,"175 Blackstock Road, London, GB, XGL, N4 2JS",Unclaimed,020 7359 6902-Astor Museum Inn,"27 Montague Street-City of London-Greater London, London, GB, XGL, WC1B 5BH",astorhostels.co.uk,020 7580 5360-St Christopher's,"28 Hammersmith Broadway, London, GB, XGL, W6 7AB",st-christophers.co.uk/london-h…,020 8748 5285-The Fun Factory,"16a Cato Road-Clapham, London, GB, XGL, SW4 7TX",Unclaimed,-Astor Kensington,"45 Queensborough Terrace, London, GB, XGL, W2 3SY",astorhostels.com,020 7229 7782-Qbic Hotel ,"42 Adler Street, London, GB, XGL, E1 1EE",qbichotels.com/london-city,020 3021 3300-YHA London St Pancras,"79-81 Euston Road, London, GB, XGL, NW1 2QE",yha.org.uk,0845 371 9344-The Royal Horseguards Hotel,"2 Whitehall Court, London, GB, XGL, SW1A 2EJ",theroyalhorseguards.com,0871 376 9033-Camden Inn Hostel,"55-57 Bayham Street-Camden, London, GB, XGL, NW1 0AA",Unclaimed,0844 472 1800-Pickwick Hall,"7 Bedford Place, London, GB, XGL, WC1B 5JE",pickwickhall.co.uk,020 7323 4958-LSE High Holborn Residence,"178 High Holborn, London, GB, XGL, WC1V 7AA",lsevacations.co.uk/Accommodati…,020 7107 5737-Smart Camden Inn,"55-57 Bayham Street-Camden Town, London, GB, XGL, NW1 0AA",Unclaimed,020 7388 8900-Travel Joy Hostel,"111Grosvenor Road, London, GB, XGL, SW1V 3LG",traveljoyhostels.com,020 7834 9689-easyHotel - Victoria,"34-40 Belgrave Road, London, GB, XGL, SW1V 1RG",Booking.com,020 7834 1379-Staysafe,"144-152 Walworth Road-Elephant & Castle, London, GB, XGL, SE17 2NG",Unclaimed,-Bowden Court,"11 Belgrave Road, London, GB, XGL, SW1V 1RB",Unclaimed,020 3553 7280-Park Villa Boutique Hostel,"51 Grove Road-Bow, London, GB, XGL, E3 4PE",Unclaimed,020 8980 1439-Astor Hyde Park,"191 Queensgate-South Kensington, London, GB, XGL, SW7 5EU",astorhostels.co.uk,020 7581 0103-Cherry Court Hotel,"23 Hugh Street, London, GB, XGL, SW1V 1QJ",cherrycourthotel.co.uk,020 7828 2840-Bird's Nest Bar and Hostel,"32 Deptford Church St, London, GB, XGL, SE8 4RZ",Unclaimed,020 8692 1928-Don Gratton House,"Don Gratton House-82 Old Montague Street, London, GB, XGL, E1 5NN",Unclaimed,0870 600 4777-Venture Hostel,"86 Tanners Hill, London, GB, XGL, SE8 4PN",Unclaimed,020 8692 6219-Alliance House,"44-45 Newington Green, London, GB, XGL, N16 9QH",Unclaimed,020 7288 8060-Northfields Hostel,"264 Northfield Avenue-Acton, London, GB, XGL, W3 0NW",Unclaimed,0844 472 1800-Baden Powell House,"65-67 Queen's Gate, London, GB, XGL, SW7 5JS",Unclaimed,020 7590 6909-Earl's Court Youth Hostel,"38 Bolton Gardens, London, GB, XGL, SW5 0AQ",yha.org.uk/find-accommodation/…,0345 371 9114-Safestay,"144-152 Walworth Road-Elephant and Castle, London, GB, XGL, SE17 1RL",Booking.com,020 7703 8000-YHA Holland Park,"Holland Walk Kensington, London, GB, XGL, W8 7QU",Unclaimed,0845 371 9122-London Backpackers,"8-10 Queens Parade-Queens Road-Hendon, London, GB, XGL, NW4 3NS",ukhostels.com,020 8203 1319-Palmers Lodge Hillspring,"233 Willesden Lane, London, GB, XGL, NW2 5RP",palmerslodges.com/hillspring,020 7099 2435-High Holborn Hall - Early Summer,"178 High Holborn-Covent Garden, London, GB, XGL, WC1V 7AA",Unclaimed,0844 472 1800-The RE London Shoreditch,"419 - 437 Hackney Road, London, GB, XGL, E2 8PP",hotelshoreditch.com,020 7613 6500-Sally London dfd,"142 London New-Padd, London, GB, XGL, NW7 2RR",Unclaimed,-Restup,"172 New Kent Road, London, GB, XGL, SE1 4YT",restup.co.uk,020 3642 4549-Astor Kensington,"138 Cromwell Road, London, GB, XGL, SW7 4HA",Unclaimed,020 7373 5138-YHA London St. Paul's,"36 Carter Lane, London, GB, XGL, EC4V 5AB",yha.org.uk,0845 371 9012-Clink 261,"261-265 Gray's Inn Road, London, GB, XGL, WC1X 8QT",hostelworld.com,020 7183 9400-LSE Bankside House,"24 Sumner Street, London, GB, XGL, SE1 9JA",lsevacations.co.uk,020 7107 5750-City Ymca London,"8 Errol Street, London, GB, XGL, EC1Y 8SE",Unclaimed,020 7628 8832-Northumberland Park House,"143 Northumberland Park, London, GB, XGL, N17 0TR",Unclaimed,020 8808 8839-The W14 Kensington,"16-22 Gunterstone Road, West Kensington, GB, XGL, W14 9BX",Hostels.com,020 7602 6600-Fairway Lodge,"400-408 Stanstead Road-Catford, London, GB, XGL, SE6 4XB",Unclaimed,020 8690 6992-Megaro Hotel,"23-27 Euston Road-St Pancras, London, GB, XGL, WC1H 8AB",hotelmegaro.co.uk,020 7843 2222-Kensington House Hotel,"15-16 Prince of Wales Terrace, London, GB, XGL, W8 5PQ",Unclaimed,020 7937 2345-Alinka Hostel,"52A Brixton Road London, London, GB, XGL, SW9 8HL",Unclaimed,-Smart Hostels,"71-72 Guilford Street, London, GB, XGL, WC1N 1DF",Unclaimed,020 7833 8818-Smart Hyde Park Inn,"48-50 Inverness Terrace, London, GB, XGL, W2 3JA",Unclaimed,-ST. Mungos,"45 Killieser Avenue, London, GB, XGL, SW2 4NX",Unclaimed,020 8674 4597-Revive Lodge Heathrow,"89 Hounslow Road-Feltham, London, GB, XGL, TW14 0BB",Unclaimed,0844 472 1800-Inner Park Road Hostel,"27 Inner Park Road, London, GB, XGL, SW19 6DF",Unclaimed,020 8789 7957-The Nadler Kensington,"25 Courtfield Gardens, London, GB, XGL, SW5 0PG",thenadler.com,020 7244 2255-all star hostel,"Chatsworth Road London, London, GB, XGL, E5 0PS",Unclaimed,-Amro Vantage,"24 Berkeley Square Mayfair, London, GB, XGL, W1J 6HE",amrovantage.com,020 3709 0274-Phoenix Hostel,"6 Daventry Street, London, GB, XGL, NW1 5NX",Unclaimed,020 7724 0705-Shepiston Lodge Heathrow,"31 Shepiston Lane-Middlesex, London, GB, XGL, UB3 1LJ",Unclaimed,020 8573 0266-Harlingford Hotel,"61-63 Cartwright Gardens, London, GB, XGL, WC1H 9EL",Unclaimed,020 7387 1551-InterContinental London Park Lane,"One Hamilton Place-Park Lane, London, GB, XGL, W1J 7QY",Claimed,020 7409 3131-Piccadilly Hotel,"12 Sherwood Street, London, GB, XGL, W1F 7BZ",Unclaimed,020 7434 9009-Abercorn House,"28-30 Bute Gardens, London, GB, XGL, W6 7DS",speedtest.net/result/267…,020 8563 8692-Youth Hostel Association,"104-108 Bolsover Street-Marylebone, London, GB, XGL, W1W 5QQ",-,0870 770 6144-O'Callaghan's Hostel,"205 Earls Court Road-Kensington, London, GB, XGL, SW5 9AN",Unclaimed,020 7370 3000-Dellow Centre,"82 Wentworth Street, London, GB, XGL, E1 7SA",Unclaimed,020 7375 3915-Eastwards Trust Hostels,"217 Romford Road, London, GB, XGL, E7 9HL",Unclaimed,020 8555 6421-Belgrave Court,"83-87 Prince Regent Lane, London, GB, XGL, E13 8RP",Unclaimed,020 7511 5429-Thames Reach Bondway,"62 Bondway, London, GB, XGL, SW8 1SF",Unclaimed,020 7840 9820-St Christopher's Inns,"48-50 Camden High Street, London, GB, XGL, NW1 0LT",st-christophers.co.uk,020 7388 1012-Ridley Villas Hostel,"35 Ridley Road, London, GB, XGL, E8 2NP",Unclaimed,020 7275 7852-Journeys London Bridge,"204 Manor Place-Walworth, London, GB, XGL, SE17 3BN",Unclaimed,020 7735 6581-Via Limehouse,"628-634 Commercial Road-Tower Hamlets, London, GB, XGL, E14 7HS",Unclaimed,04420 3588 8484-4 Star Hostel,"91 Jermyn Street, London, GB, XGL, SW1Y 6JB",Unclaimed,020 7930 3111-Barmy Badger Backpackers,"17 Longridge Road, London, GB, XGL, SW5 9SB",Unclaimed,020 7370 5213-High Street Kensington Hostel,"1 Warwick Gardens, London, GB, XGL, W14 8PH",Unclaimed,020 7602 0444-Belsize House,"40 Belsize Park Gardens, London, GB, XGL, NW3 4NA",Unclaimed,020 3740 9804-Dalco Homes,"20 Dukes Avenue, London, GB, XGL, N10 2PT",Unclaimed,020 8444 5138-Roomzzz London Stratford,"66 West Ham Lane, London, GB, XGL, E15 4PT",Unclaimed,020 3909 6400-More House,"53 Cromwell Road, London, GB, XGL, SW7 2EH",Unclaimed,020 7584 2040-Equity Point London,"100-102 Westbourne Ter-Paddington W2 6QE, London, GB, XGL, W2 6",Unclaimed,04420 7087 8001-Caplin House,"330 Commercial Road, London, GB, XGL, E1 2PY",Unclaimed,020 7780 9078-Piccadilly Guest House,"91 Jermyn Street-St Jame's, London, GB, XGL, SW1Y 6JB",Unclaimed,0845 838 4645-Kipper Project,"15 Old Ford Road, London, GB, XGL, E2 9PJ",Unclaimed,020 8980 1401-Travelodge Hotel,"195 -203 Waterloo Road, London, GB, XGL, SE1 8UX",Unclaimed,0871 984 6291-Yha England & Wales,"Holland House Holland Park, London, GB, XGL, W8 6LU",Unclaimed,020 7937 0748-Yotel,"Heathrow Airport-Terminal 4, Hounslow, GB, XGL, TW6 3XA",yotel.com,020 7100 1100-Atlas Hostels,"16 Talbot Square, London, GB, XGL, W2 1TS",Unclaimed,0845 519 9448-Irish Centre Housing,"20-22 Quex Road, London, GB, XGL, NW6 4PG",Unclaimed,020 7372 6633-Kabo Hostel,"32 Saltoun Road, London, GB, XGL, SW2 1EP",Unclaimed,020 7733 9466-Chester House,"Pages Lane, London, GB, XGL, N10 1PR",Unclaimed,020 8883 8204-Hyde Park Hostel,"26 Inverness Terrace-Bayswater, London, GB, XGL, W2 3HU",Unclaimed,020 7727 9163-"Globetrotter Inns, London","Ashlar Court-Ravenscourt Gardens, London, GB, XWY, W6",Unclaimed,020 8746 3112-Pembury Hotel,"326 - 328 Seven Sisters Road-Finsbury Park, London, GB, XGL, N4 2AP",Unclaimed,0844 472 1800-Moullin House Memorial Hostel Trust,"24-26 Mount Park Road, London, GB, XGL, W5 2RT",Unclaimed,020 8997 4343-The Louisianna,"611 Green Lanes, London, GB, XGL, N8 0RE",Unclaimed,020 8341 0726-St Christopher's Inn Greenwich,"189 Greenwich High Road, London, GB, XGL, SE10 8JA",st-christophers.co.uk/london-h…,020 8858 3591-641 Fulham Road,"641 Fulham Road, London, GB, XGL, SW6 5PU",Unclaimed,020 7581 5970-Hytyme Managment Services,"261 Barry Road, London, GB, XGL, SE22 0JT",Unclaimed,020 8299 3333-St Christopher's Village London Bridge,"161 Borough High Street, Southwark, GB, XGL, SE1 1HR",st-christophers.co.uk,020 7939 9710-University of Westminster International House,"1-5 Lambeth Rd, London, GB, XGL, SE1 7DG",Unclaimed,020 7911 5181-Central Hostel,"26 Prince's Square-Bayswater, London, GB, XGL, W2 4NJ",Unclaimed,020 7727 5807-Sandeman-Allen House,"40 Inverness Terrace, London, GB, XGL, W2 3HB",Unclaimed,020 7727 2719-International Students House,"229 Great Portland Street, London, GB, XGL, W1W 5PN",ish.org.uk,020 7631 8300-Hub Premier Inn,"15 Dacre Street, London, GB, XGL, SW1H 0DJ",Unclaimed,0871 527 9608-Threshold Tennant Trust,"10 Fontenoy Road, London, GB, XGL, SW12 9LU",Unclaimed,020 8675 1000-Casa Dello Studente,"174 Clapham Road, London, GB, XGL, SW9 0LA",Unclaimed,020 7735 3058-Arlington House,"220 Arlington Road, London, GB, XGL, NW1 7HE",Unclaimed,020 7482 3374-Pathway Housing Association,"77 Plough Road, London, GB, XGL, SW11 2BJ",Unclaimed,020 7924 3009-The Rathbone,"30 Rathbone Street, London, GB, XGL, W1T 1LB",rathbonehotel.co.uk,020 7636 2001-Alone In London,"77 Lansdowne Road, London, GB, XGL, N17 0NN",Unclaimed,020 8493 0239-Ackee Housing Project,"103 Stoke Newington Road, London, GB, XGL, N16 8BX",Unclaimed,020 7249 5659-Globe Trott Inns,"Boleyn Tavern-1 Barking Road, London, GB, XGL, E6 1PW",-,020 8471 4284-Journeys Kings Cross Hostel,"54-58 Caledonian Rd-Kings Cross, London, GB, XGL, N1 9DP",Unclaimed,020 7833 3893-Young Women's Christian Association Of Great Britain,"6-8 Kemplay Road, London, GB, XGL, NW3 1SY",Unclaimed,020 7435 5730-Journeys London East Hostel,"118 Wellington Street, London, GB, XGL, SE18 6XY",Unclaimed,020 8853 9352-St Christopher's Inn Shepherd's Bush,"13-15 Shepherds Bush Green-Shepherds Bush, London, GB, XGL, W12 8PH",st-christophers.co.uk/london-h…,020 8735 0270-Downham Lodge,"284 Amhurst Road, London, GB, XGL, N16 7UE",Unclaimed,020 7923 2900-Camden Lock Hotel,"89 Chalk Farm Road-Camden, London, GB, XGL, NW1 8AR",Unclaimed,020 7267 3912-Bed Station,"1st Floor Offices-189-193 Earls Court Road, London, GB, XGL, SW5 9AN",bedstation.co.uk,020 7835 1867-Strand Palace,"372 Strand, London, GB, XGL, WC2R 0JJ",strandpalacehotel.co.uk,020 7379 4737-Newington Court Hostel,"Newington Court-1 Collins Road, London, GB, XGL, N5 2UF",Unclaimed,020 7359 2495-Ibis Budget London Whitechapel,"100 Whitechapel Road, London, GB, XGL, E1 1JG",Unclaimed,020 7655 4620-Tune Hotel,"118-120 Westminster Bridge Road, London, GB, XGL, SE1 7RW",Unclaimed,-Youth Hostels Association,"4 Wellgarth Road, London, GB, XGL, NW11 7HR",Unclaimed,020 8458 7196-Mindset Services,"684a High Road, London, GB, XGL, N17 0AE",Unclaimed,020 8365 0777-The Stafford London,"16-18 St James's Place, London, GB, XGL, SW1A 1NJ",thestaffordlondon.com/english/…,020 7493 0111-Salvation Army,"153-175 Whitechapel Road, London, GB, XGL, E1",Unclaimed,020 7392 9490-London Hostels Association,"97 Queens Gate, London, GB, XGL, SW7 5AB",Unclaimed,020 7373 4180-Shian Housing Association,"152 Bethune Road, London, GB, XGL, N16 5DS",Unclaimed,020 8800 5060-St Mungos,"188 Vartry Road, London, GB, XGL, N15 6HA",Unclaimed,020 8809 5742-Paradice Homes,"39 Ingleton Road, London, GB, XGL, N18 2RS",Unclaimed,020 8803 1460-Dover Castle,"6 Great Dover Street, London, GB, XGL, SE1 4XW",Unclaimed,020 7403 7773-Bartrams Convent Hostel,"Rowland Hill Street, London, GB, XGL, NW3 2AD",Unclaimed,020 7431 5680-Susan Lawrence Hostel,"170 Kings Cross Road, London, GB, XGL, WC1X 9DH",Unclaimed,-Hyde Park Inn,"48-50 Inverness Terrace, London, GB, XGL, W2 3JA",Unclaimed,020 7229 0000-Euro Tower,"Courland Grove, London, GB, XGL, SW8 2PX",Unclaimed,020 7720 5191-Hostel Acacia London,"14 Queensbury Place-South Kensington London, London, GB, XGL, SW7 2EA",Unclaimed,020 7823 7103-St Georges House,"263 Camden Road, London, GB, XGL, N7 0HS",Unclaimed,020 7607 7989-St Mungon,"146 Mare Street, London, GB, XGL, E8 3SG",Unclaimed,020 8525 0007-Helen Graham House,"57 Great Russell Street-Holborn, London, GB, XGL, WC1B 3BD",Unclaimed,020 8307 3300-The Tower Hotel,"Saint Katharine's Way, London, GB, XGL, E1W 1LD",guoman.com,0800 330 8005-Youth Hostel Association,"Holland Walk-Kensington, London, GB, XGL, W8 7QU",Unclaimed,0870 770 5866-Hillspring Lodge,"233 Willesden Lane-Willesden, London, GB, XGL, NW2",Unclaimed,0844 472 1800-Keystone House Kings Cross,"272-276 Pentonville Road, London, GB, XGL, N1 9JY",Unclaimed,020 7837 6444-Ymca,"5 Shernhall Street, London, GB, XGL, E17 3EU",Unclaimed,020 8520 1667-UoL Accommodates,"ULU 4th Floor-Malet Street, London, GB, XGL, WC1E 7HY",Unclaimed,04402 0786 28881-Equinox,"100 Queens Road, London, GB, XGL, SE15 2QT",Unclaimed,020 7639 8008-hostelsconnections,"3 coborn rd-london, London, GB, XGL, E3 2DA",Unclaimed,020 8123 0817-Arosfa Hotel,"83 Gower Street, London, GB, XGL, WC1E 6HJ",Unclaimed,020 7636 2115-Regina House,"110 Gloucester Road-South Kensington, London, GB, XGL, SW7 4RJ",Unclaimed,020 7373 5151-Celtic Hotel,"62 Guilford Street, London, GB, XGL, WC1N 1DD",Unclaimed,020 7837 9258-BW Premier Park Grand,"27/37 Hogarth Road-Kensington, London, GB, XGL, SW5 0QQ",Unclaimed,020 7370 6831-Coram Leaving Care Services,"1 Yonge Park, London, GB, XGL, N4 3NU",Unclaimed,020 7607 4341-IQ Kingston,"15 Wood Street, Kingston Upon Thames, GB, XGL, KT1 1TY",Unclaimed,020 8547 3139-TravelStay Hyde Park,", London, GB, XGL, W2",Unclaimed,020 7298 2100-Thames Reach Bondway Housing Association Hostels,"73-75 Stamford Street, London, GB, XGL, SE1 9NB",Unclaimed,020 7928 2497-Emily Bowes Court,"Lebus Street, London, GB, XGL, N17 9FD",Unclaimed,020 8801 3335-Belgrave Hotel,"9-13 Clapham Road, London, GB, XGL, SW9 0JD",Unclaimed,020 7793 0142-Youth Hostels Association,"38 Bolton Gardens, London, GB, XGL, SW5 0AQ",Unclaimed,0870 770 5804-Nothing,"123 Great Garden-Bayswater, London, GB, XGL, W2 1UB",Unclaimed,-MEININGER Hotel London Hyde Park,"65-67 Queen's Gate, London, GB, XGL, SW7 5JS",meininger-hotels.com,020 3318 1407-Woodthorpe Road Hostel,"3 Woodthorpe Road, London, GB, XGL, SW15 6UQ",Unclaimed,020 8789 5801-St Mungo's,"113-117 Cedars Road, London, GB, XGL, SW4 0PJ",Unclaimed,020 7720 7377-Sundial Court,"38-42 Chiswell Street, London, GB, XGL, EC1Y 4SB",Unclaimed,020 7588 0417-RMA Accommodation,"156-158 Mile End Road, London, GB, XGL, E1 4LJ",Unclaimed,020 7790 1884-Homeless Young Peoples Project,"43-45 ST. Georges Avenue, London, GB, XGL, E7 8HR",Unclaimed,020 8471 5111-Edmonton Hostel,"1 Church Street, London, GB, XGL, N9 9DR",Unclaimed,020 8345 5415-West Two Hostel London,"22-23 Kensington Garden Square-Bayswater, London, GB, XGL, W2 4BE",Unclaimed,0844 472 1800-Tamworth House,"188 Tamworth Lane, Mitcham, GB, SRY, CR4 1DE",Unclaimed,020 8687 1319-London Eye Hostel,"73 Lambeth Walk-Lambeth, London, GB, XGL, SE11 6DX",Unclaimed,-Grow,"36 Rickmansworth Road, Watford, GB, HRT, WD18 7HT",Unclaimed,01923 256189-Z Hotel Victoria,"5 Lower Belgrave Street, London, GB, XGL, SW1W 0NR",Unclaimed,020 3589 3990-London Hostels Association,"40 Belsize Park Gardens, London, GB, XGL, NW3 4NA",Unclaimed,020 7722 8131-London Hostels Association,"Sandeman Allen-40 Inverness Terrace, London, GB, XGL, W2 3JB",Unclaimed,020 7727 2719-Alliance Club,"44-45 Newington Green, London, GB, XGL, N16 9QH",Unclaimed,020 7226 6085-Ashwell House University Hall Of Residents,"Ashwell House-1b Shepherdess Walk, London, GB, XGL, N1 7BD",Unclaimed,020 7490 3296-Chapel Hill Hostel,"51-55 Chapel Hill, Dartford, GB, KEN, DA1 4BY",Unclaimed,01322 553201-The Court Hostel,"194-196 Earls Court Road-Earls Court, London, GB, XGL, SW5 9QF",Unclaimed,04402 0737 30027-The Lodge,"44 Norfolk Square, London, GB, XGL, W2 1RT",Unclaimed,020 7262 4406-St Mungos Housing Association,"Pagnell Street Hostel-Exeter Way, London, GB, XGL, SE14 6LL",Unclaimed,020 8692 3454-YHA London Lee Valley,"Windmill Lane-Cheshunt, Hertfordshire, GB, HRT, EN8 9AF",yha.org.uk,0845 371 9057-The Green Man - Bestplace Inn,"308 Edgware Road, London, GB, XGL, W2 1DY",Unclaimed,04402 0772 37980-Number8ight,"305 High Rd-Willesden, London, GB, XGL, NW10 2JT",Unclaimed,020 8459 8576-Hostel Fleming,"198 Brecknock Road, London, GB, XGL, N19 5BE",Unclaimed,020 7609 4282-Norwegian Ywca,"52 Holland Park, London, GB, XGL, W11 3RS",Unclaimed,020 7727 9897-Ex Service Fellowship,"New Belvedere House-White Horse Road, London, GB, XGL, E1 0ND",Unclaimed,020 7790 1474-Deep Blue Apartments,"Kingsgate House-12-50 Kingsgate Road, Kingston Upon Thames, GB, XGL, KT2 5AA",Unclaimed,0370 770 0778-Ashburnham Hostel,"75 Ashburnham Grove, London, GB, XGL, SE10 8UJ",Unclaimed,020 8692 5032-Museum Inn,"27 Montague Street, London, GB, XGL, WC1B 5BH",Unclaimed,020 7580 5360-Le Méridien Piccadilly,"21 Piccadilly, London, GB, XGL, W1J 0BH",marriott.com/hotels/travel/lon…,020 7734 8000-St Christopher's,"59-61 Borough High Street, London, GB, XGL, SE1 1NE",Unclaimed,020 7407 6266-St Louise,"33 Medway Street, London, GB, XGL, SW1P 2BE",Unclaimed,020 7222 2071-Barkston Rooms,"1 Barkston Gardens-Earls Court, London, GB, XGL, SW5 0ER",Unclaimed,0844 472 1800-Hotel Strand Continental,"143 Strand, London, GB, XGL, WC2R 1JA",strand-continental.co.uk,020 7836 4880-Queen Alexandras House,"Kensington Gore, London, GB, XGL, SW7 2QT",Unclaimed,020 7589 3635-Westmoor House Rehabilitation Centre,"244 Roehampton Lane, London, GB, XGL, SW15 4AA",Unclaimed,020 8780 5863-Pullman Hotel,"100 - 110 Euston Road, London, GB, XGL, NW1 2AJ",Unclaimed,020 7666 9000-Welcome Homestay UK,"102 Inchmery Rd, London, GB, LND, SE6 2ND",welcomehomestay.co.uk,07757 938037-Capitanio Sisters,"Nile Lodge-Queens Walk, London, GB, XGL, W5 1TJ",Unclaimed,020 8997 2296-Heritage Star,"54-58 Caledonian Road, London, GB, XGL, N1 9DP",Unclaimed,020 7833 3893-Club Quarters,"8 Northumberland Avenue-Trafalgar Square, London, GB, XGL, WC2N 5BY",clubquarters.com,020 7839 9333-The Georgian House,"37 St Georges Drive, London, GB, XGL, SW1V 4DG",georgianhousehotel.co.uk,020 7834 1438-Metrogate House,"3 Queens Gate Terrace, London, GB, XGL, SW7 5PE",Unclaimed,020 7823 7440-Columbia Hotel,"95-99 Lancaster Gate, London, GB, XGL, W2 3NS",Unclaimed,020 7402 0021-Church Army Emergency Hostel,"1-5 Cosway Street, London, GB, XGL, NW1 5NR",Unclaimed,020 7262 3818-Shenehom Housing Association,"31-32 Ranelagh Avenue, London, GB, XGL, SW13 0BN",Unclaimed,020 8878 8667-Classic Property,"Tulsi House-61 Carlton Grove, London, GB, XGL, SE15 2UD",Unclaimed,07932 656296-Keyworth Hostel,"10 Keyworth Street, London, GB, XGL, SE1 6NG",Unclaimed,020 7401 8950-YHA Rotherhithe,"20 Salter Road-Rotherhithe, London, GB, XGL, SE16 5PR",yha.org.uk/find-accommodation/…,0870 770 6010-Youth Hostels Association,"20 Salter Road, London, GB, XGL, SE16 5PR",Unclaimed,020 7232 2114-Local 33,"26 Mayflower Road, London, GB, XGL, SW9 9JZ",Unclaimed,020 7737 3107-Ned Kellys,"305 High Road, London, GB, XGL, NW10 2JT",Unclaimed,020 8459 6906diff --git a/yellow-pages/pylg.log b/yellow-pages/pylg.logdeleted file mode 100644index 0ebb59a..0000000--- a/yellow-pages/pylg.log+++ /dev/null@@ -1,52 +0,0 @@-2020-12-22 18:46:32,074 - WARNING | bufferization.py/store/42 | Got duplicate or empty dictionary {'Title': '-', 'Address': '-', 'WebSite': '-', 'PhoneNumber': '-'}-2020-12-22 18:46:33,359 - WARNING | bufferization.py/store/42 | Got duplicate or empty dictionary {'Title': '-', 'Address': '-', 'WebSite': '-', 'PhoneNumber': '-'}-2020-12-22 18:46:34,790 - WARNING | bufferization.py/store/42 | Got duplicate or empty dictionary {'Title': '-', 'Address': '-', 'WebSite': '-', 'PhoneNumber': '-'}-2020-12-22 18:46:36,292 - WARNING | bufferization.py/store/42 | Got duplicate or empty dictionary {'Title': '-', 'Address': '-', 'WebSite': '-', 'PhoneNumber': '-'}-2020-12-22 18:46:37,827 - WARNING | bufferization.py/store/42 | Got duplicate or empty dictionary {'Title': '-', 'Address': '-', 'WebSite': '-', 'PhoneNumber': '-'}-2020-12-22 18:46:39,783 - WARNING | bufferization.py/store/42 | Got duplicate or empty dictionary {'Title': '-', 'Address': '-', 'WebSite': '-', 'PhoneNumber': '-'}-2020-12-22 18:46:41,092 - WARNING | bufferization.py/store/42 | Got duplicate or empty dictionary {'Title': '-', 'Address': '-', 'WebSite': '-', 'PhoneNumber': '-'}-2020-12-22 18:46:42,481 - WARNING | bufferization.py/store/42 | Got duplicate or empty dictionary {'Title': '-', 'Address': '-', 'WebSite': '-', 'PhoneNumber': '-'}-2020-12-22 18:46:43,892 - WARNING | bufferization.py/store/42 | Got duplicate or empty dictionary {'Title': '-', 'Address': '-', 'WebSite': '-', 'PhoneNumber': '-'}-2020-12-23 08:59:24,768 - WARNING | bufferization.py/store/42 | Got duplicate or empty dictionary {'Title': '-', 'Address': '-', 'WebSite': '-', 'PhoneNumber': '-'}-2020-12-23 08:59:26,267 - WARNING | bufferization.py/store/42 | Got duplicate or empty dictionary {'Title': '-', 'Address': '-', 'WebSite': '-', 'PhoneNumber': '-'}-2020-12-23 08:59:27,700 - WARNING | bufferization.py/store/42 | Got duplicate or empty dictionary {'Title': '-', 'Address': '-', 'WebSite': '-', 'PhoneNumber': '-'}-2020-12-23 08:59:29,777 - WARNING | bufferization.py/store/42 | Got duplicate or empty dictionary {'Title': '-', 'Address': '-', 'WebSite': '-', 'PhoneNumber': '-'}-2020-12-23 08:59:31,173 - WARNING | bufferization.py/store/42 | Got duplicate or empty dictionary {'Title': '-', 'Address': '-', 'WebSite': '-', 'PhoneNumber': '-'}-2020-12-23 08:59:33,312 - WARNING | bufferization.py/store/42 | Got duplicate or empty dictionary {'Title': '-', 'Address': '-', 'WebSite': '-', 'PhoneNumber': '-'}-2020-12-23 08:59:34,782 - WARNING | bufferization.py/store/42 | Got duplicate or empty dictionary {'Title': '-', 'Address': '-', 'WebSite': '-', 'PhoneNumber': '-'}-2020-12-23 08:59:36,387 - WARNING | bufferization.py/store/42 | Got duplicate or empty dictionary {'Title': '-', 'Address': '-', 'WebSite': '-', 'PhoneNumber': '-'}-2020-12-23 08:59:38,050 - WARNING | bufferization.py/store/42 | Got duplicate or empty dictionary {'Title': '-', 'Address': '-', 'WebSite': '-', 'PhoneNumber': '-'}-2020-12-23 08:59:42,403 - WARNING | bufferization.py/store/42 | Got duplicate or empty dictionary {'Title': '-', 'Address': '-', 'WebSite': '-', 'PhoneNumber': '-'}-2020-12-23 08:59:43,932 - WARNING | bufferization.py/store/42 | Got duplicate or empty dictionary {'Title': '-', 'Address': '-', 'WebSite': '-', 'PhoneNumber': '-'}-2020-12-23 08:59:45,613 - WARNING | bufferization.py/store/42 | Got duplicate or empty dictionary {'Title': '-', 'Address': '-', 'WebSite': '-', 'PhoneNumber': '-'}-2020-12-23 08:59:46,821 - WARNING | bufferization.py/store/42 | Got duplicate or empty dictionary {'Title': '-', 'Address': '-', 'WebSite': '-', 'PhoneNumber': '-'}-2020-12-23 08:59:48,219 - WARNING | bufferization.py/store/42 | Got duplicate or empty dictionary {'Title': '-', 'Address': '-', 'WebSite': '-', 'PhoneNumber': '-'}-2020-12-23 09:11:54,715 - WARNING | bufferization.py/store/42 | Got duplicate or empty dictionary {'Title': '-', 'Address': '-', 'WebSite': '-', 'PhoneNumber': '-'}-2020-12-23 09:11:55,918 - WARNING | bufferization.py/store/42 | Got duplicate or empty dictionary {'Title': '-', 'Address': '-', 'WebSite': '-', 'PhoneNumber': '-'}-2020-12-23 09:11:57,439 - WARNING | bufferization.py/store/42 | Got duplicate or empty dictionary {'Title': '-', 'Address': '-', 'WebSite': '-', 'PhoneNumber': '-'}-2020-12-23 09:11:58,668 - WARNING | bufferization.py/store/42 | Got duplicate or empty dictionary {'Title': '-', 'Address': '-', 'WebSite': '-', 'PhoneNumber': '-'}-2020-12-23 09:11:59,815 - WARNING | bufferization.py/store/42 | Got duplicate or empty dictionary {'Title': '-', 'Address': '-', 'WebSite': '-', 'PhoneNumber': '-'}-2020-12-23 09:12:00,941 - WARNING | bufferization.py/store/42 | Got duplicate or empty dictionary {'Title': '-', 'Address': '-', 'WebSite': '-', 'PhoneNumber': '-'}-2020-12-23 09:12:02,260 - WARNING | bufferization.py/store/42 | Got duplicate or empty dictionary {'Title': '-', 'Address': '-', 'WebSite': '-', 'PhoneNumber': '-'}-2020-12-23 09:12:03,670 - WARNING | bufferization.py/store/42 | Got duplicate or empty dictionary {'Title': '-', 'Address': '-', 'WebSite': '-', 'PhoneNumber': '-'}-2020-12-23 09:12:05,242 - WARNING | bufferization.py/store/42 | Got duplicate or empty dictionary {'Title': '-', 'Address': '-', 'WebSite': '-', 'PhoneNumber': '-'}-2020-12-23 09:12:08,690 - WARNING | bufferization.py/store/42 | Got duplicate or empty dictionary {'Title': '-', 'Address': '-', 'WebSite': '-', 'PhoneNumber': '-'}-2020-12-23 09:12:10,116 - WARNING | bufferization.py/store/42 | Got duplicate or empty dictionary {'Title': '-', 'Address': '-', 'WebSite': '-', 'PhoneNumber': '-'}-2020-12-23 09:12:11,587 - WARNING | bufferization.py/store/42 | Got duplicate or empty dictionary {'Title': '-', 'Address': '-', 'WebSite': '-', 'PhoneNumber': '-'}-2020-12-23 09:12:12,800 - WARNING | bufferization.py/store/42 | Got duplicate or empty dictionary {'Title': '-', 'Address': '-', 'WebSite': '-', 'PhoneNumber': '-'}-2020-12-23 09:32:17,812 - WARNING | bufferization.py/store/42 | Got duplicate or empty dictionary {'Title': 'Sweaty Betty', 'Address': '-', 'WebSite': '-', 'PhoneNumber': '-'}-2021-02-26 17:41:06,571 - WARNING | extractor.py/parse/82 | Could not locate `raw_data` skipping current page-2021-02-26 17:43:33,045 - WARNING | extractor.py/parse/82 | Could not locate `raw_data` skipping current page-2021-02-26 17:45:36,282 - WARNING | extractor.py/parse/82 | Could not locate `raw_data` skipping current page-2021-02-26 17:46:02,580 - WARNING | extractor.py/parse/82 | Could not locate `raw_data` skipping current page-2021-02-26 17:48:31,472 - WARNING | extractor.py/parse/82 | Could not locate `raw_data` skipping current page-2021-02-26 17:50:00,979 - WARNING | extractor.py/parse/82 | Could not locate `raw_data` skipping current page-2021-02-26 17:52:36,252 - WARNING | extractor.py/parse/82 | Could not locate `raw_data` skipping current page-2021-02-26 17:53:43,714 - WARNING | extractor.py/parse/82 | Could not locate `raw_data` skipping current page-2021-02-26 17:59:13,729 - WARNING | extractor.py/parse/82 | Could not locate `raw_data` skipping current page-2021-02-26 18:00:11,256 - WARNING | bufferization.py/store/42 | Got duplicate or empty dictionary {'WebSite': 'Unclaimed', 'Title': 'Ibis Budget London Whitechapel', 'PhoneNumber': '020 7655 4620', 'Address': '100 Whitechapel Road, London, GB, XGL, E1 1JG'}-2021-02-26 18:03:21,403 - WARNING | extractor.py/parse/82 | Could not locate `raw_data` skipping current page-2021-02-26 18:03:30,916 - WARNING | extractor.py/parse/82 | Could not locate `raw_data` skipping current page-2021-02-26 18:07:32,221 - WARNING | extractor.py/parse/82 | Could not locate `raw_data` skipping current page-2021-02-26 18:10:09,790 - WARNING | extractor.py/parse/82 | Could not locate `raw_data` skipping current page-2021-02-26 18:11:26,674 - WARNING | extractor.py/parse/82 | Could not locate `raw_data` skipping current page