Skip to content

Python script, which empowers people with no programming background to generate robust leads on a mass scale. This repo will be compiled of various versatile techniques used in lead generation.

New version

Madi-S23c095ade733
diff --git a/LICENSE b/LICENSE
index 2b52355..5714e5d 100644
--- a/LICENSE
+++ b/LICENSE
@@ -1,6 +1,6 @@
MIT License
-Copyright (c) 2022 Madi Shaiken
+Copyright (c) 2024 Madi Shaiken
Permission is hereby granted, free of charge, to any person obtaining a copy
of this software and associated documentation files (the "Software"), to deal
diff --git a/README.md b/README.md
index 6bc6767..007a45b 100644
--- a/README.md
+++ b/README.md
@@ -1,2 +1,56 @@
# Lead-Generation
-Python script, which empowers people with no programming background to generate robust leads on a mass scale. This repo will be compiled of various versatile techniques used in lead generation implemented in python.
+
+The updated version of my outdated dirty clumsy and having shockingly high amount of stars repository on Python - https://github.com/Madi-S/Lead-Generation
+
+# Installation
+
+```bash
+pip install py-lead-generation
+```
+
+OR
+
+```bash
+git clone https://github.com/Madi-S/Lead-Generation
+cd Lead-Generation
+python run.py
+```
+
+# Quickstart
+
+```python
+import asyncio
+from py_lead_generation import GoogleMapsEngine, YelpEngine
+
+
+async def main() -> None:
+ q = input('Enter your search query: ').strip() or 'Barbershop'
+ addr = input('Enter the location you would like to search in: ').strip() \
+ or 'Paris'
+ zoom = float(input('[Optional] Enter google maps zoom: ').strip() or 12)
+
+ engine = GoogleMapsEngine(q, addr, zoom)
+ await engine.run()
+ engine.save_to_csv()
+
+ engine = YelpEngine('Pizza', 'Mexico, Pampanga, Philippines')
+ await engine.run()
+ engine.save_to_csv('pizza_leads.csv')
+
+if __name__ == '__main__':
+ asyncio.run(main())
+```
+
+# Current functionality
+
+ - Parse Google Maps
+ - Parse Yelp
+ - Export collected data to a CSV file
+
+# Expectations of this project:
+
+ - Parse Google Maps and Yelp for telephone number, email, address and other information by given keyword
+ - Somehow parse search results in Google Search for the same information using regex or other algorithm
+ - Export all parsed data to csv or excel
+ - For parsed emails send a message, which will be prevented from going to spam
+ - For parsed telephone numbers send an SMS, which will b prevented from going to spam as well
diff --git a/__pycache__/bufferization.cpython-38.pyc b/__pycache__/bufferization.cpython-38.pyc
deleted file mode 100644
index ce02117..0000000
Binary files a/__pycache__/bufferization.cpython-38.pyc and /dev/null differ
diff --git a/__pycache__/config.cpython-38.pyc b/__pycache__/config.cpython-38.pyc
deleted file mode 100644
index 35ba21c..0000000
Binary files a/__pycache__/config.cpython-38.pyc and /dev/null differ
diff --git a/__pycache__/driver_retry.cpython-38.pyc b/__pycache__/driver_retry.cpython-38.pyc
deleted file mode 100644
index cc0ec1e..0000000
Binary files a/__pycache__/driver_retry.cpython-38.pyc and /dev/null differ
diff --git a/__pycache__/locators.cpython-38.pyc b/__pycache__/locators.cpython-38.pyc
deleted file mode 100644
index be993d6..0000000
Binary files a/__pycache__/locators.cpython-38.pyc and /dev/null differ
diff --git a/__pycache__/logger_config.cpython-38.pyc b/__pycache__/logger_config.cpython-38.pyc
deleted file mode 100644
index 9e9cc61..0000000
Binary files a/__pycache__/logger_config.cpython-38.pyc and /dev/null differ
diff --git a/__pycache__/my_config.cpython-38.pyc b/__pycache__/my_config.cpython-38.pyc
deleted file mode 100644
index c05d077..0000000
Binary files a/__pycache__/my_config.cpython-38.pyc and /dev/null differ
diff --git a/__pycache__/webdriver.cpython-38.pyc b/__pycache__/webdriver.cpython-38.pyc
deleted file mode 100644
index 5c5781a..0000000
Binary files a/__pycache__/webdriver.cpython-38.pyc and /dev/null differ
diff --git a/app/__pycache__/app.cpython-38.pyc b/app/__pycache__/app.cpython-38.pyc
deleted file mode 100644
index dd73b8c..0000000
Binary files a/app/__pycache__/app.cpython-38.pyc and /dev/null differ
diff --git a/app/__pycache__/config.cpython-38.pyc b/app/__pycache__/config.cpython-38.pyc
deleted file mode 100644
index 37727e7..0000000
Binary files a/app/__pycache__/config.cpython-38.pyc and /dev/null differ
diff --git a/app/__pycache__/crypto.cpython-38.pyc b/app/__pycache__/crypto.cpython-38.pyc
deleted file mode 100644
index 497c7b0..0000000
Binary files a/app/__pycache__/crypto.cpython-38.pyc and /dev/null differ
diff --git a/app/__pycache__/forms.cpython-38.pyc b/app/__pycache__/forms.cpython-38.pyc
deleted file mode 100644
index 4c41f63..0000000
Binary files a/app/__pycache__/forms.cpython-38.pyc and /dev/null differ
diff --git a/app/__pycache__/models.cpython-38.pyc b/app/__pycache__/models.cpython-38.pyc
deleted file mode 100644
index dd28994..0000000
Binary files a/app/__pycache__/models.cpython-38.pyc and /dev/null differ
diff --git a/app/__pycache__/views.cpython-38.pyc b/app/__pycache__/views.cpython-38.pyc
deleted file mode 100644
index 690ca8c..0000000
Binary files a/app/__pycache__/views.cpython-38.pyc and /dev/null differ
diff --git a/app/db/lg.db b/app/db/lg.db
deleted file mode 100644
index 0dff396..0000000
Binary files a/app/db/lg.db and /dev/null differ
diff --git a/archived/LICENSE b/archived/LICENSE
new file mode 100644
index 0000000..2b52355
--- /dev/null
+++ b/archived/LICENSE
@@ -0,0 +1,21 @@
+MIT License
+
+Copyright (c) 2022 Madi Shaiken
+
+Permission is hereby granted, free of charge, to any person obtaining a copy
+of this software and associated documentation files (the "Software"), to deal
+in the Software without restriction, including without limitation the rights
+to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+copies of the Software, and to permit persons to whom the Software is
+furnished to do so, subject to the following conditions:
+
+The above copyright notice and this permission notice shall be included in all
+copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
+SOFTWARE.
diff --git a/archived/README.md b/archived/README.md
new file mode 100644
index 0000000..6bc6767
--- /dev/null
+++ b/archived/README.md
@@ -0,0 +1,2 @@
+# Lead-Generation
+Python script, which empowers people with no programming background to generate robust leads on a mass scale. This repo will be compiled of various versatile techniques used in lead generation implemented in python.
diff --git a/app/app.py b/archived/app/app.py
similarity index 99%
rename from app/app.py
rename to archived/app/app.py
index 2c704f5..d8534b3 100644
--- a/app/app.py
+++ b/archived/app/app.py
@@ -13,4 +13,3 @@ app.config.from_object(Config)
db = SQLAlchemy(app)
migrate = Migrate(app, db)
-
diff --git a/app/config.py b/archived/app/config.py
similarity index 88%
rename from app/config.py
rename to archived/app/config.py
index 068158d..a1d0ac2 100644
--- a/app/config.py
+++ b/archived/app/config.py
@@ -6,11 +6,11 @@ class Config(object):
DEBUG = True
WTF_CSRF_SECRET_KEY = 'a random string'
-
+
SECRET_KEY = '6554410059521c8b5426eba21e54a7a193aaf30e67645e8c24a2f6eb8779790711a4ca0b9d62ef43e79caddf1e5a123808a4ea370f69c59cd64003e1c9ac8659'
SQLALCHEMY_TRACK_MODIFICATIONS = False
-
+
if platform.startswith('w'):
# For Windows
SQLALCHEMY_DATABASE_URI = r'sqlite:///C:\Users\khova\Desktop\Python\Code\Lead Generation\Lead-Generation\app\db\lg.db'
@@ -19,12 +19,11 @@ class Config(object):
# For Ubuntu
SQLALCHEMY_DATABASE_URI = 'mysql+mysqlconnector://root:1234@localhost/lg'
-
- SESSION_TYPE = 'redis' # redis, memcached, filesystem or mongodb
+ SESSION_TYPE = 'redis' # redis, memcached, filesystem or mongodb
PERMANENT_SESSION_LIFETIME = timedelta(days=90)
RECAPTCHA_PUBLIC_KEY = '6LeENmoaAAAAAH1dMElRkLRGE2UiSP-CK831656U'
RECAPTCHA_PRIVATE_KEY = '6LeENmoaAAAAANfgmGgA9Of_IPlJWtppMLMoMoV4'
# RECAPTCHA_THEME = 'dark'
# RECAPTCHA_TYPE = 'audio'
- # RECAPTCHA_SIZE = 'compact'
\ No newline at end of file
+ # RECAPTCHA_SIZE = 'compact'
diff --git a/app/crypto.py b/archived/app/crypto.py
similarity index 100%
rename from app/crypto.py
rename to archived/app/crypto.py
diff --git a/app/forms.py b/archived/app/forms.py
similarity index 65%
rename from app/forms.py
rename to archived/app/forms.py
index 630c6fb..c96a10e 100644
--- a/app/forms.py
+++ b/archived/app/forms.py
@@ -4,10 +4,14 @@ from wtforms.validators import DataRequired, Length, EqualTo
class UserRegistrationForm(FlaskForm):
- first_name = StringField('First Name', [DataRequired(), Length(4, 100, 'First name should consist of 4-200 characters')])
- last_name = StringField('Last Name', [DataRequired(), Length(4, 100, 'Last name should consist of 4-200 characters')])
- username = StringField('Username (4-30)', [DataRequired(), Length(4, 30, 'Username should consist of 4-30 characters')])
- password = PasswordField('Password (6-40)', [DataRequired(), Length(6, 40, 'Passowrd should consist of 6-40 characters')])
+ first_name = StringField('First Name', [DataRequired(), Length(
+ 4, 100, 'First name should consist of 4-200 characters')])
+ last_name = StringField('Last Name', [DataRequired(), Length(
+ 4, 100, 'Last name should consist of 4-200 characters')])
+ username = StringField('Username (4-30)', [DataRequired(), Length(
+ 4, 30, 'Username should consist of 4-30 characters')])
+ password = PasswordField(
+ 'Password (6-40)', [DataRequired(), Length(6, 40, 'Passowrd should consist of 6-40 characters')])
repeat_password = PasswordField('Repeat Password')
accept_terms = BooleanField('I accept the TOS', [DataRequired()])
recaptcha = RecaptchaField()
@@ -19,11 +23,13 @@ class UserRegistrationForm(FlaskForm):
class UserLoginForm(FlaskForm):
- username = StringField('Username (4-30)', validators=[DataRequired(), Length(4, 30, 'Username should consist of 4-30 characters')])
- password = PasswordField('Password (6-40)', validators=[DataRequired(), Length(6, 40, 'Passowrd should consist of 6-40 characters')])
+ username = StringField('Username (4-30)', validators=[DataRequired(), Length(
+ 4, 30, 'Username should consist of 4-30 characters')])
+ password = PasswordField('Password (6-40)', validators=[DataRequired(), Length(
+ 6, 40, 'Passowrd should consist of 6-40 characters')])
recaptcha = RecaptchaField()
submit = SubmitField('Sign In')
-
+
@property
def visible_fields(self):
- return (self.username, self.password)
\ No newline at end of file
+ return (self.username, self.password)
diff --git a/app/main.py b/archived/app/main.py
similarity index 51%
rename from app/main.py
rename to archived/app/main.py
index eaf8bc9..5ad3ef6 100644
--- a/app/main.py
+++ b/archived/app/main.py
@@ -2,5 +2,5 @@ from app import app, db
from views import *
-if __name__ == '__main__':
- app.run()
\ No newline at end of file
+if __name__ == '__main__':
+ app.run()
diff --git a/app/models.py b/archived/app/models.py
similarity index 95%
rename from app/models.py
rename to archived/app/models.py
index 5068be6..4d20d0a 100644
--- a/app/models.py
+++ b/archived/app/models.py
@@ -4,7 +4,6 @@ from sqlalchemy.sql import func
from crypto import encrypt
-
class ReprMixin:
def __repr__(self):
return f'<{self.__class__.__name__} object id: {self.id}, username: {self.username}, created at: {self.created}>'
@@ -14,12 +13,12 @@ class SecretMixin:
@classmethod
def create(cls, pwd, **kwargs):
obj = cls(password=encrypt(pwd), **kwargs)
-
+
db.session.add(obj)
db.session.commit()
print(f'Created: {obj}')
-
+
return obj
@classmethod
@@ -28,11 +27,10 @@ class SecretMixin:
print(f'User {username} found: {obj}')
if obj and obj.password == encrypt(pwd):
return obj
-
+
@classmethod
def update_pwd(cls, id, pwd):
- pass
-
+ pass
class User(ReprMixin, SecretMixin, db.Model):
diff --git a/app/static/css/fonts.css b/archived/app/static/css/fonts.css
similarity index 100%
rename from app/static/css/fonts.css
rename to archived/app/static/css/fonts.css
diff --git a/app/templates/base.html b/archived/app/templates/base.html
similarity index 100%
rename from app/templates/base.html
rename to archived/app/templates/base.html
diff --git a/app/templates/error.html b/archived/app/templates/error.html
similarity index 100%
rename from app/templates/error.html
rename to archived/app/templates/error.html
diff --git a/app/templates/index.html b/archived/app/templates/index.html
similarity index 100%
rename from app/templates/index.html
rename to archived/app/templates/index.html
diff --git a/app/templates/login.html b/archived/app/templates/login.html
similarity index 100%
rename from app/templates/login.html
rename to archived/app/templates/login.html
diff --git a/app/templates/register.html b/archived/app/templates/register.html
similarity index 100%
rename from app/templates/register.html
rename to archived/app/templates/register.html
diff --git a/app/templates/subscriptions.html b/archived/app/templates/subscriptions.html
similarity index 100%
rename from app/templates/subscriptions.html
rename to archived/app/templates/subscriptions.html
diff --git a/app/views.py b/archived/app/views.py
similarity index 98%
rename from app/views.py
rename to archived/app/views.py
index 1c46cb7..60e8fe8 100644
--- a/app/views.py
+++ b/archived/app/views.py
@@ -5,6 +5,7 @@ from models import User
from functools import wraps
+
bad_user_agents = [
'request',
'robot',
@@ -88,7 +89,7 @@ def login():
if request.method == 'GET':
return render_template('login.html', form=form)
- if form.validate_on_submit():
+ if form.validate_on_submit():
user = User.validate_user(form.username.data, form.password.data)
if user:
session['user_id'] = user.id
diff --git a/bufferization.py b/archived/bufferization.py
similarity index 91%
rename from bufferization.py
rename to archived/bufferization.py
index a18cf83..88079db 100644
--- a/bufferization.py
+++ b/archived/bufferization.py
@@ -2,13 +2,14 @@ import csv
from logger_config import get_logger
+
logger = get_logger('buffer')
class Buffer:
lower_limit = 5
upper_limit = 200
- fn = ['Title','Address','WebSite','PhoneNumber']
+ fn = ['Title', 'Address', 'WebSite', 'PhoneNumber']
def __init__(self, filename: str = 'leads.csv', buffer_size: int = 5):
if not filename.endswith('.csv'):
@@ -34,9 +35,9 @@ class Buffer:
self._data.clear()
def store(self, info: dict):
- if info.get('Title') and not info in self._data:
+ if info.get('Title') and not info in self._data:
self._data.append(info)
if len(self._data) >= self._buffer_size:
self.dump()
else:
- logger.warning('Got duplicate or empty dictionary %s', info)
\ No newline at end of file
+ logger.warning('Got duplicate or empty dictionary %s', info)
diff --git a/emails/config.py b/archived/emails/config.py
similarity index 100%
rename from emails/config.py
rename to archived/emails/config.py
diff --git a/emails/extract_emails.py b/archived/emails/extract_emails.py
similarity index 100%
rename from emails/extract_emails.py
rename to archived/emails/extract_emails.py
diff --git a/emails/hunterio.py b/archived/emails/hunterio.py
similarity index 100%
rename from emails/hunterio.py
rename to archived/emails/hunterio.py
diff --git a/emails/locators.py b/archived/emails/locators.py
similarity index 100%
rename from emails/locators.py
rename to archived/emails/locators.py
diff --git a/google-maps/extractor.py b/archived/google-maps/extractor.py
similarity index 97%
rename from google-maps/extractor.py
rename to archived/google-maps/extractor.py
index 95bf589..8712218 100644
--- a/google-maps/extractor.py
+++ b/archived/google-maps/extractor.py
@@ -1,5 +1,7 @@
import re
import asyncio
+from webdriver import Webdriver
+from logger_config import get_logger
from locators import *
from my_config import *
@@ -12,8 +14,6 @@ from pyppeteer.errors import PageError
import sys
sys.path.append('..')
-from logger_config import get_logger
-from webdriver import Webdriver
logger = get_logger('webdriver.google_maps')
@@ -30,27 +30,26 @@ class GoogleMaps(Webdriver):
:return: returns nothing
'''
-
if not '_page' in self.__dict__:
raise ValueError(
'Initialize the browser before searching by `await *.init_broswser()`')
-
+
if location and keyword and not url:
logger.debug(
'Working based on location `%s` and keyword `%s`', location, keyword)
self._location = location
self._keyword = keyword
await self._locate()
-
+
elif not (location and keyword) and url:
await self._jump(url, search_xpath)
logger.debug('Working based on given URL `%s`', url)
-
+
else:
await self._shut_browser()
raise ValueError(
f'Specify location and keyword. Otherwise, specify URL only. Got location `{location}`, keyword `{keyword}` and URL `{url}`')
-
+
await self._scrape()
await self._shut_browser()
@@ -117,7 +116,7 @@ class GoogleMaps(Webdriver):
back = (await self._page.xpath(back_xpath))[0]
await self._do_retry(back.click, result_xpath)
except:
- logger.warning('Error occured',exc_info=True)
+ logger.warning('Error occured', exc_info=True)
next_button = (await self._page.xpath(next_xpath))[0]
if not next_button:
diff --git a/google-maps/locators.py b/archived/google-maps/locators.py
similarity index 79%
rename from google-maps/locators.py
rename to archived/google-maps/locators.py
index fde8e0e..816a8d3 100644
--- a/google-maps/locators.py
+++ b/archived/google-maps/locators.py
@@ -1,8 +1,6 @@
button_xpath = '//*[@class="iRxY3GoUYUY__button gm2-hairline-border section-action-chip-button"]'
search_xpath = '//*[@role="gridcell"]'
next_xpath = '//*[@id="n7lv7yjyC35__section-pagination-button-next"]'
-# check_xpath = '//*[@src="//www.gstatic.com/images/icons/material/system_gm/2x/share_gm_blue_18dp.png"]'
place_xpath = '//*[@class="section-hero-header-title-title GLOBAL__gm2-headline-5"]'
result_xpath = '//*[@class="section-result-content"]'
back_xpath = '//*[@class="section-back-to-list-button blue-link noprint"]'
-
diff --git a/google-maps/my_config.py b/archived/google-maps/my_config.py
similarity index 100%
rename from google-maps/my_config.py
rename to archived/google-maps/my_config.py
diff --git a/linkedin/extractor.py b/archived/linkedin/extractor.py
similarity index 79%
rename from linkedin/extractor.py
rename to archived/linkedin/extractor.py
index ddfa9fd..9089dbd 100644
--- a/linkedin/extractor.py
+++ b/archived/linkedin/extractor.py
@@ -1,5 +1,6 @@
-import re
import asyncio
+from webdriver import Webdriver
+from logger_config import get_logger
from locators import *
from my_config import *
@@ -10,12 +11,9 @@ from random import uniform, randint
from bs4 import BeautifulSoup
from fake_useragent import UserAgent
-from pyppeteer.errors import PageError
import sys
sys.path.append('..')
-from logger_config import get_logger
-from webdriver import Webdriver
logger = get_logger('webdriver.linkedin')
@@ -25,16 +23,17 @@ ua = UserAgent()
class LinkedIn(Webdriver):
-
def __init__(self, email=None, pwd=None):
if not (email and pwd):
try:
self.email = environ.get('linkedin_mail')
self.pwd = environ.get('linkedin_pwd')
except:
- logger.fatal('Cannot get email and password to login in LinkedIn account', exc_info=True)
+ logger.fatal(
+ 'Cannot get email and password to login in LinkedIn account', exc_info=True)
else:
- logger.warning('Specify email and pwd as local variables `linkedin_mail` and `linkedin_pwd` for your privacy')
+ logger.warning(
+ 'Specify email and pwd as local variables `linkedin_mail` and `linkedin_pwd` for your privacy')
self.email = email
self.pwd = pwd
@@ -42,13 +41,14 @@ class LinkedIn(Webdriver):
await self._page.goto(linkedin_link)
sleep(uniform(1.7, 3.4))
await asyncio.wait([
- self._page.click(login_locator, {'button':'left','delay': uniform(50, 100)}),
+ self._page.click(
+ login_locator, {'button': 'left', 'delay': uniform(50, 100)}),
self._page.waitForNavigation()
])
sleep(uniform(1.7, 3.4))
await self.type_(self.email)
-
+
await self._page.click(pwd_locator)
sleep(uniform(0.3, 0.9))
await self.type_(self.pwd)
@@ -62,7 +62,7 @@ class LinkedIn(Webdriver):
async def search_title(self, title):
# await self._page.click(search_locator)
# sleep(uniform(2, 3))
-
+
# await self.type_(title)
# sleep(uniform(0.5, 1))
@@ -90,7 +90,8 @@ class LinkedIn(Webdriver):
soup = BeautifulSoup(html, 'html.parser')
position = soup.find(string=head_regex)
- name = position.parent.parent.parent.find(attrs={'aria-hidden':True}).text
+ name = position.parent.parent.parent.find(
+ attrs={'aria-hidden': True}).text
return (position, name)
@@ -98,9 +99,9 @@ class LinkedIn(Webdriver):
pass
async def type_(self, string):
- for letter in string:
- self._page.keyboard.type(letter)
- sleep(uniform(0.1, 0.3))
+ for letter in string:
+ self._page.keyboard.type(letter)
+ sleep(uniform(0.1, 0.3))
async def behave_like_human(self):
- pass
\ No newline at end of file
+ pass
diff --git a/linkedin/locators.py b/archived/linkedin/locators.py
similarity index 85%
rename from linkedin/locators.py
rename to archived/linkedin/locators.py
index d4f990f..274086f 100644
--- a/linkedin/locators.py
+++ b/archived/linkedin/locators.py
@@ -4,4 +4,4 @@ show_pwd_locator = 'span#password-visibility-toggle'
search_locator = 'input.search-global-typeahead__input.always-show-placeholder'
companies_locator = '.entity-result__title-text a'
-select_bar_xpath = '//*[@d="M8 11L3 6h10z"]'
\ No newline at end of file
+select_bar_xpath = '//*[@d="M8 11L3 6h10z"]'
diff --git a/linkedin/my_config.py b/archived/linkedin/my_config.py
similarity index 53%
rename from linkedin/my_config.py
rename to archived/linkedin/my_config.py
index f5d57fe..6427b94 100644
--- a/linkedin/my_config.py
+++ b/archived/linkedin/my_config.py
@@ -1,6 +1,8 @@
import re
+
__all__ = ['linkedin_link', 'head_regex']
linkedin_link = 'https://www.linkedin.com/home'
-head_regex = re.compile(r'(ceo|owner|manager|chief|manager)', flags=re.IGNORECASE)
\ No newline at end of file
+head_regex = re.compile(
+ r'(ceo|owner|manager|chief|manager)', flags=re.IGNORECASE)
diff --git a/linkedin/test_regex.py b/archived/linkedin/test_regex.py
similarity index 52%
rename from linkedin/test_regex.py
rename to archived/linkedin/test_regex.py
index 396cdbe..5d74e1f 100644
--- a/linkedin/test_regex.py
+++ b/archived/linkedin/test_regex.py
@@ -1,12 +1,14 @@
-import re
+import re
from bs4 import BeautifulSoup
-head_regex = re.compile(r'(ceo|owner|manager|chief|manager)', flags=re.IGNORECASE)
+
+head_regex = re.compile(
+ r'(ceo|owner|manager|chief|manager)', flags=re.IGNORECASE)
html = open('test.html', encoding='utf-8').read()
soup = BeautifulSoup(html, 'html.parser')
head_man = soup.find(string=head_regex)
print(head_man)
-name = head_man.parent.parent.parent.find(attrs={'aria-hidden':True}).text
+name = head_man.parent.parent.parent.find(attrs={'aria-hidden': True}).text
print(name)
diff --git a/logger_config.py b/archived/logger_config.py
similarity index 99%
rename from logger_config.py
rename to archived/logger_config.py
index 4ceac7f..c8fcdb9 100644
--- a/logger_config.py
+++ b/archived/logger_config.py
@@ -8,7 +8,7 @@ def get_logger(name: str):
:param name: Name of the logger
:return: returns full-fledged logger
'''
-
+
if isinstance(name, str):
f_formatter = logging.Formatter(
diff --git a/webdriver.py b/archived/webdriver.py
similarity index 94%
rename from webdriver.py
rename to archived/webdriver.py
index 7572586..0aa81bf 100644
--- a/webdriver.py
+++ b/archived/webdriver.py
@@ -14,12 +14,11 @@ from bufferization import Buffer
logger = get_logger('webdriver')
-
class Webdriver:
_ua = UserAgent()
_buf = Buffer()
_monitor = get_monitors()[0]
- _viewport = {'width':_monitor.width, 'height': _monitor.height,}
+ _viewport = {'width': _monitor.width, 'height': _monitor.height, }
async def init_browser(self, hidden=False, language='en-gb'):
'''
@@ -55,7 +54,8 @@ class Webdriver:
async def _do_retry(self, operation, xpath, dest=None, retries=0):
if retries == 10:
- raise SystemError('Max 10 retries exceeded when clicking the place')
+ raise SystemError(
+ 'Max 10 retries exceeded when clicking the place')
try:
if dest:
await operation(dest)
@@ -68,7 +68,7 @@ class Webdriver:
await self._do_retry(operation, xpath, dest, retries + 1)
except:
logger.error('Exception found when retrying', exc_info=True)
-
+
async def _jump(self, url, xpath):
try:
await self._page.goto(url)
@@ -78,5 +78,3 @@ class Webdriver:
logger.warning('Webdriver got invalid URL for lead generation')
await self._shut_browser()
raise ValueError('Got invalid URL for google maps')
-
-
diff --git a/yellow-pages/extractor.py b/archived/yellow-pages/extractor.py
similarity index 78%
rename from yellow-pages/extractor.py
rename to archived/yellow-pages/extractor.py
index ec532f8..ac18f8c 100644
--- a/yellow-pages/extractor.py
+++ b/archived/yellow-pages/extractor.py
@@ -1,6 +1,8 @@
import json
-import requests
import asyncio
+import requests
+from logger_config import get_logger
+from webdriver import Webdriver
from pyppeteer.errors import TimeoutError
@@ -14,15 +16,14 @@ from my_config import *
import sys
sys.path.append('..')
-from webdriver import Webdriver
-from logger_config import get_logger
logger = get_logger('webdriver.yelp')
ua = UserAgent()
-_found = lambda loc, desc: not f'No Results for {desc}' in requests.get(yelp.format(desc, loc), headers={'User-Agent': ua.random}).text
+def _found(loc, desc): return not f'No Results for {desc}' in requests.get(
+ yelp.format(desc, loc), headers={'User-Agent': ua.random}).text
class Yelp(Webdriver):
@@ -37,11 +38,13 @@ class Yelp(Webdriver):
:return: returns nothing
'''
if not '_page' in self.__dict__:
- raise ValueError('Initialize the browser before searching by `await *.init_broswser()`')
+ raise ValueError(
+ 'Initialize the browser before searching by `await *.init_broswser()`')
if loc and desc and not url:
if _found(loc, desc):
- logger.debug('Working based on location `%s` and keyword `%s`', loc, desc)
+ logger.debug(
+ 'Working based on location `%s` and keyword `%s`', loc, desc)
self._data = [desc, loc]
await self._locate()
else:
@@ -53,7 +56,8 @@ class Yelp(Webdriver):
else:
await self._shut_browser()
- raise ValueError('Specify only location and description or only URL for yelp page')
+ raise ValueError(
+ 'Specify only location and description or only URL for yelp page')
await self._scrape()
await self._shut_browser()
@@ -77,9 +81,11 @@ class Yelp(Webdriver):
soup = BeautifulSoup(html, 'html.parser')
try:
- raw_data = json.loads(soup.find('script', attrs=json_attrs).string)
+ raw_data = json.loads(
+ soup.find('script', attrs=json_attrs).string)
except AttributeError:
- logger.warning('Could not locate `raw_data` skipping current page')
+ logger.warning(
+ 'Could not locate `raw_data` skipping current page')
return
try:
@@ -89,11 +95,11 @@ class Yelp(Webdriver):
try:
data = {
- 'WebSite': website,
- 'Title': raw_data['name'],
- 'PhoneNumber': raw_data['telephone'],
- 'Address': ', '.join(list(raw_data['address'].values())),
- }
+ 'WebSite': website,
+ 'Title': raw_data['name'],
+ 'PhoneNumber': raw_data['telephone'],
+ 'Address': ', '.join(list(raw_data['address'].values())),
+ }
logger.debug('Extracted: %s', list(data.values()))
self._buf.store(data)
except Exception:
@@ -120,7 +126,7 @@ class Yelp(Webdriver):
parse(await self._page.content())
await self._page.goBack()
- sleep(uniform(1,2))
+ sleep(uniform(1, 2))
await self._page.waitForXPath(results_xpath, {'visible': True})
next_button = await self._page.xpath(next_xpath)
diff --git a/yellow-pages/locators.py b/archived/yellow-pages/locators.py
similarity index 100%
rename from yellow-pages/locators.py
rename to archived/yellow-pages/locators.py
diff --git a/yellow-pages/my_config.py b/archived/yellow-pages/my_config.py
similarity index 95%
rename from yellow-pages/my_config.py
rename to archived/yellow-pages/my_config.py
index 7e54410..b28419a 100644
--- a/yellow-pages/my_config.py
+++ b/archived/yellow-pages/my_config.py
@@ -1 +1 @@
-yelp = 'https://www.yelp.com/search?find_desc={}&find_loc={}&ns=1'
\ No newline at end of file
+yelp = 'https://www.yelp.com/search?find_desc={}&find_loc={}&ns=1'
diff --git a/google-maps/__pycache__/config.cpython-38.pyc b/google-maps/__pycache__/config.cpython-38.pyc
deleted file mode 100644
index e190f10..0000000
Binary files a/google-maps/__pycache__/config.cpython-38.pyc and /dev/null differ
diff --git a/google-maps/__pycache__/extractor.cpython-38.pyc b/google-maps/__pycache__/extractor.cpython-38.pyc
deleted file mode 100644
index 7cdfbca..0000000
Binary files a/google-maps/__pycache__/extractor.cpython-38.pyc and /dev/null differ
diff --git a/google-maps/__pycache__/locators.cpython-38.pyc b/google-maps/__pycache__/locators.cpython-38.pyc
deleted file mode 100644
index eb8b3da..0000000
Binary files a/google-maps/__pycache__/locators.cpython-38.pyc and /dev/null differ
diff --git a/google-maps/__pycache__/my_config.cpython-38.pyc b/google-maps/__pycache__/my_config.cpython-38.pyc
deleted file mode 100644
index 5ab18e2..0000000
Binary files a/google-maps/__pycache__/my_config.cpython-38.pyc and /dev/null differ
diff --git a/google-maps/leads.csv b/google-maps/leads.csv
deleted file mode 100644
index 6828319..0000000
--- a/google-maps/leads.csv
+++ /dev/null
@@ -1,286 +0,0 @@
-Title,Address,WebSite,PhoneNumber
-Kazzhol Hotel Astana,"Balqantau St 213, Nur-Sultan 010000",hotelkazzhol.kz,8 (7172) 40 52 52
-Eco Apart Hotel Astana,"Qabanbay Batyr Avenue h. 58B pod.3 ap.50, Nur-Sultan 020000",ecoaparthotel.kz,8 (708) 311 6378
-Hampton by Hilton Astana Triumphal Arch,"Prospekt Mangilik Yel. 43A, Nur-Sultan 010000",hilton.com,8 (7172) 70 35 00
-Diplomat Hotel & Business Center,"Dinmukhamed Qonayev St 29/1, Nur-Sultan 010000",diplomathotel.kz,8 (7172) 55 00 01
-ibis Nur-Sultan,"Tauelsizdik Ave 38, Nur-Sultan 010000",all.accor.com,8 (7172) 70 82 00
-King Hotel Astana,"Shokan Valikhanov Street 7, Nur-Sultan 010000",kinghotelastana.kz,8 (7172) 70 57 05
-Astana Best Hostel,"Zhk Infiniti 2, Qabanbay Batyr Ave 42, Nur-Sultan 010000",-,8 (7172) 28 68 29
-Park Avenue,"Prospekt Mangilik Yel. 37/1, Nur-Sultan 010000",hotelparkavenue.kz,8 (701) 052 2266
-Alanda Hotel,"Tauelsizdik Ave 33, Nur-Sultan 020000",alanda-hotel.kz,8 (7172) 72 93 09
-MyLINE relax complex,"Bayimbet Maylin St 37 А, Nur-Sultan 010000",-,-
-Caps Lock Future Hotel,"Dinmukhamed Qonayev St 29, Nur-Sultan 020000",capslock.kz,8 (771) 777 8111
-Hostel S size,"Ул. Бухар Жырау 36, Nur-Sultan 020000",-,8 (775) 103 3337
-Antalya Hotel,"Balqantau St 94, Nur-Sultan 020000",antalyahotel.kz,8 (705) 555 3772
-San Marino,"Zhumabek Tashenov St 17, Nur-Sultan 010000",-,8 (7172) 41 92 92
-Come INN Hotel,"Арганаты 13, Nur-Sultan 010000",comeinnhotel.com,8 (7172) 57 93 26
-HELLO INN | Ex-Hostelland Astana,"Zhumabek Tashenov St 21/1, Nur-Sultan 010000",helloinn.kz,8 (778) 999 0932
-Manhattan Astana Hotel,"Tauelsizdik Ave 5, Nur-Sultan 020000",manhattanhotel.kz,8 (7172) 36 15 36
-Kapsula Hotel Astana,"Qabanbay Batyr Ave 7/3, Nur-Sultan 020000",kapsula.kz,8 (7172) 50 74 04
-Jelsomino Boutique Hotel,"Amangeldi Imanov Street 26, Nur-Sultan 010000",jelsominohotel.kz,8 (701) 100 0201
-G Empire,"Abay Ave 63, Nur-Sultan 010000",gempire.kz,8 (7172) 40 55 01
-Red Village Hotel,"ул. Мустафина 1/1, Nur-Sultan 010009",redvillagehotel.kz,8 (7172) 57 08 08
-MAXX inn,"46/1, Qabanbay Batyr Ave, Nur-Sultan 010000",-,8 (701) 214 0106
-Alanda Hotel,"Tauelsizdik Ave 33, Nur-Sultan 020000",alanda-hotel.kz,8 (7172) 72 93 09
-Hostel Forum,"Qabanbay Batyr Ave 9, Nur-Sultan 010000",hostelforum.business.site,8 (778) 950 8631
-Inzhu,"Aleksandr Barayev St 9, Nur-Sultan 010000",-,8 (7172) 30 23 31
-Astana Plaza,"Zhengis Ave 27А, Nur-Sultan 020000",-,8 (7172) 44 48 61
-Makenty house hotel,"Qabanbay Batyr Ave 40, Nur-Sultan 010000",-,8 (778) 135 5032
-Alanda Hotel,"Tauelsizdik Ave 33, Nur-Sultan 020000",alanda-hotel.kz,8 (7172) 72 93 09
-Bozok,"Abay Ave, Nur-Sultan 010000",-,-
-Sultan Beybars,"Abylay Khan Ave 16, Nur-Sultan 010000",sultanbeibarys.com,8 (7172) 56 21 80
-NCityHotel,"Sarayshyq St 40, Nur-Sultan 020000",-,-
-San Marino,"Zhumabek Tashenov St 17, Nur-Sultan 010000",-,8 (7172) 41 92 92
-Sky House Astana,"Dostyq St 5, Nur-Sultan 020000",-,-
-miniГостиница,"Qabanbay Batyr Ave 29, Nur-Sultan 020000",-,-
-Rixos President Astana,"Kunayeva 7, Nur-Sultan 010000",all.accor.com,8 (7172) 24 50 50
-Kapsula Hotel Astana,"Qabanbay Batyr Ave 7/3, Nur-Sultan 020000",kapsula.kz,8 (7172) 50 74 04
-Jelsomino Boutique Hotel,"Amangeldi Imanov Street 26, Nur-Sultan 010000",jelsominohotel.kz,8 (701) 100 0201
-Park Inn by Radisson Astana Hotel,"Saryarka Avenue 8, Nur-Sultan 010000",radissonhotels.com,8 (7172) 67 00 00
-Wyndham Garden Astana,"Хусейн бен Талал 25, Nur-Sultan 010000",wyndhamgardenastana.com,8 (775) 030 4444
-G Empire,"Abay Ave 63, Nur-Sultan 010000",gempire.kz,8 (7172) 40 55 01
-Orynbor,"Prospekt Mangilik Yel. 50, Nur-Sultan 010000",-,8 (707) 101 7457
-Royal Park Hotel & Spa,"Kerey and Zhanibek Khans St 18, Nur-Sultan 010000",-,8 (7172) 42 22 01
-Отель The One - Гостиница в Астане - Нур-султане,"Qadyrghali Zhalayyri St, Nur-Sultan 020000",theone-astana.kz,8 (771) 222 2400
-Gostinitsa Atlas,"Vol'fgan Gete St 12/1, Nur-Sultan 010000",-,8 (778) 699 7626
-Manhattan Astana Hotel,"Tauelsizdik Ave 5, Nur-Sultan 020000",manhattanhotel.kz,8 (7172) 36 15 36
-ibis Nur-Sultan,"Tauelsizdik Ave 38, Nur-Sultan 010000",all.accor.com,8 (7172) 70 82 00
-Manhattan Astana Hotel,"Tauelsizdik Ave 5, Nur-Sultan 020000",manhattanhotel.kz,8 (7172) 36 15 36
-East Astana Hotel,"Auezova street 12, Nur-Sultan 010000",easthotelastana.kz,8 (775) 333 0055
-Hotel Inju Hills,"Mukhtar Auezov Street 50/1, Nur-Sultan 010000",-,8 (771) 161 2290
-Sheraton Nur-Sultan Hotel,"Syghanaq St 16/1, Nur-Sultan 010000",marriott.com,8 (7172) 70 05 55
-Residenz,"Petrov St 21/1, Nur-Sultan 010000",-,8 (7172) 34 22 49
-Hotel Inju on Baraeva 9,"Aleksandr Barayev St 9, Nur-Sultan 001000",-,8 (776) 230 2335
-Prestige,"Zheltoqsan St 1, Nur-Sultan 010000",-,8 (7172) 32 51 81
-"The Ritz-Carlton, Astana","Dostyk Street 16, Nur-Sultan 010016",ritzcarlton.com,8 (7172) 73 40 00
-ЖК The Arc Residence,"ул. Мәңгілік Ел, Nur-Sultan 020000",thearc.kz,8 (777) 722 1010
-Hostel Nomad4x4,"Нуртаса Ондасынова, Чубары микрорайон 52А, Nur-Sultan 010001",hostel-nomad4x4.business.site,8 (708) 865 1940
-Elite Hostel,"Shokan Valikhanov Street 9/2, Nur-Sultan 010000",-,8 (775) 653 0013
-Goldman Empire,"Сатбаев көшесі 13/1, Nur-Sultan 010000",goldmanempire.kz,8 (7172) 68 20 20
-"Гостевой дом ""Altyn hotel#","Tsentr Mezhdunarodnoy Torgovli, Dostyq St 1, Nur-Sultan 020000",-,8 (702) 250 6000
-"Mini Hotel ""Prospect""","Syghanaq St 9, Nur-Sultan 020000",-,8 (701) 301 4074
-The St. Regis Astana,"Qabanbay Batyr Ave 1, Nur-Sultan 010000",marriott.com,8 (7172) 79 08 88
-Zhassybi Hotel,"Shokan Valikhanov Street 26, Nur-Sultan 010000",-,-
-Altyn-Adam Hotel,"Säken Seyfullin Street 38, Nur-Sultan 010000",-,-
-Lion,"Maskeu St 37/2, Nur-Sultan 010000",-,8 (7172) 39 46 20
-VIP House Apartments,"Dostyq St 5/1, Nur-Sultan 010000",-,8 (700) 375 5774
-Kazzhol Hotel Astana,"Balqantau St 213, Nur-Sultan 010000",hotelkazzhol.kz,8 (7172) 40 52 52
-Astina Inn Hotel,"Burabay St, Nur-Sultan 010000",astina.kz,8 (747) 909 9918
-Pulsar Mini-Hotel,"Tauelsizdik Ave 49/3, Nur-Sultan 010011",-,8 (771) 881 1100
-Elite Apart-Hotel,"Raqymjan Qoshqarbayev Avenue 10\1, Nur-Sultan 010017",eliteapart.kz,8 (7172) 76 97 33
-Mini-hotel Ulpan,"Khalel Dosmukhameduly St 11, Nur-Sultan 010000",hotelulpan.com,8 (7172) 24 03 44
-Отель Union Plaza,"Almaty St 1/1, Nur-Sultan 020000",-,8 (776) 425 2226
-Skyversal гостиница,"Sarayshyq St 40, Nur-Sultan 010000",-,8 (702) 136 2499
-"Radisson Hotel, Astana","Saryarka Avenue 4, Nur-Sultan 010000",radissonhotels.com,8 (7172) 67 07 77
-QonaQ hotel,"пр. Мәңгілік Ел дом 50 НП 19, Отель/кафе ""QonaQ"", Nur-Sultan 010000",-,8 (771) 099 0555
-Mini Hotel Bereket Dip,"Sarayshyq St 34, Nur-Sultan 010000",-,8 (707) 777 2310
-Astina Inn Hotel,"Burabay St, Nur-Sultan 010000",astina.kz,8 (747) 909 9918
-Tengri Hotel,"Bayimbet Maylin St 1A, Nur-Sultan 020000",tengrihotel.kz,8 (7172) 41 38 38
-Seven Inn Boutique Hotel,"36th St, Nur-Sultan 010000",-,8 (7172) 65 00 01
-ARENA Hotel,"ЖК""Sezim Qala, Turan Ave 55, Nur-Sultan 010000",-,-
-Gokai Hotel,"Bauyrzhan Momyshuly Ave 24, Nur-Sultan 010009",gokai-hotel.business.site,8 (705) 134 5533
-The Veil,"Q. Zhalayyri street, Nur-Sultan 010000",-,8 (701) 068 8800
-Torgay,"Qanysh Satbaev St 15, Nur-Sultan 020000",torgai.kz,8 (7172) 97 82 76
-Shaghyn,"Sauran Street 2, Nur-Sultan 020000",-,8 (701) 272 8773
-Nomad Hotel,"Kuyshi Dina St 25/1, Nur-Sultan 010000",-,-
-Гостиница QAINAR HOTEL,"Aqkent Ln дом 5А, Nur-Sultan 020000",-,8 (701) 543 0893
-Orion,"Alexander Pushkin St 3/2, Nur-Sultan 010000",orionhotel.kz,8 (701) 223 3636
-Astana Central Hotel,"м-н 2, 010000, Ayagoz St, Nur-Sultan",astanacentralhotel.kz,8 (7172) 99 99 22
-Alash Hotel,"Блок А, Bauyrzhan Momyshuly Ave 12, Nur-Sultan",-,8 (7172) 99 84 15
-"Апарт-отель ""MONACO"" Астана","Charles de Gaulle St 1, Nur-Sultan 010000",monacohotel.kz,8 (7172) 22 22 82
-Hostel Nomad4x4,"Нуртаса Ондасынова, Чубары микрорайон 52А, Nur-Sultan 010001",hostel-nomad4x4.business.site,8 (708) 865 1940
-Capital Hostel,"Novyy Mir, Dinmukhamed Qonayev St 35, Nur-Sultan 020000",-,8 (747) 295 4988
-Hotel Akkaiyn,"Zhayyq St 30, Nur-Sultan 020000",charleston.kz,8 (701) 888 1208
-Porta Fortuna,"Sembinov St, Nur-Sultan 010000",-,-
-Versal Hotel,"Sarayshyq St 40, Nur-Sultan 010000",versal-hotel.hoteleastana.com,8 (7172) 56 94 43
-Hotel AyKun,"Kerey and Zhanibek Khans St 12 Б, Nur-Sultan 010016",aykun.kz,8 (775) 121 1152
-Pulsar Mini-Hotel,"Tauelsizdik Ave 49/3, Nur-Sultan 010011",-,8 (771) 881 1100
-Interia,"Abylay Khan Ave 37/1, Nur-Sultan 010000",-,-
-Traditional,"Turan Ave 33, Nur-Sultan 010000",traditional.kz,8 (7172) 40 21 81
-Hostel Americana,"Baltalı 2, Nur-Sultan 010000",-,8 (775) 828 7750
-Eco Apart Hotel Astana,"Qabanbay Batyr Avenue h. 58B pod.3 ap.50, Nur-Sultan 020000",ecoaparthotel.kz,8 (708) 311 6378
-"Апарт-отель ""MONACO"" Астана","Charles de Gaulle St 1, Nur-Sultan 010000",monacohotel.kz,8 (7172) 22 22 82
-O Azamat,"Grigorii Potanin St 9, Nur-Sultan 020000",oazamat-hotel.kz,8 (7172) 57 80 80
-Minigostinitsa Bereket,"Zhk Zhagalau, Taras Shevchenko St 10, Nur-Sultan 020000",-,8 (7172) 40 40 66
-Hostel Forum,"Qabanbay Batyr Ave 9, Nur-Sultan 010000",hostelforum.business.site,8 (778) 950 8631
-Hotel Aner,"проспект М.Жумабаева 25/4 25/4, Nur-Sultan 010000",hotelaner.tilda.ws,8 (775) 000 7081
-"Mini hotel ""SULTAN""","Sarayshyq St 34, Nur-Sultan 010000",-,8 (771) 123 4145
-Rush Hotel,"Zhenis Avenue 53/1, Nur-Sultan 010000",rush-hotel.kz,8 (7172) 38 14 51
-Business Hotel,"Amangeldi Imanov Street 12, Nur-Sultan 020000",-,8 (7172) 53 73 26
-Saryarka Hotel,"Sembinov St 36, Nur-Sultan 010000",-,-
-Botakoz,"Syghanaq St 10, Nur-Sultan 020000",-,8 (7172) 51 04 92
-Hotel Akkaiyn,"Zhayyq St 30, Nur-Sultan 020000",charleston.kz,8 (701) 888 1208
-Elite Apart-Hotel,"Raqymjan Qoshqarbayev Avenue 10\1, Nur-Sultan 010017",eliteapart.kz,8 (7172) 76 97 33
-Comfort Hotel Astana,"Cosmonauts St 60, Nur-Sultan 020000",comforthotel.kz,8 (7172) 24 44 44
-Brown Hotel,"Malik Ghabdullin St 20/1, Nur-Sultan 010000",-,8 (7172) 48 55 77
-Aybat,"Ulitsa Karasakal Yerimbet 65, Nur-Sultan 020000",jumbaktas.kz,8 (7172) 57 87 77
-JJ Hotel,"Qabanbay Batyr Ave 43, Nur-Sultan 010000",-,8 (776) 001 3330
-Boutique Hotel Tomiris,"Kurgaldzhinskoye Highway 9, Nur-Sultan 010000",-,8 (707) 555 5950
-Beijing Palace Soluxe Hotel Astana,"Syghanaq St 27, Nur-Sultan 010000",soluxe-astana.kz,8 (7172) 70 15 15
-Hotel Kausar,"Bogenbay Batyr Ave 6/2, Nur-Sultan 010000",hotel-kausar.kz,8 (778) 808 1616
-Aisha bibi hotel and apart,"Aysha Bibi St 38, Nur-Sultan 020000",aisha-bibi.com,8 (771) 899 4205
-Grace Point Hotel,"Syghanaq St 2, Nur-Sultan 010000",gracepoint.kz,8 (7172) 29 98 80
-"Гостиница ""Bed and Breakfast""","Sarayshyq St 38, ВП-6, Nur-Sultan 010017",hotelbb.kz,8 (702) 387 3534
-Triumph Astana Hostel,"Проспект Кабанбай батыра 11, 6 секция , 18 этаж , 386 квартира, Nur-Sultan 010000",astana-hostel.kz,-
-Astana Transit Hotel,"Астананың Халықаралық әуежайы, Qabanbay Batyr Ave 119, Nur-Sultan",astanaairport.com,8 (7172) 77 71 69
-Hotel Grand Park Esil,"Beibitshilik Street 8, Nur-Sultan 020000",grandparkesil.kz,8 (7172) 59 19 01
-Kel Inn,"Shakarim Kudayberdiuly Avenue 24/193, Nur-Sultan 020000",-,8 (777) 176 5566
-Zhastar,"улица А. Бараева 10/3, Nur-Sultan 010000",-,8 (775) 202 8688
-Арыстан,"Арқайым көшесі 11, Nur-Sultan 010000",instagram.com,8 (702) 181 7229
-Altyn Orda,"Azerbaijan Mambetov St 16, Nur-Sultan 010000",-,8 (7172) 68 06 80
-Гостиница B&B Akbulak2,"Sargul St дом 7, Nur-Sultan 010010",-,8 (747) 941 5887
-Merkuriy I K,"Tsiolkovskiy Str., Nur-Sultan 010000",-,8 (7172) 37 43 96
-Sunkar,"Sembinov Str. 15, Nur-Sultan 010000",hotel-sunkar.kz,8 (7172) 56 96 70
-ЖК The Arc Residence,"ул. Мәңгілік Ел, Nur-Sultan 020000",thearc.kz,8 (777) 722 1010
-Premier Inn Astana,"Balqantau St 187, Nur-Sultan 010000",premierinn.kz,8 (771) 333 3336
-Zhan Villa,"Balqantau St 124, Nur-Sultan 010000",-,8 (7172) 22 20 33
-Royal M Hotel,"Alash Hwy 7, Nur-Sultan 010000",-,8 (7172) 24 84 79
-Mokka Hostel,"Magzhan Zhumabaev Avenue 17, Nur-Sultan 010000",-,8 (708) 695 0025
-Керей,"Yelubay Taybekov St, Nur-Sultan 020000",-,8 (7172) 24 29 54
-Sophia,"Abylay Khan Ave 4/4, Nur-Sultan 010000",-,8 (7172) 35 30 17
-Bayterek,"Sarayshyq St 38, Nur-Sultan 010017",baiterekpremium.com,8 (7172) 57 43 21
-Holiday Inn Express Astana - Turan,"Intersection of Turan and 24 Street, Nur-Sultan 020000",ihg.com,8 (7172) 31 29 14
-Absolute,"Beibitshilik Street 1, Nur-Sultan 010000",absolutehotel.kz,8 (7172) 31 47 42
-Shanyrak on Abaya,"Abay Ave 98, Nur-Sultan 000902",-,8 (7172) 27 43 50
-"Hotel ""Duman""","Коргальджинское шоссе, 2А, Nur-Sultan",hoteldumanastana.com,8 (7172) 79 15 00
-Astana Park Hotel,"Turan Ave 2, Nur-Sultan 020000",-,8 (7172) 55 63 33
-Otel' A1,"55, Zhankent St, Nur-Sultan 020000",-,-
-Astana Zhamilya,"Tsentr Mezhdunarodnoy Torgovli, Dostyq St 1, Nur-Sultan 020000",apartamenty.kz,8 (7172) 52 43 53
-Hostel Astana,"д., Kenesary St 42, Kentau",-,8 (702) 212 8282
-Apartments on Kerey and Zhanibek Khans 22,"Kerey and Zhanibek Khans St 22, Nur-Sultan 010000",altynecopark.com,8 (778) 999 3805
-Oasis Inn,"Bauyrzhan Momyshuly Ave 12a, Nur-Sultan 010000",oasisinn.kz,8 (778) 128 4955
-Lyubimyy,"Balqantau St 77, Nur-Sultan 020000",-,8 (7172) 53 99 39
-Urban Hostel,"Magzhan Zhumabaev Avenue, Nur-Sultan 010000",-,8 (708) 999 9191
-Luxe,"Vol'fgan Gete St 8, Nur-Sultan 010000",-,8 (775) 387 0115
-Health Club & SPA Grand Prix,"Светлый, Sarayshyq Street 1, Nur-Sultan 010000",grandprixclub.kz,8 (707) 177 7222
-ЖК The Arc Residence,"ул. Мәңгілік Ел, Nur-Sultan 020000",thearc.kz,8 (777) 722 1010
-Premier Inn Astana,"Balqantau St 187, Nur-Sultan 010000",premierinn.kz,8 (771) 333 3336
-Gostinitsa Zhasybay,"Shokan Valikhanov Street 26, Nur-Sultan 020000",-,8 (7172) 21 79 79
-Alau,"Qabanbay Batyr Ave 47, Nur-Sultan 020000",-,8 (7172) 70 67 18
-Gk Ak Bulak,"Asqar Toqpanov Street 29, Nur-Sultan 020000",-,-
-Аксу,"Yegemen Qazaqstan Gazeti St 9/3, Nur-Sultan 020000",aksu-astana.kz,8 (701) 739 1326
-Hostel infiniti,"ukl Kabanbay batyra .42 8etazh ap. 460, Nur-Sultan 010000",-,8 (700) 700 2770
-Elysian hotel,"2 Есильсеий, Yengbekshiler Street, Nur-Sultan 020000",elysian-hotel.business.site,8 (700) 269 8055
-Home Hotel Astana,"Manas 9A, Nur-Sultan 010000",-,8 (7172) 54 91 90
-Azat Hotel,"Bayimbet Maylin St 35, Nur-Sultan 010000",hotel-13016.business.site,8 (702) 902 4442
-"Гостинично-ресторанный комплекс ""Турист""","Respublika Ave 33, Nur-Sultan 010000",-,8 (7172) 65 02 00
-Myrza,"Syghanaq St 21/1, Nur-Sultan 020000",-,8 (7172) 50 40 06
-Qaton Qaragay,"18 Куйшы дина, Nur-Sultan 010009",-,8 (7172) 40 06 33
-Hotel BM,"Dostyq St 5, Nur-Sultan 020000",-,8 (7172) 64 24 70
-Алтын Дала,"Bigeldinov St 8/1, Almaty 020000",-,8 (7172) 32 33 11
-Belon Life Hotel,"Abylay Khan Ave 24/1, Nur-Sultan 010000",belon.kz,-
-Azamat Hotel,"Shakarim Kudayberdiuly Avenue 7, Nur-Sultan 010000",azamat-hotel.business.site,8 (7172) 34 21 14
-Отель Реал,"Respublika Ave 54, Nur-Sultan 010000",-,8 (778) 153 3505
-YURT Hostel Astana,"Shamshi Qaldayaqov St 15/1, Nur-Sultan 020000",-,8 (708) 354 0059
-Nurbi,"Гостиница ""Нурби, Kuyshi Dina St 18/1, Nur-Sultan 020000",-,8 (7172) 40 00 00
-"""H8"" - hostel","zhk ""Montblanc De Lux"" blok G, Kenesary St 8, Nur-Sultan 010000",-,8 (771) 800 1001
-"Condo ""Lazurny Kvartal""","Sarayshyq St 5, Nur-Sultan 020000",-,8 (700) 222 0784
-O Azamat,"Grigorii Potanin St 9, Nur-Sultan 020000",oazamat-hotel.kz,8 (7172) 57 80 80
-Business Hotel,"Amangeldi Imanov Street 12, Nur-Sultan 020000",-,8 (7172) 53 73 26
-Egorkino,-,egorkino.kz,8 (707) 230 2222
-Jasmine Hotel,"Sarayshyq St 36, Nur-Sultan 020000",-,8 (747) 197 8188
-THE SQUARE,"Turan Ave 46/1, Nur-Sultan 020000",shaheeninvest.kz,8 (747) 885 8250
-Gostinitsa Astana Art Otel',"Zheltoqsan St 42, Nur-Sultan 020000",-,8 (7172) 38 34 20
-KEMER LUX,"Baktyoraz Beysekbayev St 25/1а, Nur-Sultan 010000",-,8 (778) 999 8555
-Maminn Hostel,"Seiitkali Mendeshev St, Nur-Sultan 020000",facebook.com,8 (7172) 25 61 63
-Agidel',"Vol'fgan Gete St 1Б, Nur-Sultan 010000",-,8 (7172) 93 35 51
-Belon-Lux Hotel,"Dostyq St 13, Nur-Sultan 010016",belon.kz,8 (7172) 79 54 44
-Otyrar Hotel,"Otyrar St 10, Nur-Sultan 010000",otyrar-hotel.kz,8 (776) 111 6501
-Hotel 7 palat,"Almaty street 13 , ЖК Отандостар.Левое крыло, Nur-Sultan 020000",-,8 (7172) 51 57 59
-Zhibek Zholy Hotel,"Zhanibek Tarkhan St 2, Nur-Sultan 010000",-,-
-Nochleg,"Sarayshyq St 38, Nur-Sultan 010000",-,8 (7172) 50 88 20
-Madi,"Temirkazyk St 7, Nur-Sultan 020000",-,8 (701) 726 3766
-Bayan Sulu,"Petrov St 16/4, Nur-Sultan 020000",bayansulu.kz,8 (7172) 36 60 79
-"Hotel ""Daniyar""","Tauelsizdik Ave, Nur-Sultan 020000",-,8 (701) 416 6005
-Korsar Hotel,"B.Momyshuly Str 13, Nur-Sultan 010000",-,8 (7172) 77 60 50
-Akzhayyk,"Iliyas Yesenberlin St 7, Nur-Sultan 010000",akzhaik-hotel.kz,8 (7172) 38 00 91
-Tamgaly,"Kuyshi Dina St 25, Nur-Sultan 010000",-,-
-Hotel Tarlan,"Tупик Льва Толстого 5, гостиница Тарлан, Nur-Sultan 010000",-,8 (701) 101 4442
-Жулдыз,"Sembinov St 16, Nur-Sultan 010000",-,8 (7172) 20 09 99
-Goldman Empire,"Сатбаев көшесі 13/1, Nur-Sultan 010000",goldmanempire.kz,8 (7172) 68 20 20
-Kapsula Hotel Astana,"Qabanbay Batyr Ave 7/3, Nur-Sultan 020000",kapsula.kz,8 (7172) 50 74 04
-Magas Hotel,"Затаевича ул., Nur-Sultan 010000",magas-hotel.kz,8 (7172) 38 20 21
-Kaisar hotel,"Zhumabek Tashenov St 7/3, Nur-Sultan 020000",-,8 (7172) 22 40 34
-Apple Hostel,"Улица Генерала Сабыра Ракымова 22, Nur-Sultan 010000",-,8 (707) 847 3836
-Club Hostel,"Office 16, Infinity 2 complex, Qabanbay Batyr Ave 42, Nur-Sultan 010017",-,8 (7172) 65 01 46
-Hotel Nursat,"Zhengis Ave 63, Nur-Sultan 010000",-,-
-Uyut,"улица Cығанақ 18/1, Nur-Sultan 010000",-,8 (701) 383 1149
-Hotel Milan,"г.Астана, Акбулак 2, д 12, Daraboz St 12, Nur-Sultan",mlife1.wix.com,8 (7172) 34 23 61
-Alatau Hotel,"Zhaidarman Ln 3, Nur-Sultan 020000",-,8 (701) 111 3769
-River house,"Azerbaijan Mambetov St 4, Nur-Sultan 010000",-,8 (747) 500 0129
-Olimp Hostel,"Kerey and Zhanibek Khans St 22, Nur-Sultan 020000",-,8 (775) 000 6901
-Hostel Millenium Park,"Jumeken Najimedenov Street 17, Nur-Sultan 010000",-,8 (702) 406 2895
-Hotel Karat Camp,"Tengiz Ln, Tengiz 060000",-,8 (701) 530 0418
-Agat Hotel Astana,"020000, Nur-Sultan",agathotel.kz,8 (7172) 52 13 44
-Inn Sofia,"Prospect Abyla Khan 4/4, Nur-Sultan 010000",-,-
-Sky Lux Hotel Astana,"Zhengis Ave 43, Nur-Sultan 010000",-,-
-Ырысты,"Zh. Aymauytov St, Nur-Sultan 010000",-,8 (775) 357 2428
-"""Ақ сұнқар"" қонақ үйі","Abay Ave 29, Nur-Sultan 010000",-,8 (7172) 32 73 40
-Europe Palas,"Abay Ave 29, Nur-Sultan 010000",-,-
-Merdan Hall,"Bauyrzhan Momyshuly Ave 31a, Nur-Sultan 010000",-,8 (775) 909 6818
-Ulytau,"Iliyas Yesenberlin St 26, Nur-Sultan 010000",-,8 (7172) 38 58 40
-Kapsula Hotel Astana,"Qabanbay Batyr Ave 7/3, Nur-Sultan 020000",kapsula.kz,8 (7172) 50 74 04
-Royal Park Hotel & Spa,"Kerey and Zhanibek Khans St 18, Nur-Sultan 010000",-,8 (7172) 42 22 01
-Vintage,"Bayimbet Maylin St 16/10, Nur-Sultan 010000",-,8 (7172) 43 65 17
-Hostel Dostyk 10,"ЖК Алматы, Dostyq St 10, Nur-Sultan 010000",-,8 (775) 276 5400
-Mini Hotel Alikhan,"Dostyq St 1, Nur-Sultan 020000",-,-
-Adam I Yeva,"Qayrat Rysqulbekov St 17, Nur-Sultan 010000",inn-248.business.site,8 (778) 066 2706
-Khostel-Panda,"Imanbayeva St 8, Nur-Sultan 010000",hostel-panda.kz,8 (777) 730 1964
-Efendi,"Qabanbay Batyr Ave 8, Nur-Sultan 020000",-,8 (7172) 24 06 73
-Arlan,"Mugalzhar Street 80, Nur-Sultan 020000",-,8 (771) 222 0888
-Hostel RADA,"Respublika Ave 52/2, Nur-Sultan 010000",-,8 (7172) 73 86 86
-Economy Hotel Zhyger,"Respublika Ave 59, Nur-Sultan 010000",-,8 (7172) 39 58 79
-Hotel Tourist,"Republic Prospect 33, Nur-Sultan 010000",-,8 (777) 870 5001
-Comfort Apartments 2,"020000, Nur-Sultan",-,8 (701) 958 2655
-Hostel Baiterek Astana,"Syghanaq St 23, Nur-Sultan 010000",-,-
-"Hotel ""Akku""","Turar Ryskulov St 22, Nur-Sultan 010000",-,-
-Apartments of Astana,"улица Сауран 5G, Nur-Sultan",-,8 (775) 909 0800
-Zolotoy Fazan,"Рыскулова 35, Koyandy 020000",fazan.kz,8 (777) 077 7050
-Uyutnaya,"Seifullin Street 52/2(180), Nur-Sultan 010000",-,8 (778) 535 4824
-Izumrud,"Nurtas Ongdasynov Street 46, Nur-Sultan 020000",-,8 (7172) 24 17 95
-The Barrel Hostel,"Imanbayeva St, Nur-Sultan 020000",-,8 (747) 510 6310
-WELKOM HOSTEL,"Baurzhan Momyshuly Street 2В 9этаж, Nur-Sultan 010010",-,8 (708) 819 0727
-Hotel Shah - Гостиница сауна Шах,"Кайрат Рыскулбеков 11, Qayrat Rysqulbekov St 11, Nur-Sultan 010000",hotelinastana.wixsite.com,8 (778) 139 0188
--,-,-,-
-Samruk Hostel,"st. Bokeyhana. 35, Nur-Sultan 010000",-,8 (701) 700 0969
-Akzhol,"Bayimbet Maylin St 16/12, Nur-Sultan 020000",-,8 (7172) 49 09 09
-HomeStayTursyn Astana Apartments,"Koshkarbayev, 40, Nur-Sultan 020000",-,8 (705) 170 5555
-Mommy,"38 Sarayshyq Street 7 подъезд, 2 этаж, квартира 108, Nur-Sultan 010000",-,8 (708) 972 6234
-hotel Summer garden,Nur-Sultan 020000,-,8 (7172) 23 15 27
-Inn Dostar,"Qayrat Rysqulbekov St 16, Nur-Sultan 020000",-,8 (771) 833 3778
-Akku,"Бухарбай батыра 22, Nur-Sultan 010000",-,8 (7172) 75 27 00
-Saltanat,"Ulitsa Seyfullin 410, Nur-Sultan 050050",-,-
-Zeyd,"Agybay Batyr St 99, Nur-Sultan 020000",zeid.kz,8 (7172) 21 28 05
-Hostel Mayak on Lepsy 46,"Астана, Казахстан Лепсы 46 ВП-1, Nur-Sultan 100000",hostelmayak.simplesite.com,8 (777) 001 9845
-Hostel on Sauran,"Sauran Street 9, Nur-Sultan 010000",-,8 (701) 997 7129
-Belon-Land Hotel,"Bogenbay Batyr Ave 45, Nur-Sultan 010000",belon.kz,8 (7172) 32 83 08
-Dastan Hotel,"38, Shakarim Kudayberdiuly Avenue Здание, Nur-Sultan 010000",-,8 (701) 214 0075
-Complex o'Rosso,"проезд Абылайхана ВИП городок, 19, Nur-Sultan 010000",-,8 (701) 879 2361
-Dastan,"Qayrat Rysqulbekov St 16 корпус, Nur-Sultan 020000",-,-
-AliStar,"Zhk Meruyert, Syghanaq St 21/1, Nur-Sultan 020000",krisha.kz,8 (701) 999 6864
-Topaz,"Zheltoqsan St 6, Nur-Sultan 010000",-,8 (708) 667 5063
-Hostel North Astana,"Dostyq St 5, Nur-Sultan 010000",-,8 (702) 548 8798
-Capital Apartments Astana,"Bayanauyl St 1, Nur-Sultan 020000",-,8 (7172) 96 82 89
-Hostel A99,"99, Aynaköl St, Nur-Sultan 010000",-,-
-Residenz,"Petrov St 21/1, Nur-Sultan 010000",-,8 (7172) 34 22 49
-"Economy Hotel ""Congress Hall""","Kenesary St 30, Nur-Sultan 010000",-,-
-Imanova1,"1, Иманова",-,-
-Hostel Khan Tengri,"Prospect Saryarka 18 9 etzh 76 ap, Nur-Sultan 010000",-,8 (775) 204 6055
-Zhety Kazyna,"Bogenbay Batyr Ave 18, Nur-Sultan 010000",astana.restoran.kz,8 (7172) 23 07 72
-K40,"Qabanbay Batyr Ave 40, Nur-Sultan 020000",-,8 (701) 771 9999
-AG.KZ,"Asankayghy St 41, Nur-Sultan 010000",-,8 (702) 323 8916
-Hostel on Dostyk 2,"Dostyq St 2, Nur-Sultan 010016",-,-
-Hostel on Seifullina 4/2,"Astana Seifullina 4/2, Nur-Sultan 020000",-,8 (775) 209 5400
-"Khostel As, Zhk Infiniti","Qabanbay Batyr Ave 40, Nur-Sultan 020000",-,8 (776) 917 0404
-Shama,"Mustafin St 12, Nur-Sultan 010000",-,8 (701) 257 4888
-Hotel Uyut,"Ulitsa 187 16, Nur-Sultan 010000",-,8 (7172) 47 30 32
-Residenz Hotel,"Zhumabek Tashenov Street 25, Nur-Sultan 020000",-,-
-ХОСТЕЛ В ЦЕНТРЕ АСТАНЫ. АКЦИЯ!,"Zheltoqsan St 48/1, Nur-Sultan 010000",-,8 (700) 199 1007
-Hostel Edem Rooms,"Keensary 52, Nur-Sultan 010000",-,8 (771) 888 2223
-Espresso Bar Expo Boulevard,"Qabanbay Batyr Ave 58Б, Nur-Sultan 020000",instagram.com,8 (778) 454 1605
-"Gostinitsa ""Berlin """,Nur-Sultan 020000,-,-
-Female Hostel,"Syghanaq St 18, Nur-Sultan 020000",-,8 (708) 414 4292
-Orion,"Alexander Pushkin St 3/2, Nur-Sultan 010000",orionhotel.kz,8 (701) 223 3636
-Park Inn by Radisson Astana Hotel,"Saryarka Avenue 8, Nur-Sultan 010000",radissonhotels.com,8 (7172) 67 00 00
-Сауна Charleston,"Zhayyq St 30, Nur-Sultan 010000",charleston.kz,8 (701) 888 1208
-"Сауна Ардана, VIP Сауна в Астане (Нур-Султан)","Tulkibas St 49, Nur-Sultan 010000",-,8 (778) 260 7273
diff --git a/google-maps/pylg.log b/google-maps/pylg.log
deleted file mode 100644
index a303cf5..0000000
--- a/google-maps/pylg.log
+++ /dev/null
@@ -1,259 +0,0 @@
-2020-12-26 09:03:01,994 - WARNING | webdriver.py/_do_retry/67 | Cannot locate xpath retry #1
-2020-12-26 09:04:41,661 - WARNING | webdriver.py/_do_retry/67 | Cannot locate xpath retry #1
-2020-12-26 09:06:27,528 - WARNING | webdriver.py/_do_retry/67 | Cannot locate xpath retry #1
-2020-12-26 09:08:11,889 - WARNING | webdriver.py/_do_retry/67 | Cannot locate xpath retry #1
-2020-12-26 09:09:56,557 - WARNING | webdriver.py/_do_retry/67 | Cannot locate xpath retry #1
-2020-12-26 09:11:08,657 - WARNING | webdriver.py/_do_retry/67 | Cannot locate xpath retry #1
-2020-12-26 09:39:24,550 - ERROR | webdriver.py/_do_retry/70 | Exception found when retrying
-Traceback (most recent call last):
- File "C:\Users\khova\AppData\Local\Programs\Python\Python38-32\lib\asyncio\runners.py", line 43, in run
- return loop.run_until_complete(main)
- File "C:\Users\khova\AppData\Local\Programs\Python\Python38-32\lib\asyncio\base_events.py", line 603, in run_until_complete
- self.run_forever()
- File "C:\Users\khova\AppData\Local\Programs\Python\Python38-32\lib\asyncio\windows_events.py", line 316, in run_forever
- super().run_forever()
- File "C:\Users\khova\AppData\Local\Programs\Python\Python38-32\lib\asyncio\base_events.py", line 570, in run_forever
- self._run_once()
- File "C:\Users\khova\AppData\Local\Programs\Python\Python38-32\lib\asyncio\base_events.py", line 1823, in _run_once
- event_list = self._selector.select(timeout)
- File "C:\Users\khova\AppData\Local\Programs\Python\Python38-32\lib\asyncio\windows_events.py", line 430, in select
- self._poll(timeout)
- File "C:\Users\khova\AppData\Local\Programs\Python\Python38-32\lib\asyncio\windows_events.py", line 779, in _poll
- status = _overlapped.GetQueuedCompletionStatus(self._iocp, ms)
- File "C:\Users\khova\AppData\Local\Programs\Python\Python38-32\lib\site-packages\pyppeteer\launcher.py", line 151, in _close_process
- self._loop.run_until_complete(self.killChrome())
- File "C:\Users\khova\AppData\Local\Programs\Python\Python38-32\lib\asyncio\base_events.py", line 592, in run_until_complete
- self._check_running()
- File "C:\Users\khova\AppData\Local\Programs\Python\Python38-32\lib\asyncio\base_events.py", line 552, in _check_running
- raise RuntimeError('This event loop is already running')
-RuntimeError: This event loop is already running
-
-During handling of the above exception, another exception occurred:
-
-Traceback (most recent call last):
- File "..\webdriver.py", line 63, in _do_retry
- await operation()
- File "C:\Users\khova\AppData\Local\Programs\Python\Python38-32\lib\site-packages\pyppeteer\element_handle.py", line 177, in click
- await self._scrollIntoViewIfNeeded()
- File "C:\Users\khova\AppData\Local\Programs\Python\Python38-32\lib\site-packages\pyppeteer\element_handle.py", line 68, in _scrollIntoViewIfNeeded
- error = await self.executionContext.evaluate('''
- File "C:\Users\khova\AppData\Local\Programs\Python\Python38-32\lib\site-packages\pyppeteer\execution_context.py", line 53, in evaluate
- handle = await self.evaluateHandle(
- File "C:\Users\khova\AppData\Local\Programs\Python\Python38-32\lib\site-packages\pyppeteer\execution_context.py", line 99, in evaluateHandle
- _obj = await self._client.send('Runtime.callFunctionOn', {
-asyncio.exceptions.CancelledError
-2021-02-26 17:15:57,000 - WARNING | extractor.py/_scrape/120 | Error occured
-Traceback (most recent call last):
- File "C:\Users\khova\AppData\Local\Programs\Python\Python38-32\lib\asyncio\runners.py", line 43, in run
- return loop.run_until_complete(main)
- File "C:\Users\khova\AppData\Local\Programs\Python\Python38-32\lib\asyncio\base_events.py", line 603, in run_until_complete
- self.run_forever()
- File "C:\Users\khova\AppData\Local\Programs\Python\Python38-32\lib\asyncio\windows_events.py", line 316, in run_forever
- super().run_forever()
- File "C:\Users\khova\AppData\Local\Programs\Python\Python38-32\lib\asyncio\base_events.py", line 570, in run_forever
- self._run_once()
- File "C:\Users\khova\AppData\Local\Programs\Python\Python38-32\lib\asyncio\base_events.py", line 1823, in _run_once
- event_list = self._selector.select(timeout)
- File "C:\Users\khova\AppData\Local\Programs\Python\Python38-32\lib\asyncio\windows_events.py", line 430, in select
- self._poll(timeout)
- File "C:\Users\khova\AppData\Local\Programs\Python\Python38-32\lib\asyncio\windows_events.py", line 779, in _poll
- status = _overlapped.GetQueuedCompletionStatus(self._iocp, ms)
- File "C:\Users\khova\AppData\Local\Programs\Python\Python38-32\lib\site-packages\pyppeteer\launcher.py", line 151, in _close_process
- self._loop.run_until_complete(self.killChrome())
- File "C:\Users\khova\AppData\Local\Programs\Python\Python38-32\lib\asyncio\base_events.py", line 592, in run_until_complete
- self._check_running()
- File "C:\Users\khova\AppData\Local\Programs\Python\Python38-32\lib\asyncio\base_events.py", line 552, in _check_running
- raise RuntimeError('This event loop is already running')
-RuntimeError: This event loop is already running
-
-During handling of the above exception, another exception occurred:
-
-Traceback (most recent call last):
- File "extractor.py", line 114, in _scrape
- data = extract(await self._page.content())
- File "C:\Users\khova\AppData\Local\Programs\Python\Python38-32\lib\site-packages\pyppeteer\page.py", line 803, in content
- return await frame.content()
- File "C:\Users\khova\AppData\Local\Programs\Python\Python38-32\lib\site-packages\pyppeteer\frame_manager.py", line 384, in content
- return await self.evaluate('''
- File "C:\Users\khova\AppData\Local\Programs\Python\Python38-32\lib\site-packages\pyppeteer\frame_manager.py", line 308, in evaluate
- return await context.evaluate(
- File "C:\Users\khova\AppData\Local\Programs\Python\Python38-32\lib\site-packages\pyppeteer\execution_context.py", line 53, in evaluate
- handle = await self.evaluateHandle(
- File "C:\Users\khova\AppData\Local\Programs\Python\Python38-32\lib\site-packages\pyppeteer\execution_context.py", line 99, in evaluateHandle
- _obj = await self._client.send('Runtime.callFunctionOn', {
-asyncio.exceptions.CancelledError
-2021-02-26 17:18:00,115 - WARNING | webdriver.py/_do_retry/67 | Cannot locate xpath retry #1
-2021-02-26 17:19:42,183 - WARNING | webdriver.py/_do_retry/67 | Cannot locate xpath retry #1
-2021-02-26 17:21:25,899 - WARNING | webdriver.py/_do_retry/67 | Cannot locate xpath retry #1
-2021-02-26 17:23:08,410 - WARNING | webdriver.py/_do_retry/67 | Cannot locate xpath retry #1
-2021-02-26 17:24:53,350 - WARNING | webdriver.py/_do_retry/67 | Cannot locate xpath retry #1
-2021-02-26 17:26:40,303 - WARNING | webdriver.py/_do_retry/67 | Cannot locate xpath retry #1
-2021-02-26 17:28:27,005 - WARNING | webdriver.py/_do_retry/67 | Cannot locate xpath retry #1
-2021-02-26 17:30:13,206 - WARNING | webdriver.py/_do_retry/67 | Cannot locate xpath retry #1
-2021-02-26 17:31:58,209 - WARNING | webdriver.py/_do_retry/67 | Cannot locate xpath retry #1
-2021-02-26 17:33:46,760 - WARNING | webdriver.py/_do_retry/67 | Cannot locate xpath retry #1
-2021-02-26 17:35:35,841 - WARNING | webdriver.py/_do_retry/67 | Cannot locate xpath retry #1
-2021-02-26 17:35:35,845 - ERROR | webdriver.py/_do_retry/70 | Exception found when retrying
-Traceback (most recent call last):
- File "..\webdriver.py", line 65, in _do_retry
- await self._page.waitForXPath(xpath, {'visible': True})
- File "C:\Users\khova\AppData\Local\Programs\Python\Python38-32\lib\site-packages\pyppeteer\frame_manager.py", line 855, in __await__
- raise result
-pyppeteer.errors.TimeoutError: Waiting for XPath "//*[@class="section-hero-header-title-title GLOBAL__gm2-headline-5"]" failed: timeout 30000ms exceeds.
-
-During handling of the above exception, another exception occurred:
-
-Traceback (most recent call last):
- File "..\webdriver.py", line 63, in _do_retry
- await operation()
- File "C:\Users\khova\AppData\Local\Programs\Python\Python38-32\lib\site-packages\pyppeteer\element_handle.py", line 177, in click
- await self._scrollIntoViewIfNeeded()
- File "C:\Users\khova\AppData\Local\Programs\Python\Python38-32\lib\site-packages\pyppeteer\element_handle.py", line 99, in _scrollIntoViewIfNeeded
- raise ElementHandleError(error)
-pyppeteer.errors.ElementHandleError: Node is detached from document
-2021-02-26 17:35:36,220 - WARNING | extractor.py/_scrape/120 | Error occured
-Traceback (most recent call last):
- File "extractor.py", line 117, in _scrape
- back = (await self._page.xpath(back_xpath))[0]
-IndexError: list index out of range
-2021-02-26 17:36:45,623 - WARNING | extractor.py/_scrape/120 | Error occured
-Traceback (most recent call last):
- File "extractor.py", line 111, in _scrape
- place = (await self._page.xpath(result_xpath))[i]
-IndexError: list index out of range
-2021-02-26 17:36:45,632 - WARNING | extractor.py/_scrape/120 | Error occured
-Traceback (most recent call last):
- File "extractor.py", line 111, in _scrape
- place = (await self._page.xpath(result_xpath))[i]
-IndexError: list index out of range
-2021-02-26 17:37:17,806 - WARNING | webdriver.py/_do_retry/67 | Cannot locate xpath retry #1
-2021-02-26 17:38:59,952 - WARNING | webdriver.py/_do_retry/67 | Cannot locate xpath retry #1
-2021-02-26 17:39:16,978 - WARNING | extractor.py/_scrape/120 | Error occured
-Traceback (most recent call last):
- File "extractor.py", line 111, in _scrape
- place = (await self._page.xpath(result_xpath))[i]
-IndexError: list index out of range
-2021-02-26 17:39:16,983 - WARNING | extractor.py/_scrape/120 | Error occured
-Traceback (most recent call last):
- File "extractor.py", line 111, in _scrape
- place = (await self._page.xpath(result_xpath))[i]
-IndexError: list index out of range
-2021-02-26 17:39:16,989 - WARNING | extractor.py/_scrape/120 | Error occured
-Traceback (most recent call last):
- File "extractor.py", line 111, in _scrape
- place = (await self._page.xpath(result_xpath))[i]
-IndexError: list index out of range
-2021-02-26 17:39:16,995 - WARNING | extractor.py/_scrape/120 | Error occured
-Traceback (most recent call last):
- File "extractor.py", line 111, in _scrape
- place = (await self._page.xpath(result_xpath))[i]
-IndexError: list index out of range
-2021-02-26 17:39:17,000 - WARNING | extractor.py/_scrape/120 | Error occured
-Traceback (most recent call last):
- File "extractor.py", line 111, in _scrape
- place = (await self._page.xpath(result_xpath))[i]
-IndexError: list index out of range
-2021-02-26 17:39:17,007 - WARNING | extractor.py/_scrape/120 | Error occured
-Traceback (most recent call last):
- File "extractor.py", line 111, in _scrape
- place = (await self._page.xpath(result_xpath))[i]
-IndexError: list index out of range
-2021-02-26 17:39:17,014 - WARNING | extractor.py/_scrape/120 | Error occured
-Traceback (most recent call last):
- File "extractor.py", line 111, in _scrape
- place = (await self._page.xpath(result_xpath))[i]
-IndexError: list index out of range
-2021-02-26 17:39:17,020 - WARNING | extractor.py/_scrape/120 | Error occured
-Traceback (most recent call last):
- File "extractor.py", line 111, in _scrape
- place = (await self._page.xpath(result_xpath))[i]
-IndexError: list index out of range
-2021-02-26 17:39:17,025 - WARNING | extractor.py/_scrape/120 | Error occured
-Traceback (most recent call last):
- File "extractor.py", line 111, in _scrape
- place = (await self._page.xpath(result_xpath))[i]
-IndexError: list index out of range
-2021-02-26 17:39:17,031 - WARNING | extractor.py/_scrape/120 | Error occured
-Traceback (most recent call last):
- File "extractor.py", line 111, in _scrape
- place = (await self._page.xpath(result_xpath))[i]
-IndexError: list index out of range
-2021-02-26 17:39:17,036 - WARNING | extractor.py/_scrape/120 | Error occured
-Traceback (most recent call last):
- File "extractor.py", line 111, in _scrape
- place = (await self._page.xpath(result_xpath))[i]
-IndexError: list index out of range
-2021-02-26 17:39:17,041 - WARNING | extractor.py/_scrape/120 | Error occured
-Traceback (most recent call last):
- File "extractor.py", line 111, in _scrape
- place = (await self._page.xpath(result_xpath))[i]
-IndexError: list index out of range
-2021-02-26 17:39:17,047 - WARNING | extractor.py/_scrape/120 | Error occured
-Traceback (most recent call last):
- File "extractor.py", line 111, in _scrape
- place = (await self._page.xpath(result_xpath))[i]
-IndexError: list index out of range
-2021-02-26 17:39:17,052 - WARNING | extractor.py/_scrape/120 | Error occured
-Traceback (most recent call last):
- File "extractor.py", line 111, in _scrape
- place = (await self._page.xpath(result_xpath))[i]
-IndexError: list index out of range
-2021-02-26 17:39:17,056 - WARNING | extractor.py/_scrape/120 | Error occured
-Traceback (most recent call last):
- File "extractor.py", line 111, in _scrape
- place = (await self._page.xpath(result_xpath))[i]
-IndexError: list index out of range
-2021-02-26 17:39:17,062 - WARNING | extractor.py/_scrape/120 | Error occured
-Traceback (most recent call last):
- File "extractor.py", line 111, in _scrape
- place = (await self._page.xpath(result_xpath))[i]
-IndexError: list index out of range
-2021-02-26 17:39:17,067 - WARNING | extractor.py/_scrape/120 | Error occured
-Traceback (most recent call last):
- File "extractor.py", line 111, in _scrape
- place = (await self._page.xpath(result_xpath))[i]
-IndexError: list index out of range
-2021-02-26 17:39:49,279 - WARNING | webdriver.py/_do_retry/67 | Cannot locate xpath retry #1
-2021-02-26 17:39:51,929 - ERROR | webdriver.py/_do_retry/70 | Exception found when retrying
-Traceback (most recent call last):
- File "..\webdriver.py", line 65, in _do_retry
- await self._page.waitForXPath(xpath, {'visible': True})
- File "C:\Users\khova\AppData\Local\Programs\Python\Python38-32\lib\site-packages\pyppeteer\frame_manager.py", line 855, in __await__
- raise result
-pyppeteer.errors.TimeoutError: Waiting for XPath "//*[@class="section-hero-header-title-title GLOBAL__gm2-headline-5"]" failed: timeout 30000ms exceeds.
-
-During handling of the above exception, another exception occurred:
-
-Traceback (most recent call last):
- File "..\webdriver.py", line 63, in _do_retry
- await operation()
- File "C:\Users\khova\AppData\Local\Programs\Python\Python38-32\lib\site-packages\pyppeteer\element_handle.py", line 177, in click
- await self._scrollIntoViewIfNeeded()
- File "C:\Users\khova\AppData\Local\Programs\Python\Python38-32\lib\site-packages\pyppeteer\element_handle.py", line 99, in _scrollIntoViewIfNeeded
- raise ElementHandleError(error)
-pyppeteer.errors.ElementHandleError: Node is detached from document
-2021-02-26 17:39:52,369 - WARNING | extractor.py/_scrape/120 | Error occured
-Traceback (most recent call last):
- File "extractor.py", line 117, in _scrape
- back = (await self._page.xpath(back_xpath))[0]
-IndexError: list index out of range
-2021-02-26 17:39:52,382 - WARNING | extractor.py/_scrape/120 | Error occured
-Traceback (most recent call last):
- File "extractor.py", line 111, in _scrape
- place = (await self._page.xpath(result_xpath))[i]
-IndexError: list index out of range
-2021-02-26 17:39:56,928 - WARNING | extractor.py/_scrape/120 | Error occured
-Traceback (most recent call last):
- File "extractor.py", line 111, in _scrape
- place = (await self._page.xpath(result_xpath))[i]
-IndexError: list index out of range
-2021-02-26 17:39:56,987 - WARNING | extractor.py/_scrape/120 | Error occured
-Traceback (most recent call last):
- File "extractor.py", line 111, in _scrape
- place = (await self._page.xpath(result_xpath))[i]
-IndexError: list index out of range
-2021-02-26 17:39:57,051 - WARNING | extractor.py/_scrape/120 | Error occured
-Traceback (most recent call last):
- File "extractor.py", line 111, in _scrape
- place = (await self._page.xpath(result_xpath))[i]
-IndexError: list index out of range
diff --git a/linkedin/__pycache__/locators.cpython-38.pyc b/linkedin/__pycache__/locators.cpython-38.pyc
deleted file mode 100644
index 602624d..0000000
Binary files a/linkedin/__pycache__/locators.cpython-38.pyc and /dev/null differ
diff --git a/linkedin/__pycache__/my_config.cpython-38.pyc b/linkedin/__pycache__/my_config.cpython-38.pyc
deleted file mode 100644
index 545b78b..0000000
Binary files a/linkedin/__pycache__/my_config.cpython-38.pyc and /dev/null differ
diff --git a/linkedin/leads.csv b/linkedin/leads.csv
deleted file mode 100644
index 58ce79a..0000000
--- a/linkedin/leads.csv
+++ /dev/null
@@ -1 +0,0 @@
-Title,Address,WebSite,PhoneNumber
diff --git a/py_lead_generation/__init__.py b/py_lead_generation/__init__.py
new file mode 100644
index 0000000..93d0df3
--- /dev/null
+++ b/py_lead_generation/__init__.py
@@ -0,0 +1,2 @@
+from py_lead_generation.src.yelp.engine import YelpEngine
+from py_lead_generation.src.google_maps.engine import GoogleMapsEngine
diff --git a/py_lead_generation/requirements.txt b/py_lead_generation/requirements.txt
new file mode 100644
index 0000000..2788388
--- /dev/null
+++ b/py_lead_generation/requirements.txt
@@ -0,0 +1,22 @@
+beautifulsoup4==4.12.2
+certifi==2023.11.17
+charset-normalizer==3.3.2
+colorama==0.4.6
+geographiclib==2.0
+geopy==2.4.1
+greenlet==3.0.1
+idna==3.6
+iniconfig==2.0.0
+packaging==23.2
+playwright==1.40.0
+pluggy==1.3.0
+pyee==11.0.1
+pytest==7.4.3
+pytest-base-url==2.0.0
+pytest-playwright==0.4.3
+python-slugify==8.0.1
+setuptools==69.0.3
+soupsieve==2.5
+text-unidecode==1.3
+typing_extensions==4.9.0
+urllib3==2.1.0
\ No newline at end of file
diff --git a/linkedin/pylg.log b/py_lead_generation/src/__init__.py
similarity index 100%
rename from linkedin/pylg.log
rename to py_lead_generation/src/__init__.py
diff --git a/py_lead_generation/src/engines/__init__.py b/py_lead_generation/src/engines/__init__.py
new file mode 100644
index 0000000..e69de29
diff --git a/py_lead_generation/src/engines/abstract.py b/py_lead_generation/src/engines/abstract.py
new file mode 100644
index 0000000..7c67cb9
--- /dev/null
+++ b/py_lead_generation/src/engines/abstract.py
@@ -0,0 +1,35 @@
+from abc import ABC
+
+
+class AbstractEngine(ABC):
+ '''
+ `AbstractEngine`
+
+ Abstract class specifying main methods and attributes
+
+ [CONSTANT] `BASE_URL` - base url for google maps website with substitute search query, coordinates and zoom
+
+ [CONSTANT] `FIELD_NAMES` - field names of scraped data, extracts data entries to csv file based on these field names
+
+ [EDITABLE] `FILENAME` - file name for exporting collected leads, must have .csv extension
+ '''
+
+ BASE_URL = ''
+ FIELD_NAMES = []
+ FILENAME = 'leads.csv'
+
+ async def _get_search_results_urls(self, *args, **kwargs) -> list[str]:
+ '''
+ Retreiving search results URLs from the website (Yelp/Google Maps) method
+ '''
+ ...
+
+ def _parse_data_with_soup(self, html: str) -> list[dict]:
+ '''
+ `html: str` - html representation of the page to parse
+
+ Should be defined in child class
+
+ Returns `list[dict]` typed parsed data - `[title, addr, phone, website]`
+ '''
+ ...
diff --git a/py_lead_generation/src/engines/base.py b/py_lead_generation/src/engines/base.py
new file mode 100644
index 0000000..6cdfff4
--- /dev/null
+++ b/py_lead_generation/src/engines/base.py
@@ -0,0 +1,103 @@
+import asyncio
+from playwright.async_api import Playwright, async_playwright
+
+from py_lead_generation.src.misc.writer import CsvWriter
+from py_lead_generation.src.engines.playwright_config import PlaywrightEngineConfig
+
+
+class BaseEngine(PlaywrightEngineConfig):
+ '''
+ `BaseEngine`
+
+ Base engine class expressing methods, which are shared between all engines, does not provide implementation of `AbstractEngine` methods
+ '''
+
+ async def run(self) -> None:
+ '''
+ Uses headles webdriver powered by Playwright
+
+ Creates a web url based on initialized parameters
+
+ Parses results by given query and url one by one
+
+ Assigns collected results to `.entries`
+
+ To save the results call `.save_to_csv()` method after
+ '''
+ async with async_playwright() as playwright:
+ self.playwright: Playwright = playwright
+ await self._setup_browser()
+ await self._open_url_and_wait(self.url)
+ urls: list[str] = await self._get_search_results_urls()
+ self._entries: list[dict] = await self._get_search_results_entries(urls)
+ await self.browser.close()
+
+ def save_to_csv(self, filename: str = None) -> None:
+ '''
+ `filename: str = None` - optional parameter, by default uses `self.FILENAME`
+
+ If file with such name already exists, it will not overwrite it but append newly found entries to existing ones
+
+ If file with such name does not exist, it will create a new csv file with predetermined fieldnames
+ '''
+ if filename:
+ self.FILENAME = filename
+
+ if not self.FILENAME.endswith('.csv'):
+ raise ValueError('Use .csv file extension')
+ if not self._entries:
+ raise NotImplementedError(
+ 'Entries are empty, call .run() method first to save them'
+ )
+ csv_writer = CsvWriter(self.FILENAME, self.FIELD_NAMES)
+ csv_writer.append(self._entries)
+
+ @property
+ def entries(self) -> list[dict]:
+ '''
+ Returns `list[dict]` typed entries once `.run()` method was called
+ '''
+ if not self._entries:
+ raise NotImplementedError(
+ 'Entries are empty, call .run() method first to create them'
+ )
+ return self._entries
+
+ @entries.setter
+ def entries(self, _) -> None:
+ '''
+ You cannot do that ;)
+
+ Made for durability reasons
+ '''
+ raise ValueError('Cannot set value to data. This is not allowed')
+
+ async def _open_url_and_wait(self, url: str, sleep_duration_s: int = 3) -> None:
+ '''
+ `url: str` - url to open
+ `sleep_duration_s: int = 3` - amount of seconds to wait for the page to load before any operations on that page
+
+ Navigates to initialized `self.url` and waits for `sleep_duration_s` seconds
+ '''
+ await self.page.goto(url)
+ await asyncio.sleep(sleep_duration_s)
+
+ async def _get_search_results_entries(self, urls: list[str]) -> list[dict]:
+ '''
+ `urls: list[str]` - list of urls for google maps entities to scrape for
+
+ Goes over each url in the list and retreives data from opening page one by one
+
+ Searches for data using exact image source values selectors, no other unique way to extract data using selectors is not found yet
+
+ Returns `list[dict]` typed search result entries for each seperate given url from `urls`
+ '''
+ entries = []
+ for url in urls:
+ await self._open_url_and_wait(url, 1.5)
+ html = await self.page.content()
+ data = self._parse_data_with_soup(html)
+ entry = dict(zip(self.FIELD_NAMES, data))
+ entries.append(entry)
+
+ return entries
diff --git a/py_lead_generation/src/engines/playwright_config.py b/py_lead_generation/src/engines/playwright_config.py
new file mode 100644
index 0000000..4db09f7
--- /dev/null
+++ b/py_lead_generation/src/engines/playwright_config.py
@@ -0,0 +1,24 @@
+from playwright.async_api import Browser, Page, BrowserType
+
+
+class PlaywrightEngineConfig:
+ '''
+ `PlaywrightEngineConfig`
+
+ Playwright setup configuration parameters
+
+ [EDITABLE] `BROWSER_PARAMS` - chromiun browser parameters configuration from playwright, additional parameters can be set if needed
+
+ [EDITABLE] `PAGE_PARAMS` - browser page parameters configuration from playwright, additional parameters can be set if needed
+ '''
+
+ BROWSER_PARAMS = {'headless': False, 'proxy': None, 'slow_mo': 150}
+ PAGE_PARAMS = {'java_script_enabled': True, 'bypass_csp': True}
+
+ async def _setup_browser(self) -> None:
+ '''
+ Sets up the browser by initializing `playwright.chromiun` and `Browser` along with `Page` after
+ '''
+ chromium: BrowserType = self.playwright.chromium
+ self.browser: Browser = await chromium.launch(**self.BROWSER_PARAMS)
+ self.page: Page = await self.browser.new_page(**self.PAGE_PARAMS)
diff --git a/py_lead_generation/src/google_maps/__init__.py b/py_lead_generation/src/google_maps/__init__.py
new file mode 100644
index 0000000..e69de29
diff --git a/py_lead_generation/src/google_maps/engine.py b/py_lead_generation/src/google_maps/engine.py
new file mode 100644
index 0000000..5c20577
--- /dev/null
+++ b/py_lead_generation/src/google_maps/engine.py
@@ -0,0 +1,137 @@
+import time
+import asyncio
+from bs4 import BeautifulSoup
+
+from py_lead_generation.src.engines.base import BaseEngine
+from py_lead_generation.src.engines.abstract import AbstractEngine
+from py_lead_generation.src.misc.utils import get_coords_by_location
+
+
+class GoogleMapsEngine(BaseEngine, AbstractEngine):
+ '''
+ `GoogleMapsEngine`
+
+ [EDITABLE] `SCROLL_TIME_DURATION_S` - scroll time duration to view the search results, preferably should be not less than 150, for testing purposes can be decreased
+
+ [EDITABLE] `SLEEP_PER_SCROLL_S` - amount of seconds to wait before each scroll of search results so that google maps does not output endless loading ~ aka simulate human-like activity, preferable should be not less than 5
+
+ Usage:
+
+ `engine = GoogleMapsEngine(*args, **kwargs)`
+
+ `await engine.run()`
+
+ `await engine.save_to_csv()`
+
+ `print(engine.entries)`
+ '''
+
+ BASE_URL = 'https://www.google.com/maps/search/{query}/@{coords},{zoom}z/data=!3m1!4b1?entry=ttu'
+ FIELD_NAMES = ['Title', 'Address', 'PhoneNumber', 'WebsiteURL']
+ FILENAME = 'google_maps_leads.csv'
+
+ SLEEP_PER_SCROLL_S = 5
+ SCROLL_TIME_DURATION_S = 200
+
+ def __init__(self, query: str, location: str, zoom: int | float = 12) -> None:
+ '''
+ `query: str` - what are you looking for? e.g., `gym`
+ `location: str` - where are you looking for that query? e.g., `Astana`
+ `zoom: int | float` - google maps zoom e.g., `8.75`
+
+ Creates `GoogleMapsEngine` instance
+ '''
+ self._entries = []
+ self.zoom = zoom
+ self.query = query
+ self.location = location
+ self.coords = get_coords_by_location(self.location)
+ self.search_query = f'{self.query}%20{self.location}'
+ self.url = self.BASE_URL.format(
+ query=self.search_query, coords=','.join(self.coords), zoom=self.zoom
+ )
+
+ async def _get_search_results_urls(self) -> list[str]:
+ '''
+ Goes through the search results for `GoogleMapsEngine.SCROLL_TIME_DURATION_S` seconds
+
+ Waits for `GoogleMapsEngine.SLEEP_PER_SCROLL_S` seconds so that GoogleMaps will not output endless load, aka simulate human-like activity
+
+ Or scrapes the results urls, once the are no more results
+ '''
+ async def hover_search_results() -> None:
+ '''
+ Hovers on leftbar, where search results are located
+
+ Needed so that scroll function is used properly - not on the map but on the search results div
+ '''
+ leftbar = await self.page.query_selector('[role="main"]')
+ await leftbar.hover()
+ await asyncio.sleep(0.5)
+
+ async def scroll_and_sleep(delta_y: int = 1000) -> None:
+ '''
+ `delta_y: int = 1000` pixel units to scroll down along y-axis
+
+ Scrolls down by `delta_y` and waits for `GoogleMapsEngine.SCROLL_TIME_DURATION_S` seconds
+ '''
+ await self.page.mouse.wheel(0, delta_y)
+ await asyncio.sleep(self.SLEEP_PER_SCROLL_S)
+
+ async def end_locator_is_present() -> bool:
+ '''
+ Retunrs `end_locator` as boolean value, which correlates with the possible end of the search results
+
+ Once `end_locator` is Truthy, aka ElemenetHandle, it means that there is nothing more to scroll
+ '''
+ end_locator = await self.page.query_selector('.m6QErb.tLjsW.eKbjU')
+ return bool(end_locator)
+
+ async def scrape_urls() -> list[str]:
+ '''
+ Should be called once page is being scrolled all the way down (`end_locator` found) or `GoogleMapsEngine.SCROLL_TIME_DURATION_S` duration limit is exceeded
+
+ Returns `list[str]` typed scraped urls list using query_selector
+ '''
+ urls = []
+ link_elements = await self.page.query_selector_all('a.hfpxzc')
+ for link_element in link_elements:
+ url = await link_element.get_attribute('href')
+ urls.append(url)
+ return urls
+
+ await hover_search_results()
+ start_scroll_time = time.time()
+
+ while True:
+ await scroll_and_sleep()
+ finish_scroll_time = time.time()
+ if (await end_locator_is_present()) or (finish_scroll_time - start_scroll_time > self.SCROLL_TIME_DURATION_S):
+ break
+
+ urls = await scrape_urls()
+ return urls
+
+ def _parse_data_with_soup(self, html: str) -> list[str]:
+ '''
+ `html: str` - html representation of the page to parse
+
+ Returns `list[str]` typed parsed data - `[title, addr, phone, website]`
+ '''
+ soup = BeautifulSoup(html, 'html.parser')
+ data = []
+
+ title = soup.select_one('.DUwDvf.lfPIob').get_text()
+ data.append(title)
+
+ addr_img_src = '//www.gstatic.com/images/icons/material/system_gm/2x/place_gm_blue_24dp.png'
+ phone_img_src = '//www.gstatic.com/images/icons/material/system_gm/2x/phone_gm_blue_24dp.png'
+ website_img_src = '//www.gstatic.com/images/icons/material/system_gm/2x/public_gm_blue_24dp.png'
+
+ for img_src in (addr_img_src, phone_img_src, website_img_src):
+ img_element = soup.select_one(f'img[src="{img_src}"]')
+ info = img_element.parent.parent.parent.select_one('.Io6YTe').get_text() \
+ if img_element else '-'
+ data.append(info)
+
+ return data
diff --git a/py_lead_generation/src/google_search/README.md b/py_lead_generation/src/google_search/README.md
new file mode 100644
index 0000000..9b25634
--- /dev/null
+++ b/py_lead_generation/src/google_search/README.md
@@ -0,0 +1,11 @@
+empty for now
+but in the foreseeable future
+i will finish it
+using playwright and regex
+to scrape google search results urls
+locate leads information on them
+and export it to csv
+most importantly here
+is the regex
+which is complex
+bb for now
diff --git a/py_lead_generation/src/google_search/__init__.py b/py_lead_generation/src/google_search/__init__.py
new file mode 100644
index 0000000..e69de29
diff --git a/py_lead_generation/src/misc/__init__.py b/py_lead_generation/src/misc/__init__.py
new file mode 100644
index 0000000..e69de29
diff --git a/py_lead_generation/src/misc/utils.py b/py_lead_generation/src/misc/utils.py
new file mode 100644
index 0000000..5ac83f3
--- /dev/null
+++ b/py_lead_generation/src/misc/utils.py
@@ -0,0 +1,16 @@
+from geopy.geocoders import Nominatim
+
+
+geolocator = Nominatim(user_agent='google-leads')
+
+
+def get_coords_by_location(location: str) -> tuple[str]:
+ '''
+ `location: str` - specific location or city name
+
+ Returns a tuple of the coordinates, two strings represented as float numbers - (latitude, longitude)
+ '''
+ loc = geolocator.geocode(location)
+ coords = (loc.latitude, loc.longitude)
+ coords = list(map(str, coords))
+ return coords
diff --git a/py_lead_generation/src/misc/writer.py b/py_lead_generation/src/misc/writer.py
new file mode 100644
index 0000000..580d557
--- /dev/null
+++ b/py_lead_generation/src/misc/writer.py
@@ -0,0 +1,41 @@
+import csv
+import os.path
+
+
+class CsvWriter:
+ '''
+ CSW Writer class to create and append data to a csv file with fieldnames
+ '''
+
+ def __init__(self, filename: str, fieldnames: list[str]) -> None:
+ '''
+ `filename: str` - file name or file path to create, e.g., `'leads.csv'`
+ `filename: list[str]` - field names/headings, should match data dictionary keys, e.g., `['id', 'name', 'phone_number']`
+
+ Creates a new file or skip if file with such name already exists
+ '''
+
+ self.filename = filename
+ self.fieldnames = fieldnames
+
+ if not os.path.exists(filename):
+ self._init()
+
+ def _init(self) -> None:
+ '''
+ Initializes csv file by writing a new one with fieldnames
+ '''
+ with open(self.filename, 'w', newline='') as f:
+ writer = csv.DictWriter(f, fieldnames=self.fieldnames)
+ writer.writeheader()
+
+ def append(self, data: list[dict]) -> None:
+ '''
+ `data: list[dict]` - data to write, keys of this dictionary should match fieldnames
+
+ Appends incoming `data` to the csv file
+ '''
+ with open(self.filename, 'a', encoding='utf-8', newline='') as f:
+ writer = csv.DictWriter(f, fieldnames=self.fieldnames)
+ for d in data:
+ writer.writerow(d)
diff --git a/py_lead_generation/src/yelp/__init__.py b/py_lead_generation/src/yelp/__init__.py
new file mode 100644
index 0000000..e69de29
diff --git a/py_lead_generation/src/yelp/engine.py b/py_lead_generation/src/yelp/engine.py
new file mode 100644
index 0000000..d65e1db
--- /dev/null
+++ b/py_lead_generation/src/yelp/engine.py
@@ -0,0 +1,90 @@
+import time
+import asyncio
+from bs4 import BeautifulSoup
+
+from py_lead_generation.src.engines.base import BaseEngine
+from py_lead_generation.src.engines.abstract import AbstractEngine
+
+
+class YelpEngine(BaseEngine, AbstractEngine):
+ '''
+ `YelpEngine`
+
+ Usage:
+
+ `engine = YelpEngine(*args, **kwargs)`
+
+ `await engine.run()`
+
+ `await engine.save_to_csv()`
+
+ `print(engine.entries)`
+ '''
+
+ BASE_URL = 'https://www.yelp.com/search?find_desc={query}&find_loc={location}%2C+Philippines&start=0'
+ FIELD_NAMES = ['Title', 'Address', 'PhoneNumber', 'Tags']
+ FILENAME = 'yelp_leads.csv'
+
+ def __init__(self, query: str, location: str) -> None:
+ '''
+ '''
+ self._entries = []
+ self.query = query
+ self.location = location
+ self.url = self.BASE_URL.format(
+ query=self.query, location=self.location
+ )
+
+ async def _get_search_results_urls(self) -> list[str]:
+ '''
+ '''
+ urls = []
+ host = 'https://www.yelp.com'
+ while True:
+ await asyncio.sleep(3)
+ link_elements = await self.page.query_selector_all('.css-1hqkluu')
+ for link_element in link_elements:
+ url = await link_element.get_attribute('href')
+ url = host + url
+ urls.append(url)
+
+ next_page_btn = await self.page.query_selector(
+ '.next-link.navigation-button__09f24__m9qRz.css-ahgoya'
+ )
+ if not next_page_btn:
+ break
+
+ await next_page_btn.scroll_into_view_if_needed()
+ await next_page_btn.click()
+
+ return urls
+
+ def _parse_data_with_soup(self, html: str) -> list[str]:
+ '''
+ `html: str` - html representation of the page to parse
+
+ Returns `list[str]` typed parsed data - `[title, addr, phone, tags]`
+ '''
+ soup = BeautifulSoup(html, 'html.parser')
+ data = []
+
+ title_selector = '.css-1se8maq'
+ addr_selector = '.css-qyp8bo'
+ phone_selector = '.css-djo2w .css-1p9ibgf'
+ tags_selector = '.css-1xfc281 span.css-1fdy0l5'
+
+ for selector in (title_selector, addr_selector, phone_selector):
+ element = soup.select_one(selector)
+ info = element.get_text() if element else '-'
+ data.append(info)
+
+ tags = ''
+ tag_elements = soup.select(tags_selector)
+ for tag_element in tag_elements:
+ el = tag_element.find('a')
+ tag = el.get_text() if el else '-'
+ tags += tag + ','
+ tags = tags[:-1]
+ data.append(tags)
+
+ return data
diff --git a/run.py b/run.py
new file mode 100644
index 0000000..304f2ba
--- /dev/null
+++ b/run.py
@@ -0,0 +1,20 @@
+import asyncio
+from py_lead_generation import GoogleMapsEngine, YelpEngine
+
+
+async def main() -> None:
+ q = input('Enter your search query: ').strip() or 'Barbershop'
+ addr = input('Enter the location you would like to search in: ').strip() \
+ or 'Paris'
+ zoom = float(input('[Optional] Enter google maps zoom: ').strip() or 12)
+
+ engine = GoogleMapsEngine(q, addr, zoom)
+ await engine.run()
+ engine.save_to_csv()
+
+ engine = YelpEngine('Pizza', 'Mexico, Pampanga, Philippines')
+ await engine.run()
+ engine.save_to_csv('pizza_leads.csv')
+
+if __name__ == '__main__':
+ asyncio.run(main())
diff --git a/setup.py b/setup.py
new file mode 100644
index 0000000..ac8c77c
--- /dev/null
+++ b/setup.py
@@ -0,0 +1,32 @@
+from setuptools import setup, find_packages
+
+
+with open('README.md', 'r') as f:
+ long_description = f.read()
+
+VERSION = '0.3.0'
+
+setup(
+ name='py_lead_generation',
+ version=VERSION,
+ license = 'MIT',
+ author='Madi-S (Madi Shaiken)',
+ author_email='<[email protected]>',
+ description='Lead generation scripts',
+ long_description=long_description,
+ long_description_content_type='text/markdown',
+ url='https://github.com/Madi-S/Lead-Generation',
+ packages=find_packages(),
+ install_requires=['playwright', 'beautifulsoup4', 'geopy'],
+ python_requires='>=3.10',
+ keywords=['python', 'lead generation', 'web automation',
+ 'playwright', 'google maps', 'yelp'],
+ classifiers=[
+ 'Development Status :: 1 - Planning',
+ 'Intended Audience :: Developers',
+ 'Programming Language :: Python :: 3.10',
+ 'Operating System :: Unix',
+ 'Operating System :: MacOS :: MacOS X',
+ 'Operating System :: Microsoft :: Windows',
+ ]
+)
diff --git a/yellow-pages/__pycache__/config.cpython-38.pyc b/yellow-pages/__pycache__/config.cpython-38.pyc
deleted file mode 100644
index d46a9a4..0000000
Binary files a/yellow-pages/__pycache__/config.cpython-38.pyc and /dev/null differ
diff --git a/yellow-pages/__pycache__/locators.cpython-38.pyc b/yellow-pages/__pycache__/locators.cpython-38.pyc
deleted file mode 100644
index f0f5da1..0000000
Binary files a/yellow-pages/__pycache__/locators.cpython-38.pyc and /dev/null differ
diff --git a/yellow-pages/__pycache__/my_config.cpython-38.pyc b/yellow-pages/__pycache__/my_config.cpython-38.pyc
deleted file mode 100644
index e61acb0..0000000
Binary files a/yellow-pages/__pycache__/my_config.cpython-38.pyc and /dev/null differ
diff --git a/yellow-pages/leads.csv b/yellow-pages/leads.csv
deleted file mode 100644
index 2740b1b..0000000
--- a/yellow-pages/leads.csv
+++ /dev/null
@@ -1,289 +0,0 @@
-Title,Address,WebSite,PhoneNumber
-Sohostel,"91 Dean Street, London, GB, XGL, W1D 3SY",Unclaimed,020 8821 5154
-YHA London Central,"104 Bolsover Street, London, GB, XGL, W1W 5NU",yha.org.uk,0845 371 9154
-The Hoxton - Shoreditch,"81-83 Great Eastern Street, London, GB, XGL, EC2A 3HU",Unclaimed,020 7550 1000
-Wombat&apos;s City Hostel London,"7 Dock Street, London, GB, XGL, E1 8LL",wombats-hostels.com,020 7680 7600
-Palmers Lodge,"Swiss Cottage
-40 College Crescent, London, GB, XGL, NW3 5LB",palmerslodges.com,020 7483 8470
-The Dictionary Hostel,"12-20 Kingsland Road, London, GB, XGL, E2 8DA",thedictionaryhostel.com,020 7613 2784
-Woodville Guesthouse,"35 Woodville Road
-Walthamstow, London, GB, XGL, E17 7ER",Unclaimed,07901 155500
-Generator London,"37 Tavistock Place, London, GB, XGL, WC1H 9SE",generatorhostels.com,020 7388 7666
-St Christophers Inn,"121 Borough High Street, London, GB, XGL, SE1 1NP",Unclaimed,020 7407 2392
-Astor Victoria,"71 Belgrave Road, London, GB, XGL, SW1V 2BG",Unclaimed,020 7834 3077
-Holland House,"53 Eccleston Square, London, GB, XGL, SW1V 1PG",Unclaimed,020 7834 9104
-Bestplace,"41-42 Cosser Street, London, GB, XGL, SE1 7BU",bestplaceinns.com,020 7928 0720
-Clink 78,"78 Kings Cross Road, London, GB, XGL, WC1X 9QG",clinkhostels.com,020 7183 9400
-Queen Elizabeth Hostel,"58 Bagleys Lane
-Hammersmith, London, GB, XGL, SW6 2BH",Unclaimed,0844 472 1800
-The Walrus Bar and Hostel,"172 Westminster Bridge Road, London, GB, XGL, SE1 7RW",Menus | The Walrus Bar and Hostel http://thewalrusbarandhostel.co.uk/bar/assets/walrus_bh_menu.pdf,020 7928 4368
-Arsenal Tavern Backpackers,"175 Blackstock Road, London, GB, XGL, N4 2JS",Unclaimed,020 7359 6902
-Astor Museum Inn,"27 Montague Street
-City of London
-Greater London, London, GB, XGL, WC1B 5BH",astorhostels.co.uk,020 7580 5360
-St Christopher&apos;s,"28 Hammersmith Broadway, London, GB, XGL, W6 7AB",st-christophers.co.uk/london-h…,020 8748 5285
-The Fun Factory,"16a Cato Road
-Clapham, London, GB, XGL, SW4 7TX",Unclaimed,
-Astor Kensington,"45 Queensborough Terrace, London, GB, XGL, W2 3SY",astorhostels.com,020 7229 7782
-Qbic Hotel ,"42 Adler Street, London, GB, XGL, E1 1EE",qbichotels.com/london-city,020 3021 3300
-YHA London St Pancras,"79-81 Euston Road, London, GB, XGL, NW1 2QE",yha.org.uk,0845 371 9344
-The Royal Horseguards Hotel,"2 Whitehall Court, London, GB, XGL, SW1A 2EJ",theroyalhorseguards.com,0871 376 9033
-Camden Inn Hostel,"55-57 Bayham Street
-Camden, London, GB, XGL, NW1 0AA",Unclaimed,0844 472 1800
-Pickwick Hall,"7 Bedford Place, London, GB, XGL, WC1B 5JE",pickwickhall.co.uk,020 7323 4958
-LSE High Holborn Residence,"178 High Holborn, London, GB, XGL, WC1V 7AA",lsevacations.co.uk/Accommodati…,020 7107 5737
-Smart Camden Inn,"55-57 Bayham Street
-Camden Town, London, GB, XGL, NW1 0AA",Unclaimed,020 7388 8900
-Travel Joy Hostel,"111Grosvenor Road, London, GB, XGL, SW1V 3LG",traveljoyhostels.com,020 7834 9689
-easyHotel - Victoria,"34-40 Belgrave Road, London, GB, XGL, SW1V 1RG",Booking.com,020 7834 1379
-Staysafe,"144-152 Walworth Road
-Elephant &amp; Castle, London, GB, XGL, SE17 2NG",Unclaimed,
-Bowden Court,"11 Belgrave Road, London, GB, XGL, SW1V 1RB",Unclaimed,020 3553 7280
-Park Villa Boutique Hostel,"51 Grove Road
-Bow, London, GB, XGL, E3 4PE",Unclaimed,020 8980 1439
-Astor Hyde Park,"191 Queensgate
-South Kensington, London, GB, XGL, SW7 5EU",astorhostels.co.uk,020 7581 0103
-Cherry Court Hotel,"23 Hugh Street, London, GB, XGL, SW1V 1QJ",cherrycourthotel.co.uk,020 7828 2840
-Bird&apos;s Nest Bar and Hostel,"32 Deptford Church St, London, GB, XGL, SE8 4RZ",Unclaimed,020 8692 1928
-Don Gratton House,"Don Gratton House
-82 Old Montague Street, London, GB, XGL, E1 5NN",Unclaimed,0870 600 4777
-Venture Hostel,"86 Tanners Hill, London, GB, XGL, SE8 4PN",Unclaimed,020 8692 6219
-Alliance House,"44-45 Newington Green, London, GB, XGL, N16 9QH",Unclaimed,020 7288 8060
-Northfields Hostel,"264 Northfield Avenue
-Acton, London, GB, XGL, W3 0NW",Unclaimed,0844 472 1800
-Baden Powell House,"65-67 Queen&apos;s Gate, London, GB, XGL, SW7 5JS",Unclaimed,020 7590 6909
-Earl&apos;s Court Youth Hostel,"38 Bolton Gardens, London, GB, XGL, SW5 0AQ",yha.org.uk/find-accommodation/…,0345 371 9114
-Safestay,"144-152 Walworth Road
-Elephant and Castle, London, GB, XGL, SE17 1RL",Booking.com,020 7703 8000
-YHA Holland Park,"Holland Walk Kensington, London, GB, XGL, W8 7QU",Unclaimed,0845 371 9122
-London Backpackers,"8-10 Queens Parade
-Queens Road
-Hendon, London, GB, XGL, NW4 3NS",ukhostels.com,020 8203 1319
-Palmers Lodge Hillspring,"233 Willesden Lane, London, GB, XGL, NW2 5RP",palmerslodges.com/hillspring,020 7099 2435
-High Holborn Hall - Early Summer,"178 High Holborn
-Covent Garden, London, GB, XGL, WC1V 7AA",Unclaimed,0844 472 1800
-The RE London Shoreditch,"419 - 437 Hackney Road, London, GB, XGL, E2 8PP",hotelshoreditch.com,020 7613 6500
-Sally London dfd,"142 London New
-Padd, London, GB, XGL, NW7 2RR",Unclaimed,
-Restup,"172 New Kent Road, London, GB, XGL, SE1 4YT",restup.co.uk,020 3642 4549
-Astor Kensington,"138 Cromwell Road, London, GB, XGL, SW7 4HA",Unclaimed,020 7373 5138
-YHA London St. Paul&apos;s,"36 Carter Lane, London, GB, XGL, EC4V 5AB",yha.org.uk,0845 371 9012
-Clink 261,"261-265 Gray&apos;s Inn Road, London, GB, XGL, WC1X 8QT",hostelworld.com,020 7183 9400
-LSE Bankside House,"24 Sumner Street, London, GB, XGL, SE1 9JA",lsevacations.co.uk,020 7107 5750
-City Ymca London,"8 Errol Street, London, GB, XGL, EC1Y 8SE",Unclaimed,020 7628 8832
-Northumberland Park House,"143 Northumberland Park, London, GB, XGL, N17 0TR",Unclaimed,020 8808 8839
-The W14 Kensington,"16-22 Gunterstone Road, West Kensington, GB, XGL, W14 9BX",Hostels.com,020 7602 6600
-Fairway Lodge,"400-408 Stanstead Road
-Catford, London, GB, XGL, SE6 4XB",Unclaimed,020 8690 6992
-Megaro Hotel,"23-27 Euston Road
-St Pancras, London, GB, XGL, WC1H 8AB",hotelmegaro.co.uk,020 7843 2222
-Kensington House Hotel,"15-16 Prince of Wales Terrace, London, GB, XGL, W8 5PQ",Unclaimed,020 7937 2345
-Alinka Hostel,"52A Brixton Road London, London, GB, XGL, SW9 8HL",Unclaimed,
-Smart Hostels,"71-72 Guilford Street, London, GB, XGL, WC1N 1DF",Unclaimed,020 7833 8818
-Smart Hyde Park Inn,"48-50 Inverness Terrace, London, GB, XGL, W2 3JA",Unclaimed,
-ST. Mungos,"45 Killieser Avenue, London, GB, XGL, SW2 4NX",Unclaimed,020 8674 4597
-Revive Lodge Heathrow,"89 Hounslow Road
-Feltham, London, GB, XGL, TW14 0BB",Unclaimed,0844 472 1800
-Inner Park Road Hostel,"27 Inner Park Road, London, GB, XGL, SW19 6DF",Unclaimed,020 8789 7957
-The Nadler Kensington,"25 Courtfield Gardens, London, GB, XGL, SW5 0PG",thenadler.com,020 7244 2255
-all star hostel,"Chatsworth Road London, London, GB, XGL, E5 0PS",Unclaimed,
-Amro Vantage,"24 Berkeley Square Mayfair, London, GB, XGL, W1J 6HE",amrovantage.com,020 3709 0274
-Phoenix Hostel,"6 Daventry Street, London, GB, XGL, NW1 5NX",Unclaimed,020 7724 0705
-Shepiston Lodge Heathrow,"31 Shepiston Lane
-Middlesex, London, GB, XGL, UB3 1LJ",Unclaimed,020 8573 0266
-Harlingford Hotel,"61-63 Cartwright Gardens, London, GB, XGL, WC1H 9EL",Unclaimed,020 7387 1551
-InterContinental London Park Lane,"One Hamilton Place
-Park Lane, London, GB, XGL, W1J 7QY",Claimed,020 7409 3131
-Piccadilly Hotel,"12 Sherwood Street, London, GB, XGL, W1F 7BZ",Unclaimed,020 7434 9009
-Abercorn House,"28-30 Bute Gardens, London, GB, XGL, W6 7DS",speedtest.net/result/267…,020 8563 8692
-Youth Hostel Association,"104-108 Bolsover Street
-Marylebone, London, GB, XGL, W1W 5QQ",-,0870 770 6144
-O&apos;Callaghan&apos;s Hostel,"205 Earls Court Road
-Kensington, London, GB, XGL, SW5 9AN",Unclaimed,020 7370 3000
-Dellow Centre,"82 Wentworth Street, London, GB, XGL, E1 7SA",Unclaimed,020 7375 3915
-Eastwards Trust Hostels,"217 Romford Road, London, GB, XGL, E7 9HL",Unclaimed,020 8555 6421
-Belgrave Court,"83-87 Prince Regent Lane, London, GB, XGL, E13 8RP",Unclaimed,020 7511 5429
-Thames Reach Bondway,"62 Bondway, London, GB, XGL, SW8 1SF",Unclaimed,020 7840 9820
-St Christopher&apos;s Inns,"48-50 Camden High Street, London, GB, XGL, NW1 0LT",st-christophers.co.uk,020 7388 1012
-Ridley Villas Hostel,"35 Ridley Road, London, GB, XGL, E8 2NP",Unclaimed,020 7275 7852
-Journeys London Bridge,"204 Manor Place
-Walworth, London, GB, XGL, SE17 3BN",Unclaimed,020 7735 6581
-Via Limehouse,"628-634 Commercial Road
-Tower Hamlets, London, GB, XGL, E14 7HS",Unclaimed,04420 3588 8484
-4 Star Hostel,"91 Jermyn Street, London, GB, XGL, SW1Y 6JB",Unclaimed,020 7930 3111
-Barmy Badger Backpackers,"17 Longridge Road, London, GB, XGL, SW5 9SB",Unclaimed,020 7370 5213
-High Street Kensington Hostel,"1 Warwick Gardens, London, GB, XGL, W14 8PH",Unclaimed,020 7602 0444
-Belsize House,"40 Belsize Park Gardens, London, GB, XGL, NW3 4NA",Unclaimed,020 3740 9804
-Dalco Homes,"20 Dukes Avenue, London, GB, XGL, N10 2PT",Unclaimed,020 8444 5138
-Roomzzz London Stratford,"66 West Ham Lane, London, GB, XGL, E15 4PT",Unclaimed,020 3909 6400
-More House,"53 Cromwell Road, London, GB, XGL, SW7 2EH",Unclaimed,020 7584 2040
-Equity Point London,"100-102 Westbourne Ter
-Paddington W2 6QE, London, GB, XGL, W2 6",Unclaimed,04420 7087 8001
-Caplin House,"330 Commercial Road, London, GB, XGL, E1 2PY",Unclaimed,020 7780 9078
-Piccadilly Guest House,"91 Jermyn Street
-St Jame&apos;s, London, GB, XGL, SW1Y 6JB",Unclaimed,0845 838 4645
-Kipper Project,"15 Old Ford Road, London, GB, XGL, E2 9PJ",Unclaimed,020 8980 1401
-Travelodge Hotel,"195 -203 Waterloo Road, London, GB, XGL, SE1 8UX",Unclaimed,0871 984 6291
-Yha England &amp; Wales,"Holland House Holland Park, London, GB, XGL, W8 6LU",Unclaimed,020 7937 0748
-Yotel,"Heathrow Airport
-Terminal 4, Hounslow, GB, XGL, TW6 3XA",yotel.com,020 7100 1100
-Atlas Hostels,"16 Talbot Square, London, GB, XGL, W2 1TS",Unclaimed,0845 519 9448
-Irish Centre Housing,"20-22 Quex Road, London, GB, XGL, NW6 4PG",Unclaimed,020 7372 6633
-Kabo Hostel,"32 Saltoun Road, London, GB, XGL, SW2 1EP",Unclaimed,020 7733 9466
-Chester House,"Pages Lane, London, GB, XGL, N10 1PR",Unclaimed,020 8883 8204
-Hyde Park Hostel,"26 Inverness Terrace
-Bayswater, London, GB, XGL, W2 3HU",Unclaimed,020 7727 9163
-"Globetrotter Inns, London","Ashlar Court
-Ravenscourt Gardens, London, GB, XWY, W6",Unclaimed,020 8746 3112
-Pembury Hotel,"326 - 328 Seven Sisters Road
-Finsbury Park, London, GB, XGL, N4 2AP",Unclaimed,0844 472 1800
-Moullin House Memorial Hostel Trust,"24-26 Mount Park Road, London, GB, XGL, W5 2RT",Unclaimed,020 8997 4343
-The Louisianna,"611 Green Lanes, London, GB, XGL, N8 0RE",Unclaimed,020 8341 0726
-St Christopher&apos;s Inn Greenwich,"189 Greenwich High Road, London, GB, XGL, SE10 8JA",st-christophers.co.uk/london-h…,020 8858 3591
-641 Fulham Road,"641 Fulham Road, London, GB, XGL, SW6 5PU",Unclaimed,020 7581 5970
-Hytyme Managment Services,"261 Barry Road, London, GB, XGL, SE22 0JT",Unclaimed,020 8299 3333
-St Christopher&apos;s Village London Bridge,"161 Borough High Street, Southwark, GB, XGL, SE1 1HR",st-christophers.co.uk,020 7939 9710
-University of Westminster International House,"1-5 Lambeth Rd, London, GB, XGL, SE1 7DG",Unclaimed,020 7911 5181
-Central Hostel,"26 Prince&apos;s Square
-Bayswater, London, GB, XGL, W2 4NJ",Unclaimed,020 7727 5807
-Sandeman-Allen House,"40 Inverness Terrace, London, GB, XGL, W2 3HB",Unclaimed,020 7727 2719
-International Students House,"229 Great Portland Street, London, GB, XGL, W1W 5PN",ish.org.uk,020 7631 8300
-Hub Premier Inn,"15 Dacre Street, London, GB, XGL, SW1H 0DJ",Unclaimed,0871 527 9608
-Threshold Tennant Trust,"10 Fontenoy Road, London, GB, XGL, SW12 9LU",Unclaimed,020 8675 1000
-Casa Dello Studente,"174 Clapham Road, London, GB, XGL, SW9 0LA",Unclaimed,020 7735 3058
-Arlington House,"220 Arlington Road, London, GB, XGL, NW1 7HE",Unclaimed,020 7482 3374
-Pathway Housing Association,"77 Plough Road, London, GB, XGL, SW11 2BJ",Unclaimed,020 7924 3009
-The Rathbone,"30 Rathbone Street, London, GB, XGL, W1T 1LB",rathbonehotel.co.uk,020 7636 2001
-Alone In London,"77 Lansdowne Road, London, GB, XGL, N17 0NN",Unclaimed,020 8493 0239
-Ackee Housing Project,"103 Stoke Newington Road, London, GB, XGL, N16 8BX",Unclaimed,020 7249 5659
-Globe Trott Inns,"Boleyn Tavern
-1 Barking Road, London, GB, XGL, E6 1PW",-,020 8471 4284
-Journeys Kings Cross Hostel,"54-58 Caledonian Rd
-Kings Cross, London, GB, XGL, N1 9DP",Unclaimed,020 7833 3893
-Young Women&apos;s Christian Association Of Great Britain,"6-8 Kemplay Road, London, GB, XGL, NW3 1SY",Unclaimed,020 7435 5730
-Journeys London East Hostel,"118 Wellington Street, London, GB, XGL, SE18 6XY",Unclaimed,020 8853 9352
-St Christopher&apos;s Inn Shepherd&apos;s Bush,"13-15 Shepherds Bush Green
-Shepherds Bush, London, GB, XGL, W12 8PH",st-christophers.co.uk/london-h…,020 8735 0270
-Downham Lodge,"284 Amhurst Road, London, GB, XGL, N16 7UE",Unclaimed,020 7923 2900
-Camden Lock Hotel,"89 Chalk Farm Road
-Camden, London, GB, XGL, NW1 8AR",Unclaimed,020 7267 3912
-Bed Station,"1st Floor Offices
-189-193 Earls Court Road, London, GB, XGL, SW5 9AN",bedstation.co.uk,020 7835 1867
-Strand Palace,"372 Strand, London, GB, XGL, WC2R 0JJ",strandpalacehotel.co.uk,020 7379 4737
-Newington Court Hostel,"Newington Court
-1 Collins Road, London, GB, XGL, N5 2UF",Unclaimed,020 7359 2495
-Ibis Budget London Whitechapel,"100 Whitechapel Road, London, GB, XGL, E1 1JG",Unclaimed,020 7655 4620
-Tune Hotel,"118-120 Westminster Bridge Road, London, GB, XGL, SE1 7RW",Unclaimed,
-Youth Hostels Association,"4 Wellgarth Road, London, GB, XGL, NW11 7HR",Unclaimed,020 8458 7196
-Mindset Services,"684a High Road, London, GB, XGL, N17 0AE",Unclaimed,020 8365 0777
-The Stafford London,"16-18 St James&apos;s Place, London, GB, XGL, SW1A 1NJ",thestaffordlondon.com/english/…,020 7493 0111
-Salvation Army,"153-175 Whitechapel Road, London, GB, XGL, E1",Unclaimed,020 7392 9490
-London Hostels Association,"97 Queens Gate, London, GB, XGL, SW7 5AB",Unclaimed,020 7373 4180
-Shian Housing Association,"152 Bethune Road, London, GB, XGL, N16 5DS",Unclaimed,020 8800 5060
-St Mungos,"188 Vartry Road, London, GB, XGL, N15 6HA",Unclaimed,020 8809 5742
-Paradice Homes,"39 Ingleton Road, London, GB, XGL, N18 2RS",Unclaimed,020 8803 1460
-Dover Castle,"6 Great Dover Street, London, GB, XGL, SE1 4XW",Unclaimed,020 7403 7773
-Bartrams Convent Hostel,"Rowland Hill Street, London, GB, XGL, NW3 2AD",Unclaimed,020 7431 5680
-Susan Lawrence Hostel,"170 Kings Cross Road, London, GB, XGL, WC1X 9DH",Unclaimed,
-Hyde Park Inn,"48-50 Inverness Terrace, London, GB, XGL, W2 3JA",Unclaimed,020 7229 0000
-Euro Tower,"Courland Grove, London, GB, XGL, SW8 2PX",Unclaimed,020 7720 5191
-Hostel Acacia London,"14 Queensbury Place
-South Kensington London, London, GB, XGL, SW7 2EA",Unclaimed,020 7823 7103
-St Georges House,"263 Camden Road, London, GB, XGL, N7 0HS",Unclaimed,020 7607 7989
-St Mungon,"146 Mare Street, London, GB, XGL, E8 3SG",Unclaimed,020 8525 0007
-Helen Graham House,"57 Great Russell Street
-Holborn, London, GB, XGL, WC1B 3BD",Unclaimed,020 8307 3300
-The Tower Hotel,"Saint Katharine&apos;s Way, London, GB, XGL, E1W 1LD",guoman.com,0800 330 8005
-Youth Hostel Association,"Holland Walk
-Kensington, London, GB, XGL, W8 7QU",Unclaimed,0870 770 5866
-Hillspring Lodge,"233 Willesden Lane
-Willesden, London, GB, XGL, NW2",Unclaimed,0844 472 1800
-Keystone House Kings Cross,"272-276 Pentonville Road, London, GB, XGL, N1 9JY",Unclaimed,020 7837 6444
-Ymca,"5 Shernhall Street, London, GB, XGL, E17 3EU",Unclaimed,020 8520 1667
-UoL Accommodates,"ULU 4th Floor
-Malet Street, London, GB, XGL, WC1E 7HY",Unclaimed,04402 0786 28881
-Equinox,"100 Queens Road, London, GB, XGL, SE15 2QT",Unclaimed,020 7639 8008
-hostelsconnections,"3 coborn rd
-london, London, GB, XGL, E3 2DA",Unclaimed,020 8123 0817
-Arosfa Hotel,"83 Gower Street, London, GB, XGL, WC1E 6HJ",Unclaimed,020 7636 2115
-Regina House,"110 Gloucester Road
-South Kensington, London, GB, XGL, SW7 4RJ",Unclaimed,020 7373 5151
-Celtic Hotel,"62 Guilford Street, London, GB, XGL, WC1N 1DD",Unclaimed,020 7837 9258
-BW Premier Park Grand,"27/37 Hogarth Road
-Kensington, London, GB, XGL, SW5 0QQ",Unclaimed,020 7370 6831
-Coram Leaving Care Services,"1 Yonge Park, London, GB, XGL, N4 3NU",Unclaimed,020 7607 4341
-IQ Kingston,"15 Wood Street, Kingston Upon Thames, GB, XGL, KT1 1TY",Unclaimed,020 8547 3139
-TravelStay Hyde Park,", London, GB, XGL, W2",Unclaimed,020 7298 2100
-Thames Reach Bondway Housing Association Hostels,"73-75 Stamford Street, London, GB, XGL, SE1 9NB",Unclaimed,020 7928 2497
-Emily Bowes Court,"Lebus Street, London, GB, XGL, N17 9FD",Unclaimed,020 8801 3335
-Belgrave Hotel,"9-13 Clapham Road, London, GB, XGL, SW9 0JD",Unclaimed,020 7793 0142
-Youth Hostels Association,"38 Bolton Gardens, London, GB, XGL, SW5 0AQ",Unclaimed,0870 770 5804
-Nothing,"123 Great Garden
-Bayswater, London, GB, XGL, W2 1UB",Unclaimed,
-MEININGER Hotel London Hyde Park,"65-67 Queen&apos;s Gate, London, GB, XGL, SW7 5JS",meininger-hotels.com,020 3318 1407
-Woodthorpe Road Hostel,"3 Woodthorpe Road, London, GB, XGL, SW15 6UQ",Unclaimed,020 8789 5801
-St Mungo&apos;s,"113-117 Cedars Road, London, GB, XGL, SW4 0PJ",Unclaimed,020 7720 7377
-Sundial Court,"38-42 Chiswell Street, London, GB, XGL, EC1Y 4SB",Unclaimed,020 7588 0417
-RMA Accommodation,"156-158 Mile End Road, London, GB, XGL, E1 4LJ",Unclaimed,020 7790 1884
-Homeless Young Peoples Project,"43-45 ST. Georges Avenue, London, GB, XGL, E7 8HR",Unclaimed,020 8471 5111
-Edmonton Hostel,"1 Church Street, London, GB, XGL, N9 9DR",Unclaimed,020 8345 5415
-West Two Hostel London,"22-23 Kensington Garden Square
-Bayswater, London, GB, XGL, W2 4BE",Unclaimed,0844 472 1800
-Tamworth House,"188 Tamworth Lane, Mitcham, GB, SRY, CR4 1DE",Unclaimed,020 8687 1319
-London Eye Hostel,"73 Lambeth Walk
-Lambeth, London, GB, XGL, SE11 6DX",Unclaimed,
-Grow,"36 Rickmansworth Road, Watford, GB, HRT, WD18 7HT",Unclaimed,01923 256189
-Z Hotel Victoria,"5 Lower Belgrave Street, London, GB, XGL, SW1W 0NR",Unclaimed,020 3589 3990
-London Hostels Association,"40 Belsize Park Gardens, London, GB, XGL, NW3 4NA",Unclaimed,020 7722 8131
-London Hostels Association,"Sandeman Allen
-40 Inverness Terrace, London, GB, XGL, W2 3JB",Unclaimed,020 7727 2719
-Alliance Club,"44-45 Newington Green, London, GB, XGL, N16 9QH",Unclaimed,020 7226 6085
-Ashwell House University Hall Of Residents,"Ashwell House
-1b Shepherdess Walk, London, GB, XGL, N1 7BD",Unclaimed,020 7490 3296
-Chapel Hill Hostel,"51-55 Chapel Hill, Dartford, GB, KEN, DA1 4BY",Unclaimed,01322 553201
-The Court Hostel,"194-196 Earls Court Road
-Earls Court, London, GB, XGL, SW5 9QF",Unclaimed,04402 0737 30027
-The Lodge,"44 Norfolk Square, London, GB, XGL, W2 1RT",Unclaimed,020 7262 4406
-St Mungos Housing Association,"Pagnell Street Hostel
-Exeter Way, London, GB, XGL, SE14 6LL",Unclaimed,020 8692 3454
-YHA London Lee Valley,"Windmill Lane
-Cheshunt, Hertfordshire, GB, HRT, EN8 9AF",yha.org.uk,0845 371 9057
-The Green Man - Bestplace Inn,"308 Edgware Road, London, GB, XGL, W2 1DY",Unclaimed,04402 0772 37980
-Number8ight,"305 High Rd
-Willesden, London, GB, XGL, NW10 2JT",Unclaimed,020 8459 8576
-Hostel Fleming,"198 Brecknock Road, London, GB, XGL, N19 5BE",Unclaimed,020 7609 4282
-Norwegian Ywca,"52 Holland Park, London, GB, XGL, W11 3RS",Unclaimed,020 7727 9897
-Ex Service Fellowship,"New Belvedere House
-White Horse Road, London, GB, XGL, E1 0ND",Unclaimed,020 7790 1474
-Deep Blue Apartments,"Kingsgate House
-12-50 Kingsgate Road, Kingston Upon Thames, GB, XGL, KT2 5AA",Unclaimed,0370 770 0778
-Ashburnham Hostel,"75 Ashburnham Grove, London, GB, XGL, SE10 8UJ",Unclaimed,020 8692 5032
-Museum Inn,"27 Montague Street, London, GB, XGL, WC1B 5BH",Unclaimed,020 7580 5360
-Le Méridien Piccadilly,"21 Piccadilly, London, GB, XGL, W1J 0BH",marriott.com/hotels/travel/lon…,020 7734 8000
-St Christopher&apos;s,"59-61 Borough High Street, London, GB, XGL, SE1 1NE",Unclaimed,020 7407 6266
-St Louise,"33 Medway Street, London, GB, XGL, SW1P 2BE",Unclaimed,020 7222 2071
-Barkston Rooms,"1 Barkston Gardens
-Earls Court, London, GB, XGL, SW5 0ER",Unclaimed,0844 472 1800
-Hotel Strand Continental,"143 Strand, London, GB, XGL, WC2R 1JA",strand-continental.co.uk,020 7836 4880
-Queen Alexandras House,"Kensington Gore, London, GB, XGL, SW7 2QT",Unclaimed,020 7589 3635
-Westmoor House Rehabilitation Centre,"244 Roehampton Lane, London, GB, XGL, SW15 4AA",Unclaimed,020 8780 5863
-Pullman Hotel,"100 - 110 Euston Road, London, GB, XGL, NW1 2AJ",Unclaimed,020 7666 9000
-Welcome Homestay UK,"102 Inchmery Rd, London, GB, LND, SE6 2ND",welcomehomestay.co.uk,07757 938037
-Capitanio Sisters,"Nile Lodge
-Queens Walk, London, GB, XGL, W5 1TJ",Unclaimed,020 8997 2296
-Heritage Star,"54-58 Caledonian Road, London, GB, XGL, N1 9DP",Unclaimed,020 7833 3893
-Club Quarters,"8 Northumberland Avenue
-Trafalgar Square, London, GB, XGL, WC2N 5BY",clubquarters.com,020 7839 9333
-The Georgian House,"37 St Georges Drive, London, GB, XGL, SW1V 4DG",georgianhousehotel.co.uk,020 7834 1438
-Metrogate House,"3 Queens Gate Terrace, London, GB, XGL, SW7 5PE",Unclaimed,020 7823 7440
-Columbia Hotel,"95-99 Lancaster Gate, London, GB, XGL, W2 3NS",Unclaimed,020 7402 0021
-Church Army Emergency Hostel,"1-5 Cosway Street, London, GB, XGL, NW1 5NR",Unclaimed,020 7262 3818
-Shenehom Housing Association,"31-32 Ranelagh Avenue, London, GB, XGL, SW13 0BN",Unclaimed,020 8878 8667
-Classic Property,"Tulsi House
-61 Carlton Grove, London, GB, XGL, SE15 2UD",Unclaimed,07932 656296
-Keyworth Hostel,"10 Keyworth Street, London, GB, XGL, SE1 6NG",Unclaimed,020 7401 8950
-YHA Rotherhithe,"20 Salter Road
-Rotherhithe, London, GB, XGL, SE16 5PR",yha.org.uk/find-accommodation/…,0870 770 6010
-Youth Hostels Association,"20 Salter Road, London, GB, XGL, SE16 5PR",Unclaimed,020 7232 2114
-Local 33,"26 Mayflower Road, London, GB, XGL, SW9 9JZ",Unclaimed,020 7737 3107
-Ned Kellys,"305 High Road, London, GB, XGL, NW10 2JT",Unclaimed,020 8459 6906
diff --git a/yellow-pages/pylg.log b/yellow-pages/pylg.log
deleted file mode 100644
index 0ebb59a..0000000
--- a/yellow-pages/pylg.log
+++ /dev/null
@@ -1,52 +0,0 @@
-2020-12-22 18:46:32,074 - WARNING | bufferization.py/store/42 | Got duplicate or empty dictionary {'Title': '-', 'Address': '-', 'WebSite': '-', 'PhoneNumber': '-'}
-2020-12-22 18:46:33,359 - WARNING | bufferization.py/store/42 | Got duplicate or empty dictionary {'Title': '-', 'Address': '-', 'WebSite': '-', 'PhoneNumber': '-'}
-2020-12-22 18:46:34,790 - WARNING | bufferization.py/store/42 | Got duplicate or empty dictionary {'Title': '-', 'Address': '-', 'WebSite': '-', 'PhoneNumber': '-'}
-2020-12-22 18:46:36,292 - WARNING | bufferization.py/store/42 | Got duplicate or empty dictionary {'Title': '-', 'Address': '-', 'WebSite': '-', 'PhoneNumber': '-'}
-2020-12-22 18:46:37,827 - WARNING | bufferization.py/store/42 | Got duplicate or empty dictionary {'Title': '-', 'Address': '-', 'WebSite': '-', 'PhoneNumber': '-'}
-2020-12-22 18:46:39,783 - WARNING | bufferization.py/store/42 | Got duplicate or empty dictionary {'Title': '-', 'Address': '-', 'WebSite': '-', 'PhoneNumber': '-'}
-2020-12-22 18:46:41,092 - WARNING | bufferization.py/store/42 | Got duplicate or empty dictionary {'Title': '-', 'Address': '-', 'WebSite': '-', 'PhoneNumber': '-'}
-2020-12-22 18:46:42,481 - WARNING | bufferization.py/store/42 | Got duplicate or empty dictionary {'Title': '-', 'Address': '-', 'WebSite': '-', 'PhoneNumber': '-'}
-2020-12-22 18:46:43,892 - WARNING | bufferization.py/store/42 | Got duplicate or empty dictionary {'Title': '-', 'Address': '-', 'WebSite': '-', 'PhoneNumber': '-'}
-2020-12-23 08:59:24,768 - WARNING | bufferization.py/store/42 | Got duplicate or empty dictionary {'Title': '-', 'Address': '-', 'WebSite': '-', 'PhoneNumber': '-'}
-2020-12-23 08:59:26,267 - WARNING | bufferization.py/store/42 | Got duplicate or empty dictionary {'Title': '-', 'Address': '-', 'WebSite': '-', 'PhoneNumber': '-'}
-2020-12-23 08:59:27,700 - WARNING | bufferization.py/store/42 | Got duplicate or empty dictionary {'Title': '-', 'Address': '-', 'WebSite': '-', 'PhoneNumber': '-'}
-2020-12-23 08:59:29,777 - WARNING | bufferization.py/store/42 | Got duplicate or empty dictionary {'Title': '-', 'Address': '-', 'WebSite': '-', 'PhoneNumber': '-'}
-2020-12-23 08:59:31,173 - WARNING | bufferization.py/store/42 | Got duplicate or empty dictionary {'Title': '-', 'Address': '-', 'WebSite': '-', 'PhoneNumber': '-'}
-2020-12-23 08:59:33,312 - WARNING | bufferization.py/store/42 | Got duplicate or empty dictionary {'Title': '-', 'Address': '-', 'WebSite': '-', 'PhoneNumber': '-'}
-2020-12-23 08:59:34,782 - WARNING | bufferization.py/store/42 | Got duplicate or empty dictionary {'Title': '-', 'Address': '-', 'WebSite': '-', 'PhoneNumber': '-'}
-2020-12-23 08:59:36,387 - WARNING | bufferization.py/store/42 | Got duplicate or empty dictionary {'Title': '-', 'Address': '-', 'WebSite': '-', 'PhoneNumber': '-'}
-2020-12-23 08:59:38,050 - WARNING | bufferization.py/store/42 | Got duplicate or empty dictionary {'Title': '-', 'Address': '-', 'WebSite': '-', 'PhoneNumber': '-'}
-2020-12-23 08:59:42,403 - WARNING | bufferization.py/store/42 | Got duplicate or empty dictionary {'Title': '-', 'Address': '-', 'WebSite': '-', 'PhoneNumber': '-'}
-2020-12-23 08:59:43,932 - WARNING | bufferization.py/store/42 | Got duplicate or empty dictionary {'Title': '-', 'Address': '-', 'WebSite': '-', 'PhoneNumber': '-'}
-2020-12-23 08:59:45,613 - WARNING | bufferization.py/store/42 | Got duplicate or empty dictionary {'Title': '-', 'Address': '-', 'WebSite': '-', 'PhoneNumber': '-'}
-2020-12-23 08:59:46,821 - WARNING | bufferization.py/store/42 | Got duplicate or empty dictionary {'Title': '-', 'Address': '-', 'WebSite': '-', 'PhoneNumber': '-'}
-2020-12-23 08:59:48,219 - WARNING | bufferization.py/store/42 | Got duplicate or empty dictionary {'Title': '-', 'Address': '-', 'WebSite': '-', 'PhoneNumber': '-'}
-2020-12-23 09:11:54,715 - WARNING | bufferization.py/store/42 | Got duplicate or empty dictionary {'Title': '-', 'Address': '-', 'WebSite': '-', 'PhoneNumber': '-'}
-2020-12-23 09:11:55,918 - WARNING | bufferization.py/store/42 | Got duplicate or empty dictionary {'Title': '-', 'Address': '-', 'WebSite': '-', 'PhoneNumber': '-'}
-2020-12-23 09:11:57,439 - WARNING | bufferization.py/store/42 | Got duplicate or empty dictionary {'Title': '-', 'Address': '-', 'WebSite': '-', 'PhoneNumber': '-'}
-2020-12-23 09:11:58,668 - WARNING | bufferization.py/store/42 | Got duplicate or empty dictionary {'Title': '-', 'Address': '-', 'WebSite': '-', 'PhoneNumber': '-'}
-2020-12-23 09:11:59,815 - WARNING | bufferization.py/store/42 | Got duplicate or empty dictionary {'Title': '-', 'Address': '-', 'WebSite': '-', 'PhoneNumber': '-'}
-2020-12-23 09:12:00,941 - WARNING | bufferization.py/store/42 | Got duplicate or empty dictionary {'Title': '-', 'Address': '-', 'WebSite': '-', 'PhoneNumber': '-'}
-2020-12-23 09:12:02,260 - WARNING | bufferization.py/store/42 | Got duplicate or empty dictionary {'Title': '-', 'Address': '-', 'WebSite': '-', 'PhoneNumber': '-'}
-2020-12-23 09:12:03,670 - WARNING | bufferization.py/store/42 | Got duplicate or empty dictionary {'Title': '-', 'Address': '-', 'WebSite': '-', 'PhoneNumber': '-'}
-2020-12-23 09:12:05,242 - WARNING | bufferization.py/store/42 | Got duplicate or empty dictionary {'Title': '-', 'Address': '-', 'WebSite': '-', 'PhoneNumber': '-'}
-2020-12-23 09:12:08,690 - WARNING | bufferization.py/store/42 | Got duplicate or empty dictionary {'Title': '-', 'Address': '-', 'WebSite': '-', 'PhoneNumber': '-'}
-2020-12-23 09:12:10,116 - WARNING | bufferization.py/store/42 | Got duplicate or empty dictionary {'Title': '-', 'Address': '-', 'WebSite': '-', 'PhoneNumber': '-'}
-2020-12-23 09:12:11,587 - WARNING | bufferization.py/store/42 | Got duplicate or empty dictionary {'Title': '-', 'Address': '-', 'WebSite': '-', 'PhoneNumber': '-'}
-2020-12-23 09:12:12,800 - WARNING | bufferization.py/store/42 | Got duplicate or empty dictionary {'Title': '-', 'Address': '-', 'WebSite': '-', 'PhoneNumber': '-'}
-2020-12-23 09:32:17,812 - WARNING | bufferization.py/store/42 | Got duplicate or empty dictionary {'Title': 'Sweaty Betty', 'Address': '-', 'WebSite': '-', 'PhoneNumber': '-'}
-2021-02-26 17:41:06,571 - WARNING | extractor.py/parse/82 | Could not locate `raw_data` skipping current page
-2021-02-26 17:43:33,045 - WARNING | extractor.py/parse/82 | Could not locate `raw_data` skipping current page
-2021-02-26 17:45:36,282 - WARNING | extractor.py/parse/82 | Could not locate `raw_data` skipping current page
-2021-02-26 17:46:02,580 - WARNING | extractor.py/parse/82 | Could not locate `raw_data` skipping current page
-2021-02-26 17:48:31,472 - WARNING | extractor.py/parse/82 | Could not locate `raw_data` skipping current page
-2021-02-26 17:50:00,979 - WARNING | extractor.py/parse/82 | Could not locate `raw_data` skipping current page
-2021-02-26 17:52:36,252 - WARNING | extractor.py/parse/82 | Could not locate `raw_data` skipping current page
-2021-02-26 17:53:43,714 - WARNING | extractor.py/parse/82 | Could not locate `raw_data` skipping current page
-2021-02-26 17:59:13,729 - WARNING | extractor.py/parse/82 | Could not locate `raw_data` skipping current page
-2021-02-26 18:00:11,256 - WARNING | bufferization.py/store/42 | Got duplicate or empty dictionary {'WebSite': 'Unclaimed', 'Title': 'Ibis Budget London Whitechapel', 'PhoneNumber': '020 7655 4620', 'Address': '100 Whitechapel Road, London, GB, XGL, E1 1JG'}
-2021-02-26 18:03:21,403 - WARNING | extractor.py/parse/82 | Could not locate `raw_data` skipping current page
-2021-02-26 18:03:30,916 - WARNING | extractor.py/parse/82 | Could not locate `raw_data` skipping current page
-2021-02-26 18:07:32,221 - WARNING | extractor.py/parse/82 | Could not locate `raw_data` skipping current page
-2021-02-26 18:10:09,790 - WARNING | extractor.py/parse/82 | Could not locate `raw_data` skipping current page
-2021-02-26 18:11:26,674 - WARNING | extractor.py/parse/82 | Could not locate `raw_data` skipping current page