🕷️ The BEST Web Scraping framework that handles everything from a single request to a full-scale crawl! Don't be shy, join here: https://discord.gg/EMgGbDceNQ and follow here for daily tips and tricks: https://x.com/Scrapling_dev
Python aiai-scrapingautomationcrawlercrawlingcrawling-pythondatadata-extractionmcpmcp-serverplaywrightpythonscrapingselectorsstealthweb-scraperweb-scrapingweb-scraping-pythonwebscrapingxpath
fix: remove dead fetchall() after INSERT and correct retrieve() error message (#377)
Semianchuk Vitalii2f400555c3cd
diff --git a/README.md b/README.mdindex 8d95c33..2f39ac9 100644--- a/README.md+++ b/README.md@@ -98,11 +98,11 @@ MySpider().start()</tr><tr><td width="200">- <a href="https://coldproxy.com/" target="_blank" title="Residential, IPv6 & Datacenter Proxies for Web Scraping">+ <a href="https://coldproxy.com/?utm_source=scrapling&utm_medium=github&utm_campaign=coldproxy&utm_content=platinum_sponsor" target="_blank" title="Residential, IPv6 & Datacenter Proxies for Web Scraping"><img src="https://raw.githubusercontent.com/D4Vinci/Scrapling/main/images/coldproxy.png"></a></td>- <td> <a href="https://coldproxy.com/" target="_blank"><b>ColdProxy</b></a> provides residential and datacenter proxies for stable web scraping, public data collection, and geo-targeted testing across 195+ countries.+ <td> <a href="https://coldproxy.com/?utm_source=scrapling&utm_medium=github&utm_campaign=coldproxy&utm_content=platinum_sponsor" target="_blank"><b>ColdProxy</b></a> provides residential and datacenter proxies for stable web scraping, public data collection, and geo-targeted testing across 195+ countries.</td></tr><tr>diff --git a/docs/README_AR.md b/docs/README_AR.mdindex 972c562..4a43fea 100644--- a/docs/README_AR.md+++ b/docs/README_AR.md@@ -94,11 +94,11 @@ MySpider().start()</tr><tr><td width="200">- <a href="https://coldproxy.com/" target="_blank" title="Residential, IPv6 & Datacenter Proxies for Web Scraping">+ <a href="https://coldproxy.com/?utm_source=scrapling&utm_medium=github&utm_campaign=coldproxy&utm_content=platinum_sponsor" target="_blank" title="Residential, IPv6 & Datacenter Proxies for Web Scraping"><img src="https://raw.githubusercontent.com/D4Vinci/Scrapling/main/images/coldproxy.png"></a></td>- <td> توفر <a href="https://coldproxy.com/" target="_blank"><b>ColdProxy</b></a> وكلاء سكنيين ووكلاء مراكز بيانات لاستخراج بيانات الويب بشكل مستقر، وجمع البيانات العامة، والاختبار الموجَّه جغرافياً في أكثر من 195 دولة.+ <td> توفر <a href="https://coldproxy.com/?utm_source=scrapling&utm_medium=github&utm_campaign=coldproxy&utm_content=platinum_sponsor" target="_blank"><b>ColdProxy</b></a> وكلاء سكنيين ووكلاء مراكز بيانات لاستخراج بيانات الويب بشكل مستقر، وجمع البيانات العامة، والاختبار الموجَّه جغرافياً في أكثر من 195 دولة.</td></tr><tr>diff --git a/docs/README_CN.md b/docs/README_CN.mdindex 9a144e2..c1ec984 100644--- a/docs/README_CN.md+++ b/docs/README_CN.md@@ -94,11 +94,11 @@ MySpider().start()</tr><tr><td width="200">- <a href="https://coldproxy.com/" target="_blank" title="Residential, IPv6 & Datacenter Proxies for Web Scraping">+ <a href="https://coldproxy.com/?utm_source=scrapling&utm_medium=github&utm_campaign=coldproxy&utm_content=platinum_sponsor" target="_blank" title="Residential, IPv6 & Datacenter Proxies for Web Scraping"><img src="https://raw.githubusercontent.com/D4Vinci/Scrapling/main/images/coldproxy.png"></a></td>- <td> <a href="https://coldproxy.com/" target="_blank"><b>ColdProxy</b></a> 提供住宅代理和数据中心代理,用于稳定的网络抓取、公共数据收集,以及覆盖 195 多个国家/地区的地理定向测试。+ <td> <a href="https://coldproxy.com/?utm_source=scrapling&utm_medium=github&utm_campaign=coldproxy&utm_content=platinum_sponsor" target="_blank"><b>ColdProxy</b></a> 提供住宅代理和数据中心代理,用于稳定的网络抓取、公共数据收集,以及覆盖 195 多个国家/地区的地理定向测试。</td></tr><tr>diff --git a/docs/README_DE.md b/docs/README_DE.mdindex aad2ad2..c2d5079 100644--- a/docs/README_DE.md+++ b/docs/README_DE.md@@ -94,11 +94,11 @@ MySpider().start()</tr><tr><td width="200">- <a href="https://coldproxy.com/" target="_blank" title="Residential, IPv6 & Datacenter Proxies for Web Scraping">+ <a href="https://coldproxy.com/?utm_source=scrapling&utm_medium=github&utm_campaign=coldproxy&utm_content=platinum_sponsor" target="_blank" title="Residential, IPv6 & Datacenter Proxies for Web Scraping"><img src="https://raw.githubusercontent.com/D4Vinci/Scrapling/main/images/coldproxy.png"></a></td>- <td> <a href="https://coldproxy.com/" target="_blank"><b>ColdProxy</b></a> bietet Residential- und Datacenter-Proxies für stabiles Web Scraping, öffentliche Datenerfassung und geografisch gezielte Tests in über 195 Ländern.+ <td> <a href="https://coldproxy.com/?utm_source=scrapling&utm_medium=github&utm_campaign=coldproxy&utm_content=platinum_sponsor" target="_blank"><b>ColdProxy</b></a> bietet Residential- und Datacenter-Proxies für stabiles Web Scraping, öffentliche Datenerfassung und geografisch gezielte Tests in über 195 Ländern.</td></tr><tr>diff --git a/docs/README_ES.md b/docs/README_ES.mdindex ce1513d..e5a7b22 100644--- a/docs/README_ES.md+++ b/docs/README_ES.md@@ -94,11 +94,11 @@ MySpider().start()</tr><tr><td width="200">- <a href="https://coldproxy.com/" target="_blank" title="Residential, IPv6 & Datacenter Proxies for Web Scraping">+ <a href="https://coldproxy.com/?utm_source=scrapling&utm_medium=github&utm_campaign=coldproxy&utm_content=platinum_sponsor" target="_blank" title="Residential, IPv6 & Datacenter Proxies for Web Scraping"><img src="https://raw.githubusercontent.com/D4Vinci/Scrapling/main/images/coldproxy.png"></a></td>- <td> <a href="https://coldproxy.com/" target="_blank"><b>ColdProxy</b></a> proporciona proxies residenciales y de centros de datos para web scraping estable, recopilación de datos públicos y pruebas con segmentación geográfica en más de 195 países.+ <td> <a href="https://coldproxy.com/?utm_source=scrapling&utm_medium=github&utm_campaign=coldproxy&utm_content=platinum_sponsor" target="_blank"><b>ColdProxy</b></a> proporciona proxies residenciales y de centros de datos para web scraping estable, recopilación de datos públicos y pruebas con segmentación geográfica en más de 195 países.</td></tr><tr>diff --git a/docs/README_FR.md b/docs/README_FR.mdindex e1694c6..f27ff24 100644--- a/docs/README_FR.md+++ b/docs/README_FR.md@@ -94,11 +94,11 @@ MySpider().start()</tr><tr><td width="200">- <a href="https://coldproxy.com/" target="_blank" title="Residential, IPv6 & Datacenter Proxies for Web Scraping">+ <a href="https://coldproxy.com/?utm_source=scrapling&utm_medium=github&utm_campaign=coldproxy&utm_content=platinum_sponsor" target="_blank" title="Residential, IPv6 & Datacenter Proxies for Web Scraping"><img src="https://raw.githubusercontent.com/D4Vinci/Scrapling/main/images/coldproxy.png"></a></td>- <td> <a href="https://coldproxy.com/" target="_blank"><b>ColdProxy</b></a> fournit des proxies résidentiels et de datacenter pour un web scraping stable, la collecte de données publiques et des tests géolocalisés dans plus de 195 pays.+ <td> <a href="https://coldproxy.com/?utm_source=scrapling&utm_medium=github&utm_campaign=coldproxy&utm_content=platinum_sponsor" target="_blank"><b>ColdProxy</b></a> fournit des proxies résidentiels et de datacenter pour un web scraping stable, la collecte de données publiques et des tests géolocalisés dans plus de 195 pays.</td></tr><tr>diff --git a/docs/README_JP.md b/docs/README_JP.mdindex f0e30dc..a9f0e10 100644--- a/docs/README_JP.md+++ b/docs/README_JP.md@@ -94,11 +94,11 @@ MySpider().start()</tr><tr><td width="200">- <a href="https://coldproxy.com/" target="_blank" title="Residential, IPv6 & Datacenter Proxies for Web Scraping">+ <a href="https://coldproxy.com/?utm_source=scrapling&utm_medium=github&utm_campaign=coldproxy&utm_content=platinum_sponsor" target="_blank" title="Residential, IPv6 & Datacenter Proxies for Web Scraping"><img src="https://raw.githubusercontent.com/D4Vinci/Scrapling/main/images/coldproxy.png"></a></td>- <td> <a href="https://coldproxy.com/" target="_blank"><b>ColdProxy</b></a> は、安定したウェブスクレイピング、公開データ収集、195以上の国・地域でのジオターゲティングテストのために、レジデンシャルおよびデータセンタープロキシを提供します。+ <td> <a href="https://coldproxy.com/?utm_source=scrapling&utm_medium=github&utm_campaign=coldproxy&utm_content=platinum_sponsor" target="_blank"><b>ColdProxy</b></a> は、安定したウェブスクレイピング、公開データ収集、195以上の国・地域でのジオターゲティングテストのために、レジデンシャルおよびデータセンタープロキシを提供します。</td></tr><tr>diff --git a/docs/README_KR.md b/docs/README_KR.mdindex 6b013a7..193530a 100644--- a/docs/README_KR.md+++ b/docs/README_KR.md@@ -94,11 +94,11 @@ MySpider().start()</tr><tr><td width="200">- <a href="https://coldproxy.com/" target="_blank" title="Residential, IPv6 & Datacenter Proxies for Web Scraping">+ <a href="https://coldproxy.com/?utm_source=scrapling&utm_medium=github&utm_campaign=coldproxy&utm_content=platinum_sponsor" target="_blank" title="Residential, IPv6 & Datacenter Proxies for Web Scraping"><img src="https://raw.githubusercontent.com/D4Vinci/Scrapling/main/images/coldproxy.png"></a></td>- <td> <a href="https://coldproxy.com/" target="_blank"><b>ColdProxy</b></a>는 안정적인 웹 스크래핑, 공개 데이터 수집, 195개 이상의 국가에서의 지역 타겟팅 테스트를 위한 주거용 및 데이터센터 프록시를 제공합니다.+ <td> <a href="https://coldproxy.com/?utm_source=scrapling&utm_medium=github&utm_campaign=coldproxy&utm_content=platinum_sponsor" target="_blank"><b>ColdProxy</b></a>는 안정적인 웹 스크래핑, 공개 데이터 수집, 195개 이상의 국가에서의 지역 타겟팅 테스트를 위한 주거용 및 데이터센터 프록시를 제공합니다.</td></tr><tr>diff --git a/docs/README_PT_BR.md b/docs/README_PT_BR.mdindex 65de3a2..e1defaa 100644--- a/docs/README_PT_BR.md+++ b/docs/README_PT_BR.md@@ -96,11 +96,11 @@ MySpider().start()</tr><tr><td width="200">- <a href="https://coldproxy.com/" target="_blank" title="Residential, IPv6 & Datacenter Proxies for Web Scraping">+ <a href="https://coldproxy.com/?utm_source=scrapling&utm_medium=github&utm_campaign=coldproxy&utm_content=platinum_sponsor" target="_blank" title="Residential, IPv6 & Datacenter Proxies for Web Scraping"><img src="https://raw.githubusercontent.com/D4Vinci/Scrapling/main/images/coldproxy.png"></a></td>- <td> A <a href="https://coldproxy.com/" target="_blank"><b>ColdProxy</b></a> oferece proxies residenciais e de datacenter para web scraping estável, coleta de dados públicos e testes com segmentação geográfica em mais de 195 países.+ <td> A <a href="https://coldproxy.com/?utm_source=scrapling&utm_medium=github&utm_campaign=coldproxy&utm_content=platinum_sponsor" target="_blank"><b>ColdProxy</b></a> oferece proxies residenciais e de datacenter para web scraping estável, coleta de dados públicos e testes com segmentação geográfica em mais de 195 países.</td></tr><tr>diff --git a/docs/README_RU.md b/docs/README_RU.mdindex fdb708e..c153482 100644--- a/docs/README_RU.md+++ b/docs/README_RU.md@@ -94,11 +94,11 @@ MySpider().start()</tr><tr><td width="200">- <a href="https://coldproxy.com/" target="_blank" title="Residential, IPv6 & Datacenter Proxies for Web Scraping">+ <a href="https://coldproxy.com/?utm_source=scrapling&utm_medium=github&utm_campaign=coldproxy&utm_content=platinum_sponsor" target="_blank" title="Residential, IPv6 & Datacenter Proxies for Web Scraping"><img src="https://raw.githubusercontent.com/D4Vinci/Scrapling/main/images/coldproxy.png"></a></td>- <td> <a href="https://coldproxy.com/" target="_blank"><b>ColdProxy</b></a> предоставляет резидентные и дата-центровые прокси для стабильного веб-скрейпинга, сбора публичных данных и гео-таргетированного тестирования в более чем 195 странах.+ <td> <a href="https://coldproxy.com/?utm_source=scrapling&utm_medium=github&utm_campaign=coldproxy&utm_content=platinum_sponsor" target="_blank"><b>ColdProxy</b></a> предоставляет резидентные и дата-центровые прокси для стабильного веб-скрейпинга, сбора публичных данных и гео-таргетированного тестирования в более чем 195 странах.</td></tr><tr>diff --git a/docs/index.md b/docs/index.mdindex fdf604f..4a886fa 100644--- a/docs/index.md+++ b/docs/index.md@@ -59,7 +59,7 @@ MySpider().start()<a href="https://proxidize.com/?utm_source=github&utm_medium=sponsorship&utm_campaign=scrapling&utm_content=d4vinci" target="_blank" title="Clean Proxies with No Nonsense."><img src="https://raw.githubusercontent.com/D4Vinci/Scrapling/main/images/proxidize.png" class="ad"></a>- <a href="https://coldproxy.com/" target="_blank" title="Residential, IPv6 & Datacenter Proxies for Web Scraping">+ <a href="https://coldproxy.com/?utm_source=scrapling&utm_medium=github&utm_campaign=coldproxy&utm_content=platinum_sponsor" target="_blank" title="Residential, IPv6 & Datacenter Proxies for Web Scraping"><img src="https://raw.githubusercontent.com/D4Vinci/Scrapling/main/images/coldproxy.png" class="ad"></a><a href="https://hypersolutions.co/?utm_source=github&utm_medium=readme&utm_campaign=scrapling" target="_blank" title="Bot Protection Bypass API for Akamai, DataDome, Incapsula & Kasada">diff --git a/scrapling/core/storage.py b/scrapling/core/storage.pyindex d93a320..1fccabe 100644--- a/scrapling/core/storage.py+++ b/scrapling/core/storage.py@@ -56,7 +56,7 @@ class StorageSystemMixin(ABC): # pragma: no coverthe docs for more info.:return: A dictionary of the unique properties"""- raise NotImplementedError("Storage system must implement `save` method")+ raise NotImplementedError("Storage system must implement `retrieve` method")@staticmethod@lru_cache(128, typed=True)@@ -123,7 +123,6 @@ class SQLiteStorageSystem(StorageSystemMixin):""",(url, identifier, dumps(element_data)),)- self.cursor.fetchall()self.connection.commit()def retrieve(self, identifier: str) -> Optional[Dict[str, Any]]: