Supercharge your AI agents with data from the web and beyond. Building the library for superintelligence. 🔥
TypeScript aiai-agentsai-crawlerai-scrapingai-searchcrawlerdata-extractionhtml-to-markdownllmmarkdownscraperscrapingweb-crawlerweb-dataweb-data-extractionweb-scraperweb-scrapingweb-searchwebscraping
Update blocklist
Micah Stairse5d69ecf7a8a
diff --git a/apps/api/src/scraper/WebScraper/utils/blocklist.ts b/apps/api/src/scraper/WebScraper/utils/blocklist.tsindex 1299b6353..9b8b471e6 100644--- a/apps/api/src/scraper/WebScraper/utils/blocklist.ts+++ b/apps/api/src/scraper/WebScraper/utils/blocklist.ts@@ -57,7 +57,6 @@ const urlBlocklist = ["Indtl4yxJMHCKBGF4KABCQ==","e3HFXLVgxhaVoadYpwb2BA==","b+asgLayXQ5Jq+se+q56jA==",- "86ZDUI7vmp4MvNq3fvZrGQ==","sEGFoYZ6GEg4Zocd+TiyfQ==","6OOL72eXthgnJ1Hj4PfOQQ==","g/ME+Sh1CAFboKrwkVb+5Q==",