Supercharge your AI agents with data from the web and beyond. Building the library for superintelligence. 🔥
TypeScript aiai-agentsai-crawlerai-scrapingai-searchcrawlerdata-extractionhtml-to-markdownllmmarkdownscraperscrapingweb-crawlerweb-dataweb-data-extractionweb-scraperweb-scrapingweb-searchwebscraping
Add exceptions to blocklist (#2156)
Micah Stairs175c5a393204
diff --git a/apps/api/src/scraper/WebScraper/utils/blocklist.ts b/apps/api/src/scraper/WebScraper/utils/blocklist.tsindex fdf1f538f..90896d746 100644--- a/apps/api/src/scraper/WebScraper/utils/blocklist.ts+++ b/apps/api/src/scraper/WebScraper/utils/blocklist.ts@@ -90,6 +90,8 @@ const allowedKeywords = ["://creditcards.aa.com","://aa.org","://www.aa.org",+ "://www.reddit.com/dev/api",+ "://ads-api.reddit.com",];function decryptedBlocklist(list: string[]): string[] {