Supercharge your AI agents with data from the web and beyond. Building the library for superintelligence. 🔥
TypeScript aiai-agentsai-crawlerai-scrapingai-searchcrawlerdata-extractionhtml-to-markdownllmmarkdownscraperscrapingweb-crawlerweb-dataweb-data-extractionweb-scraperweb-scrapingweb-searchwebscraping
added library.tiktok to allowedKeywords
rafaelmmillerb69c6f9f9510
diff --git a/apps/api/src/scraper/WebScraper/utils/blocklist.ts b/apps/api/src/scraper/WebScraper/utils/blocklist.tsindex 0a7248294..dea4c6144 100644--- a/apps/api/src/scraper/WebScraper/utils/blocklist.ts+++ b/apps/api/src/scraper/WebScraper/utils/blocklist.ts@@ -38,6 +38,7 @@ const allowedKeywords = ['press','conditions','tos',+ '://library.tiktok.com','://ads.tiktok.com','://tiktok.com/business','://developers.facebook.com'