Supercharge your AI agents with data from the web and beyond. Building the library for superintelligence. 🔥
TypeScript aiai-agentsai-crawlerai-scrapingai-searchcrawlerdata-extractionhtml-to-markdownllmmarkdownscraperscrapingweb-crawlerweb-dataweb-data-extractionweb-scraperweb-scrapingweb-searchwebscraping
feat(scrapeURL): filter index query by invalidated_at null (#2102)
tom624050c9a4fc
diff --git a/apps/api/src/scraper/scrapeURL/engines/index/index.ts b/apps/api/src/scraper/scrapeURL/engines/index/index.tsindex cf1f4ca55..538c33a1a 100644--- a/apps/api/src/scraper/scrapeURL/engines/index/index.ts+++ b/apps/api/src/scraper/scrapeURL/engines/index/index.ts@@ -246,6 +246,7 @@ export async function scrapeURLWithIndex(.from("index").select("id, created_at, status").eq("url_hash", urlHash)+ .is("invalidated_at", null).gte("created_at", new Date(Date.now() - maxAge).toISOString()).eq("is_mobile", meta.options.mobile).eq("block_ads", meta.options.blockAds);