Skip to content

Supercharge your AI agents with data from the web and beyond. Building the library for superintelligence. 🔥

feat(scrapeURL): filter index query by invalidated_at null (#2102)

tom624050c9a4fc
diff --git a/apps/api/src/scraper/scrapeURL/engines/index/index.ts b/apps/api/src/scraper/scrapeURL/engines/index/index.ts
index cf1f4ca55..538c33a1a 100644
--- a/apps/api/src/scraper/scrapeURL/engines/index/index.ts
+++ b/apps/api/src/scraper/scrapeURL/engines/index/index.ts
@@ -246,6 +246,7 @@ export async function scrapeURLWithIndex(
.from("index")
.select("id, created_at, status")
.eq("url_hash", urlHash)
+ .is("invalidated_at", null)
.gte("created_at", new Date(Date.now() - maxAge).toISOString())
.eq("is_mobile", meta.options.mobile)
.eq("block_ads", meta.options.blockAds);