Supercharge your AI agents with data from the web and beyond. Building the library for superintelligence. 🔥
TypeScript aiai-agentsai-crawlerai-scrapingai-searchcrawlerdata-extractionhtml-to-markdownllmmarkdownscraperscrapingweb-crawlerweb-dataweb-data-extractionweb-scraperweb-scrapingweb-searchwebscraping
Update types.ts
Nicolas64546f1259b2
diff --git a/apps/api/src/controllers/v1/types.ts b/apps/api/src/controllers/v1/types.tsindex f03d014ae..da8847a8b 100644--- a/apps/api/src/controllers/v1/types.ts+++ b/apps/api/src/controllers/v1/types.ts@@ -232,7 +232,7 @@ const crawlerOptions = z.object({allowExternalLinks: z.boolean().default(false),allowSubdomains: z.boolean().default(false),ignoreRobotsTxt: z.boolean().default(false),- ignoreSitemap: z.boolean().default(true),+ ignoreSitemap: z.boolean().default(false),deduplicateSimilarURLs: z.boolean().default(true),ignoreQueryParameters: z.boolean().default(false),}).strict(strictMessage);