Supercharge your AI agents with data from the web and beyond. Building the library for superintelligence. 🔥
TypeScript aiai-agentsai-crawlerai-scrapingai-searchcrawlerdata-extractionhtml-to-markdownllmmarkdownscraperscrapingweb-crawlerweb-dataweb-data-extractionweb-scraperweb-scrapingweb-searchwebscraping
fix(queue-worker): proper concurrency
Gergő Móriczcf9010a1268e
diff --git a/apps/api/src/services/queue-worker.ts b/apps/api/src/services/queue-worker.tsindex 9524d37b5..20b42a524 100644--- a/apps/api/src/services/queue-worker.ts+++ b/apps/api/src/services/queue-worker.ts@@ -338,7 +338,7 @@ const separateWorkerFun = (lockDuration: 30 * 1000, // 30 secondsstalledInterval: 30 * 1000, // 30 secondsmaxStalledCount: 10, // 10 times- concurrency: 6, // from k8s setup+ concurrency: Math.ceil(6 / scrapeQueueNames.length), // from k8s setupuseWorkerThreads: true,});