Supercharge your AI agents with data from the web and beyond. Building the library for superintelligence. 🔥
TypeScript aiai-agentsai-crawlerai-scrapingai-searchcrawlerdata-extractionhtml-to-markdownllmmarkdownscraperscrapingweb-crawlerweb-dataweb-data-extractionweb-scraperweb-scrapingweb-searchwebscraping
feat(api/worker): better queue selection
Gergő Móricz105e7d3606c2
diff --git a/apps/api/src/services/queue-worker.ts b/apps/api/src/services/queue-worker.tsindex 3ee0928ed..e7b0b2e53 100644--- a/apps/api/src/services/queue-worker.ts+++ b/apps/api/src/services/queue-worker.ts@@ -510,7 +510,11 @@ app.listen(workerPort, () => {_logger.info(`Liveness endpoint is running on port ${workerPort}`);});-const myScrapeQueue = Math.floor(Math.random() * queueMultiplexWidth);+const myScrapeQueue = process.env.QUEUE_MULTIPLEX_INDEX !== undefined+ ? parseInt(process.env.QUEUE_MULTIPLEX_INDEX)+ : Math.floor(Math.random() * queueMultiplexWidth);++_logger.info(`Using scrape queue ${myScrapeQueue}`);(async () => {async function failedListener(args: { jobId: string; failedReason: string; prev?: string | undefined; }) {