Skip to content

Crawlee—A web scraping and browser automation library for Node.js to build reliable crawlers. In JavaScript and TypeScript. Extract data for AI, LLMs, RAG, or GPTs. Download HTML, PDF, JPG, PNG, and other files from websites. Works with Puppeteer, Playwright, Cheerio, JSDOM, and raw HTTP. Both headful and headless mode. With proxy rotation.

chore: disable worker for memory storage (#1420)

Vlad Frangu8fbb3e81cf7f
diff --git a/packages/memory-storage/src/workers/instance.ts b/packages/memory-storage/src/workers/instance.ts
index 18c628987..f26544c88 100644
--- a/packages/memory-storage/src/workers/instance.ts
+++ b/packages/memory-storage/src/workers/instance.ts
@@ -1,4 +1,3 @@
-import { existsSync } from 'node:fs';
import { resolve } from 'node:path';
import { Worker } from 'node:worker_threads';
import type { WorkerReceivedMessage } from '../utils';
@@ -21,7 +20,9 @@ export function initWorkerIfNeeded() {
});
const workerPath = resolve(__dirname, './file-storage-worker.js');
- const exists = existsSync(workerPath);
+ // vladfrangu: The worker is temporarily disabled due to node/v8 having internal bugs that sometimes cause hard crashes when the process exits.
+ // const exists = existsSync(workerPath);
+ const exists = false;
if (exists) {
workerInstance = new Worker(workerPath);