Skip to content

Crawlee—A web scraping and browser automation library for Node.js to build reliable crawlers. In JavaScript and TypeScript. Extract data for AI, LLMs, RAG, or GPTs. Download HTML, PDF, JPG, PNG, and other files from websites. Works with Puppeteer, Playwright, Cheerio, JSDOM, and raw HTTP. Both headful and headless mode. With proxy rotation.

chore(utils): re-add tldts dependency for enqueue-strategy URL filtering

Martin Adámekbd20a49f8a0a
diff --git a/packages/utils/package.json b/packages/utils/package.json
index 1b0e96b11..70c93d2d8 100644
--- a/packages/utils/package.json
+++ b/packages/utils/package.json
@@ -51,6 +51,7 @@
"file-type": "^21.0.0",
"robots-parser": "^3.0.1",
"sax": "^1.4.1",
+ "tldts": "^7.0.6",
"tslib": "^2.8.1",
"whatwg-mimetype": "^4.0.0",
"zod": "catalog:"
diff --git a/pnpm-lock.yaml b/pnpm-lock.yaml
index cea73cdf8..d4796eca7 100644
--- a/pnpm-lock.yaml
+++ b/pnpm-lock.yaml
@@ -934,6 +934,9 @@ importers:
sax:
specifier: ^1.4.1
version: 1.6.0
+ tldts:
+ specifier: ^7.0.6
+ version: 7.0.28
tslib:
specifier: ^2.8.1
version: 2.8.1