Crawlee—A web scraping and browser automation library for Node.js to build reliable crawlers. In JavaScript and TypeScript. Extract data for AI, LLMs, RAG, or GPTs. Download HTML, PDF, JPG, PNG, and other files from websites. Works with Puppeteer, Playwright, Cheerio, JSDOM, and raw HTTP. Both headful and headless mode. With proxy rotation.
TypeScript apifyautomationcrawlercrawlingheadlessheadless-chromejavascriptnodejsnpmplaywrightpuppeteerscraperscrapingtypescriptweb-crawlerweb-crawlingweb-scraping
chore(utils): re-add tldts dependency for enqueue-strategy URL filtering
Martin Adámekbd20a49f8a0a
diff --git a/packages/utils/package.json b/packages/utils/package.jsonindex 1b0e96b11..70c93d2d8 100644--- a/packages/utils/package.json+++ b/packages/utils/package.json@@ -51,6 +51,7 @@"file-type": "^21.0.0","robots-parser": "^3.0.1","sax": "^1.4.1",+ "tldts": "^7.0.6","tslib": "^2.8.1","whatwg-mimetype": "^4.0.0","zod": "catalog:"diff --git a/pnpm-lock.yaml b/pnpm-lock.yamlindex cea73cdf8..d4796eca7 100644--- a/pnpm-lock.yaml+++ b/pnpm-lock.yaml@@ -934,6 +934,9 @@ importers:sax:specifier: ^1.4.1version: 1.6.0+ tldts:+ specifier: ^7.0.6+ version: 7.0.28tslib:specifier: ^2.8.1version: 2.8.1