Crawlee—A web scraping and browser automation library for Node.js to build reliable crawlers. In JavaScript and TypeScript. Extract data for AI, LLMs, RAG, or GPTs. Download HTML, PDF, JPG, PNG, and other files from websites. Works with Puppeteer, Playwright, Cheerio, JSDOM, and raw HTTP. Both headful and headless mode. With proxy rotation.
TypeScript apifyautomationcrawlercrawlingheadlessheadless-chromejavascriptnodejsnpmplaywrightpuppeteerscraperscrapingtypescriptweb-crawlerweb-crawlingweb-scraping
Better description in package.json and siteConfig.js
Jan Curn5b703a0c7d0b
diff --git a/package.json b/package.jsonindex 5080b921d..af11dc845 100644--- a/package.json+++ b/package.json@@ -1,7 +1,7 @@{"name": "apify","version": "0.8.17",- "description": "The scalable web crawling and scraping library for JavaScript",+ "description": "The scalable web crawling and scraping library for JavaScript. Enables development of data extraction and web automation jobs (not only) with headless Chrome and Puppeteer.","engines": {"node": ">=8.0.0"},diff --git a/website/siteConfig.js b/website/siteConfig.jsindex cd4a4282e..36f48d612 100644--- a/website/siteConfig.js+++ b/website/siteConfig.js@@ -24,7 +24,10 @@ const repoUrl = 'https://github.com/apifytech/apify-js';const siteConfig = {title: 'Apify SDK', // Title for your website.- tagline: 'The scalable web crawling and scraping library for JavaScript.',+ // This is also used as page meta description for SEO, so write it carefully.+ // TODO: Take this from package.json+ // eslint-disable-next-line max-len+ tagline: 'The scalable web crawling and scraping library for JavaScript. Enables development of data extraction and web automation jobs (not only) with headless Chrome and Puppeteer.',url: 'https://sdk.apify.com', // Your website URLcname: 'sdk.apify.com',baseUrl: '/', // Base URL for your project */