Crawlee—A web scraping and browser automation library for Node.js to build reliable crawlers. In JavaScript and TypeScript. Extract data for AI, LLMs, RAG, or GPTs. Download HTML, PDF, JPG, PNG, and other files from websites. Works with Puppeteer, Playwright, Cheerio, JSDOM, and raw HTTP. Both headful and headless mode. With proxy rotation.
TypeScript apifyautomationcrawlercrawlingheadlessheadless-chromejavascriptnodejsnpmplaywrightpuppeteerscraperscrapingtypescriptweb-crawlerweb-crawlingweb-scraping
Still WIP - paused until later
davidjohnbarton11dff4bfe18b
diff --git a/README.md b/README.mdindex 53da9d3a4..9216ae4f0 100644--- a/README.md+++ b/README.md@@ -76,7 +76,7 @@ since the same language is used to write the scripts as well as the data extractThe goal of the Apify SDK is to fill this gap and provide a toolboxfor generic web scraping, crawling and automation tasks in JavaScript.So don't reinvent the wheel every time you need data from the web,-and focus on writing the code specific to the target website, rather than developing commonalities.+and focus on writing code specific to the target website, rather than developing commonalities.## Overview@@ -125,7 +125,7 @@ The Apify SDK is available as the <a href="https://www.npmjs.com/package/apify"><a href="https://www.apify.com/docs/sdk/apify-runtime-js/latest#Dataset"><code>Dataset</code></a>- Provides a store for structured data and enables theirexport to formats like JSON, JSONL, CSV, Excel or HTML.- The data is stored on local filesystem or in the cloud.+ The data is stored on a local filesystem or in the cloud.Datasets are useful for storing and sharing large tabular crawling results,such as a list of products or real estate offers.</li>@@ -133,7 +133,7 @@ The Apify SDK is available as the <a href="https://www.npmjs.com/package/apify"><a href="https://www.apify.com/docs/sdk/apify-runtime-js/latest#KeyValueStore"><code>KeyValueStore</code></a>- A simple key-value store for arbitrary data records or files, along with their MIME content type.It is ideal for saving screenshots of web pages, PDFs or to persist the state of your crawlers.- The data is stored on local filesystem or in the cloud.+ The data is stored on a local filesystem or in the cloud.</li><li><a href="https://www.apify.com/docs/sdk/apify-runtime-js/latest#AutoscaledPool"><code>AutoscaledPool</code></a>@@ -231,7 +231,7 @@ By default, the crawling data will be stored in a local directory at `./apify_stFor example, the input JSON file for the actor is expected to be in the default key-value storein `./apify_storage/key_value_stores/default/INPUT.json`.-Now you can easily deploy your code to Apify cloud by running:+Now you can easily deploy your code to the Apify cloud by running:```bashapify login@@ -246,7 +246,7 @@ and [Apify Actor](https://www.apify.com/docs/actor) documentation.### Usage on the Apify cloud platformYou can also develop your web scraping project-in an online code editor directly on the Apify cloud. You'll need to have an Apify Account.+in an online code editor directly on the Apify cloud. You'll need to have an Apify account.Go to [Actors](https://my.apify.com/actors)page in the app, click <i>Create new</i> and then go to the<i>Source</i> tab and start writing your code or paste one of the code examples below.@@ -261,10 +261,10 @@ It can run for a few seconds, hours or even infinitely.An actor can perform anything from a simple action such as filling out a web form or sending an email,to complex operations such as crawling an entire website and removing duplicates from a large dataset.-To run an actor, you need to have an [Apify Account](https://my.apify.com/).+To run an actor, you need to have an [Apify account](https://my.apify.com/).Actors can be shared in the [Apify Library](https://www.apify.com/library?&type=acts)so that other people can use them.-But no worries, if you share your actor in the library+But don't worry, if you share your actor in the libraryand somebody uses it, it runs under their account, not yours.**Related links**@@ -450,7 +450,7 @@ This example demonstrates how to use [PuppeteerCrawler](https://www.apify.com/doin combination with [RequestList](https://www.apify.com/docs/sdk/apify-runtime-js/latest#RequestList)and [RequestQueue](https://www.apify.com/docs/sdk/apify-runtime-js/latest#RequestQueue) to recursively scrape the[Hacker News](https://news.ycombinator.com) website using headless Chrome / Puppeteer.-The crawlers starts with a single URL, finds links to next pages,+The crawler starts with a single URL, finds links to next pages,enqueues them and continues until no more desired links are available.The results are stored to the default dataset. In local configuration, the results are represented as JSON files in `./apify_storage/datasets/default`