Crawlee—A web scraping and browser automation library for Node.js to build reliable crawlers. In JavaScript and TypeScript. Extract data for AI, LLMs, RAG, or GPTs. Download HTML, PDF, JPG, PNG, and other files from websites. Works with Puppeteer, Playwright, Cheerio, JSDOM, and raw HTTP. Both headful and headless mode. With proxy rotation.
TypeScript apifyautomationcrawlercrawlingheadlessheadless-chromejavascriptnodejsnpmplaywrightpuppeteerscraperscrapingtypescriptweb-crawlerweb-crawlingweb-scraping
@crawlee/got-scraping-client
This package provides a Crawlee-compliant HttpClient interface for the got-scraping package.
To use the got-scraping package directly without Crawlee, check out got-scraping on NPM.
Example usage
Simply pass the GotScrapingHttpClient instance to the httpClient option of the crawler constructor:
import { CheerioCrawler, Dictionary } from '@crawlee/cheerio';
import { GotScrapingHttpClient } from '@crawlee/got-scraping-client';
const crawler = new CheerioCrawler({
httpClient: new GotScrapingHttpClient(),
async requestHandler({ $, request }) {
// Extract the title of the page.
const title = $('title').text();
console.log(`Title of the page ${request.url}: ${title}`);
},
});
crawler.run([
'http://www.example.com/page-1',
'http://www.example.com/page-2',
]);