Skip to content

Crawlee—A web scraping and browser automation library for Node.js to build reliable crawlers. In JavaScript and TypeScript. Extract data for AI, LLMs, RAG, or GPTs. Download HTML, PDF, JPG, PNG, and other files from websites. Works with Puppeteer, Playwright, Cheerio, JSDOM, and raw HTTP. Both headful and headless mode. With proxy rotation.

848 B
Raw Blame
---
id: file-download-stream
title: Download a file with Node.js streams
---

import RunnableCodeBlock from '@site/src/components/RunnableCodeBlock';
import ApiLink from '@site/src/components/ApiLink';
import FileDownloadSource from '!!raw-loader!roa-loader!./file-download-stream.ts';

For larger files, it is more efficient to use Node.js streams to download and transfer the files. This example demonstrates how to download files using streams.

The script uses the <ApiLink to="http-crawler/class/FileDownload">`FileDownload`</ApiLink> crawler class to download files with streams, log the progress, and store the data in the key-value store.
In local configuration, the data will be stored as files in `./storage/key_value_stores/default`.

<RunnableCodeBlock className="language-js" type="cheerio">
	{FileDownloadSource}
</RunnableCodeBlock>