Crawlee—A web scraping and browser automation library for Node.js to build reliable crawlers. In JavaScript and TypeScript. Extract data for AI, LLMs, RAG, or GPTs. Download HTML, PDF, JPG, PNG, and other files from websites. Works with Puppeteer, Playwright, Cheerio, JSDOM, and raw HTTP. Both headful and headless mode. With proxy rotation.
TypeScript apifyautomationcrawlercrawlingheadlessheadless-chromejavascriptnodejsnpmplaywrightpuppeteerscraperscrapingtypescriptweb-crawlerweb-crawlingweb-scraping
fix: remove old system info implementation
Martin Adámekdb404a1f0190
diff --git a/docs/experiments/systemInfoV2.mdx b/docs/experiments/systemInfoV2.mdxdeleted file mode 100644index fb9a481c4..000000000--- a/docs/experiments/systemInfoV2.mdx+++ /dev/null@@ -1,95 +0,0 @@-----id: experiments-system-information-v2-title: System Information V2-description: Improved autoscaling through cgroup aware metric collection.------import ApiLink from '@site/src/components/ApiLink';--:::caution--This is an experimental feature. While we welcome testers, keep in mind that it is currently not recommended to use this in production.--The API is subject to change, and we might introduce breaking changes in the future.--Should you be using this, feel free to open issues on our [GitHub repository](https://github.com/apify/crawlee), and we'll take a look.--:::--Starting with the newest `crawlee` beta, we have introduced a new crawler option that enables an improved metric collection system.-This new system should collect cpu and memory metrics more accurately in containerised environments by checking for cgroup enforce limits.--## How to enable the experiment--:::note--This example shows how to enable the experiment in the <ApiLink to="cheerio-crawler/class/CheerioCrawler">`CheerioCrawler`</ApiLink>,-but you can apply this to any crawler type.--:::--```ts-import { CheerioCrawler, Configuration } from 'crawlee';--Configuration.set('systemInfoV2', true);--const crawler = new CheerioCrawler({- async requestHandler({ $, request }) {- const title = $('title').text();- console.log(`The title of "${request.url}" is: ${title}.`);- },-});--await crawler.run(['https://crawlee.dev']);-```--## Other changes--:::info--This section is only useful if you're a tinkerer and want to see what's going on under the hood.--:::--The existing solution checked the bare metal metrics for how much cpu and memory was being used and how much headroom was available.-This is an intuitive solution but unfortunately doesnt account for when there is an external limit on the amount of resources a process can consume.-This is often the case in containerized environments where each container will have a quota for its cpu and memory usage.--This experiment attempts to address this issue by introducing a new `isContainerized()` utility function and changing the way resources are collected-when a container is detected.--:::note--This `isContainerized()` function is very similar to the existing `isDocker()` function however for now they both work side by side.-If this experiment is successful, `isDocker()` may eventually be deprecated in favour of `isContainerized()`.--:::--### Cgroup detection--On linux, to detect if cgroup is available, we check if there is a directory at `/sys/fs/cgroup`.-If the directory exists, a version of cgroup is installed.-Next we check the version of cgroup installed by checking for a directory at `/sys/fs/cgroup/memory/`.-If it exists, cgroup V1 is installed. If it is missing, it is assumed cgroup V2 is installed.--### CPU metric collection--The existing solution worked by checking the fraction of cpu idle ticks to the total number of cpu ticks since the last profile.-If 100000 ticks elapse and 5000 were idle, the cpu is at 95% utilisation.--In this experiment, the method of cpu load calculation depends on the result of `isContainerized()` or if set, the `CRAWLEE_CONTAINERIZED` environment variable.-If `isContainerized()` returns true, the new cgroup aware metric collection will be used over the "bare metal" numbers.-This works by inspecting the `/sys/fs/cgroup/cpuacct/cpuacct.usage`, `/sys/fs/cgroup/cpu/cpu.cfs_quota_us` and `/sys/fs/cgroup/cpu/cpu.cfs_period_us`-files for cgroup V1 and the `/sys/fs/cgroup/cpu.stat` and `/sys/fs/cgroup/cpu.max` files for cgroup V2.-The actual cpu usage figure is calculated in the same manner as the "bare metal" figure by comparing the total number of ticks elapsed to the number-of idle ticks between profiles but by using the figures from the cgroup files.-If no cgroup quota is enforced, the "bare metal" numbers will be used.--### Memory metric collection--The existing solution was already cgroup aware however an improvement has been made to memory metric collection when running on windows.-The existing solution used an external package `apify/ps-tree` to find the amount of memory crawlee and any child processes were using.-On Windows, this package used the depreciated "WMIC" command line utility to determine memory usage.--In this experiment, `apify/ps-tree` has been removed and replaced by the `packages/utils/src/internals/ps-tree.ts` file. This works in much the-same manner however, instead of using "WMIC", it uses "powershell" to collect the same data.\ No newline at end of filediff --git a/docs/guides/configuration.mdx b/docs/guides/configuration.mdxindex 597c3dcc2..22ce4ec2e 100644--- a/docs/guides/configuration.mdx+++ b/docs/guides/configuration.mdx@@ -94,7 +94,6 @@ Storage directories are purged by default. If set to `false` - local storage dir#### `CRAWLEE_CONTAINERIZED`-This variable is only effective when the systemInfoV2 experiment is enabled.Changes how crawlee measures its CPU and Memory usage and limits. If unset, crawlee will determine if it is containerised using common features of containerized environments using the `isContainerized` utility function.- A file at `/.dockerenv`.- A file at `/proc/self/cgroup` containing `docker`.diff --git a/docs/upgrading/upgrading_v4.md b/docs/upgrading/upgrading_v4.mdindex de696db05..56f527752 100644--- a/docs/upgrading/upgrading_v4.md+++ b/docs/upgrading/upgrading_v4.md@@ -47,3 +47,7 @@ Previously, the crawling context extended a `Record` type, allowing to access an## Remove `experimentalContainers` optionThis experimental option relied on an outdated manifest version for browser extensions, it is not possible to achieve this with the currently supported versions.++## Available resource detection++In v3, we introduced a new way to detect available resources for the crawler, available via `systemInfoV2` flag. In v4, this is the default way to detect available resources. The old way is removed completely together with the `systemInfoV2` flag.diff --git a/packages/core/src/configuration.ts b/packages/core/src/configuration.tsindex 3dbd66f64..7155efe3a 100644--- a/packages/core/src/configuration.ts+++ b/packages/core/src/configuration.ts@@ -167,13 +167,6 @@ export interface ConfigurationOptions {*/persistStorage?: boolean;- /**- * Defines whether to use the systemInfoV2 metric collection experiment.- *- * Alternative to `CRAWLEE_SYSTEM_INFO_V2` environment variable.- */- systemInfoV2?: boolean;-/*** Used in place of `isContainerized()` when collecting system metrics.*@@ -245,7 +238,6 @@ export interface ConfigurationOptions {* `defaultBrowserPath` | `CRAWLEE_DEFAULT_BROWSER_PATH` | -* `disableBrowserSandbox` | `CRAWLEE_DISABLE_BROWSER_SANDBOX` | -* `availableMemoryRatio` | `CRAWLEE_AVAILABLE_MEMORY_RATIO` | `0.25`- * `systemInfoV2` | `CRAWLEE_SYSTEM_INFO_V2` | false* `containerized | `CRAWLEE_CONTAINERIZED | -*/export class Configuration {@@ -268,7 +260,6 @@ export class Configuration {CRAWLEE_DISABLE_BROWSER_SANDBOX: 'disableBrowserSandbox',CRAWLEE_LOG_LEVEL: 'logLevel',CRAWLEE_PERSIST_STORAGE: 'persistStorage',- CRAWLEE_SYSTEM_INFO_V2: 'systemInfoV2',CRAWLEE_CONTAINERIZED: 'containerized',};@@ -278,7 +269,6 @@ export class Configuration {'xvfb','disableBrowserSandbox','persistStorage',- 'systemInfoV2','containerized',];@@ -299,7 +289,6 @@ export class Configuration {persistStateIntervalMillis: 60_000,systemInfoIntervalMillis: 1_000,persistStorage: true,- systemInfoV2: true,};/**diff --git a/packages/core/src/events/local_event_manager.ts b/packages/core/src/events/local_event_manager.tsindex 01e89b6cf..88baa0b99 100644--- a/packages/core/src/events/local_event_manager.ts+++ b/packages/core/src/events/local_event_manager.ts@@ -1,6 +1,4 @@-import os from 'node:os';--import { getCurrentCpuTicksV2, getMemoryInfo, getMemoryInfoV2, isContainerized } from '@crawlee/utils';+import { getCurrentCpuTicksV2, getMemoryInfo, isContainerized } from '@crawlee/utils';import log from '@apify/log';import { betterClearInterval, betterSetInterval } from '@apify/utilities';@@ -9,8 +7,6 @@ import type { SystemInfo } from '../autoscaling/system_status.js';import { EventManager, EventType } from './event_manager.js';export class LocalEventManager extends EventManager {- private previousTicks = { idle: 0, total: 0 };-/*** Initializes the EventManager and sets up periodic `systemInfo` and `persistState` events.* This is automatically called at the beginning of `crawler.run()`.@@ -57,20 +53,6 @@ export class LocalEventManager extends EventManager {return this.config.get('containerized', await isContainerized());}- private getCurrentCpuTicks() {- const cpus = os.cpus();- return cpus.reduce(- (acc, cpu) => {- const cpuTimes = Object.values(cpu.times);- return {- idle: acc.idle + cpu.times.idle,- total: acc.total + cpuTimes.reduce((sum, num) => sum + num),- };- },- { idle: 0, total: 0 },- );- }-/*** Creates a SystemInfo object based on local metrics.*/@@ -83,19 +65,7 @@ export class LocalEventManager extends EventManager {}private async createCpuInfo(options: { maxUsedCpuRatio: number }) {- if (this.config.get('systemInfoV2')) {- const usedCpuRatio = await getCurrentCpuTicksV2(await this.isContainerizedWrapper());- return {- cpuCurrentUsage: usedCpuRatio * 100,- isCpuOverloaded: usedCpuRatio > options.maxUsedCpuRatio,- };- }- const ticks = this.getCurrentCpuTicks();- const idleTicksDelta = ticks.idle - this.previousTicks!.idle;- const totalTicksDelta = ticks.total - this.previousTicks!.total;- const usedCpuRatio = totalTicksDelta ? 1 - idleTicksDelta / totalTicksDelta : 0;- Object.assign(this.previousTicks, ticks);-+ const usedCpuRatio = await getCurrentCpuTicksV2(await this.isContainerizedWrapper());return {cpuCurrentUsage: usedCpuRatio * 100,isCpuOverloaded: usedCpuRatio > options.maxUsedCpuRatio,@@ -104,7 +74,7 @@ export class LocalEventManager extends EventManager {private async createMemoryInfo() {try {- const memInfo = await this.getMemoryInfo();+ const memInfo = await this.getMemoryInfo(this.isContainerizedWrapper());return {memTotalBytes: memInfo.totalBytes,memCurrentBytes: memInfo.mainProcessBytes + memInfo.childProcessesBytes,diff --git a/packages/utils/src/index.ts b/packages/utils/src/index.tsindex 6fbd03a72..d6f6bf243 100644--- a/packages/utils/src/index.ts+++ b/packages/utils/src/index.ts@@ -3,7 +3,6 @@ export * from './internals/cheerio.js';export * from './internals/chunk.js';export * from './internals/extract-urls.js';export * from './internals/general.js';-export * from './internals/memory-info.js';export * from './internals/debug.js';export * as social from './internals/social.js';export * from './internals/typedefs.js';@@ -15,7 +14,7 @@ export * from './internals/robots.js';export * from './internals/sitemap.js';export * from './internals/url.js';-export { getCurrentCpuTicksV2 } from './internals/systemInfoV2/cpu-info.js';-export { getMemoryInfoV2 } from './internals/systemInfoV2/memory-info.js';+export { getCurrentCpuTicksV2, CpuSample } from './internals/system-info/cpu-info.js';+export { getMemoryInfo, MemoryInfo } from './internals/system-info/memory-info.js';export { Dictionary, Awaitable, Constructor } from '@crawlee/types';diff --git a/packages/utils/src/internals/memory-info.ts b/packages/utils/src/internals/memory-info.tsdeleted file mode 100644index da142f05c..000000000--- a/packages/utils/src/internals/memory-info.ts+++ /dev/null@@ -1,165 +0,0 @@-import { execSync } from 'node:child_process';-import { access, readFile } from 'node:fs/promises';-import { freemem, totalmem } from 'node:os';-import util from 'node:util';--import type { Dictionary } from '@crawlee/types';--import log from '@apify/log';-// @ts-expect-error We need to add typings for @apify/ps-tree-import psTree from '@apify/ps-tree';--import { isDocker } from './general.js';--const MEMORY_FILE_PATHS = {- TOTAL: {- V1: '/sys/fs/cgroup/memory/memory.limit_in_bytes',- V2: '/sys/fs/cgroup/memory.max',- },- USED: {- V1: '/sys/fs/cgroup/memory/memory.usage_in_bytes',- V2: '/sys/fs/cgroup/memory.current',- },-};--/**- * Describes memory usage of the process.- */-export interface MemoryInfo {- /** Total memory available in the system or container */- totalBytes: number;-- /** Amount of free memory in the system or container */- freeBytes: number;-- /** Amount of memory used (= totalBytes - freeBytes) */- usedBytes: number;-- /** Amount of memory used the current Node.js process */- mainProcessBytes: number;-- /** Amount of memory used by child processes of the current Node.js process */- childProcessesBytes: number;-}--/**- * Returns memory statistics of the process and the system, see {@apilink MemoryInfo}.- *- * If the process runs inside of Docker, the `getMemoryInfo` gets container memory limits,- * otherwise it gets system memory limits.- *- * Beware that the function is quite inefficient because it spawns a new process.- * Therefore you shouldn't call it too often, like more than once per second.- */-export async function getMemoryInfo(): Promise<MemoryInfo> {- const psTreePromised = util.promisify(psTree);-- // lambda does *not* have `ps` and other command line tools- // required to extract memory usage.- const isLambdaEnvironment = process.platform === 'linux' && !!process.env.AWS_LAMBDA_FUNCTION_MEMORY_SIZE;-- const isDockerVar = !isLambdaEnvironment && (await isDocker());-- let mainProcessBytes = -1;- let childProcessesBytes = 0;-- if (isLambdaEnvironment) {- // reported in bytes- mainProcessBytes = process.memoryUsage().rss;-- // https://stackoverflow.com/a/55914335/129415- const memInfo = execSync('cat /proc/meminfo').toString();- const values = memInfo.split(/[\n: ]/).filter((val) => val.trim());- // /proc/meminfo reports in kb, not bytes, the total used memory is reported by meminfo- // subtract memory used by the main node process in order to infer memory used by any child processes- childProcessesBytes = +values[19] * 1000 - mainProcessBytes;- } else {- // Query both root and child processes- const processes = await psTreePromised(process.pid, true);-- processes.forEach((rec: Dictionary<string>) => {- // Skip the 'ps' or 'wmic' commands used by ps-tree to query the processes- if (rec.COMMAND === 'ps' || rec.COMMAND === 'WMIC.exe') {- return;- }- const bytes = parseInt(rec.RSS, 10);- // Obtain main process' memory separately- if (rec.PID === `${process.pid}`) {- mainProcessBytes = bytes;- return;- }- childProcessesBytes += bytes;- });- }-- let totalBytes: number;- let usedBytes: number;- let freeBytes: number;-- if (isLambdaEnvironment) {- // memory size is defined in megabytes- totalBytes = parseInt(process.env.AWS_LAMBDA_FUNCTION_MEMORY_SIZE!, 10) * 1000000;- usedBytes = mainProcessBytes + childProcessesBytes;- freeBytes = totalBytes - usedBytes;-- log.debug(`lambda size of ${totalBytes} with ${freeBytes} free bytes`);- } else if (isDockerVar) {- // When running inside Docker container, use container memory limits-- // Check whether cgroups V1 or V2 is used- let cgroupsVersion: keyof typeof MEMORY_FILE_PATHS.TOTAL = 'V1';- try {- // If this directory does not exists, assume docker is using cgroups V2- await access('/sys/fs/cgroup/memory/');- } catch {- cgroupsVersion = 'V2';- }-- try {- let [totalBytesStr, usedBytesStr] = await Promise.all([- readFile(MEMORY_FILE_PATHS.TOTAL[cgroupsVersion], 'utf8'),- readFile(MEMORY_FILE_PATHS.USED[cgroupsVersion], 'utf8'),- ]);-- // Cgroups V2 files contains newline character. Getting rid of it for better handling in later part of the code.- totalBytesStr = totalBytesStr.replace(/[^a-zA-Z0-9 ]/g, '');- usedBytesStr = usedBytesStr.replace(/[^a-zA-Z0-9 ]/g, '');-- // Cgroups V2 contains 'max' string if memory is not limited- // See https://git.kernel.org/pub/scm/linux/kernel/git/tj/cgroup.git/tree/Documentation/admin-guide/cgroup-v2.rst (see "memory.max")- if (totalBytesStr === 'max') {- totalBytes = totalmem();- // Cgroups V1 is set to number related to platform and page size if memory is not limited- // See https://unix.stackexchange.com/q/420906- } else {- totalBytes = parseInt(totalBytesStr, 10);- const containerRunsWithUnlimitedMemory = totalBytes > Number.MAX_SAFE_INTEGER;- if (containerRunsWithUnlimitedMemory) totalBytes = totalmem();- }- usedBytes = parseInt(usedBytesStr, 10);- freeBytes = totalBytes - usedBytes;- } catch (err) {- // log.deprecated logs a warning only once- log.deprecated(- 'Your environment is Docker, but your system does not support memory cgroups. ' +- "If you're running containers with limited memory, memory auto-scaling will not work properly.\n\n" +- `Cause: ${(err as Error).message}`,- );- totalBytes = totalmem();- freeBytes = freemem();- usedBytes = totalBytes - freeBytes;- }- } else {- totalBytes = totalmem();- freeBytes = freemem();- usedBytes = totalBytes - freeBytes;- }-- return {- totalBytes,- freeBytes,- usedBytes,- mainProcessBytes,- childProcessesBytes,- };-}diff --git a/packages/utils/src/internals/systemInfoV2/cpu-info.ts b/packages/utils/src/internals/system-info/cpu-info.tssimilarity index 100%rename from packages/utils/src/internals/systemInfoV2/cpu-info.tsrename to packages/utils/src/internals/system-info/cpu-info.tsdiff --git a/packages/utils/src/internals/systemInfoV2/memory-info.ts b/packages/utils/src/internals/system-info/memory-info.tssimilarity index 98%rename from packages/utils/src/internals/systemInfoV2/memory-info.tsrename to packages/utils/src/internals/system-info/memory-info.tsindex b1c57d72d..855fde459 100644--- a/packages/utils/src/internals/systemInfoV2/memory-info.ts+++ b/packages/utils/src/internals/system-info/memory-info.ts@@ -49,7 +49,7 @@ export interface MemoryInfo {* @returns An object containing the free and used memory metrics.* @internal*/-export async function getMemoryInfoV2(containerized = false): Promise<MemoryInfo> {+export async function getMemoryInfo(containerized = false): Promise<MemoryInfo> {let mainProcessBytes = -1;let childProcessesBytes = 0;diff --git a/packages/utils/src/internals/systemInfoV2/ps-tree.ts b/packages/utils/src/internals/system-info/ps-tree.tssimilarity index 100%rename from packages/utils/src/internals/systemInfoV2/ps-tree.tsrename to packages/utils/src/internals/system-info/ps-tree.tsdiff --git a/test/utils/cpu-infoV2.test.ts b/test/utils/cpu-infoV2.test.tsindex efa5381a9..5f76af90c 100644--- a/test/utils/cpu-infoV2.test.ts+++ b/test/utils/cpu-infoV2.test.ts@@ -11,7 +11,7 @@ import {getCurrentCpuTicksV2,getSystemCpuUsage,sampleCpuUsage,-} from '../../packages/utils/src/internals/systemInfoV2/cpu-info.js';+} from '../../packages/utils/src/internals/system-info/cpu-info.js';vitest.mock('@crawlee/utils/src/internals/general', async (importActual) => {const original: typeof import('@crawlee/utils') = await importActual();diff --git a/test/utils/memory-info.test.ts b/test/utils/memory-info.test.tsdeleted file mode 100644index ffac5a9e2..000000000--- a/test/utils/memory-info.test.ts+++ /dev/null@@ -1,253 +0,0 @@-import { access, readFile } from 'node:fs/promises';-import { freemem, totalmem } from 'node:os';--import { launchPuppeteer } from '@crawlee/puppeteer';-import { getMemoryInfo, isDocker } from '@crawlee/utils';--vitest.mock('node:os', async (importActual) => {- const originalOs: typeof import('node:os') = await importActual();- return {- ...originalOs,- freemem: vitest.fn(),- totalmem: vitest.fn(),- };-});--vitest.mock('@crawlee/utils/src/internals/general', async (importActual) => {- const original: typeof import('@crawlee/utils') = await importActual();-- return {- ...original,- isDocker: vitest.fn(),- };-});--vitest.mock('node:fs/promises', async (importActual) => {- const originalFs: typeof import('node:fs/promises') = await importActual();- return {- ...originalFs,- readFile: vitest.fn(originalFs.readFile),- access: vitest.fn(originalFs.access),- };-});--const isDockerSpy = vitest.mocked(isDocker);-const freememSpy = vitest.mocked(freemem);-const totalmemSpy = vitest.mocked(totalmem);-const accessSpy = vitest.mocked(access);-// If you use this spy, make sure to reset it to the original implementation at the end of the test.-const readFileSpy = vitest.mocked(readFile);--describe('getMemoryInfo()', () => {- test('works WITHOUT child process outside the container', async () => {- isDockerSpy.mockResolvedValueOnce(false);- freememSpy.mockReturnValueOnce(222);- totalmemSpy.mockReturnValueOnce(333);-- const data = await getMemoryInfo();-- expect(freememSpy).toHaveBeenCalled();- expect(totalmemSpy).toHaveBeenCalled();-- expect(data).toMatchObject({- totalBytes: 333,- freeBytes: 222,- usedBytes: 111,- });-- expect(data.mainProcessBytes).toBeGreaterThanOrEqual(20_000_000);- });-- test('works WITHOUT child process inside the container', async () => {- isDockerSpy.mockResolvedValueOnce(true);- accessSpy.mockResolvedValueOnce();-- readFileSpy.mockImplementation(async (path) => {- if (path === '/sys/fs/cgroup/memory/memory.limit_in_bytes') {- return Promise.resolve('333');- }-- if (path === '/sys/fs/cgroup/memory/memory.usage_in_bytes') {- return Promise.resolve('111');- }-- throw new Error(`Unexpected path ${path}`);- });-- const data = await getMemoryInfo();-- expect(data).toMatchObject({- totalBytes: 333,- freeBytes: 222,- usedBytes: 111,- });-- expect(data.mainProcessBytes).toBeGreaterThanOrEqual(20_000_000);- });-- // TODO: check if this comment is still accurate- // this test hangs because we launch the browser, closing is apparently not enough?- test('works WITH child process outside the container', async () => {- process.env.CRAWLEE_HEADLESS = '1';- isDockerSpy.mockResolvedValueOnce(false);- freememSpy.mockReturnValueOnce(222);- totalmemSpy.mockReturnValueOnce(333);-- let browser!: Awaited<ReturnType<typeof launchPuppeteer>>;-- try {- browser = await launchPuppeteer();- const data = await getMemoryInfo();-- expect(freememSpy).toHaveBeenCalled();- expect(totalmemSpy).toHaveBeenCalled();- expect(data).toMatchObject({- totalBytes: 333,- freeBytes: 222,- usedBytes: 111,- });- expect(data.mainProcessBytes).toBeGreaterThanOrEqual(20_000_000);- expect(data.childProcessesBytes).toBeGreaterThanOrEqual(20_000_000);- } finally {- delete process.env.CRAWLEE_HEADLESS;- await browser?.close();- }- });-- // TODO: check if this comment is still accurate- // this test hangs because we launch the browser, closing is apparently not enough?- test('works WITH child process inside the container', async () => {- process.env.CRAWLEE_HEADLESS = '1';- isDockerSpy.mockResolvedValueOnce(true);- accessSpy.mockResolvedValueOnce();-- readFileSpy.mockImplementation(async (path) => {- if (path === '/sys/fs/cgroup/memory/memory.limit_in_bytes') {- return Promise.resolve('333');- }-- if (path === '/sys/fs/cgroup/memory/memory.usage_in_bytes') {- return Promise.resolve('111');- }-- throw new Error(`Unexpected path ${path}`);- });-- let browser!: Awaited<ReturnType<typeof launchPuppeteer>>;- try {- browser = await launchPuppeteer();- const data = await getMemoryInfo();-- expect(data).toMatchObject({- totalBytes: 333,- freeBytes: 222,- usedBytes: 111,- });- expect(data.mainProcessBytes).toBeGreaterThanOrEqual(20_000_000);- expect(data.childProcessesBytes).toBeGreaterThanOrEqual(20_000_000);- } finally {- delete process.env.CRAWLEE_HEADLESS;- await browser?.close();- }- });-- test('works with cgroup V1 with LIMITED memory', async () => {- isDockerSpy.mockResolvedValueOnce(true);- accessSpy.mockResolvedValueOnce();-- readFileSpy.mockImplementation(async (path) => {- if (path === '/sys/fs/cgroup/memory/memory.limit_in_bytes') {- return Promise.resolve('333');- }-- if (path === '/sys/fs/cgroup/memory/memory.usage_in_bytes') {- return Promise.resolve('111');- }-- throw new Error(`Unexpected path ${path}`);- });-- const data = await getMemoryInfo();- expect(data).toMatchObject({- totalBytes: 333,- freeBytes: 222,- usedBytes: 111,- });- });-- test('works with cgroup V1 with UNLIMITED memory', async () => {- isDockerSpy.mockResolvedValueOnce(true);- accessSpy.mockResolvedValueOnce();-- readFileSpy.mockImplementation(async (path) => {- if (path === '/sys/fs/cgroup/memory/memory.limit_in_bytes') {- return Promise.resolve('9223372036854771712');- }-- if (path === '/sys/fs/cgroup/memory/memory.usage_in_bytes') {- return Promise.resolve('111');- }-- throw new Error(`Unexpected path ${path}`);- });-- totalmemSpy.mockReturnValueOnce(333);-- const data = await getMemoryInfo();- expect(data).toMatchObject({- totalBytes: 333,- freeBytes: 222,- usedBytes: 111,- });- });-- test('works with cgroup V2 with LIMITED memory', async () => {- isDockerSpy.mockResolvedValueOnce(true);- accessSpy.mockRejectedValueOnce(new Error('ENOENT'));-- readFileSpy.mockImplementation(async (path) => {- if (path === '/sys/fs/cgroup/memory.max') {- return Promise.resolve('333\n');- }-- if (path === '/sys/fs/cgroup/memory.current') {- return Promise.resolve('111\n');- }-- throw new Error(`Unexpected path ${path}`);- });-- const data = await getMemoryInfo();- expect(data).toMatchObject({- totalBytes: 333,- freeBytes: 222,- usedBytes: 111,- });- });-- test('works with cgroup V2 with UNLIMITED memory', async () => {- isDockerSpy.mockResolvedValueOnce(true);- accessSpy.mockRejectedValueOnce(new Error('ENOENT'));-- readFileSpy.mockImplementation(async (path) => {- if (path === '/sys/fs/cgroup/memory.max') {- return Promise.resolve('max\n');- }-- if (path === '/sys/fs/cgroup/memory.current') {- return Promise.resolve('111\n');- }-- throw new Error(`Unexpected path ${path}`);- });-- totalmemSpy.mockReturnValueOnce(333);-- const data = await getMemoryInfo();- expect(data).toMatchObject({- totalBytes: 333,- freeBytes: 222,- usedBytes: 111,- });- });-});diff --git a/test/utils/memory-infoV2.test.ts b/test/utils/memory-infoV2.test.tsindex df5e9b458..1623336a9 100644--- a/test/utils/memory-infoV2.test.ts+++ b/test/utils/memory-infoV2.test.ts@@ -2,7 +2,7 @@ import { access, readFile } from 'node:fs/promises';import { freemem, totalmem } from 'node:os';import { launchPuppeteer } from '@crawlee/puppeteer';-import { getCgroupsVersion, getMemoryInfoV2 } from '@crawlee/utils';+import { getCgroupsVersion, getMemoryInfo } from '@crawlee/utils';vitest.mock('node:os', async (importActual) => {const originalOs: typeof import('node:os') = await importActual();@@ -43,7 +43,7 @@ describe('getMemoryInfoV2()', () => {freememSpy.mockReturnValueOnce(222);totalmemSpy.mockReturnValueOnce(333);- const data = await getMemoryInfoV2();+ const data = await getMemoryInfo();expect(freememSpy).toHaveBeenCalled();expect(totalmemSpy).toHaveBeenCalled();@@ -73,7 +73,7 @@ describe('getMemoryInfoV2()', () => {throw new Error(`Unexpected path ${path}`);});- const data = await getMemoryInfoV2(true);+ const data = await getMemoryInfo(true);expect(data).toMatchObject({totalBytes: 333,@@ -93,7 +93,7 @@ describe('getMemoryInfoV2()', () => {try {browser = await launchPuppeteer();- const data = await getMemoryInfoV2();+ const data = await getMemoryInfo();expect(freememSpy).toHaveBeenCalled();expect(totalmemSpy).toHaveBeenCalled();@@ -130,7 +130,7 @@ describe('getMemoryInfoV2()', () => {let browser!: Awaited<ReturnType<typeof launchPuppeteer>>;try {browser = await launchPuppeteer();- const data = await getMemoryInfoV2(true);+ const data = await getMemoryInfo(true);expect(data).toMatchObject({totalBytes: 333,@@ -161,7 +161,7 @@ describe('getMemoryInfoV2()', () => {throw new Error(`Unexpected path ${path}`);});- const data = await getMemoryInfoV2(true);+ const data = await getMemoryInfo(true);expect(data).toMatchObject({totalBytes: 333,freeBytes: 222,@@ -187,7 +187,7 @@ describe('getMemoryInfoV2()', () => {totalmemSpy.mockReturnValueOnce(333);- const data = await getMemoryInfoV2(true);+ const data = await getMemoryInfo(true);expect(data).toMatchObject({totalBytes: 333,freeBytes: 222,@@ -211,7 +211,7 @@ describe('getMemoryInfoV2()', () => {throw new Error(`Unexpected path ${path}`);});- const data = await getMemoryInfoV2(true);+ const data = await getMemoryInfo(true);expect(data).toMatchObject({totalBytes: 333,freeBytes: 222,@@ -237,7 +237,7 @@ describe('getMemoryInfoV2()', () => {totalmemSpy.mockReturnValueOnce(333);- const data = await getMemoryInfoV2(true);+ const data = await getMemoryInfo(true);expect(data).toMatchObject({totalBytes: 333,freeBytes: 222,diff --git a/test/utils/psTree.test.ts b/test/utils/psTree.test.tsindex 1d5719f16..1f24d9088 100644--- a/test/utils/psTree.test.ts+++ b/test/utils/psTree.test.ts@@ -1,7 +1,7 @@import { exec } from 'node:child_process';import path from 'node:path';-import { psTree } from '../../packages/utils/src/internals/systemInfoV2/ps-tree.js';+import { psTree } from '../../packages/utils/src/internals/system-info/ps-tree.js';const scripts = {parent: path.join(import.meta.dirname, 'fixtures', 'parent.js'),