Crawlee—A web scraping and browser automation library for Node.js to build reliable crawlers. In JavaScript and TypeScript. Extract data for AI, LLMs, RAG, or GPTs. Download HTML, PDF, JPG, PNG, and other files from websites. Works with Puppeteer, Playwright, Cheerio, JSDOM, and raw HTTP. Both headful and headless mode. With proxy rotation.
TypeScript apifyautomationcrawlercrawlingheadlessheadless-chromejavascriptnodejsnpmplaywrightpuppeteerscraperscrapingtypescriptweb-crawlerweb-crawlingweb-scraping
feat: improved cross platform metric collection (#2834)
NathanSavageKaimaie41b2f744513
diff --git a/docs/experiments/systemInfoV2.mdx b/docs/experiments/systemInfoV2.mdxnew file mode 100644index 000000000..93f8f27e1--- /dev/null+++ b/docs/experiments/systemInfoV2.mdx@@ -0,0 +1,95 @@+---+id: experiments-system-infomation-v2+title: System Infomation V2+description: Improved autoscaling through cgroup aware metric collection.+---++import ApiLink from '@site/src/components/ApiLink';++:::caution++This is an experimental feature. While we welcome testers, keep in mind that it is currently not recommended to use this in production.++The API is subject to change, and we might introduce breaking changes in the future.++Should you be using this, feel free to open issues on our [GitHub repository](https://github.com/apify/crawlee), and we'll take a look.++:::++Starting with the newest `crawlee` beta, we have introduced a new crawler option that enables an improved metric collection system.+This new system should collect cpu and memory metrics more accurately in containerised environments by checking for cgroup enforce limits.++## How to enable the experiment++:::note++This example shows how to enable the experiment in the <ApiLink to="cheerio-crawler/class/CheerioCrawler">`CheerioCrawler`</ApiLink>,+but you can apply this to any crawler type.++:::++```ts+import { CheerioCrawler, Configuration } from 'crawlee';++Configuration.set('systemInfoV2', true);++const crawler = new CheerioCrawler({+ async requestHandler({ $, request }) {+ const title = $('title').text();+ console.log(`The title of "${request.url}" is: ${title}.`);+ },+});++await crawler.run(['https://crawlee.dev']);+```++## Other changes++:::info++This section is only useful if you're a tinkerer and want to see what's going on under the hood.++:::++The existing solution checked the bare metal metrics for how much cpu and memory was being used and how much headroom was available.+This is an intuitive solution but unfortunately doesnt account for when there is an external limit on the amount of resources a process can consume.+This is often the case in containerized environments where each container will have a quota for its cpu and memory usage.++This experiment attempts to address this issue by introducing a new `isContainerized()` utility function and changing the way resources are collected+when a container is detected.++:::note++This `isContainerized()` function is very similar to the existing `isDocker()` function however for now they both work side by side.+If this experiment is successful, eventualy `isDocker()` may eventually be depreciated in favour of `isContainerized()`.++:::++### Cgroup detection++On linux, to detect if cgroup is available, we check if there is a directory at `/sys/fs/cgroup`.+If the directory exists, a version of cgroup is installed.+Next we check the version of cgroup installed by checking for a directory at `/sys/fs/cgroup/memory/`.+If it exists, cgroup V1 is installed. If it is missing, it is assumed cgroup V2 is installed.++### CPU metric collection++The existing solution worked by checking the fraction of cpu idle ticks to the total number of cpu ticks since the last profile.+If 100000 ticks elapse and 5000 were idle, the cpu is at 95% utilisation.++In this experiment, the method of cpu load calculation depends on the result of `isContainerized()` or if set, the `CRAWLEE_CONTAINERIZED` environment variable.+If `isContainerized()` returns true, the new cgroup aware metric collection will be used over the "bare metal" numbers.+This works by inspecting the `/sys/fs/cgroup/cpuacct/cpuacct.usage`, `/sys/fs/cgroup/cpu/cpu.cfs_quota_us` and `/sys/fs/cgroup/cpu/cpu.cfs_period_us`+files for cgroup V1 and the `/sys/fs/cgroup/cpu.stat` and `/sys/fs/cgroup/cpu.max` files for cgroup V2.+The actual cpu usage figure is calculated in the same manner as the "bare metal" figure by comparing the total number of ticks elapsed to the number+of idle ticks between profiles but by using the figures from the cgroup files.+If no cgroup quota is enforced, the "bare metal" numbers will be used.++### Memory metric collection++The existing solution was already cgroup aware however an improvement has been made to memory metric collection when running on windows.+The existing solution used an external package `apify/ps-tree` to find the amount of memory crawlee and any child processes were using.+On Windows, this package used the depreciated "WMIC" command line utility to determine memory usage.++In this experiment, `apify/ps-tree` has been removed and replaced by the `packages/utils/src/internals/ps-tree.ts` file. This works in much the+same manner however, instead of using "WMIC", it uses "powershell" to collect the same data.\ No newline at end of filediff --git a/docs/guides/configuration.mdx b/docs/guides/configuration.mdxindex f5977e330..597c3dcc2 100644--- a/docs/guides/configuration.mdx+++ b/docs/guides/configuration.mdx@@ -92,6 +92,17 @@ The default request queue has ID `default`. Setting this environment variable ovStorage directories are purged by default. If set to `false` - local storage directories would not be purged automatically at the start of the crawler run or before opening of some storage explicitly (e.g. via `Dataset.open()`). Useful if we're trying e.g. to add more items to dataset with each next run (and keep the previously saved/scraped items).+#### `CRAWLEE_CONTAINERIZED`++This variable is only effective when the systemInfoV2 experiment is enabled.+Changes how crawlee measures its CPU and Memory usage and limits. If unset, crawlee will determine if it is containerised using common features of containerized environments using the `isContainerized` utility function.+ - A file at `/.dockerenv`.+ - A file at `/proc/self/cgroup` containing `docker`.+ - A value for the `KUBERNETES_SERVICE_HOST` environment variable.+ If `isLambda` returns true, `isContainerized` will return false regardless of these other checks.++When this variable is set, it is used in place of `isContainerized`.+### Convenience env varsThe next group includes env vars that can help achieve certain goals without having to changediff --git a/packages/core/src/autoscaling/snapshotter.ts b/packages/core/src/autoscaling/snapshotter.tsindex 8bff9944d..b718a48dd 100644--- a/packages/core/src/autoscaling/snapshotter.ts+++ b/packages/core/src/autoscaling/snapshotter.ts@@ -194,7 +194,7 @@ export class Snapshotter {if (memoryMbytes > 0) {this.maxMemoryBytes = memoryMbytes * 1024 * 1024;} else {- const { totalBytes } = await this._getMemoryInfo();+ const { totalBytes } = await getMemoryInfo();this.maxMemoryBytes = Math.ceil(totalBytes * this.config.get('availableMemoryRatio')!);this.log.debug(`Setting max memory of this run to ${Math.round(this.maxMemoryBytes / 1024 / 1024)} MB. ` +@@ -416,11 +416,4 @@ export class Snapshotter {}snapshots.splice(0, oldCount);}-- /**- * Helper method for easier mocking.- */- private async _getMemoryInfo() {- return getMemoryInfo();- }}diff --git a/packages/core/src/configuration.ts b/packages/core/src/configuration.tsindex d5ed9b4f5..626fb3120 100644--- a/packages/core/src/configuration.ts+++ b/packages/core/src/configuration.ts@@ -161,6 +161,20 @@ export interface ConfigurationOptions {* Alternative to `CRAWLEE_PERSIST_STORAGE` environment variable.*/persistStorage?: boolean;++ /**+ * Defines whether to use the systemInfoV2 metric collection experiment.+ *+ * Alternative to `CRAWLEE_SYSTEM_INFO_V2` environment variable.+ */+ systemInfoV2?: boolean;++ /**+ * Used in place of `isContainerized()` when collecting system metrics.+ *+ * Alternative to `CRAWLEE_CONTAINERIZED` environment variable.+ */+ containerized?: boolean;}/**@@ -226,6 +240,8 @@ export interface ConfigurationOptions {* `defaultBrowserPath` | `CRAWLEE_DEFAULT_BROWSER_PATH` | -* `disableBrowserSandbox` | `CRAWLEE_DISABLE_BROWSER_SANDBOX` | -* `availableMemoryRatio` | `CRAWLEE_AVAILABLE_MEMORY_RATIO` | `0.25`+ * `systemInfoV2` | `CRAWLEE_SYSTEM_INFO_V2` | false+ * `containerized | `CRAWLEE_CONTAINERIZED | -*/export class Configuration {/**@@ -247,9 +263,19 @@ export class Configuration {CRAWLEE_DISABLE_BROWSER_SANDBOX: 'disableBrowserSandbox',CRAWLEE_LOG_LEVEL: 'logLevel',CRAWLEE_PERSIST_STORAGE: 'persistStorage',+ CRAWLEE_SYSTEM_INFO_V2: 'systemInfoV2',+ CRAWLEE_CONTAINERIZED: 'containerized',};- protected static BOOLEAN_VARS = ['purgeOnStart', 'headless', 'xvfb', 'disableBrowserSandbox', 'persistStorage'];+ protected static BOOLEAN_VARS = [+ 'purgeOnStart',+ 'headless',+ 'xvfb',+ 'disableBrowserSandbox',+ 'persistStorage',+ 'systemInfoV2',+ 'containerized',+ ];protected static INTEGER_VARS = ['memoryMbytes', 'persistStateIntervalMillis', 'systemInfoIntervalMillis'];@@ -268,6 +294,7 @@ export class Configuration {persistStateIntervalMillis: 60_000,systemInfoIntervalMillis: 1_000,persistStorage: true,+ systemInfoV2: false,};/**diff --git a/packages/core/src/events/local_event_manager.ts b/packages/core/src/events/local_event_manager.tsindex da7b5ed02..c30a3b50e 100644--- a/packages/core/src/events/local_event_manager.ts+++ b/packages/core/src/events/local_event_manager.ts@@ -2,7 +2,7 @@ import os from 'node:os';import log from '@apify/log';import { betterClearInterval, betterSetInterval } from '@apify/utilities';-import { getMemoryInfo } from '@crawlee/utils';+import { getMemoryInfo, getMemoryInfoV2, getCurrentCpuTicksV2, isContainerized } from '@crawlee/utils';import { EventManager, EventType } from './event_manager';import type { SystemInfo } from '../autoscaling';@@ -49,6 +49,17 @@ export class LocalEventManager extends EventManager {intervalCallback();}+ /**+ * @internal+ */+ async isContainerizedWrapper() {+ const config = this.config.get('containerized');+ if (config !== undefined) {+ return config;+ }+ return isContainerized();+ }+private getCurrentCpuTicks() {const cpus = os.cpus();return cpus.reduce(@@ -69,12 +80,19 @@ export class LocalEventManager extends EventManager {private async createSystemInfo(options: { maxUsedCpuRatio: number }) {return {createdAt: new Date(),- ...this.createCpuInfo(options),+ ...(await this.createCpuInfo(options)),...(await this.createMemoryInfo()),} as SystemInfo;}- private createCpuInfo(options: { maxUsedCpuRatio: number }) {+ private async createCpuInfo(options: { maxUsedCpuRatio: number }) {+ if (this.config.get('systemInfoV2')) {+ const usedCpuRatio = await getCurrentCpuTicksV2(await this.isContainerizedWrapper());+ return {+ cpuCurrentUsage: usedCpuRatio * 100,+ isCpuOverloaded: usedCpuRatio > options.maxUsedCpuRatio,+ };+ }const ticks = this.getCurrentCpuTicks();const idleTicksDelta = ticks.idle - this.previousTicks!.idle;const totalTicksDelta = ticks.total - this.previousTicks!.total;@@ -89,22 +107,19 @@ export class LocalEventManager extends EventManager {private async createMemoryInfo() {try {- const memInfo = await this._getMemoryInfo();- const { mainProcessBytes, childProcessesBytes } = memInfo;-+ if (this.config.get('systemInfoV2')) {+ const memInfo = await getMemoryInfoV2(await this.isContainerizedWrapper());+ return {+ memCurrentBytes: memInfo.mainProcessBytes + memInfo.childProcessesBytes,+ };+ }+ const memInfo = await getMemoryInfo();return {- memCurrentBytes: mainProcessBytes + childProcessesBytes,+ memCurrentBytes: memInfo.mainProcessBytes + memInfo.childProcessesBytes,};} catch (err) {log.exception(err as Error, 'Memory snapshot failed.');return {};}}-- /**- * Helper method for easier mocking.- */- private async _getMemoryInfo() {- return getMemoryInfo();- }}diff --git a/packages/utils/src/index.ts b/packages/utils/src/index.tsindex 6ca3e8e7a..29d9c91bc 100644--- a/packages/utils/src/index.ts+++ b/packages/utils/src/index.ts@@ -13,4 +13,7 @@ export * from './internals/robots';export * from './internals/sitemap';export * from './internals/url';+export { getCurrentCpuTicksV2 } from './internals/systemInfoV2/cpu-info';+export { getMemoryInfoV2 } from './internals/systemInfoV2/memory-info';+export { Dictionary, Awaitable, Constructor } from '@crawlee/types';diff --git a/packages/utils/src/internals/general.ts b/packages/utils/src/internals/general.tsindex 3234a9830..e1cd8e7ef 100644--- a/packages/utils/src/internals/general.ts+++ b/packages/utils/src/internals/general.ts@@ -43,6 +43,70 @@ export async function isDocker(forceReset?: boolean): Promise<boolean> {return isDockerPromiseCache;}+let isContainerizedResult: boolean | undefined;++/**+ * Detects if crawlee is running in a containerized environment.+ */+export async function isContainerized() {+ // Value is very unlikley to change. Cache the result after the first execution.+ if (isContainerizedResult !== undefined) {+ return isContainerizedResult;+ }++ // return false if running in aws lambda+ if (isLambda()) {+ isContainerizedResult = false;+ return isContainerizedResult;+ }++ const dockerenvCheck = fs+ .stat('/.dockerenv')+ .then(() => true)+ .catch(() => false);++ const cgroupCheck = fs+ .readFile('/proc/self/cgroup', 'utf8')+ .then((content) => content.includes('docker'))+ .catch(() => false);++ const [dockerenvResult, cgroupResult] = await Promise.all([dockerenvCheck, cgroupCheck]);++ isContainerizedResult = dockerenvResult || cgroupResult || !!process.env.KUBERNETES_SERVICE_HOST;+ return isContainerizedResult;+}++export function isLambda() {+ return !!process.env.AWS_LAMBDA_FUNCTION_MEMORY_SIZE;+}++let _cgroupsVersion: null | 'V1' | 'V2';+/**+ * gets the cgroup version by checking for a file at /sys/fs/cgroup/memory+ * @returns "V1" or "V2" for the version of cgroup or null if cgroup is not found.+ */+export async function getCgroupsVersion(forceReset?: boolean) {+ // Parameter forceReset is just internal for unit tests.+ if (_cgroupsVersion !== undefined && !forceReset) {+ return _cgroupsVersion;+ }+ try {+ // If this directory does not exists, cgroups are not available+ await fs.access('/sys/fs/cgroup/');+ } catch (e) {+ _cgroupsVersion = null;+ return null;+ }+ _cgroupsVersion = 'V1';+ try {+ // If this directory does not exists, assume the container is using cgroups V2+ await fs.access('/sys/fs/cgroup/memory/');+ } catch (e) {+ _cgroupsVersion = 'V2';+ }+ return _cgroupsVersion;+}+/*** Computes a weighted average of an array of numbers, complemented by an array of weights.* @ignorediff --git a/packages/utils/src/internals/systemInfoV2/cpu-info.ts b/packages/utils/src/internals/systemInfoV2/cpu-info.tsnew file mode 100644index 000000000..94f55d9b8--- /dev/null+++ b/packages/utils/src/internals/systemInfoV2/cpu-info.ts@@ -0,0 +1,240 @@+import { execSync } from 'node:child_process';+import { readFile } from 'node:fs/promises';+import os from 'node:os';++import log from '@apify/log';++import { getCgroupsVersion } from '../general';++const CPU_FILE_PATHS = {+ STAT: {+ V1: '/sys/fs/cgroup/cpuacct/cpuacct.usage',+ V2: '/sys/fs/cgroup/cpu.stat',+ },+ QUOTA: {+ V1: '/sys/fs/cgroup/cpu/cpu.cfs_quota_us',+ V2: '/sys/fs/cgroup/cpu.max',+ },+ PERIOD: {+ V1: '/sys/fs/cgroup/cpu/cpu.cfs_period_us',+ V2: '/sys/fs/cgroup/cpu.max',+ },+};++let CLOCK_TICKS_PER_SECOND = 100;+let CLOCK_TICKS_CHECKED = false;++const NANOSECONDS_PER_SECOND = 1e9;++const previousTicks = { idle: 0, total: 0 };+/**+ * Gets the "bare metal" cpu load.+ * Used in+ * - AWS Lambda+ * - Containers without a cGroup quota+ * - Uncontainerized environments+ * @returns a number between 0 and 1 for the cpu load+ * @internal+ */+export function getCurrentCpuTicks() {+ const cpusCores = os.cpus();+ const ticks = cpusCores.reduce(+ (acc, cpu) => {+ const cpuTimes = Object.values(cpu.times);+ return {+ idle: acc.idle + cpu.times.idle,+ total: acc.total + cpuTimes.reduce((sum, num) => sum + num),+ };+ },+ { idle: 0, total: 0 },+ );+ const idleTicksDelta = ticks.idle - previousTicks!.idle;+ const totalTicksDelta = ticks.total - previousTicks!.total;+ return totalTicksDelta ? 1 - idleTicksDelta / totalTicksDelta : 0;+}++/**+ * Reads the linux tick rate+ * @returns the number of ticks per second+ */+function getClockTicks(): number {+ try {+ const result = execSync('getconf CLK_TCK').toString().trim();+ return parseInt(result, 10);+ } catch (err) {+ log.warningOnce('Failed to get clock ticks; defaulting to 100');+ return 100;+ }+}++/**+ * Reads the cgroup cpu quota.+ * In V1, a quota of -1 means “unlimited.”+ * In V2, a first field of "max" means unlimited.+ * @param cgroupsVersion the cGroup version+ * @returns The Cpu Quota+ * @internal+ */+export async function getCpuQuota(cgroupsVersion: string): Promise<number | null> {+ if (cgroupsVersion === 'V1') {+ const quotaStr = await readFile(CPU_FILE_PATHS.QUOTA.V1, 'utf8');+ const quota = parseInt(quotaStr.trim(), 10);+ return quota === -1 ? null : quota;+ }+ // cgroup v2+ const maxStr = await readFile(CPU_FILE_PATHS.QUOTA.V2, 'utf8');+ const parts = maxStr.trim().split(/\s+/);+ if (parts[0] === 'max') {+ return null;+ }+ return parseInt(parts[0], 10);+}++/**+ * Reads the cgroup cpu period.+ * @param cgroupsVersion the cGroup version+ * @returns The Cpu quota period+ * @internal+ */+export async function getCpuPeriod(cgroupsVersion: string): Promise<number> {+ if (cgroupsVersion === 'V1') {+ const quotaStr = await readFile(CPU_FILE_PATHS.PERIOD.V1, 'utf8');+ const quota = parseInt(quotaStr.trim(), 10);+ return quota;+ }+ // cgroup v2+ const maxStr = await readFile(CPU_FILE_PATHS.PERIOD.V2, 'utf8');+ const parts = maxStr.trim().split(/\s+/);+ return parseInt(parts[1], 10);+}++/**+ * Reads the cgroup cpu usage of the container+ *+ * @param cgroupsVersion the cGroup version+ * @returns the cpu usage+ * @internal+ */+export async function getContainerCpuUsage(cgroupsVersion: string): Promise<number> {+ if (cgroupsVersion === 'V1') {+ const data = await readFile(CPU_FILE_PATHS.STAT.V1, 'utf8');+ return Number(data.trim());+ }+ // cgroup v2+ const data = await readFile(CPU_FILE_PATHS.STAT.V2, 'utf8');+ const lines = data.split('\n');+ let usageUsec = 0;+ for (const line of lines) {+ const parts = line.trim().split(/\s+/);+ if (parts[0] === 'usage_usec') {+ usageUsec = Number(parts[1]);+ break;+ }+ }+ // Convert microseconds to nanoseconds.+ return usageUsec * 1000;+}++/**+ * Reads the cgroup cpu usage of the system from cgroup+ *+ * @returns the cpu usage+ * @internal+ */+export async function getSystemCpuUsage() {+ const statData = await readFile('/proc/stat', 'utf8');+ const lines = statData.split('\n');+ for (const line of lines) {+ if (line.startsWith('cpu ')) {+ // Split the line and extract the first seven numeric fields:+ // user, nice, system, idle, iowait, irq, softirq+ const parts = line.split(/\s+/).slice(1, 8);+ let totalTicks = 0;+ for (const part of parts) {+ totalTicks += Number(part);+ }+ // Convert clock ticks to nanoseconds.+ return (totalTicks * NANOSECONDS_PER_SECOND) / CLOCK_TICKS_PER_SECOND;+ }+ }+ throw new Error('no cpu line'); // shouldnt ever happen+}++/**+ * a cpu sample with the container usage and system usage+ */+export interface CpuSample {+ containerUsage: number; // in nanoseconds+ systemUsage: number; // in nanoseconds+}++/**+ * Takes a CPU usage sample for both the container and the system.+ *+ * @returns An object containing the container and system CPU usage.+ * @internal+ */+export async function sampleCpuUsage(cGroupsVersion: string): Promise<CpuSample> {+ const [containerUsage, systemUsage] = await Promise.all([+ getContainerCpuUsage(cGroupsVersion),+ getSystemCpuUsage(),+ ]);+ return { containerUsage, systemUsage };+}++let previousSample: CpuSample = { containerUsage: 0, systemUsage: 0 };++/**+ * Gets the cpu usage of the system.+ * If the crawler is running in a containerized environment, crawlee will check for a cgroup enforced cpu limit.+ * If a cgroup limit is found, it will be taken as the maximum load against which the current load will be gauged.+ * @returns a number between 0 and 1 for the cpu load+ * @internal+ */+export async function getCurrentCpuTicksV2(containerized = false): Promise<number> {+ try {+ // if not containerized+ if (!containerized) {+ // bare metal cpu limit+ return getCurrentCpuTicks();+ }+ if (!CLOCK_TICKS_CHECKED) {+ CLOCK_TICKS_PER_SECOND = getClockTicks();+ CLOCK_TICKS_CHECKED = true;+ }+ const cgroupsVersion = await getCgroupsVersion();+ // if cgroup is not detected, return bare metal cpu limit+ if (cgroupsVersion === null) {+ log.deprecated(+ 'Your environment is containerized, but your system does not support cgroups.\n' ++ "If you're running containers with limited cpu, cpu auto-scaling will not work properly.",+ );+ return getCurrentCpuTicks();+ }+ // cgroup aware cpu limit. If no limits are set, default to returning getCurrentCpuTicks.+ const quota = await getCpuQuota(cgroupsVersion!);+ if (quota === null) {+ // no cgroup limit, return host cpu load+ return getCurrentCpuTicks();+ }+ const period = await getCpuPeriod(cgroupsVersion!);+ // eg. having a 200000us quots per 100000us means the cGroup can fully use 2 cores+ const cpuAllowance = quota / period;++ const sample = await sampleCpuUsage(cgroupsVersion!);++ const containerDelta = sample.containerUsage - previousSample.containerUsage;+ const systemDelta = sample.systemUsage - previousSample.systemUsage;++ previousSample = sample;++ const numCpus = os.cpus().length;++ // Calculate the CPU usage percentage.+ return ((containerDelta / systemDelta) * numCpus) / cpuAllowance;+ } catch (err) {+ // if anything fails, default to bare metal metrics+ log.exception(err as Error, 'Cpu snapshot failed.');+ return getCurrentCpuTicks();+ }+}diff --git a/packages/utils/src/internals/systemInfoV2/memory-info.ts b/packages/utils/src/internals/systemInfoV2/memory-info.tsnew file mode 100644index 000000000..96c005bf2--- /dev/null+++ b/packages/utils/src/internals/systemInfoV2/memory-info.ts@@ -0,0 +1,148 @@+import { execSync } from 'node:child_process';+import { readFile } from 'node:fs/promises';+import { freemem, totalmem } from 'node:os';++import log from '@apify/log';++import { getCgroupsVersion, isLambda } from '../general';+import { psTree } from './ps-tree';++const MEMORY_FILE_PATHS = {+ TOTAL: {+ V1: '/sys/fs/cgroup/memory/memory.limit_in_bytes',+ V2: '/sys/fs/cgroup/memory.max',+ },+ USED: {+ V1: '/sys/fs/cgroup/memory/memory.usage_in_bytes',+ V2: '/sys/fs/cgroup/memory.current',+ },+};++/**+ * Describes memory usage of the process.+ */+export interface MemoryInfo {+ /** Total memory available in the system or container */+ totalBytes: number;++ /** Amount of free memory in the system or container */+ freeBytes: number;++ /** Amount of memory used (= totalBytes - freeBytes) */+ usedBytes: number;++ /** Amount of memory used the current Node.js process */+ mainProcessBytes: number;++ /** Amount of memory used by child processes of the current Node.js process */+ childProcessesBytes: number;+}++/**+ * Returns memory statistics of the process and the system, see {@apilink MemoryInfo}.+ *+ * If the process runs inside of a container, the `getMemoryInfo` gets container memory limits,+ * otherwise it gets system memory limits.+ *+ * Beware that the function is quite inefficient because it spawns a new process.+ * Therefore you shouldn't call it too often, like more than once per second.+ * @returns An object containing the free and used memory metrics.+ * @internal+ */+export async function getMemoryInfoV2(containerized = false): Promise<MemoryInfo> {+ let mainProcessBytes = -1;+ let childProcessesBytes = 0;++ // lambda does *not* have `ps` and other command line tools+ // required to extract memory usage.+ if (isLambda()) {+ // reported in bytes+ mainProcessBytes = process.memoryUsage().rss;++ // https://stackoverflow.com/a/55914335/129415+ const memInfo = execSync('cat /proc/meminfo').toString();+ const values = memInfo.split(/[\n: ]/).filter((val) => val.trim());+ // /proc/meminfo reports in kb, not bytes, the total used memory is reported by meminfo+ // subtract memory used by the main node process in order to infer memory used by any child processes+ childProcessesBytes = +values[19] * 1000 - mainProcessBytes;+ } else {+ // Query both root and child processes+ const processes = await psTree(process.pid, true);++ processes.forEach((rec) => {+ // Obtain main process' memory separately+ if (rec.PID === `${process.pid}`) {+ mainProcessBytes = rec.RSS;+ return;+ }+ childProcessesBytes += rec.RSS;+ });+ }++ let totalBytes: number;+ let usedBytes: number;+ let freeBytes: number;++ if (isLambda()) {+ // memory size is defined in megabytes+ totalBytes = parseInt(process.env.AWS_LAMBDA_FUNCTION_MEMORY_SIZE!, 10) * 1000000;+ usedBytes = mainProcessBytes + childProcessesBytes;+ freeBytes = totalBytes - usedBytes;++ log.debug(`lambda size of ${totalBytes} with ${freeBytes} free bytes`);+ } else if (containerized) {+ // When running inside a container, use container memory limits++ const cgroupsVersion = await getCgroupsVersion();++ try {+ if (cgroupsVersion === null) {+ throw new Error('cgroup not available');+ }+ let [totalBytesStr, usedBytesStr] = await Promise.all([+ readFile(MEMORY_FILE_PATHS.TOTAL[cgroupsVersion], 'utf8'),+ readFile(MEMORY_FILE_PATHS.USED[cgroupsVersion], 'utf8'),+ ]);++ // Cgroups V2 files contains newline character. Getting rid of it for better handling in later part of the code.+ totalBytesStr = totalBytesStr.replace(/[^a-zA-Z0-9 ]/g, '');+ usedBytesStr = usedBytesStr.replace(/[^a-zA-Z0-9 ]/g, '');++ // Cgroups V2 contains 'max' string if memory is not limited+ // See https://git.kernel.org/pub/scm/linux/kernel/git/tj/cgroup.git/tree/Documentation/admin-guide/cgroup-v2.rst (see "memory.max")+ if (totalBytesStr === 'max') {+ totalBytes = totalmem();+ // Cgroups V1 is set to number related to platform and page size if memory is not limited+ // See https://unix.stackexchange.com/q/420906+ } else {+ totalBytes = parseInt(totalBytesStr, 10);+ const containerRunsWithUnlimitedMemory = totalBytes > Number.MAX_SAFE_INTEGER;+ if (containerRunsWithUnlimitedMemory) totalBytes = totalmem();+ }+ usedBytes = parseInt(usedBytesStr, 10);+ freeBytes = totalBytes - usedBytes;+ } catch (err) {+ // log.deprecated logs a warning only once+ log.deprecated(+ 'Your environment is containerized, but your system does not support memory cgroups. ' ++ "If you're running containers with limited memory, memory auto-scaling will not work properly.\n\n" ++ `Cause: ${(err as Error).message}`,+ );+ totalBytes = totalmem();+ freeBytes = freemem();+ usedBytes = totalBytes - freeBytes;+ }+ } else {+ totalBytes = totalmem();+ freeBytes = freemem();+ usedBytes = totalBytes - freeBytes;+ }++ return {+ totalBytes,+ freeBytes,+ usedBytes,+ mainProcessBytes,+ childProcessesBytes,+ };+}diff --git a/packages/utils/src/internals/systemInfoV2/ps-tree.ts b/packages/utils/src/internals/systemInfoV2/ps-tree.tsnew file mode 100644index 000000000..395539d55--- /dev/null+++ b/packages/utils/src/internals/systemInfoV2/ps-tree.ts@@ -0,0 +1,173 @@+import type { ChildProcess } from 'child_process';+import { spawn } from 'child_process';+import * as readline from 'readline';++export interface ProcessInfo {+ PPID: string;+ PID: string;+ STAT: string | null;+ RSS: number;+ COMMAND: string;+}++// Type of only the string keys in ProcessInfo.+type ProcessInfoStringKeys = {+ [K in keyof ProcessInfo]: ProcessInfo[K] extends string | null ? K : never;+}[keyof ProcessInfo];++/**+ * Returns a promise that resolves with an array of ProcessInfo objects representing+ * the children of the given PID.+ *+ * @param pid - The PID (number or string) for which to list child processes.+ * @param includeRoot - Optional flag. When true, include the process with the given PID if found.+ * Defaults to false.+ * @internal+ */+export async function psTree(pid: number | string, includeRoot: boolean = false): Promise<ProcessInfo[]> {+ return new Promise((resolve, reject) => {+ if (typeof pid === 'number') {+ pid = pid.toString();+ }++ let processLister: ChildProcess;+ if (process.platform === 'win32') {+ processLister = spawn('powershell', [+ '-NoProfile',+ '-Command',+ 'Get-CimInstance Win32_Process | Format-Table ProcessId,ParentProcessId,WorkingSetSize,Name',+ ]);+ } else {+ processLister = spawn('ps', ['-A', '-o', 'ppid,pid,stat,rss,comm']);+ }++ processLister.on('error', reject);++ if (!processLister.stdout) {+ reject(new Error('Child process stdout is null'));+ return;+ }++ // Create a readline interface to process stdout line-by-line.+ const rl = readline.createInterface({+ input: processLister.stdout,+ });++ const rows: ProcessInfo[] = [];+ let headers: string[] | null = null;++ rl.on('line', (line: string) => {+ const trimmed = line.trim();+ if (trimmed === '') {+ return; // Skip empty lines.+ }++ // When headers have been set, skip a dashed separator line.+ const fields = trimmed.split(/\s+/);+ if (headers !== null && fields.every((field) => /^-+$/.test(field))) {+ return;+ }++ // The first nonempty line is assumed to be the header row.+ if (!headers) {+ headers = fields.map(normalizeHeader);+ return;+ }++ // Copy the fields into an array to process.+ const columns = fields.slice();++ // Build the process row object.+ const row: Partial<ProcessInfo> = {};+ const hdrs = headers.slice();+ // For all headers except the last one, assign one column per header.+ // The last header gets all remaining columns joined (in case the command name contains spaces).+ for (const [index, header] of hdrs.entries()) {+ let value: string;+ if (index === hdrs.length - 1) {+ value = columns.join(' ');+ } else {+ value = columns.shift()!;+ }++ if (header === 'RSS') {+ row[header] = Number.parseInt(value, 10);+ if (process.platform !== 'win32') {+ // On Unix like systems, convert RSS (in KB) to bytes.+ row[header] *= 1024;+ }+ } else {+ row[header as ProcessInfoStringKeys] = value;+ }+ }++ // On Windows, add STAT with a null value for compatibility.+ if (process.platform === 'win32') {+ row.STAT = null;+ }++ rows.push(row as ProcessInfo);+ });++ rl.on('close', () => {+ const parents: { [key: string]: boolean } = {};+ const children: ProcessInfo[] = [];++ // Seed with the provided PID.+ parents[pid] = true;++ // Build the list of child processes.+ rows.forEach((proc) => {+ // Skip the 'ps' or 'powershell' commands used by psTree to query the processes+ if (proc.COMMAND === 'ps' || proc.COMMAND === 'powershell.exe') {+ return;+ }+ if (parents[proc.PPID]) {+ parents[proc.PID] = true;+ children.push(proc);+ } else if (includeRoot && pid === proc.PID) {+ children.push(proc);+ }+ });+ resolve(children);+ });++ // Also listen for errors on the stdout stream.+ processLister.stdout.on('error', reject);+ });+}++/**+ * Normalizes the header names so that the rest of the code can work uniformly.+ *+ * On non‑Windows systems, we only adjust "COMM" to "COMMAND" (e.g. on macOS).+ * On Windows, the headers from Get-CimInstance + Format-Table are:+ *+ * ProcessId, ParentProcessId, WorkingSetSize, Name+ *+ * which are mapped to:+ *+ * PID, PPID, RSS, COMMAND+ */+function normalizeHeader(str: string): string {+ if (process.platform !== 'win32') {+ // macOS may output "COMM" instead of "COMMAND"+ if (str === 'COMM') return 'COMMAND';+ return str;+ }++ switch (str) {+ case 'Name':+ return 'COMMAND';+ case 'ParentProcessId':+ return 'PPID';+ case 'ProcessId':+ return 'PID';+ case 'Status':+ return 'STAT';+ case 'WorkingSetSize':+ return 'RSS';+ default:+ throw new Error(`Unknown process listing header: ${str}`);+ }+}diff --git a/test/browser-pool/browser-plugins/plugins.test.ts b/test/browser-pool/browser-plugins/plugins.test.tsindex cf417fa1e..beae39ca5 100644--- a/test/browser-pool/browser-plugins/plugins.test.ts+++ b/test/browser-pool/browser-plugins/plugins.test.ts@@ -325,9 +325,7 @@ describe('Plugins', () => {const response = await page.goto(`http://127.0.0.1:${(target.address() as AddressInfo).port}`);const text = await response!.text();- // FAILING. It should give 127.0.0.3 for all platforms.- // See https://github.com/puppeteer/puppeteer/issues/7698- expect(text).toBe(process.platform === 'win32' ? '127.0.0.1' : '127.0.0.3');+ expect(text).toBe('127.0.0.3');await page.close();});diff --git a/test/core/autoscaling/snapshotter.test.ts b/test/core/autoscaling/snapshotter.test.tsindex 8b5699ec4..e6a776c89 100644--- a/test/core/autoscaling/snapshotter.test.ts+++ b/test/core/autoscaling/snapshotter.test.ts@@ -3,6 +3,7 @@ import os from 'os';import log from '@apify/log';import { Configuration, EventType, LocalEventManager, Snapshotter } from '@crawlee/core';import type { MemoryInfo } from '@crawlee/utils';+import * as utils from '@crawlee/utils';import { sleep } from '@crawlee/utils';const toBytes = (x: number) => x * 1024 * 1024;@@ -208,15 +209,11 @@ describe('Snapshotter', () => {test('correctly marks memoryOverloaded using OS metrics', async () => {const noop = () => {};const memoryData = {+ totalBytes: toBytes(10000),mainProcessBytes: toBytes(1000),childProcessesBytes: toBytes(1000),} as MemoryInfo;- const getMemoryInfo = async () => ({ ...memoryData });- vitest.spyOn(LocalEventManager.prototype as any, '_getMemoryInfo').mockImplementation(getMemoryInfo);- vitest- .spyOn(Snapshotter.prototype as any, '_getMemoryInfo')- .mockResolvedValueOnce({ totalBytes: toBytes(10000) });-+ vitest.spyOn(utils, 'getMemoryInfo').mockResolvedValue(memoryData);const config = new Configuration({ availableMemoryRatio: 1 });const snapshotter = new Snapshotter({ config, maxUsedMemoryRatio: 0.5 });// do not initialize the event intervals as we will fire them manually@@ -247,9 +244,7 @@ describe('Snapshotter', () => {});test('correctly logs critical memory overload', async () => {- vitest- .spyOn(Snapshotter.prototype as any, '_getMemoryInfo')- .mockResolvedValueOnce({ totalBytes: toBytes(10000) });+ vitest.spyOn(utils, 'getMemoryInfo').mockResolvedValueOnce({ totalBytes: toBytes(10000) } as MemoryInfo);const config = new Configuration({ availableMemoryRatio: 1 });const snapshotter = new Snapshotter({ config, maxUsedMemoryRatio: 0.5 });await snapshotter.start();@@ -313,7 +308,7 @@ describe('Snapshotter', () => {});await snapshotter.start();await config.getEventManager().init();- await sleep(1e3);+ await sleep(1.5e3);await snapshotter.stop();await config.getEventManager().close();const memorySnapshots = snapshotter.getMemorySample();diff --git a/test/tsconfig.json b/test/tsconfig.jsonindex 43f94a0c4..7fa113996 100644--- a/test/tsconfig.json+++ b/test/tsconfig.json@@ -1,7 +1,7 @@{"extends": "../tsconfig.json","include": ["**/*", "../packages/*/src/**/*"],- "exclude": ["e2e"],+ "exclude": ["e2e", "**/fixtures/*"],"compilerOptions": {"sourceMap": true,"noUnusedLocals": false,diff --git a/test/utils/cpu-infoV2.test.ts b/test/utils/cpu-infoV2.test.tsnew file mode 100644index 000000000..6107bbdb2--- /dev/null+++ b/test/utils/cpu-infoV2.test.ts@@ -0,0 +1,286 @@+import { readFile } from 'node:fs/promises';+import os from 'os';++import { getCgroupsVersion } from '@crawlee/utils';++import {+ getCurrentCpuTicks,+ getCpuQuota,+ getCpuPeriod,+ getContainerCpuUsage,+ getSystemCpuUsage,+ sampleCpuUsage,+ getCurrentCpuTicksV2,+} from '../../packages/utils/src/internals/systemInfoV2/cpu-info';++vitest.mock('@crawlee/utils/src/internals/general', async (importActual) => {+ const original: typeof import('@crawlee/utils') = await importActual();+ return {+ ...original,+ getCgroupsVersion: vitest.fn(),+ };+});++vitest.mock('node:fs/promises', async (importActual) => {+ const originalFs: typeof import('node:fs/promises') = await importActual();+ return {+ ...originalFs,+ readFile: vitest.fn(originalFs.readFile),+ };+});++const getCgroupsVersionSpy = vitest.mocked(getCgroupsVersion);+const readFileSpy = vitest.mocked(readFile);++describe('getCurrentCpuTicks()', () => {+ test('calculates cpu load based on os.cpus', () => {+ // For two CPUs, we simulate:+ // CPU 1: { user: 100, nice: 0, sys: 50, idle: 50, irq: 0 } → total = 200, idle = 50+ // CPU 2: { user: 200, nice: 0, sys: 100, idle: 100, irq: 0 } → total = 400, idle = 100+ // Combined: total = 600, idle = 150 → load = 1 - (150/600) = 0.75+ const cpusMock = vitest+ .spyOn(os, 'cpus')+ .mockReturnValue([+ { times: { user: 100, nice: 0, sys: 50, idle: 50, irq: 0 } },+ { times: { user: 200, nice: 0, sys: 100, idle: 100, irq: 0 } },+ ] as os.CpuInfo[]);+ const load = getCurrentCpuTicks();+ expect(load).toBeCloseTo(0.75);+ cpusMock.mockRestore();+ });+});++describe('getCpuQuota()', () => {+ test('V1: returns null for unlimited quota (-1)', async () => {+ readFileSpy.mockImplementation(async (path) => {+ if (path === '/sys/fs/cgroup/cpu/cpu.cfs_quota_us') {+ return Promise.resolve('-1\n');+ }+ throw new Error(`Unexpected path ${path}`);+ });+ const quota = await getCpuQuota('V1');+ expect(quota).toBeNull();+ });++ test('V1: returns numeric quota', async () => {+ readFileSpy.mockImplementation(async (path) => {+ if (path === '/sys/fs/cgroup/cpu/cpu.cfs_quota_us') {+ return Promise.resolve('200000\n');+ }+ throw new Error(`Unexpected path ${path}`);+ });+ const quota = await getCpuQuota('V1');+ expect(quota).toEqual(200000);+ });++ test('V2: returns null for unlimited quota ("max")', async () => {+ readFileSpy.mockImplementation(async (path) => {+ if (path === '/sys/fs/cgroup/cpu.max') {+ return Promise.resolve('max 100000');+ }+ throw new Error(`Unexpected path ${path}`);+ });+ const quota = await getCpuQuota('V2');+ expect(quota).toBeNull();+ });++ test('V2: returns numeric quota', async () => {+ readFileSpy.mockImplementation(async (path) => {+ if (path === '/sys/fs/cgroup/cpu.max') {+ return Promise.resolve('200000 100000');+ }+ throw new Error(`Unexpected path ${path}`);+ });+ const quota = await getCpuQuota('V2');+ expect(quota).toBe(200000);+ });+});++describe('getCpuPeriod()', () => {+ test('V1: returns period', async () => {+ readFileSpy.mockImplementation(async (path) => {+ if (path === '/sys/fs/cgroup/cpu/cpu.cfs_period_us') {+ return Promise.resolve('100000\n');+ }+ throw new Error(`Unexpected path ${path}`);+ });+ const period = await getCpuPeriod('V1');+ expect(period).toBe(100000);+ });++ test('V2: returns period from the second field', async () => {+ readFileSpy.mockImplementation(async (path) => {+ if (path === '/sys/fs/cgroup/cpu.max') {+ return Promise.resolve('200000 100000');+ }+ throw new Error(`Unexpected path ${path}`);+ });+ const period = await getCpuPeriod('V2');+ expect(period).toBe(100000);+ });+});++describe('getContainerCpuUsage()', () => {+ test('V1: returns container cpu usage', async () => {+ readFileSpy.mockImplementation(async (path) => {+ if (path === '/sys/fs/cgroup/cpuacct/cpuacct.usage') {+ return Promise.resolve('123456789\n');+ }+ throw new Error(`Unexpected path ${path}`);+ });+ const usage = await getContainerCpuUsage('V1');+ expect(usage).toBe(123456789);+ });++ test('V2: parses usage_usec and converts to nanoseconds', async () => {+ // Simulate a file with a line containing "usage_usec"+ const fileContent = 'other 0\nusage_usec 5000\nmoreinfo 0';+ readFileSpy.mockImplementation(async (path) => {+ if (path === '/sys/fs/cgroup/cpu.stat') {+ return Promise.resolve(fileContent);+ }+ throw new Error(`Unexpected path ${path}`);+ });+ const usage = await getContainerCpuUsage('V2');+ expect(usage).toBe(5000 * 1000);+ });+});++describe('getSystemCpuUsage()', () => {+ test('parses /proc/stat correctly', async () => {+ // Provide a fake /proc/stat file with a proper "cpu" line.+ // Example: "cpu 100 0 50 150 0 0 0" → total ticks = 300,+ // so systemUsage = (300 * 1e9) / 100 = 3000000000.+ const statContent = 'cpu 100 0 50 150 0 0 0\notherline';+ readFileSpy.mockImplementation(async (path) => {+ if (path === '/proc/stat') {+ return Promise.resolve(statContent);+ }+ throw new Error(`Unexpected path ${path}`);+ });+ const usage = await getSystemCpuUsage();+ expect(usage).toBeCloseTo(3000000000);+ });++ test('throws error if no cpu line is found', async () => {+ readFileSpy.mockImplementation(async (path) => {+ if (path === '/proc/stat') {+ return Promise.resolve('no cpu info');+ }+ throw new Error(`Unexpected path ${path}`);+ });+ await expect(getSystemCpuUsage()).rejects.toThrow('no cpu line');+ });+});++describe('sampleCpuUsage()', () => {+ test('returns a valid CpuSample for V1', async () => {+ // For V1, getContainerCpuUsage reads from its stat file and+ // getSystemCpuUsage reads from /proc/stat.+ readFileSpy.mockImplementation(async (path) => {+ if (path === '/sys/fs/cgroup/cpuacct/cpuacct.usage') {+ return Promise.resolve('1000000\n');+ }+ if (path === '/proc/stat') {+ return Promise.resolve('cpu 100 0 50 150 0 0 0\n');+ }++ throw new Error(`Unexpected path ${path}`);+ });+ const sample = await sampleCpuUsage('V1');+ expect(sample).toEqual({+ containerUsage: 1000000,+ systemUsage: 3000000000, // as computed above.+ });+ });++ test('returns a valid CpuSample for V2', async () => {+ // For V1, getContainerCpuUsage reads from its stat file and+ // getSystemCpuUsage reads from /proc/stat.+ readFileSpy.mockImplementation(async (path) => {+ if (path === '/sys/fs/cgroup/cpu.stat') {+ return Promise.resolve('other 0\nusage_usec 1000\nmoreinfo 0');+ }+ if (path === '/proc/stat') {+ return Promise.resolve('cpu 100 0 50 150 0 0 0\n');+ }++ throw new Error(`Unexpected path ${path}`);+ });+ const sample = await sampleCpuUsage('V2');+ expect(sample).toEqual({+ containerUsage: 1000000,+ systemUsage: 3000000000, // as computed above.+ });+ });+});++describe('getCpuInfo()', () => {+ test('returns bare metal cpu ticks in AWS Lambda environment', async () => {+ // Simulate AWS Lambda by setting the env variable.+ process.env.AWS_LAMBDA_FUNCTION_MEMORY_SIZE = '128';+ const cpusMock = vitest+ .spyOn(os, 'cpus')+ .mockReturnValue([{ times: { user: 100, nice: 0, sys: 50, idle: 50, irq: 0 } }] as os.CpuInfo[]);+ const load = await getCurrentCpuTicksV2();+ // For one CPU: total = 100+0+50+50 = 200, idle = 50 → load = 0.75.+ expect(load).toBeCloseTo(0.75);+ cpusMock.mockRestore();+ delete process.env.AWS_LAMBDA_FUNCTION_MEMORY_SIZE;+ });++ test('returns bare metal cpu ticks when not containerized', async () => {+ const cpusMock = vitest+ .spyOn(os, 'cpus')+ .mockReturnValue([{ times: { user: 200, nice: 0, sys: 100, idle: 100, irq: 0 } }] as os.CpuInfo[]);+ const load = await getCurrentCpuTicksV2();+ // For one CPU: total = 200+0+100+100 = 400, idle = 100 → load = 0.75.+ expect(load).toBeCloseTo(0.75);+ cpusMock.mockRestore();+ });++ test('returns container-aware cpu usage when containerized with quota set', async () => {+ getCgroupsVersionSpy.mockResolvedValueOnce('V1');+ // For V1:+ // - getCpuQuota: return 200000 → quota = 200000.+ // - getCpuPeriod: return 100000 → period = 100000.+ // cpuAllowance = quota/period = 2.+ // - sampleCpuUsage: returns container usage and system usage.+ // For container usage, simulate "1000000000\n"+ // For system usage, simulate a /proc/stat line with 300 total ticks,+ // so systemUsage = (300 * 1e9) / 100 = 3000000000.+ readFileSpy+ .mockResolvedValueOnce('200000\n') // for getCpuQuota+ .mockResolvedValueOnce('100000\n') // for getCpuPeriod+ .mockResolvedValueOnce('1000000000\n') // for getContainerCpuUsage+ .mockResolvedValueOnce('cpu 300 0 0 0 0 0 0\n'); // for getSystemCpuUsage+ // Simulate a system with 2 CPUs.+ const cpusMock = vitest+ .spyOn(os, 'cpus')+ .mockReturnValue([+ { times: { user: 200, nice: 0, sys: 100, idle: 100, irq: 0 } },+ { times: { user: 200, nice: 0, sys: 100, idle: 100, irq: 0 } },+ ] as os.CpuInfo[]);+ // Initially, previousSample is { containerUsage: 0, systemUsage: 0 }.+ const result = await getCurrentCpuTicksV2(true);+ // Calculation:+ // containerDelta = 1000000, systemDelta = 3000000000, numCpus = 2, cpuAllowance = 2.+ // So: ((1000000000 / 3000000000) * 2) / 2 ≈ 0.3333+ expect(result).toBeCloseTo(0.3333, 4);+ cpusMock.mockRestore();+ });++ test('returns bare metal cpu ticks when containerized but no cgroup quota', async () => {+ getCgroupsVersionSpy.mockResolvedValueOnce('V1');+ // For V1, a quota of "-1" signals no limit → quota becomes null.+ readFileSpy.mockResolvedValueOnce('-1\n'); // getCpuQuota returns null+ // In this branch, getCpuInfo falls back to getCurrentCpuTicks.+ const cpusMock = vitest+ .spyOn(os, 'cpus')+ .mockReturnValue([{ times: { user: 300, nice: 0, sys: 150, idle: 150, irq: 0 } }] as os.CpuInfo[]);+ const result = await getCurrentCpuTicksV2(true);+ // For one CPU: total = 300+0+150+150 = 600, idle = 150 → load = 0.75.+ expect(result).toBeCloseTo(0.75);+ cpusMock.mockRestore();+ });+});diff --git a/test/utils/fixtures/child.js b/test/utils/fixtures/child.jsnew file mode 100644index 000000000..fb2b7d031--- /dev/null+++ b/test/utils/fixtures/child.js@@ -0,0 +1,4 @@+// dummy script for testing ps-tree.ts+setTimeout(() => {+ /* Does nothing, but prevents exit */+}, 1000);diff --git a/test/utils/fixtures/parent.js b/test/utils/fixtures/parent.jsnew file mode 100644index 000000000..6d0e510cb--- /dev/null+++ b/test/utils/fixtures/parent.js@@ -0,0 +1,5 @@+const cp = require('child_process');++for (let count = 1; count < 10; count++) {+ cp.exec('node ./test/utils/fixtures/child.js');+}diff --git a/test/utils/general.test.ts b/test/utils/general.test.tsindex b546367fc..ef668bb21 100644--- a/test/utils/general.test.ts+++ b/test/utils/general.test.ts@@ -1,7 +1,6 @@import asyncFs from 'node:fs/promises';-import type { IncomingMessage } from 'node:http';-import { isDocker, weightedAvg, sleep, snakeCaseToCamelCase } from '@crawlee/utils';+import { getCgroupsVersion, isDocker, isLambda, weightedAvg, sleep, snakeCaseToCamelCase } from '@crawlee/utils';describe('isDocker()', () => {test('works for dockerenv && cgroup', async () => {@@ -53,6 +52,98 @@ describe('isDocker()', () => {});});+describe('isContainerized()', () => {+ afterEach(() => {+ delete process.env.KUBERNETES_SERVICE_HOST;+ delete process.env.CRAWLEE_CONTAINERIZED;+ // resets the `isContainerizedResult` module scoped variable+ vi.resetModules();+ });++ test('returns true when a "/.dockerenv" file exists', async () => {+ // @ts-ignore flaky linting of dynamic import. Some environments throw ts(2307), others not.+ const { isContainerized } = await import('@crawlee/utils');+ const statMock = vitest.spyOn(asyncFs, 'stat').mockImplementationOnce(async () => null as any);+ const result = await isContainerized();+ expect(result).toBe(true);+ });++ test('returns false when isLambda is true', async () => {+ // @ts-ignore flaky linting of dynamic import. Some environments throw ts(2307), others not.+ const utils = await import('@crawlee/utils');+ const lambdaMock = vitest.spyOn(utils, 'isLambda').mockReturnValue(true);+ const result = await utils.isContainerized();+ expect(result).toBe(false);+ });++ test('returns true when a "/proc/stat/cgroup" file contains "docker"', async () => {+ // @ts-ignore flaky linting of dynamic import. Some environments throw ts(2307), others not.+ const { isContainerized } = await import('@crawlee/utils');+ const readFileMock = vitest+ .spyOn(asyncFs, 'readFile')+ .mockResolvedValue("'something ... docker ... something'");+ const result = await isContainerized();+ expect(result).toBe(true);+ });++ test('returns true when KUBERNETES_SERVICE_HOST environment variable is set', async () => {+ // @ts-ignore flaky linting of dynamic import. Some environments throw ts(2307), others not.+ const { isContainerized } = await import('@crawlee/utils');+ process.env.KUBERNETES_SERVICE_HOST = 'some-host';+ const result = await isContainerized();+ expect(result).toBe(true);+ });++ test('returns false when no other conditions are met', async () => {+ // @ts-ignore flaky linting of dynamic import. Some environments throw ts(2307), others not.+ const { isContainerized } = await import('@crawlee/utils');+ const result = await isContainerized();+ expect(result).toBe(false);+ });+});++describe('isLambda()', () => {+ afterEach(() => {+ delete process.env.AWS_LAMBDA_FUNCTION_MEMORY_SIZE;+ });++ test('returns true when AWS_LAMBDA_FUNCTION_MEMORY_SIZE environment variable is set', () => {+ process.env.AWS_LAMBDA_FUNCTION_MEMORY_SIZE = '12345';+ const result = isLambda();+ expect(result).toBe(true);+ });+});++describe('getCgroupsVersion()', () => {+ // Reset the module cache so that _cgroupsVersion is not retained across tests.+ beforeEach(() => {+ vitest.resetModules();+ });++ test('returns null when access to /sys/fs/cgroup/ fails', async () => {+ vitest.spyOn(asyncFs, 'access').mockRejectedValue(new Error('not found'));+ const version = await getCgroupsVersion(true);+ expect(version).toBe(null);+ });++ test('returns V2 when access to /sys/fs/cgroup/memory/ fails', async () => {+ vitest.spyOn(asyncFs, 'access').mockImplementation(async (path) => {+ if (path === '/sys/fs/cgroup/') {+ return;+ }+ throw new Error('not found');+ });+ const version = await getCgroupsVersion(true);+ expect(version).toBe('V2');+ });++ test('returns V1 when access to /sys/fs/cgroup/memory/ succeeds', async () => {+ vitest.spyOn(asyncFs, 'access').mockResolvedValue();+ const version = await getCgroupsVersion(true);+ expect(version).toBe('V1');+ });+});+describe('weightedAvg()', () => {test('works', () => {expect(weightedAvg([10, 10, 10], [1, 1, 1])).toBe(10);diff --git a/test/utils/memory-infoV2.test.ts b/test/utils/memory-infoV2.test.tsnew file mode 100644index 000000000..5130ccca1--- /dev/null+++ b/test/utils/memory-infoV2.test.ts@@ -0,0 +1,247 @@+import { readFile, access } from 'node:fs/promises';+import { freemem, totalmem } from 'node:os';++import { launchPuppeteer } from '@crawlee/puppeteer';+import { getCgroupsVersion, getMemoryInfoV2 } from '@crawlee/utils';++vitest.mock('node:os', async (importActual) => {+ const originalOs: typeof import('node:os') = await importActual();+ return {+ ...originalOs,+ freemem: vitest.fn(),+ totalmem: vitest.fn(),+ };+});++vitest.mock('@crawlee/utils/src/internals/general', async (importActual) => {+ const original: typeof import('@crawlee/utils') = await importActual();++ return {+ ...original,+ getCgroupsVersion: vitest.fn(),+ };+});++vitest.mock('node:fs/promises', async (importActual) => {+ const originalFs: typeof import('node:fs/promises') = await importActual();+ return {+ ...originalFs,+ readFile: vitest.fn(originalFs.readFile),+ access: vitest.fn(originalFs.access),+ };+});++const getCgroupsVersionSpy = vitest.mocked(getCgroupsVersion);+const freememSpy = vitest.mocked(freemem);+const totalmemSpy = vitest.mocked(totalmem);+const accessSpy = vitest.mocked(access);+// If you use this spy, make sure to reset it to the original implementation at the end of the test.+const readFileSpy = vitest.mocked(readFile);++describe('getMemoryInfoV2()', () => {+ test('works WITHOUT child process outside the container', async () => {+ freememSpy.mockReturnValueOnce(222);+ totalmemSpy.mockReturnValueOnce(333);++ const data = await getMemoryInfoV2();++ expect(freememSpy).toHaveBeenCalled();+ expect(totalmemSpy).toHaveBeenCalled();++ expect(data).toMatchObject({+ totalBytes: 333,+ freeBytes: 222,+ usedBytes: 111,+ });++ expect(data.mainProcessBytes).toBeGreaterThanOrEqual(20_000_000);+ });++ test('works WITHOUT child process inside the container', async () => {+ getCgroupsVersionSpy.mockResolvedValueOnce('V1');+ accessSpy.mockResolvedValueOnce();++ readFileSpy.mockImplementation(async (path) => {+ if (path === '/sys/fs/cgroup/memory/memory.limit_in_bytes') {+ return Promise.resolve('333');+ }++ if (path === '/sys/fs/cgroup/memory/memory.usage_in_bytes') {+ return Promise.resolve('111');+ }++ throw new Error(`Unexpected path ${path}`);+ });++ const data = await getMemoryInfoV2(true);++ expect(data).toMatchObject({+ totalBytes: 333,+ freeBytes: 222,+ usedBytes: 111,+ });++ expect(data.mainProcessBytes).toBeGreaterThanOrEqual(20_000_000);+ });++ test('works WITH child process outside the container', async () => {+ process.env.CRAWLEE_HEADLESS = '1';+ freememSpy.mockReturnValueOnce(222);+ totalmemSpy.mockReturnValueOnce(333);++ let browser!: Awaited<ReturnType<typeof launchPuppeteer>>;++ try {+ browser = await launchPuppeteer();+ const data = await getMemoryInfoV2();++ expect(freememSpy).toHaveBeenCalled();+ expect(totalmemSpy).toHaveBeenCalled();+ expect(data).toMatchObject({+ totalBytes: 333,+ freeBytes: 222,+ usedBytes: 111,+ });+ expect(data.mainProcessBytes).toBeGreaterThanOrEqual(20_000_000);+ expect(data.childProcessesBytes).toBeGreaterThanOrEqual(20_000_000);+ } finally {+ delete process.env.CRAWLEE_HEADLESS;+ await browser?.close();+ }+ });++ test('works WITH child process inside the container', async () => {+ process.env.CRAWLEE_HEADLESS = '1';+ getCgroupsVersionSpy.mockResolvedValueOnce('V1');+ accessSpy.mockResolvedValueOnce();++ readFileSpy.mockImplementation(async (path) => {+ if (path === '/sys/fs/cgroup/memory/memory.limit_in_bytes') {+ return Promise.resolve('333');+ }++ if (path === '/sys/fs/cgroup/memory/memory.usage_in_bytes') {+ return Promise.resolve('111');+ }++ throw new Error(`Unexpected path ${path}`);+ });++ let browser!: Awaited<ReturnType<typeof launchPuppeteer>>;+ try {+ browser = await launchPuppeteer();+ const data = await getMemoryInfoV2(true);++ expect(data).toMatchObject({+ totalBytes: 333,+ freeBytes: 222,+ usedBytes: 111,+ });+ expect(data.mainProcessBytes).toBeGreaterThanOrEqual(20_000_000);+ expect(data.childProcessesBytes).toBeGreaterThanOrEqual(20_000_000);+ } finally {+ delete process.env.CRAWLEE_HEADLESS;+ await browser?.close();+ }+ });++ test('works with cgroup V1 with LIMITED memory', async () => {+ getCgroupsVersionSpy.mockResolvedValueOnce('V1');+ accessSpy.mockResolvedValueOnce();++ readFileSpy.mockImplementation(async (path) => {+ if (path === '/sys/fs/cgroup/memory/memory.limit_in_bytes') {+ return Promise.resolve('333');+ }++ if (path === '/sys/fs/cgroup/memory/memory.usage_in_bytes') {+ return Promise.resolve('111');+ }++ throw new Error(`Unexpected path ${path}`);+ });++ const data = await getMemoryInfoV2(true);+ expect(data).toMatchObject({+ totalBytes: 333,+ freeBytes: 222,+ usedBytes: 111,+ });+ });++ test('works with cgroup V1 with UNLIMITED memory', async () => {+ getCgroupsVersionSpy.mockResolvedValueOnce('V1');+ accessSpy.mockResolvedValueOnce();++ readFileSpy.mockImplementation(async (path) => {+ if (path === '/sys/fs/cgroup/memory/memory.limit_in_bytes') {+ return Promise.resolve('9223372036854771712');+ }++ if (path === '/sys/fs/cgroup/memory/memory.usage_in_bytes') {+ return Promise.resolve('111');+ }++ throw new Error(`Unexpected path ${path}`);+ });++ totalmemSpy.mockReturnValueOnce(333);++ const data = await getMemoryInfoV2(true);+ expect(data).toMatchObject({+ totalBytes: 333,+ freeBytes: 222,+ usedBytes: 111,+ });+ });++ test('works with cgroup V2 with LIMITED memory', async () => {+ getCgroupsVersionSpy.mockResolvedValueOnce('V2');+ accessSpy.mockRejectedValueOnce(new Error('ENOENT'));++ readFileSpy.mockImplementation(async (path) => {+ if (path === '/sys/fs/cgroup/memory.max') {+ return Promise.resolve('333\n');+ }++ if (path === '/sys/fs/cgroup/memory.current') {+ return Promise.resolve('111\n');+ }++ throw new Error(`Unexpected path ${path}`);+ });++ const data = await getMemoryInfoV2(true);+ expect(data).toMatchObject({+ totalBytes: 333,+ freeBytes: 222,+ usedBytes: 111,+ });+ });++ test('works with cgroup V2 with UNLIMITED memory', async () => {+ getCgroupsVersionSpy.mockResolvedValueOnce('V2');+ accessSpy.mockRejectedValueOnce(new Error('ENOENT'));++ readFileSpy.mockImplementation(async (path) => {+ if (path === '/sys/fs/cgroup/memory.max') {+ return Promise.resolve('max\n');+ }++ if (path === '/sys/fs/cgroup/memory.current') {+ return Promise.resolve('111\n');+ }++ throw new Error(`Unexpected path ${path}`);+ });++ totalmemSpy.mockReturnValueOnce(333);++ const data = await getMemoryInfoV2(true);+ expect(data).toMatchObject({+ totalBytes: 333,+ freeBytes: 222,+ usedBytes: 111,+ });+ });+});diff --git a/test/utils/psTree.test.ts b/test/utils/psTree.test.tsnew file mode 100644index 000000000..2b55aac04--- /dev/null+++ b/test/utils/psTree.test.ts@@ -0,0 +1,49 @@+import { exec } from 'child_process';+import path from 'path';++import { psTree } from '../../packages/utils/src/internals/systemInfoV2/ps-tree';++const scripts = {+ parent: path.join(__dirname, 'fixtures', 'parent.js'),+ child: path.join(__dirname, 'fixtures', 'child.js'),+};++describe('psTree()', () => {+ test('Spawn a Parent process which has Ten Child Processes', async () => {+ const parent = exec(`node ${scripts.parent}`);++ // Wait for the child process(es) to spawn.+ await new Promise((resolve) => setTimeout(resolve, 500));++ const children = await psTree(parent.pid!);++ expect(children.length).toBeGreaterThan(0);++ // Allow time for the processes to be terminated.+ await new Promise((resolve) => setTimeout(resolve, 2000));++ const postKillChildren = await psTree(parent.pid!);++ expect(postKillChildren.length).toEqual(0);+ });++ test('Includes itself if includeRoot is true', async () => {+ const parent = exec(`node ${scripts.parent}`);++ // Wait for the child process(es) to spawn.+ await new Promise((resolve) => setTimeout(resolve, 500));++ const processes = await psTree(parent.pid!, true);++ const parentProcess = processes.find((process) => Number.parseInt(process.PID, 10) === parent.pid!);++ expect(parentProcess).toBeDefined();++ // Allow time for the processes to be terminated.+ await new Promise((resolve) => setTimeout(resolve, 2000));++ const postKillChildren = await psTree(parent.pid!);++ expect(postKillChildren.length).toEqual(0);+ });+});