Skip to content

A self-hostable bookmark-everything app (links, notes and images) with AI-based automatic tagging and full text search

feat(crawler): discard oversized full-page archives via CRAWLER_FULL_PAGE_ARCHIVE_MAX_SIZE_MB (#3054)

Chandan P1afa26ad3cf2
diff --git a/apps/workers/workers/crawler/assetStorage.ts b/apps/workers/workers/crawler/assetStorage.ts
index a52e3aab..6f059fb9 100644
--- a/apps/workers/workers/crawler/assetStorage.ts
+++ b/apps/workers/workers/crawler/assetStorage.ts
@@ -446,6 +446,18 @@ export async function archiveWebpage(
const stats = await fs.stat(assetPath);
const fileSize = stats.size;
+ // Discard oversized archives: media-rich pages can produce 1GB+
+ // monolith files that never render and only hang/crash browsers.
+ // 0 (default) disables the limit.
+ const maxArchiveSizeMb = serverConfig.crawler.fullPageArchiveMaxSizeMb;
+ if (maxArchiveSizeMb > 0 && fileSize > maxArchiveSizeMb * 1024 * 1024) {
+ logger.warn(
+ `[Crawler][${jobId}] Discarding page archive of ${fileSize} bytes as it exceeds CRAWLER_FULL_PAGE_ARCHIVE_MAX_SIZE_MB=${maxArchiveSizeMb}.`,
+ );
+ await tryCatch(fs.unlink(assetPath));
+ return null;
+ }
+
const { data: quotaApproved, error: quotaError } = await tryCatch(
QuotaService.checkStorageQuota(db, userId, fileSize),
);
diff --git a/docs/docs/03-configuration/01-environment-variables.md b/docs/docs/03-configuration/01-environment-variables.md
index fa83364c..33c560bf 100644
--- a/docs/docs/03-configuration/01-environment-variables.md
+++ b/docs/docs/03-configuration/01-environment-variables.md
@@ -146,6 +146,7 @@ Either `OPENAI_API_KEY` or `OLLAMA_BASE_URL` need to be set for automatic taggin
| CRAWLER_SCREENSHOT_TIMEOUT_SEC | No | 5 | How long to wait for the screenshot finish before timing out. If you are capturing full-page screenshots of long webpages, consider increasing this value. |
| CRAWLER_STORE_PDF | No | false | Whether to store a PDF snapshot of the crawled page. Disabled by default, as it can lead to much higher disk usage. When enabled, a PDF version of each crawled page will be captured and stored as an asset, which can be viewed in the bookmark preview. |
| CRAWLER_FULL_PAGE_ARCHIVE | No | false | Whether to store a full local copy of the page or not. Disabled by default, as it can lead to much higher disk usage. If disabled, only the readable text of the page is archived. |
+| CRAWLER_FULL_PAGE_ARCHIVE_MAX_SIZE_MB | No | 0 | Maximum size in MB for a stored full-page archive. Archives larger than this are discarded instead of saved, since media-rich pages can produce 1GB+ files that never render. 0 disables the limit. |
| CRAWLER_JOB_TIMEOUT_SEC | No | 60 | How long to wait for the crawler job to finish before timing out. If you have a slow internet connection or a low powered device, you might want to bump this up a bit |
| CRAWLER_NAVIGATE_TIMEOUT_SEC | No | 30 | How long to spend navigating to the page (along with its redirects). Increase this if you have a slow internet connection |
| CRAWLER_PREFLIGHT_USER_AGENT | No | Not set | The `User-Agent` header to send on the crawler's preflight request (the plain HTTP request that determines a link's content type and extracts early preview metadata before the full crawl). If not set, no `User-Agent` header is sent. |
diff --git a/packages/shared/config.ts b/packages/shared/config.ts
index 4ec9de94..f20bca51 100644
--- a/packages/shared/config.ts
+++ b/packages/shared/config.ts
@@ -139,6 +139,7 @@ const allEnv = z.object({
CRAWLER_FULL_PAGE_SCREENSHOT: stringBool("false"),
CRAWLER_STORE_PDF: stringBool("false"),
CRAWLER_FULL_PAGE_ARCHIVE: stringBool("false"),
+ CRAWLER_FULL_PAGE_ARCHIVE_MAX_SIZE_MB: z.coerce.number().default(0),
CRAWLER_VIDEO_DOWNLOAD: stringBool("false"),
CRAWLER_VIDEO_DOWNLOAD_MAX_SIZE: z.coerce.number().default(50),
CRAWLER_VIDEO_DOWNLOAD_TIMEOUT_SEC: z.coerce.number().default(10 * 60),
@@ -409,6 +410,7 @@ const serverConfigSchema = allEnv.transform((val, ctx) => {
fullPageScreenshot: val.CRAWLER_FULL_PAGE_SCREENSHOT,
storePdf: val.CRAWLER_STORE_PDF,
fullPageArchive: val.CRAWLER_FULL_PAGE_ARCHIVE,
+ fullPageArchiveMaxSizeMb: val.CRAWLER_FULL_PAGE_ARCHIVE_MAX_SIZE_MB,
downloadVideo: val.CRAWLER_VIDEO_DOWNLOAD,
maxVideoDownloadSize: val.CRAWLER_VIDEO_DOWNLOAD_MAX_SIZE,
downloadVideoTimeout: val.CRAWLER_VIDEO_DOWNLOAD_TIMEOUT_SEC,