Skip to content

A self-hostable bookmark-everything app (links, notes and images) with AI-based automatic tagging and full text search

fix: skip url getContentType fetch if precrawledArchiveAssetId exists (#2580)

GSAlexb7c9fe862235
diff --git a/apps/workers/workers/crawlerWorker.ts b/apps/workers/workers/crawlerWorker.ts
index 5d71826e..a7cbd32a 100644
--- a/apps/workers/workers/crawlerWorker.ts
+++ b/apps/workers/workers/crawlerWorker.ts
@@ -2138,12 +2138,14 @@ async function runCrawler(
`[Crawler][${jobId}] Will crawl "${truncateUrl(url)}" for link with id "${bookmarkId}"`,
);
- const contentType = await getContentType(
- url,
- jobId,
- job.abortSignal,
- runProxy,
- );
+ if (precrawledArchiveAssetId) {
+ logger.info(
+ `[Crawler][${jobId}] Skipped fetching content-type for the url ${url} as precrawledArchiveAssetId exists`,
+ );
+ }
+ const contentType = precrawledArchiveAssetId
+ ? ASSET_TYPES.TEXT_HTML
+ : await getContentType(url, jobId, job.abortSignal, runProxy);
job.abortSignal.throwIfAborted();
// Link bookmarks get transformed into asset bookmarks if they point to a supported asset instead of a webpage