Skip to content

Crawlee—A web scraping and browser automation library for Node.js to build reliable crawlers. In JavaScript and TypeScript. Extract data for AI, LLMs, RAG, or GPTs. Download HTML, PDF, JPG, PNG, and other files from websites. Works with Puppeteer, Playwright, Cheerio, JSDOM, and raw HTTP. Both headful and headless mode. With proxy rotation.

fix: prevent race condition in KeyValueStore.getAutoSavedValue() (#2193)

Matěj Volfe340e2b87649
diff --git a/packages/core/src/storages/key_value_store.ts b/packages/core/src/storages/key_value_store.ts
index 6d013a856..ac01963aa 100644
--- a/packages/core/src/storages/key_value_store.ts
+++ b/packages/core/src/storages/key_value_store.ts
@@ -227,6 +227,15 @@ export class KeyValueStore {
}
const value = await this.getValue<T>(key, defaultValue);
+
+ // The await above could have run in parallel with another call to this function. If the other call finished more quickly,
+ // the value will in cache at this point, and returning the new fetched value would introduce two different instances of
+ // the auto-saved object, and only the latter one would be persisted.
+ // Therefore we re-check the cache here, and if such race condition happened, we drop the fetched value and return the cached one.
+ if (this.cache.has(key)) {
+ return this.cache.get(key) as T;
+ }
+
this.cache.set(key, value!);
this.ensurePersistStateEvent();