Skip to content

:hedgehog: PostHog is the leading platform for building self-driving products. Our developer tools – AI observability, analytics, session replay, flags, experiments, error tracking, logs, and more – capture all the context agents need to diagnose problems, uncover opportunities, and ship fixes. Steer it all from Slack, web, desktop, or the MCP.

fix(ai-research-v4): retry an image store failure instead of failing the batch (#124956)

Nicholas Waltz8aa05b4eb034
diff --git a/nodejs/src/ingestion/pipelines/sessionreplay/ai-research/image-fetch-v4/fetch-runner.test.ts b/nodejs/src/ingestion/pipelines/sessionreplay/ai-research/image-fetch-v4/fetch-runner.test.ts
index 1e27daa0cbb1..d28d2385e5ad 100644
--- a/nodejs/src/ingestion/pipelines/sessionreplay/ai-research/image-fetch-v4/fetch-runner.test.ts
+++ b/nodejs/src/ingestion/pipelines/sessionreplay/ai-research/image-fetch-v4/fetch-runner.test.ts
@@ -602,7 +602,7 @@ describe('FetchRunner', () => {
it.each([
['a fetch pass', OPTIONS],
['the pod candidate pool', POOL_OPTIONS],
- ])('marks an image publish failure as lost after all candidate work settles in %s', async (_mode, options) => {
+ ])('retries an image publish failure through the frontier in %s', async (_mode, options) => {
const harness = build({ bytes: Buffer.from('image'), contentType: 'image/png' }, {}, 'queued', options)
harness.publishImage.mockRejectedValue(
new S3ServiceException({
@@ -620,9 +620,16 @@ describe('FetchRunner', () => {
)
expect(attempts).toEqual([
- expect.objectContaining({ outcome: 'publish_failed', finished: false, lost: true }),
- expect.objectContaining({ outcome: 'publish_failed', finished: false, lost: true }),
+ expect.objectContaining({ outcome: 'publish_failed', finished: false, lost: false }),
+ expect.objectContaining({ outcome: 'publish_failed', finished: false, lost: false }),
])
+ expect(harness.republish).toHaveBeenCalledTimes(2)
+ expect(harness.republish).toHaveBeenCalledWith(
+ expect.objectContaining({ lastBlockReason: 'transient_backoff' }),
+ expect.any(Object),
+ 'retry',
+ 60_000
+ )
// One log for the burst, with the error name and status and nothing that names the image.
const logged = warn.mock.calls.filter(([, event]) => event === 'ai_research_v4_image_fetch_storage_failed')
expect(logged).toEqual([
diff --git a/nodejs/src/ingestion/pipelines/sessionreplay/ai-research/image-fetch-v4/fetch-runner.ts b/nodejs/src/ingestion/pipelines/sessionreplay/ai-research/image-fetch-v4/fetch-runner.ts
index 90d631cfa175..bcfd4908c5f8 100644
--- a/nodejs/src/ingestion/pipelines/sessionreplay/ai-research/image-fetch-v4/fetch-runner.ts
+++ b/nodejs/src/ingestion/pipelines/sessionreplay/ai-research/image-fetch-v4/fetch-runner.ts
@@ -743,13 +743,18 @@ export class FetchRunner implements FetchPass {
}
} catch (error) {
this.logStorageFailure(result.outcome === 'ok' ? 'publish' : 'refresh', error)
- return {
- candidate: attemptedCandidate,
- outcome: 'publish_failed',
- finished: false,
- lost: true,
+ // The republish keeps the URL on the frontier, so one failed write does not fail the whole batch. The
+ // origin did nothing wrong, so its budget records no failure. The block reason is one that pods on
+ // older code can still parse, because they dead-letter a record with a reason they do not know.
+ return await this.republish(
+ republishBatch,
+ attemptedCandidate,
+ 'publish_failed',
+ 'retry',
+ ONE_MINUTE_MS,
configurationUpdates,
- }
+ 'transient_backoff'
+ )
}
if (!stored) {
return await this.refetchWithoutValidators(