🪢 Open source agent evals & observability: Trace, evaluate, and improve LLM applications with one open platform.
Replay Ingestion Events from S3 (v2)
Replays failed ingestion events by reading S3 keys from a CSV and submitting them to Langfuse via an admin API endpoint. This replaces the v1 script, which required direct Redis, ClickHouse, and PostgreSQL access plus a full repo clone.
v2 only requires the Langfuse host URL, an admin API key, and a CSV file exported from Athena.
Athena (S3 access logs)
│
â–¼
events.csv ──► replay script ──► POST /api/admin/ingestion-replay
│
â–¼
Redis queues (IngestionSecondaryQueue / OtelIngestionQueue)
│
â–¼
Worker processing
Prerequisites
- S3 server access logging enabled on the Langfuse events bucket (see Initial setup)
- Athena configured to query the access logs (see Initial setup)
- Node.js 18+ with
npx tsxavailable (no repo clone orpnpmneeded) events.csvexported from Athena (see Export events)LANGFUSE_HOSTURL of the target Langfuse instance (e.g.https://cloud.langfuse.com)ADMIN_API_KEYfor authenticating against the admin API- Network access from your machine to the Langfuse host
1. Initial setup (one-time, on AWS)
Before you can query S3 access logs via Athena, you need to configure S3 server access logging and set up Athena. Skip this section if your environment already has Athena configured for the events bucket. If you're not using AWS as a cloud vendor, consult your cloud provider's documentation on how to store blob storage access logs and efficiently retrieve the events.csv.
1a. Enable S3 server access logging
Create a dedicated S3 bucket to store access logs for your Langfuse events bucket. Then enable server access logging on the events bucket with the new bucket as the destination.
AWS docs: Enabling Amazon S3 server access logging
Set a key prefix (e.g. logs/<events-bucket-name>/) to keep logs organized if you reuse the bucket for multiple sources.
1b. Set up Athena query result location
Athena requires an S3 location to store query results. Configure this in the Athena console under Settings → Query result location.
AWS docs: Specifying a query result location
1c. Create an Athena database
CREATE DATABASE s3_access_logs_db
1d. Create the S3 access logs table
Create an external table that maps the S3 server access log format. Replace the LOCATION with the S3 URI of your access log bucket and prefix from step 1a.
AWS docs: Querying access logs for requests using Amazon Athena
CREATE EXTERNAL TABLE `s3_access_logs_db.events_bucket_logs`(
`bucketowner` STRING,
`bucket_name` STRING,
`requestdatetime` STRING,
`remoteip` STRING,
`requester` STRING,
`requestid` STRING,
`operation` STRING,
`key` STRING,
`request_uri` STRING,
`httpstatus` STRING,
`errorcode` STRING,
`bytessent` BIGINT,
`objectsize` BIGINT,
`totaltime` STRING,
`turnaroundtime` STRING,
`referrer` STRING,
`useragent` STRING,
`versionid` STRING,
`hostid` STRING,
`sigv` STRING,
`ciphersuite` STRING,
`authtype` STRING,
`endpoint` STRING,
`tlsversion` STRING,
`accesspointarn` STRING,
`aclrequired` STRING)
ROW FORMAT SERDE
'org.apache.hadoop.hive.serde2.RegexSerDe'
WITH SERDEPROPERTIES (
'input.regex'='([^ ]*) ([^ ]*) \\[(.*?)\\] ([^ ]*) ([^ ]*) ([^ ]*) ([^ ]*) ([^ ]*) (\"[^\"]*\"|-) (-|[0-9]*) ([^ ]*) ([^ ]*) ([^ ]*) ([^ ]*) ([^ ]*) ([^ ]*) (\"[^\"]*\"|-) ([^ ]*)(?: ([^ ]*) ([^ ]*) ([^ ]*) ([^ ]*) ([^ ]*) ([^ ]*) ([^ ]*) ([^ ]*))?.*$')
STORED AS INPUTFORMAT
'org.apache.hadoop.mapred.TextInputFormat'
OUTPUTFORMAT
'org.apache.hadoop.hive.ql.io.HiveIgnoreKeyTextOutputFormat'
LOCATION
's3://<your-access-log-bucket>/logs/<your-events-bucket>/'
1e. Verify the setup
Run a simple query to confirm logs are being collected and the table is configured correctly:
SELECT key, operation, httpstatus, requestdatetime
FROM s3_access_logs_db.events_bucket_logs
WHERE operation = 'REST.PUT.OBJECT'
LIMIT 10
If this returns results, the setup is complete. Note that it may take some time after enabling access logging for the first logs to appear.
2. Export events from Athena
Query S3 access logs to identify the events that need to be replayed. Adjust the time range, table name, and bucket name to match your environment and incident window.
SELECT operation, key
FROM s3_access_logs_db.events_bucket_logs
WHERE operation = 'REST.PUT.OBJECT'
AND bucket_name = '<your-events-bucket>'
AND parse_datetime(requestdatetime, 'dd/MMM/yyyy:HH:mm:ss Z')
BETWEEN parse_datetime('2025-07-09:00:30:00', 'yyyy-MM-dd:HH:mm:ss')
AND parse_datetime('2025-07-09:07:45:00', 'yyyy-MM-dd:HH:mm:ss')
Download the result as CSV. The expected format:
"operation","key"
"REST.PUT.OBJECT","projectId/trace/eventBodyId/eventId.json"
"REST.PUT.OBJECT","otel/projectId/2025/07/09/14/30/some-uuid.json"
Two S3 key formats are supported:
| Format | Pattern | Target queue |
|---|---|---|
| Standard | {projectId}/{type}/{eventBodyId}/{eventId}.json |
IngestionSecondaryQueue |
| OTEL | otel/{projectId}/{yyyy}/{mm}/{dd}/{hh}/{mm}/{eventId}.json |
OtelIngestionQueue |
Keys that don't match either pattern are skipped and logged.
3. Run the replay script
LANGFUSE_HOST=https://cloud.langfuse.com \
ADMIN_API_KEY=your-admin-api-key \
npx tsx replay.ts --file events.csv
Configuration
| Flag / env var | Default | Description |
|---|---|---|
--file |
events.csv |
Path to the CSV file |
--batch-size |
500 |
Number of keys per API request |
--concurrency |
4 |
Maximum parallel API requests |
--rate-limit |
50 |
Maximum requests per second |
--dry-run |
false |
Parse and validate without sending requests |
--resume |
false |
Resume from the last checkpoint (skips already-processed rows) |
LANGFUSE_HOST |
- | Target Langfuse instance URL (required) |
ADMIN_API_KEY |
- | Admin API key for authentication (required) |
Admin API endpoint
POST /api/admin/ingestion-replay
Accepts a batch of S3 keys and enqueues them for reprocessing.
Authentication: Authorization: Bearer {ADMIN_API_KEY} header, validated by AdminApiAuthService.
Request:
{
"keys": [
"projectId/trace/eventBodyId/eventId.json",
"otel/projectId/2025/07/09/14/30/some-uuid.json"
]
}
Response (200 OK):
{
"queued": 498,
"skipped": 2,
"errors": []
}
| Status | Meaning |
|---|---|
200 |
Batch accepted (check skipped/errors for partial failures) |
401 |
Missing or invalid ADMIN_API_KEY |
400 |
Malformed request body |
429 |
Rate limited, retry after backoff |
Event transformation
The endpoint parses each S3 key and constructs the queue payload:
Standard keys ({projectId}/{type}/{eventBodyId}/{eventId}.json):
{
"authCheck": {
"validKey": true,
"scope": { "projectId": "<projectId>" }
},
"data": {
"eventBodyId": "<eventBodyId>",
"fileKey": "<eventId>",
"type": "<type>-create"
}
}
Enqueued to IngestionSecondaryQueue.
OTEL keys (otel/{projectId}/{yyyy}/{mm}/{dd}/{hh}/{mm}/{eventId}.json):
{
"authCheck": {
"validKey": true,
"scope": { "projectId": "<projectId>", "accessLevel": "project" }
},
"data": {
"fileKey": "otel/<projectId>/<yyyy>/<mm>/<dd>/<hh>/<mm>/<eventId>.json"
}
}
Enqueued to OtelIngestionQueue.
Progress tracking and error handling
- Progress: The script logs progress after each batch (e.g.
[1200/45000] 2.7% — 498 queued, 2 skipped). - Checkpoints: After each successful batch, the current CSV row offset is written to a
.checkpointfile next to the input CSV. Use--resumeto continue from the last checkpoint after a failure. - Rate limiting: The script respects
--rate-limitlocally and backs off on429responses from the server using exponential backoff with jitter. - Retries: Transient failures (
429,5xx) are retried up to 3 times per batch. Permanent failures (4xxother than429) are logged and skipped. - Error log: Failed keys are appended to
errors.csvnext to the input file for manual inspection.
Differences from v1
| v1 | v2 | |
|---|---|---|
| Infrastructure access | Redis, ClickHouse, PostgreSQL, S3 | Langfuse host URL only |
| Setup | Full repo clone, pnpm install, .env file |
npx tsx + env vars |
| Event delivery | Direct BullMQ addBulk to Redis |
HTTP POST to admin API |
| Resume support | Manual (split files, rerun) | Built-in checkpoint/resume |
| Rate limiting | None (can overwhelm Redis) | Client-side + server-side rate limiting |