Skip to content

The only tool you need to know what is happening and how to fix it.

add: parallelized move over

narrow-adventure4f7f2f93fa8e
diff --git a/CLAUDE.md b/CLAUDE.md
index 32ad3cd7..fef85dc9 100644
--- a/CLAUDE.md
+++ b/CLAUDE.md
@@ -315,6 +315,7 @@ PROFILE_RETENTION_DAYS=30 # 0 to disable; on-disk archive TTL, only
# V2 trace tables move-over (see "Trace Storage (V2 tables)" below)
V2_MOVE_OVER= # "true" copies the history of the tables V2 replaced into the V2 tables, in the background, newest day first. Runs at startup and one hour after each pass; resumable (progress in the transactional DB v2_move_over_days, sqlite/0077 + pg/0143), so it can stay set. Finish upgrading old writers before enabling it: completed source-table/date pairs are skipped. With several backend instances set it on exactly one. On SQLite and DuckDB it stops at the telemetry retention window, since older days would be pruned at once.
V2_MOVE_OVER_OLDEST= # optional YYYY-MM-DD; days before it stay in the old tables. An unparsable value logs and does not start the move.
+V2_MOVE_OVER_WORKERS=1 # source days copied concurrently inside the one enabled instance (max 16; invalid/non-positive = 1). Each day keeps table order; days are scheduled newest first. Increase to 4 for parallel backfill when the telemetry DB has spare capacity. Progress rows and logs update after successful batches roughly every 10s; slow queries/inserts can delay updates. Errors cancel and join all workers before the next hourly pass.
# Session recording uploads (see "Session Recording Uploader" section below)
SESSION_RECORDING_UPLOAD_WORKERS=32 # 0 to disable uploads entirely
@@ -1017,7 +1018,7 @@ A trace is a trace and a span is a span. Five tables hold traces on every backen
- **Frontend: large traces.** A trace over 500 spans opens with every node that has more than 50 children folded (`initiallyCollapsed` in `utils/span-tree.ts`); smaller traces open fully expanded. A folded row shows a pill with the number of hidden spans and an error marker when one of them failed, and the header has Expand all and Collapse all. The waterfall keeps what was clicked as the difference from that default, so there is no effect to reset. Pages must hold span payloads in `$state.raw`: a deep `$state` proxy over 20,000 spans took four minutes to read in the browser, against 0.4 seconds raw. The payload is only ever replaced, never edited.
- **Frontend.** `span-waterfall.svelte` names the service on each row and colours bars by service once a trace crosses services (one colour per row otherwise), marks error spans and missing parents, renders rows 500 at a time so a 20,000 span trace cannot stall the page, caps the name column to half the visible width on narrow screens, and links to the whole trace in the explorer. The span popover shows service, kind, status and scope (`utils/span-fields.ts`). Span payloads are held in `$state.raw`. The tree is keyed `traceId:spanId` (`utils/span-tree.ts`), and the issue pages build their related entity from the backend's `relatedEntity`.
- `GET /api/health/deep` also reports `spanPartitionTimeFallbacks`: spans whose source start was missing or unrepresentable, so their partition time is the ingest time (`shared.OtelStorageTimes`).
-- **Move-over** (`repositories/telemetry/move_over.go`, started by `retention.startMoveOver` when `V2_MOVE_OVER=true`). Copies the old tables into the V2 tables inside the backend (an embedded DuckDB file cannot be opened by a second process), newest day first, one UTC day of one table at a time, recorded in the transactional DB `v2_move_over_days` (migrations `sqlite/0077`, `pg/0143`) so it stops and resumes. The scheduler runs at startup and one hour after each pass ends, including failures; passes do not overlap. `started` is committed before telemetry writes, `done` after all batches succeed, with no main-DB transaction held during the copy. `moved_rows` counts scanned source rows, including already-copied rows skipped during recovery. Finish upgrading old writers first: completed table/date pairs are not revisited. A resumed day skips rows already written. It reads through `LegacyRepository` (one per backend, the old ids parked in the V2 model fields, see `shared/legacy.go`) and writes through the V2 repositories, so the id mapping lives in one place: the trace id comes from the `traceway.otel.trace_id` attribute the old ingest kept, else the old `distributed_trace_id`, else the row id; an old zero padded span UUID becomes its last 16 hex characters; spans and exceptions find their old owner by id, nearest in time, and take its trace; each entity also gets its own span row. Slices are halved when a read comes back full, so no day is sorted, and only a single overfull second is paged by stable full-row order and offset. Operator notes: `docs/pages/server/upgrade-v2-tables.mdx`.
+- **Move-over** (`repositories/telemetry/move_over.go`, started by `retention.startMoveOver` when `V2_MOVE_OVER=true`). Copies the old tables into the V2 tables inside the backend (an embedded DuckDB file cannot be opened by a second process), UTC days scheduled newest first with configurable concurrency (`V2_MOVE_OVER_WORKERS`, default 1, cap 16), each day keeping table order, recorded in the transactional DB `v2_move_over_days` (migrations `sqlite/0077`, `pg/0143`) so it stops and resumes. The scheduler runs at startup and one hour after each pass ends, including failures; passes do not overlap. `started` is committed before telemetry writes, `done` after all batches succeed, with no main-DB transaction held during the copy. `moved_rows` counts scanned source rows in the current attempt, including already-copied rows skipped during recovery; it resets on resume. Progress and logs update after successful batches roughly every 10s (a slow query or insert can delay this). A failed worker cancels the pass, and all workers are joined before return; separate backend instances are still not coordinated. Finish upgrading old writers first: completed table/date pairs are not revisited. A resumed day skips rows already written. It reads through `LegacyRepository` (one per backend, the old ids parked in the V2 model fields, see `shared/legacy.go`) and writes through the V2 repositories, so the id mapping lives in one place: the trace id comes from the `traceway.otel.trace_id` attribute the old ingest kept, else the old `distributed_trace_id`, else the row id; an old zero padded span UUID becomes its last 16 hex characters; spans and exceptions find their old owner by id, nearest in time, and take its trace; each entity also gets its own span row. Slices are halved when a read comes back full, so no day is sorted, and only a single overfull second is paged by stable full-row order and offset. Operator notes: `docs/pages/server/upgrade-v2-tables.mdx`.
ClickHouse `ch/0090_add_span_version` adds a materialized SHA-256 fingerprint over length-prefixed protobuf components. All span readers rank by duration then this fingerprint, so topology never reads payload columns for new parts. Old V2 parts calculate it on read until merged or explicitly materialized; no rewrite is scheduled by migration. The column costs 32 uncompressed bytes per physical row.
diff --git a/backend/app/config/config.go b/backend/app/config/config.go
index c47681a0..6df44bda 100644
--- a/backend/app/config/config.go
+++ b/backend/app/config/config.go
@@ -48,8 +48,9 @@ type Cfg struct {
ProfileArchiveRaw string
ProfileRetentionDays string
- V2MoveOver string
- V2MoveOverOldest string
+ V2MoveOver string
+ V2MoveOverOldest string
+ V2MoveOverWorkers string
SourceMapCacheMaxEntries string
SourceMapCacheMaxBytesMB string
@@ -245,8 +246,9 @@ func LoadFromEnv() *Cfg {
ProfileArchiveRaw: os.Getenv("PROFILE_ARCHIVE_RAW"),
ProfileRetentionDays: os.Getenv("PROFILE_RETENTION_DAYS"),
- V2MoveOver: os.Getenv("V2_MOVE_OVER"),
- V2MoveOverOldest: os.Getenv("V2_MOVE_OVER_OLDEST"),
+ V2MoveOver: os.Getenv("V2_MOVE_OVER"),
+ V2MoveOverOldest: os.Getenv("V2_MOVE_OVER_OLDEST"),
+ V2MoveOverWorkers: os.Getenv("V2_MOVE_OVER_WORKERS"),
SourceMapCacheMaxEntries: os.Getenv("SOURCEMAP_CACHE_MAX_ENTRIES"),
SourceMapCacheMaxBytesMB: os.Getenv("SOURCEMAP_CACHE_MAX_BYTES_MB"),
diff --git a/backend/app/repositories/telemetry/move_over.go b/backend/app/repositories/telemetry/move_over.go
index 1072a192..620e533b 100644
--- a/backend/app/repositories/telemetry/move_over.go
+++ b/backend/app/repositories/telemetry/move_over.go
@@ -5,6 +5,7 @@ import (
"fmt"
"log"
"strings"
+ "sync"
"time"
"github.com/google/uuid"
@@ -17,14 +18,17 @@ import (
type MoveOverOptions struct {
// PageSize bounds the rows held in memory at once. Zero means the default.
PageSize int
+ // Workers copies this many distinct source days concurrently. Zero means one.
+ Workers int
// Oldest, when set, is the last day moved. History before it stays in the old tables.
Oldest time.Time
Log func(format string, args ...any)
}
const (
- defaultPageSize = 20000
- moveOverDay = 24 * time.Hour
+ defaultPageSize = 20000
+ moveOverDay = 24 * time.Hour
+ moveOverProgressInterval = 10 * time.Second
// An owner is recorded near its spans and exceptions, not at the same instant: an endpoint at its start, an old
// OTel task at its end.
ownerWindow = moveOverDay
@@ -33,20 +37,23 @@ const (
// Entities before what hangs off them, so a day is usable as soon as its endpoints are in.
var tableOrder = []string{"endpoints", "tasks", "ai_traces", "exception_stack_traces", "spans"}
+type moveOverLegacy interface {
+ OccurrenceTime(table string, at time.Time) time.Time
+ FindEndpoints(ctx context.Context, from, to time.Time, limit int, offset *int) ([]models.Endpoint, error)
+ FindTasks(ctx context.Context, from, to time.Time, limit int, offset *int) ([]models.Task, error)
+ FindAiTraces(ctx context.Context, from, to time.Time, limit int, offset *int) ([]models.AiTrace, error)
+ FindExceptions(ctx context.Context, from, to time.Time, limit int, offset *int) ([]shared.LegacyException, error)
+ FindSpans(ctx context.Context, from, to time.Time, limit int, offset *int) ([]shared.LegacySpan, error)
+ FindOwners(ctx context.Context, ids []uuid.UUID, from, to time.Time) ([]models.Endpoint, []models.Task, []models.AiTrace, error)
+ FindBounds(ctx context.Context, table string) (oldest, newest time.Time, found bool, err error)
+ FindMovedIds(ctx context.Context, table string, ids []uuid.UUID, from, to time.Time) (map[shared.MovedOccurrence]bool, error)
+ FindMovedSpans(ctx context.Context, traceIds []string, from, to time.Time) (map[string]bool, error)
+}
+
type mover struct {
MoveOverOptions
- legacy interface {
- OccurrenceTime(table string, at time.Time) time.Time
- FindEndpoints(ctx context.Context, from, to time.Time, limit int, offset *int) ([]models.Endpoint, error)
- FindTasks(ctx context.Context, from, to time.Time, limit int, offset *int) ([]models.Task, error)
- FindAiTraces(ctx context.Context, from, to time.Time, limit int, offset *int) ([]models.AiTrace, error)
- FindExceptions(ctx context.Context, from, to time.Time, limit int, offset *int) ([]shared.LegacyException, error)
- FindSpans(ctx context.Context, from, to time.Time, limit int, offset *int) ([]shared.LegacySpan, error)
- FindOwners(ctx context.Context, ids []uuid.UUID, from, to time.Time) ([]models.Endpoint, []models.Task, []models.AiTrace, error)
- FindBounds(ctx context.Context, table string) (oldest, newest time.Time, found bool, err error)
- FindMovedIds(ctx context.Context, table string, ids []uuid.UUID, from, to time.Time) (map[shared.MovedOccurrence]bool, error)
- FindMovedSpans(ctx context.Context, traceIds []string, from, to time.Time) (map[string]bool, error)
- }
+ logMu sync.Mutex
+ legacy moveOverLegacy
progress interface {
FindProgress(ctx context.Context) ([]transactional.MoveOverDay, error)
SaveProgress(ctx context.Context, day transactional.MoveOverDay) error
@@ -54,8 +61,8 @@ type mover struct {
}
// RunMoveOver copies history from the tables V2 replaced into the V2 tables. It runs inside the backend, because the
-// embedded DuckDB file cannot be opened by a second process. It goes newest day first, so recent data is back first,
-// one day of one table at a time, and records each finished day so it can stop and resume. It returns when history is
+// embedded DuckDB file cannot be opened by a second process. It schedules distinct days newest first, keeping the
+// table order within each day, and records each finished table/day so it can stop and resume. It returns when history is
// exhausted, the context ends, or a read or a write fails. Running it again picks up where it stopped.
func RunMoveOver(ctx context.Context, options MoveOverOptions) error {
m := &mover{MoveOverOptions: options, legacy: LegacyRepository, progress: moveOverProgress{}}
@@ -69,6 +76,9 @@ func (m *mover) run(ctx context.Context) error {
if m.PageSize <= 0 {
m.PageSize = defaultPageSize
}
+ if m.Workers <= 0 {
+ m.Workers = 1
+ }
if m.Log == nil {
m.Log = func(format string, args ...any) { log.Printf("[tracewaybackend] move-over: "+format, args...) }
}
@@ -102,14 +112,14 @@ func (m *mover) run(ctx context.Context) error {
}
}
if len(bounds) == 0 {
- m.Log("the old tables are empty, nothing to move")
+ m.log("the old tables are empty, nothing to move")
return nil
}
if limit := m.Oldest.UTC().Truncate(moveOverDay); !m.Oldest.IsZero() && limit.After(oldest) {
oldest = limit
}
- for current := newest; !current.Before(oldest); current = current.Add(-moveOverDay) {
+ moveSourceDay := func(ctx context.Context, current time.Time) error {
for _, table := range tableOrder {
held, known := bounds[table]
label := current.Format(time.DateOnly)
@@ -124,6 +134,7 @@ func (m *mover) run(ctx context.Context) error {
return fmt.Errorf("move-over progress of %s %s: %w", table, label, err)
}
began := time.Now()
+ m.log("%s %s: starting (resumed=%t)", label, table, resumed)
rows, err := m.moveDay(ctx, table, current, resumed)
if err != nil {
return fmt.Errorf("move-over of %s %s: %w", table, label, err)
@@ -131,17 +142,78 @@ func (m *mover) run(ctx context.Context) error {
if err := m.progress.SaveProgress(ctx, transactional.MoveOverDay{Table: table, Day: label, State: transactional.MoveOverDone, Rows: rows}); err != nil {
return fmt.Errorf("move-over progress of %s %s: %w", table, label, err)
}
- m.Log("%s %s: %d rows in %s", label, table, rows, time.Since(began).Round(time.Millisecond))
+ m.log("%s %s: %d rows in %s", label, table, rows, time.Since(began).Round(time.Millisecond))
}
+ return nil
+ }
+ m.log("starting pass from %s back to %s with %d workers", newest.Format(time.DateOnly), oldest.Format(time.DateOnly), m.Workers)
+ if err := runMoveOverDays(ctx, newest, oldest, m.Workers, moveSourceDay); err != nil {
+ return err
}
- m.Log("done, every day from %s back to %s is in the V2 tables", newest.Format(time.DateOnly), oldest.Format(time.DateOnly))
+ m.log("done, every day from %s back to %s is in the V2 tables", newest.Format(time.DateOnly), oldest.Format(time.DateOnly))
return nil
}
+func (m *mover) log(format string, args ...any) {
+ m.logMu.Lock()
+ defer m.logMu.Unlock()
+ m.Log(format, args...)
+}
+
+func runMoveOverDays(ctx context.Context, newest, oldest time.Time, concurrency int, move func(context.Context, time.Time) error) error {
+ ctx, cancel := context.WithCancelCause(ctx)
+ defer cancel(nil)
+ days := make(chan time.Time)
+ var workers sync.WaitGroup
+ for range concurrency {
+ workers.Go(func() {
+ defer func() {
+ if recovered := recover(); recovered != nil {
+ cancel(fmt.Errorf("move-over worker panicked: %v", recovered))
+ }
+ }()
+ for day := range days {
+ if ctx.Err() != nil {
+ return
+ }
+ if err := move(ctx, day); err != nil {
+ cancel(err)
+ return
+ }
+ }
+ })
+ }
+schedule:
+ for day := newest; !day.Before(oldest); day = day.Add(-moveOverDay) {
+ select {
+ case <-ctx.Done():
+ break schedule
+ case days <- day:
+ }
+ }
+ close(days)
+ workers.Wait()
+ return context.Cause(ctx)
+}
+
// moveDay walks one day in slices narrow enough to hold in memory. A slice that comes back full is halved and read
// again, so no read sorts a day. Only a single second that is still too full is paged, in a stable full-row order.
func (m *mover) moveDay(ctx context.Context, table string, start time.Time, resumed bool) (int64, error) {
var total int64
+ began := time.Now()
+ lastProgress := began
+ reportProgress := func() error {
+ if time.Since(lastProgress) < moveOverProgressInterval {
+ return nil
+ }
+ label := start.Format(time.DateOnly)
+ if err := m.progress.SaveProgress(ctx, transactional.MoveOverDay{Table: table, Day: label, State: transactional.MoveOverStarted, Rows: total}); err != nil {
+ return fmt.Errorf("save progress: %w", err)
+ }
+ m.log("%s %s: in progress, %d source rows processed in %s", label, table, total, time.Since(began).Round(time.Millisecond))
+ lastProgress = time.Now()
+ return nil
+ }
width, end := time.Hour, start.Add(moveOverDay)
for cursor := start; cursor.Before(end); {
if err := ctx.Err(); err != nil {
@@ -162,16 +234,23 @@ func (m *mover) moveDay(ctx context.Context, table string, start time.Time, resu
width = max(to.Sub(cursor)/2, time.Second).Truncate(time.Second)
continue
}
+ total += moved
} else {
for offset, full := 0, true; full; {
page, more, err := m.move(ctx, table, cursor, to, &offset, resumed)
if err != nil {
return total, err
}
- moved, full, offset = moved+page, more, offset+int(page)
+ moved, total, full, offset = moved+page, total+page, more, offset+int(page)
+ if err := reportProgress(); err != nil {
+ return total, err
+ }
}
}
- total, cursor = total+moved, to
+ cursor = to
+ if err := reportProgress(); err != nil {
+ return total, err
+ }
// A sparse stretch widens the slice again, or one busy second would leave the rest of the day read second by second.
if moved*2 <= int64(m.PageSize) {
width = min(width*2, moveOverDay)
diff --git a/backend/app/repositories/telemetry/move_over_test.go b/backend/app/repositories/telemetry/move_over_test.go
index b9706b4c..dd3dddcf 100644
--- a/backend/app/repositories/telemetry/move_over_test.go
+++ b/backend/app/repositories/telemetry/move_over_test.go
@@ -250,6 +250,14 @@ func seedLegacyDay(t *testing.T, project uuid.UUID, at time.Time, trace uuid.UUI
}
func TestMoveOverBringsHistoryIntoTheV2Tables(t *testing.T) {
+ for _, workers := range []int{1, 3} {
+ t.Run(fmt.Sprintf("workers=%d", workers), func(t *testing.T) {
+ testMoveOverBringsHistoryIntoTheV2Tables(t, workers)
+ })
+ }
+}
+
+func testMoveOverBringsHistoryIntoTheV2Tables(t *testing.T, workers int) {
setupTestDB(t)
setupMoveOverProgress(t)
legacyReset(t)
@@ -263,8 +271,10 @@ func TestMoveOverBringsHistoryIntoTheV2Tables(t *testing.T) {
}
var moved []string
- options := MoveOverOptions{PageSize: 2, Oldest: today.AddDate(0, 0, -1), Log: func(format string, args ...any) {
- moved = append(moved, strings.SplitN(fmt.Sprintf(format, args...), ":", 2)[0])
+ options := MoveOverOptions{PageSize: 2, Workers: workers, Oldest: today.AddDate(0, 0, -1), Log: func(format string, args ...any) {
+ if format == "%s %s: %d rows in %s" {
+ moved = append(moved, strings.SplitN(fmt.Sprintf(format, args...), ":", 2)[0])
+ }
}}
if err := RunMoveOver(ctx, options); err != nil {
t.Fatal(err)
@@ -275,9 +285,37 @@ func TestMoveOverBringsHistoryIntoTheV2Tables(t *testing.T) {
order = append(order, strings.Fields(line)[0])
}
}
+ if workers > 1 {
+ sort.Sort(sort.Reverse(sort.StringSlice(order)))
+ }
if want := []string{today.Format(time.DateOnly), today.AddDate(0, 0, -1).Format(time.DateOnly)}; fmt.Sprint(order) != fmt.Sprint(want) {
t.Fatalf("newest day first, and nothing older than asked for: %v", order)
}
+ for _, day := range order {
+ var tables []string
+ for _, line := range moved {
+ fields := strings.Fields(line)
+ if fields[0] == day {
+ tables = append(tables, fields[1])
+ }
+ }
+ if got := strings.Join(tables, ","); got != "endpoints,tasks,exception_stack_traces,spans" {
+ t.Fatalf("table order within %s: %s", day, got)
+ }
+ }
+ progress, err := (moveOverProgress{}).FindProgress(ctx)
+ if err != nil || len(progress) != 8 {
+ t.Fatalf("want progress for four tables on two days: %+v, %v", progress, err)
+ }
+ for _, day := range progress {
+ wantRows := int64(1)
+ if day.Table == "spans" {
+ wantRows = 3
+ }
+ if day.State != transactional.MoveOverDone || day.Rows != wantRows {
+ t.Errorf("incorrect completed source-row count: %+v", day)
+ }
+ }
window := today
endpoints, err := EndpointRepository.FindByTraceIds(ctx, []string{hexId(traces[0])}, []uuid.UUID{project}, &window)
diff --git a/backend/app/repositories/telemetry/move_over_workers_test.go b/backend/app/repositories/telemetry/move_over_workers_test.go
new file mode 100644
index 00000000..a0042dfd
--- /dev/null
+++ b/backend/app/repositories/telemetry/move_over_workers_test.go
@@ -0,0 +1,168 @@
+package telemetry
+
+import (
+ "context"
+ "errors"
+ "strings"
+ "testing"
+ "testing/synctest"
+ "time"
+
+ "github.com/tracewayapp/traceway/backend/app/models"
+ "github.com/tracewayapp/traceway/backend/app/repositories/transactional"
+)
+
+func TestMoveOverDaysRunConcurrentlyWithoutOverlapping(t *testing.T) {
+ synctest.Test(t, func(t *testing.T) {
+ newest := time.Now().UTC().Truncate(moveOverDay)
+ started := make(chan time.Time, 6)
+ release := make(chan struct{})
+ defer close(release)
+ done := make(chan error, 1)
+ go func() {
+ done <- runMoveOverDays(context.Background(), newest, newest.Add(-5*moveOverDay), 2, func(ctx context.Context, day time.Time) error {
+ started <- day
+ <-release
+ return nil
+ })
+ }()
+ synctest.Wait()
+ if len(started) != 2 {
+ t.Fatalf("want two days running concurrently, got %d", len(started))
+ }
+ seen := map[time.Time]bool{<-started: true, <-started: true}
+ if !seen[newest] || !seen[newest.Add(-moveOverDay)] {
+ t.Fatalf("the newest days must start first: %v", seen)
+ }
+ for range 4 {
+ release <- struct{}{}
+ synctest.Wait()
+ if len(started) != 1 {
+ t.Fatalf("one available worker should start exactly one day: %d", len(started))
+ }
+ day := <-started
+ if seen[day] {
+ t.Fatalf("day %v scheduled more than once", day)
+ }
+ seen[day] = true
+ }
+ release <- struct{}{}
+ release <- struct{}{}
+ if err := <-done; err != nil || len(seen) != 6 {
+ t.Fatalf("want six completed days: %v, %v", seen, err)
+ }
+ })
+}
+
+func TestMoveOverDaysCancelAndJoinWorkers(t *testing.T) {
+ for _, failure := range []string{"error", "panic", "cancel"} {
+ t.Run(failure, func(t *testing.T) {
+ synctest.Test(t, func(t *testing.T) {
+ ctx, cancel := context.WithCancel(context.Background())
+ defer cancel()
+ newest := time.Now().UTC().Truncate(moveOverDay)
+ peerStarted := make(chan struct{})
+ peerFinished := make(chan struct{})
+ injected := errors.New("copy failed")
+ err := runMoveOverDays(ctx, newest, newest.Add(-5*moveOverDay), 2, func(ctx context.Context, day time.Time) error {
+ if day.Equal(newest) {
+ <-peerStarted
+ switch failure {
+ case "panic":
+ panic("copy panicked")
+ case "cancel":
+ cancel()
+ return ctx.Err()
+ default:
+ return injected
+ }
+ }
+ if !day.Equal(newest.Add(-moveOverDay)) {
+ t.Errorf("started another day after failure: %v", day)
+ return errors.New("unexpected day")
+ }
+ close(peerStarted)
+ <-ctx.Done()
+ time.Sleep(time.Second)
+ close(peerFinished)
+ return ctx.Err()
+ })
+ select {
+ case <-peerFinished:
+ default:
+ t.Fatal("returned before the cancelled worker finished")
+ }
+ switch failure {
+ case "error":
+ if !errors.Is(err, injected) {
+ t.Fatalf("lost the original failure: %v", err)
+ }
+ case "panic":
+ if err == nil || !strings.Contains(err.Error(), "copy panicked") {
+ t.Fatalf("lost the worker panic: %v", err)
+ }
+ case "cancel":
+ if !errors.Is(err, context.Canceled) {
+ t.Fatalf("lost external cancellation: %v", err)
+ }
+ }
+ })
+ })
+ }
+}
+
+type slowMoveOverLegacy struct {
+ moveOverLegacy
+}
+
+func (slowMoveOverLegacy) FindEndpoints(context.Context, time.Time, time.Time, int, *int) ([]models.Endpoint, error) {
+ time.Sleep(11 * time.Second)
+ return nil, nil
+}
+
+type recordingMoveOverProgress struct {
+ moveOverProgress
+ days chan transactional.MoveOverDay
+ err error
+}
+
+func (p *recordingMoveOverProgress) SaveProgress(ctx context.Context, day transactional.MoveOverDay) error {
+ p.days <- day
+ return p.err
+}
+
+func TestMoveOverReportsProgressBeforeDayCompletes(t *testing.T) {
+ synctest.Test(t, func(t *testing.T) {
+ start := time.Now().UTC().Truncate(moveOverDay)
+ progress := &recordingMoveOverProgress{days: make(chan transactional.MoveOverDay, 16)}
+ m := &mover{
+ MoveOverOptions: MoveOverOptions{PageSize: 2, Log: func(string, ...any) {}},
+ legacy: slowMoveOverLegacy{}, progress: progress,
+ }
+ done := make(chan error, 1)
+ go func() {
+ _, err := m.moveDay(context.Background(), "endpoints", start, false)
+ done <- err
+ }()
+ time.Sleep(12 * time.Second)
+ synctest.Wait()
+ if len(progress.days) != 1 {
+ t.Fatalf("want one intermediate progress update: %d", len(progress.days))
+ }
+ if day := <-progress.days; day.State != transactional.MoveOverStarted || day.Day != start.Format(time.DateOnly) {
+ t.Fatalf("want intermediate progress for the current day: %+v", day)
+ }
+ select {
+ case <-done:
+ t.Fatal("the day should still be copying")
+ default:
+ }
+ if err := <-done; err != nil {
+ t.Fatal(err)
+ }
+ progress.err = errProgressUnavailable
+ if _, err := m.moveDay(context.Background(), "endpoints", start, true); !errors.Is(err, progress.err) {
+ t.Fatalf("a failed progress update must stop the copy: %v", err)
+ }
+ })
+}
diff --git a/backend/app/retention/move_over.go b/backend/app/retention/move_over.go
index 010ceda4..00aeca85 100644
--- a/backend/app/retention/move_over.go
+++ b/backend/app/retention/move_over.go
@@ -23,7 +23,7 @@ func startMoveOver(ctx context.Context, telemetryRetentionDays int) {
if err != nil || !enabled {
return
}
- options := telemetry.MoveOverOptions{}
+ options := telemetry.MoveOverOptions{Workers: parseMoveOverWorkers(cfg.V2MoveOverWorkers)}
if oldest := strings.TrimSpace(cfg.V2MoveOverOldest); oldest != "" {
if options.Oldest, err = time.Parse(time.DateOnly, oldest); err != nil {
log.Printf("[tracewaybackend] move-over: V2_MOVE_OVER_OLDEST %q is not a YYYY-MM-DD date, not starting", oldest)
@@ -42,6 +42,14 @@ func startMoveOver(ctx context.Context, telemetryRetentionDays int) {
}()
}
+func parseMoveOverWorkers(value string) int {
+ workers, err := strconv.Atoi(strings.TrimSpace(value))
+ if err != nil || workers < 1 {
+ return 1
+ }
+ return min(workers, 16)
+}
+
func moveOverOptionsForRun(options telemetry.MoveOverOptions, retentionDays int, now time.Time) telemetry.MoveOverOptions {
// Recompute for every pass: a failed migration can outlive the retention window it started with.
if retentionDays > 0 {
diff --git a/backend/app/retention/move_over_test.go b/backend/app/retention/move_over_test.go
index 1ec2753e..b56b3f61 100644
--- a/backend/app/retention/move_over_test.go
+++ b/backend/app/retention/move_over_test.go
@@ -55,11 +55,11 @@ func TestMoveOverScheduleDoesNotStartWhenCancelled(t *testing.T) {
func TestMoveOverRecomputesRetentionForEachPass(t *testing.T) {
now := time.Date(2026, 9, 20, 1, 0, 0, 0, time.FixedZone("UTC+2", 2*60*60))
- options := telemetry.MoveOverOptions{Oldest: now.AddDate(0, 0, -90), PageSize: 25}
+ options := telemetry.MoveOverOptions{Oldest: now.AddDate(0, 0, -90), PageSize: 25, Workers: 4}
for _, elapsed := range []int{0, 10} {
current := now.AddDate(0, 0, elapsed)
got := moveOverOptionsForRun(options, 30, current)
- if !got.Oldest.Equal(current.UTC().AddDate(0, 0, -30)) || got.Oldest.Location() != time.UTC || got.PageSize != 25 {
+ if !got.Oldest.Equal(current.UTC().AddDate(0, 0, -30)) || got.Oldest.Location() != time.UTC || got.PageSize != 25 || got.Workers != 4 {
t.Fatalf("retention window at day %d: %+v", elapsed, got)
}
}
@@ -71,3 +71,11 @@ func TestMoveOverRecomputesRetentionForEachPass(t *testing.T) {
t.Fatal("a newer operator lower bound must win")
}
}
+
+func TestMoveOverWorkers(t *testing.T) {
+ for value, want := range map[string]int{"": 1, "0": 1, "-1": 1, "invalid": 1, "1": 1, " 4 ": 4, "16": 16, "100": 16} {
+ if got := parseMoveOverWorkers(value); got != want {
+ t.Errorf("workers %q = %d, want %d", value, got, want)
+ }
+ }
+}
diff --git a/docs/pages/server/upgrade-v2-tables.mdx b/docs/pages/server/upgrade-v2-tables.mdx
index 384cd2d3..3754f0a0 100644
--- a/docs/pages/server/upgrade-v2-tables.mdx
+++ b/docs/pages/server/upgrade-v2-tables.mdx
@@ -62,14 +62,17 @@ Add these settings to your existing backend configuration and restart or recreat
V2_MOVE_OVER=true
# Optional: copy this UTC source date and newer dates.
V2_MOVE_OVER_OLDEST=2026-09-01
+# Optional: copy four source days concurrently in this instance.
+V2_MOVE_OVER_WORKERS=4
```
The date is an example; replace it with your desired oldest day or omit it. For Docker Compose, add the settings to the backend service's existing `environment` block and recreate that service. A plain container restart does not change its environment. Preserve the service's other settings and volume mounts.
| Variable | Default | Behavior |
|----------|---------|----------|
-| `V2_MOVE_OVER` | unset; disabled | `true` enables one background worker in this backend process |
+| `V2_MOVE_OVER` | unset; disabled | `true` enables background copying in this backend process |
| `V2_MOVE_OVER_OLDEST` | unset | Inclusive oldest UTC source day, formatted `YYYY-MM-DD`; older days are left in the old tables |
+| `V2_MOVE_OVER_WORKERS` | `1` | Number of source days copied concurrently in this instance; capped at `16`. Unset, invalid or non-positive values use `1` |
An invalid oldest date is logged and the worker does not start. An unset, false or unparseable `V2_MOVE_OVER` leaves the worker disabled.
@@ -79,7 +82,7 @@ On SQLite and DuckDB, the oldest source day is constrained by both the configure
## How copying works
-The worker processes the newest UTC source day first, then walks backward. Within each day it copies:
+The worker schedules UTC source days newest first, then walks backward. `V2_MOVE_OVER_WORKERS=4` allows four distinct days to copy at once; when one finishes, that worker takes the next day. Completed table/day pairs are skipped, and interrupted pairs are resumed. Within each day it copies:
```text
endpoints → tasks → ai_traces → exception_stack_traces → spans
@@ -87,9 +90,11 @@ endpoints → tasks → ai_traces → exception_stack_traces → spans
History becomes visible progressively as batches are inserted. An occurrence can appear before its child spans and exceptions have been copied; a day is not published atomically.
+Parallel days use the same conversion and recovery rules as a single worker. Spans and exceptions resolve their owners from the legacy tables, so they do not need another day's V2 copy to finish first. Keep the setting on exactly one backend instance: day assignments are coordinated inside that process, not across replicas.
+
Each primary source read returns at most 20,000 rows. It starts with a one-hour window. A full result may be truncated, so the worker halves the window and reads again before writing. Once a window is only one second wide, a full result is paged using a stable order over all copied source values. Sparse windows grow again.
-This bounds the source page size, not total memory or I/O. Large attributes, owner lookups, sorting and destination inserts still compete with live traffic. There is no configurable throughput limiter. Keep legacy source rows unchanged while copying: do not import, manually delete or mutate them during a pass. Account for automatic retention when selecting the history to preserve.
+This bounds each worker's source page size, not total memory or I/O. Large attributes, owner lookups, sorting and destination inserts still compete with live traffic. For faster backfill, start with `V2_MOVE_OVER_WORKERS=4` and watch database CPU, memory, disk and ingest latency before increasing it. More workers help only while the database has spare capacity; SQLite's single writer limits the benefit there. There is no configurable throughput limiter. Keep legacy source rows unchanged while copying: do not import, manually delete or mutate them during a pass. Account for automatic retention when selecting the history to preserve.
## Progress and verification
@@ -107,12 +112,15 @@ ORDER BY day DESC, source_table;
| `started` | The table/day is in progress or was interrupted; inspect backend logs to distinguish them |
| `done` | The pass completed this table/day without a reported error; later passes skip it |
-`updated_at` changes when a table/day is started or completed; it is not a per-batch heartbeat. `moved_rows` is recorded on completion and counts processed source rows, including rows skipped during recovery. It is not the number of newly inserted V2 rows or a live progress counter.
+`updated_at` and `moved_rows` update at the start, after successful batches roughly every 10 seconds, and on completion. A slow query or insert can delay the next update; this is not a heartbeat independent of copying. `moved_rows` counts processed source rows in the current attempt, including rows skipped during recovery. It resets when an interrupted table/day is resumed and is not the number of newly inserted V2 rows. With multiple workers, several dates can be `started` at the same time.
Backend logs use the `move-over:` prefix. For example:
```text
-move-over: 2026-09-19 spans: 23677 rows in 1.0s
+move-over: starting pass from 2026-09-19 back to 2026-09-01 with 4 workers
+move-over: 2026-09-19 spans: starting (resumed=false)
+move-over: 2026-09-19 spans: in progress, 18000 source rows processed in 10.1s
+move-over: 2026-09-19 spans: 23677 rows in 13.0s
move-over: done, every day from 2026-09-19 back to 2026-09-01 is in the V2 tables
```
@@ -124,7 +132,7 @@ Leave the setting enabled to retry remaining work hourly, or disable it and rest
Set `V2_MOVE_OVER=false` or remove it, then restart/recreate the worker instance to stop background copying. Already copied rows remain available. Re-enable it later to resume; retain the main database and its progress records along with the telemetry database.
-Before writing a table/day, the worker commits a `started` marker. It writes spans before their corresponding endpoint, task or AI rows, and records `done` after the day completes. A read, write or progress error stops the pass, logs the failure and leaves unfinished work resumable. A rejected canonical span also prevents that day from being marked complete. Persistent errors require correcting the cause; retrying does not bypass invalid data or insufficient disk space.
+Before writing a table/day, the worker commits a `started` marker. It writes spans before their corresponding endpoint, task or AI rows, and records `done` after the day completes. A read, write or progress error cancels the pass and waits for all workers to stop before retrying. Completed pairs remain `done`; unfinished pairs remain resumable. A rejected canonical span also prevents that day from being marked complete. Persistent errors require correcting the cause; retrying does not bypass invalid data or insufficient disk space.
Recovery rescans the unfinished day rather than resuming a saved page offset. It skips destination rows already found using these identities: