Skip to content

A platform for community discussion. Free, open, simple.

MT: Run conversion steps concurrently and split the heavy ones across cores

Gerhard Schlager12a065662765
diff --git a/Gemfile.lock b/Gemfile.lock
index 9d084599b39..389c38a25a7 100644
--- a/Gemfile.lock
+++ b/Gemfile.lock
@@ -15,11 +15,12 @@ PATH
migrations-core (0.0.1)
activesupport
colored2
+ concurrent-ruby
digest-xxhash
extralite-bundle
i18n
+ json
lru_redux
- oj
samovar
unicode-display_width
zeitwerk
diff --git a/migrations/README.md b/migrations/README.md
index 18730b6751d..8db0d6b2465 100644
--- a/migrations/README.md
+++ b/migrations/README.md
@@ -45,6 +45,54 @@ the same pattern. The discard mechanism itself is library-specific — mysql2 ha
needs one at all: connect, fork an empty child that exits normally, wait for it, and query
again from the parent (see the fork-safety specs in `postgres_spec.rb`).
+### Partitioning large steps
+
+Most steps run in a single worker. A handful are large enough that it's worth
+splitting them across CPU cores, so the framework can run one worker per chunk of
+the source. A step opts in from its `source` block:
+
+```ruby
+source do
+ reads_table "topic_users", where: "user_id > 0"
+ partition_by :topic_id
+end
+```
+
+`reads_table` is the part that reads a whole table: it defines `items`
+(`SELECT * FROM topic_users WHERE …`) and `max_progress` (the row count), filtered
+by `where`. It works on its own, without partitioning — a plain table-copy step
+declares just `reads_table` and writes neither method. `partition_by` adds the
+split: it takes the key (normally a single indexed column, so each chunk is an
+index range scan; pass an array for a composite key) and reuses the table and
+filter from `reads_table`, so it only needs the column. When both are present the
+generated queries add the chunk to their `WHERE` automatically.
+
+Override `items` when you need specific columns, a join, or a particular order —
+and then add `partition_slice` to its `WHERE` yourself:
+
+```ruby
+def items
+ @source_db.query("SELECT id, name FROM topic_users WHERE #{partition_slice} AND name IS NOT NULL")
+end
+```
+
+The framework does the rest. Before forking, it asks the adapter for the chunk
+boundaries — evenly sized chunks over a numeric key, or a sorted-key scan for a
+text/UUID/composite key. It then forks one worker per chunk; each worker reads
+its `[lower, upper)` slice (that's what `partition_slice` expands to), writes its
+own SQLite shard, and the shards are merged back into the run database.
+
+Two things to get right:
+
+- **In a custom query, add `partition_slice` to the `WHERE`.** Miss it and each
+ worker reads the whole source instead of its slice — duplicated work and wrong
+ counts.
+- **Only partition order-independent steps.** Workers run concurrently and their
+ output is merged, so there is no global order across the step. A running total
+ or a sequence number across all rows can't be partitioned. Deduplication can,
+ but do it in the source query (`DISTINCT ON`, a window function, a view) and
+ partition on the dedup key, rather than keeping state in `process`.
+
## Schema DSL
The schema DSL lives in `migrations/tooling/lib/migrations/tooling/schema/dsl/`. Config sources
@@ -100,11 +148,3 @@ bin/lint --fix path/to/file
```
Uses both rubocop and syntax_tree. Always lint changed files.
-
-## Known issues
-
-- **Parallel step items must be hashes (or other non-scalar JSON values).** Worker processes
- receive their items as an Oj stream over a pipe, and Oj's stream parser can only detect the
- end of a bare scalar (a number or string) once the next byte arrives — so a step whose
- `items` yields scalars stalls the worker pipeline. Wrap scalar items in a hash
- (e.g. `{ id: ... }`). This should go away if we switch the worker pipes from Oj to JSON.
diff --git a/migrations/converters/lib/migrations-converters.rb b/migrations/converters/lib/migrations-converters.rb
index 15bb421d3c9..467d8a67ae3 100644
--- a/migrations/converters/lib/migrations-converters.rb
+++ b/migrations/converters/lib/migrations-converters.rb
@@ -1,4 +1,12 @@
# frozen_string_literal: true
require "migrations-core"
+
+# `pg` isn't safe to `require` from several threads at once: doing so races
+# pg.rb's own body and it re-initializes its constants with a warning. The
+# Postgres adapter that pulls it in is autoloaded lazily and first referenced
+# inside the scheduler's concurrent coordinator threads and forks, so load it
+# here on the main thread, before any of that starts.
+require "pg"
+
require_relative "migrations/converters"
diff --git a/migrations/converters/lib/migrations/converters/adapter/postgres.rb b/migrations/converters/lib/migrations/converters/adapter/postgres.rb
index 274076d6c8a..0401ad8b51f 100644
--- a/migrations/converters/lib/migrations/converters/adapter/postgres.rb
+++ b/migrations/converters/lib/migrations/converters/adapter/postgres.rb
@@ -6,6 +6,8 @@ module Migrations
module Converters
module Adapter
class Postgres
+ include Conversion::PartitionSource
+
class DiscardedError < StandardError
end
@@ -34,27 +36,24 @@ module Migrations
end
end
- def query_first_row(sql)
- connection.exec(sql).first
- end
-
- def query_value(sql, column = nil)
- if (row = query_first_row(sql))
- column ? row[column.to_sym] : row.values.first
- else
- nil
- end
+ def query_value(sql)
+ query_first_row(sql)&.values&.first
end
def count(sql)
query_value(sql).to_i
end
- # Reads every row of `table`, optionally narrowed by the `where` body
- # (nil reads the whole table). Backs the `reads_table` default in a step's
- # source; the table name is quoted here so the SQL stays in the adapter.
- def select_all(table, where: nil)
- query("SELECT * FROM #{connection.quote_ident(table)}#{where_clause(where)}")
+ # Reads every row of `table`, optionally narrowed by the `where` body (nil
+ # reads the whole table) and sorted by `order` (a column or array of them).
+ # Backs the `reads_table` default in a step's source; a partitioned step
+ # orders by its key so each worker writes its shard in key order — sequential
+ # SQLite index inserts in the shard and, later, in the merge.
+ def select_all(table, where: nil, order: nil)
+ query(
+ "SELECT * FROM #{connection.quote_ident(table)}" \
+ "#{where_clause(where)}#{order_clause(order)}",
+ )
end
# The row count of `table`, optionally narrowed by `where`. Backs
@@ -63,6 +62,59 @@ module Migrations
count("SELECT COUNT(*) FROM #{connection.quote_ident(table)}#{where_clause(where)}")
end
+ # The following back {Conversion::Partitioner}; see its primitive list.
+
+ # The key's min and max. Answered from the key's index, so this stays cheap
+ # and doesn't scan the table.
+ def partition_bounds(column, from, base)
+ row =
+ query_first_row(
+ "SELECT MIN(#{column}) AS min, MAX(#{column}) AS max " \
+ "FROM #{connection.quote_ident(from)}#{where_clause(base)}",
+ )
+ [row[:min], row[:max]]
+ end
+
+ # The planner's row estimate (`pg_class.reltuples`), so the partitioner can
+ # tell a dense key from a sparse one without a COUNT(*) scan. It's the whole
+ # table's estimate and ignores `base`, which is fine for a magnitude check;
+ # it is -1 before the table is analysed.
+ def estimated_row_count(from)
+ query_value(
+ "SELECT reltuples::bigint FROM pg_class WHERE oid = #{quote(from)}::regclass",
+ ).to_i
+ end
+
+ # The chunk lower bounds, computed in SQL: `NTILE` splits the sorted key
+ # into `count` equal-sized buckets and `DISTINCT ON` takes each bucket's
+ # first value, so only the boundaries cross the wire, not the whole key
+ # column. Works the same for a scalar or a composite key. This is the
+ # optional fast path the {Conversion::Partitioner} prefers over streaming.
+ def boundaries_by_scan(key, from, base, count)
+ select = Array(key).join(", ")
+ rows = query(<<~SQL)
+ SELECT DISTINCT ON (bucket) #{select}
+ FROM (
+ SELECT #{select}, NTILE(#{count}) OVER (ORDER BY #{select}) AS bucket
+ FROM #{connection.quote_ident(from)}#{where_clause(base)}
+ ) buckets
+ ORDER BY bucket, #{select}
+ SQL
+ rows.map { |row| boundary_value(row, key) }.uniq
+ end
+
+ # The WHERE body limiting a query to the chunk `[lower, upper)` of `key`
+ # (one column, or an array for a composite key), AND-ed with `base`. A
+ # composite key compares as a row value: `(a, b) >= (a0, b0)`.
+ def chunk_filter(key, lower, upper, base: nil)
+ return base if lower.nil?
+
+ expression = key_expression(key)
+ conditions = ["#{expression} >= #{value_expression(key, lower)}"]
+ conditions << "#{expression} < #{value_expression(key, upper)}" unless upper.nil?
+ [base, *conditions].compact.join(" AND ")
+ end
+
def close
@connection.finish if @connection && [email protected]?
@connection = nil
@@ -92,15 +144,6 @@ module Migrations
@discarded = true
end
- def reset
- connection.reset
- configure_connection
- end
-
- def escape_string(str)
- connection.escape_string(str)
- end
-
def encode_array(array)
@array_encoder ||= PG::TextEncoder::Array.new
@@ -109,6 +152,14 @@ module Migrations
private
+ def query_first_row(sql)
+ connection.exec(sql).first
+ end
+
+ def escape_string(str)
+ connection.escape_string(str)
+ end
+
# `" WHERE <filter>"`, or "" when there's no filter — so a missing filter
# reads the whole table instead of leaning on a `WHERE TRUE` that not every
# dialect accepts.
@@ -116,6 +167,29 @@ module Migrations
filter ? " WHERE #{filter}" : ""
end
+ # `" ORDER BY <cols>"`, or "" when unordered. Accepts a column or an array.
+ def order_clause(order)
+ order ? " ORDER BY #{Array(order).join(", ")}" : ""
+ end
+
+ def boundary_value(row, key)
+ return row.values.first unless key.is_a?(Array)
+ key.map { |part| row[part.to_sym] }
+ end
+
+ def key_expression(key)
+ key.is_a?(Array) ? "(#{key.join(", ")})" : key.to_s
+ end
+
+ def value_expression(key, value)
+ return quote(value) unless key.is_a?(Array)
+ "(#{value.map { |part| quote(part) }.join(", ")})"
+ end
+
+ def quote(value)
+ value.is_a?(Numeric) ? value.to_s : "'#{escape_string(value.to_s)}'"
+ end
+
def connection
if @discarded
raise DiscardedError,
diff --git a/migrations/converters/lib/migrations/converters/cli/convert_command.rb b/migrations/converters/lib/migrations/converters/cli/convert_command.rb
index 448fdbf724d..da5874b4fe9 100644
--- a/migrations/converters/lib/migrations/converters/cli/convert_command.rb
+++ b/migrations/converters/lib/migrations/converters/cli/convert_command.rb
@@ -22,6 +22,15 @@ module Migrations
"Skip the specified steps (comma-separated).",
default: [],
type: STEP_LIST
+ option "--max-parallel-steps <count>",
+ "Maximum number of steps to run at the same time. " \
+ "Caps the source DB connections a run opens; " \
+ "defaults to the worker budget (usable CPUs minus one).",
+ type: Integer
+ option "--no-fork",
+ "Run each step inline in one process instead of forking workers. " \
+ "Forces serial execution; meant for debugging (a breakpoint in a " \
+ "step's `process` stops in the main run, not an unreachable child)."
end
one :converter_type, "The converter to run (e.g. discourse)."
@@ -39,10 +48,15 @@ module Migrations
settings = load_settings(type)
- Database.reset!(settings[:intermediate_db][:path]) if @options[:reset]
+ Database.delete_database(settings[:intermediate_db][:path]) if @options[:reset]
converter = "migrations/converters/#{type}/converter".camelize.constantize
- converter.new(settings).run(only_steps: @options[:only], skip_steps: @options[:skip])
+ converter.new(settings).run(
+ only_steps: @options[:only],
+ skip_steps: @options[:skip],
+ max_parallel_steps: @options[:max_parallel_steps],
+ no_fork: @options[:no_fork] || false,
+ )
end
private
diff --git a/migrations/converters/lib/migrations/converters/discourse/converter.rb b/migrations/converters/lib/migrations/converters/discourse/converter.rb
index 1dcd91eb7f5..093f699f23c 100644
--- a/migrations/converters/lib/migrations/converters/discourse/converter.rb
+++ b/migrations/converters/lib/migrations/converters/discourse/converter.rb
@@ -4,13 +4,10 @@ module Migrations
module Converters
module Discourse
class Converter < Conversion::Base
- def initialize(settings)
- super
- @source_db = Adapter::Postgres.new(settings[:source_db])
- end
-
+ # Steps run concurrently and a Postgres connection can't be shared, so each
+ # step gets its own adapter; the step's source closes it in its `cleanup`.
def step_args(step_class)
- { source_db: @source_db }
+ { source_db: Adapter::Postgres.new(settings[:source_db]) }
end
end
end
diff --git a/migrations/converters/lib/migrations/converters/discourse/steps/badge_groupings.rb b/migrations/converters/lib/migrations/converters/discourse/steps/badge_groupings.rb
index cc0d47f1949..66d1a9b2cd8 100644
--- a/migrations/converters/lib/migrations/converters/discourse/steps/badge_groupings.rb
+++ b/migrations/converters/lib/migrations/converters/discourse/steps/badge_groupings.rb
@@ -3,7 +3,7 @@
module Migrations
module Converters
module Discourse
- class BadgeGroupings < Conversion::ProgressStep
+ class BadgeGroupings < Conversion::Step
source do
# Skip the system badge groupings.
reads_table "badge_groupings", where: "id NOT IN (1, 2, 3, 4, 5)"
diff --git a/migrations/converters/lib/migrations/converters/discourse/steps/badges.rb b/migrations/converters/lib/migrations/converters/discourse/steps/badges.rb
index 2724bb81420..bac2243edab 100644
--- a/migrations/converters/lib/migrations/converters/discourse/steps/badges.rb
+++ b/migrations/converters/lib/migrations/converters/discourse/steps/badges.rb
@@ -3,7 +3,7 @@
module Migrations
module Converters
module Discourse
- class Badges < Conversion::ProgressStep
+ class Badges < Conversion::Step
source do
def max_progress
@source_db.count <<~SQL
diff --git a/migrations/converters/lib/migrations/converters/discourse/steps/categories.rb b/migrations/converters/lib/migrations/converters/discourse/steps/categories.rb
index 1afa697cd02..d30f5ba3d6c 100644
--- a/migrations/converters/lib/migrations/converters/discourse/steps/categories.rb
+++ b/migrations/converters/lib/migrations/converters/discourse/steps/categories.rb
@@ -3,7 +3,7 @@
module Migrations
module Converters
module Discourse
- class Categories < Conversion::ProgressStep
+ class Categories < Conversion::Step
SEEDED_CATEGORY_SETTINGS = %w[
uncategorized_category_id
meta_category_id
diff --git a/migrations/converters/lib/migrations/converters/discourse/steps/category_custom_fields.rb b/migrations/converters/lib/migrations/converters/discourse/steps/category_custom_fields.rb
index 9f93e53adfd..34823f1aea6 100644
--- a/migrations/converters/lib/migrations/converters/discourse/steps/category_custom_fields.rb
+++ b/migrations/converters/lib/migrations/converters/discourse/steps/category_custom_fields.rb
@@ -3,7 +3,7 @@
module Migrations
module Converters
module Discourse
- class CategoryCustomFields < Conversion::ProgressStep
+ class CategoryCustomFields < Conversion::Step
source { reads_table "category_custom_fields" }
processor do
diff --git a/migrations/converters/lib/migrations/converters/discourse/steps/category_moderation_groups.rb b/migrations/converters/lib/migrations/converters/discourse/steps/category_moderation_groups.rb
index 2fca4046e9d..d5c35cc08f6 100644
--- a/migrations/converters/lib/migrations/converters/discourse/steps/category_moderation_groups.rb
+++ b/migrations/converters/lib/migrations/converters/discourse/steps/category_moderation_groups.rb
@@ -3,7 +3,7 @@
module Migrations
module Converters
module Discourse
- class CategoryModerationGroups < Conversion::ProgressStep
+ class CategoryModerationGroups < Conversion::Step
source { reads_table "category_moderation_groups", where: "group_id > 0" }
processor do
diff --git a/migrations/converters/lib/migrations/converters/discourse/steps/category_users.rb b/migrations/converters/lib/migrations/converters/discourse/steps/category_users.rb
index 06df06dfc86..858fafc8d0b 100644
--- a/migrations/converters/lib/migrations/converters/discourse/steps/category_users.rb
+++ b/migrations/converters/lib/migrations/converters/discourse/steps/category_users.rb
@@ -3,7 +3,7 @@
module Migrations
module Converters
module Discourse
- class CategoryUsers < Conversion::ProgressStep
+ class CategoryUsers < Conversion::Step
source { reads_table "category_users", where: "user_id > 0" }
processor do
diff --git a/migrations/converters/lib/migrations/converters/discourse/steps/group_users.rb b/migrations/converters/lib/migrations/converters/discourse/steps/group_users.rb
index 554a2db2b2e..97701b46794 100644
--- a/migrations/converters/lib/migrations/converters/discourse/steps/group_users.rb
+++ b/migrations/converters/lib/migrations/converters/discourse/steps/group_users.rb
@@ -3,7 +3,7 @@
module Migrations
module Converters
module Discourse
- class GroupUsers < Conversion::ProgressStep
+ class GroupUsers < Conversion::Step
source do
# Skip the automatic Trust Level groups.
reads_table "group_users", where: "group_id NOT IN (10, 11, 12, 13, 14) AND user_id > 0"
diff --git a/migrations/converters/lib/migrations/converters/discourse/steps/groups.rb b/migrations/converters/lib/migrations/converters/discourse/steps/groups.rb
index b1f6a596d3a..239e08dad06 100644
--- a/migrations/converters/lib/migrations/converters/discourse/steps/groups.rb
+++ b/migrations/converters/lib/migrations/converters/discourse/steps/groups.rb
@@ -3,7 +3,7 @@
module Migrations
module Converters
module Discourse
- class Groups < Conversion::ProgressStep
+ class Groups < Conversion::Step
source do
def max_progress
@source_db.count <<~SQL
diff --git a/migrations/converters/lib/migrations/converters/discourse/steps/muted_users.rb b/migrations/converters/lib/migrations/converters/discourse/steps/muted_users.rb
index 8af34ee4516..1a9d422b855 100644
--- a/migrations/converters/lib/migrations/converters/discourse/steps/muted_users.rb
+++ b/migrations/converters/lib/migrations/converters/discourse/steps/muted_users.rb
@@ -3,7 +3,7 @@
module Migrations
module Converters
module Discourse
- class MutedUsers < Conversion::ProgressStep
+ class MutedUsers < Conversion::Step
source { reads_table "muted_users" }
processor do
diff --git a/migrations/converters/lib/migrations/converters/discourse/steps/permalink_normalizations.rb b/migrations/converters/lib/migrations/converters/discourse/steps/permalink_normalizations.rb
index a181a451a4f..a351b4cc83d 100644
--- a/migrations/converters/lib/migrations/converters/discourse/steps/permalink_normalizations.rb
+++ b/migrations/converters/lib/migrations/converters/discourse/steps/permalink_normalizations.rb
@@ -4,18 +4,35 @@ module Migrations
module Converters
module Discourse
class PermalinkNormalizations < Conversion::Step
- def execute
- normalizations = @source_db.query_value <<~SQL
- SELECT value
- FROM site_settings
- WHERE name = 'permalink_normalizations'
- SQL
+ source do
+ def max_progress
+ normalizations.size
+ end
- return if normalizations.blank?
+ def items
+ normalizations
+ end
- normalizations
- .split("|")
- .each { |normalization| IntermediateDB::PermalinkNormalization.create(normalization:) }
+ private
+
+ def normalizations
+ @normalizations ||=
+ begin
+ value = @source_db.query_value(<<~SQL)
+ SELECT value
+ FROM site_settings
+ WHERE name = 'permalink_normalizations'
+ SQL
+
+ value.present? ? value.split("|") : []
+ end
+ end
+ end
+
+ processor do
+ def process(item)
+ IntermediateDB::PermalinkNormalization.create(normalization: item)
+ end
end
end
end
diff --git a/migrations/converters/lib/migrations/converters/discourse/steps/site_settings.rb b/migrations/converters/lib/migrations/converters/discourse/steps/site_settings.rb
index 963eabb6fd9..ec127beaa95 100644
--- a/migrations/converters/lib/migrations/converters/discourse/steps/site_settings.rb
+++ b/migrations/converters/lib/migrations/converters/discourse/steps/site_settings.rb
@@ -3,7 +3,7 @@
module Migrations
module Converters
module Discourse
- class SiteSettings < Conversion::ProgressStep
+ class SiteSettings < Conversion::Step
source do
def max_progress
@source_db.count <<~SQL
diff --git a/migrations/converters/lib/migrations/converters/discourse/steps/tag_group_memberships.rb b/migrations/converters/lib/migrations/converters/discourse/steps/tag_group_memberships.rb
index 11b4d223dee..5966218e6cd 100644
--- a/migrations/converters/lib/migrations/converters/discourse/steps/tag_group_memberships.rb
+++ b/migrations/converters/lib/migrations/converters/discourse/steps/tag_group_memberships.rb
@@ -3,7 +3,7 @@
module Migrations
module Converters
module Discourse
- class TagGroupMemberships < Conversion::ProgressStep
+ class TagGroupMemberships < Conversion::Step
source { reads_table "tag_group_memberships" }
processor do
diff --git a/migrations/converters/lib/migrations/converters/discourse/steps/tag_group_permissions.rb b/migrations/converters/lib/migrations/converters/discourse/steps/tag_group_permissions.rb
index 0d3b341b93e..6506da03102 100644
--- a/migrations/converters/lib/migrations/converters/discourse/steps/tag_group_permissions.rb
+++ b/migrations/converters/lib/migrations/converters/discourse/steps/tag_group_permissions.rb
@@ -3,7 +3,7 @@
module Migrations
module Converters
module Discourse
- class TagGroupPermissions < Conversion::ProgressStep
+ class TagGroupPermissions < Conversion::Step
source { reads_table "tag_group_permissions" }
processor do
diff --git a/migrations/converters/lib/migrations/converters/discourse/steps/tag_groups.rb b/migrations/converters/lib/migrations/converters/discourse/steps/tag_groups.rb
index 68bbdcd2925..db912b6f5cb 100644
--- a/migrations/converters/lib/migrations/converters/discourse/steps/tag_groups.rb
+++ b/migrations/converters/lib/migrations/converters/discourse/steps/tag_groups.rb
@@ -3,7 +3,7 @@
module Migrations
module Converters
module Discourse
- class TagGroups < Conversion::ProgressStep
+ class TagGroups < Conversion::Step
source { reads_table "tag_groups" }
processor do
diff --git a/migrations/converters/lib/migrations/converters/discourse/steps/tag_users.rb b/migrations/converters/lib/migrations/converters/discourse/steps/tag_users.rb
index f2fbe4507d1..3951db5eec7 100644
--- a/migrations/converters/lib/migrations/converters/discourse/steps/tag_users.rb
+++ b/migrations/converters/lib/migrations/converters/discourse/steps/tag_users.rb
@@ -3,7 +3,7 @@
module Migrations
module Converters
module Discourse
- class TagUsers < Conversion::ProgressStep
+ class TagUsers < Conversion::Step
source { reads_table "tag_users", where: "user_id > 0" }
processor do
diff --git a/migrations/converters/lib/migrations/converters/discourse/steps/tags.rb b/migrations/converters/lib/migrations/converters/discourse/steps/tags.rb
index a75f12d45c5..06aed00f817 100644
--- a/migrations/converters/lib/migrations/converters/discourse/steps/tags.rb
+++ b/migrations/converters/lib/migrations/converters/discourse/steps/tags.rb
@@ -3,7 +3,7 @@
module Migrations
module Converters
module Discourse
- class Tags < Conversion::ProgressStep
+ class Tags < Conversion::Step
source { reads_table "tags" }
processor do
diff --git a/migrations/converters/lib/migrations/converters/discourse/steps/topic_allowed_groups.rb b/migrations/converters/lib/migrations/converters/discourse/steps/topic_allowed_groups.rb
index 923170d3649..d28106a4abc 100644
--- a/migrations/converters/lib/migrations/converters/discourse/steps/topic_allowed_groups.rb
+++ b/migrations/converters/lib/migrations/converters/discourse/steps/topic_allowed_groups.rb
@@ -3,7 +3,7 @@
module Migrations
module Converters
module Discourse
- class TopicAllowedGroups < Conversion::ProgressStep
+ class TopicAllowedGroups < Conversion::Step
source { reads_table "topic_allowed_groups" }
processor do
diff --git a/migrations/converters/lib/migrations/converters/discourse/steps/topic_allowed_users.rb b/migrations/converters/lib/migrations/converters/discourse/steps/topic_allowed_users.rb
index dcc24f45eb4..7421904d440 100644
--- a/migrations/converters/lib/migrations/converters/discourse/steps/topic_allowed_users.rb
+++ b/migrations/converters/lib/migrations/converters/discourse/steps/topic_allowed_users.rb
@@ -3,7 +3,7 @@
module Migrations
module Converters
module Discourse
- class TopicAllowedUsers < Conversion::ProgressStep
+ class TopicAllowedUsers < Conversion::Step
source { reads_table "topic_allowed_users", where: "user_id > 0" }
processor do
diff --git a/migrations/converters/lib/migrations/converters/discourse/steps/topic_tags.rb b/migrations/converters/lib/migrations/converters/discourse/steps/topic_tags.rb
index 5d281c950c2..5f5e33161ce 100644
--- a/migrations/converters/lib/migrations/converters/discourse/steps/topic_tags.rb
+++ b/migrations/converters/lib/migrations/converters/discourse/steps/topic_tags.rb
@@ -3,7 +3,7 @@
module Migrations
module Converters
module Discourse
- class TopicTags < Conversion::ProgressStep
+ class TopicTags < Conversion::Step
source { reads_table "topic_tags" }
processor do
diff --git a/migrations/converters/lib/migrations/converters/discourse/steps/topic_users.rb b/migrations/converters/lib/migrations/converters/discourse/steps/topic_users.rb
index dfc4f1bcd6b..b7ee85cc9b8 100644
--- a/migrations/converters/lib/migrations/converters/discourse/steps/topic_users.rb
+++ b/migrations/converters/lib/migrations/converters/discourse/steps/topic_users.rb
@@ -3,8 +3,11 @@
module Migrations
module Converters
module Discourse
- class TopicUsers < Conversion::ProgressStep
- source { reads_table "topic_users", where: "user_id > 0" }
+ class TopicUsers < Conversion::Step
+ source do
+ reads_table "topic_users", where: "user_id > 0"
+ partition_by :topic_id
+ end
processor do
def process(item)
diff --git a/migrations/converters/lib/migrations/converters/discourse/steps/topics.rb b/migrations/converters/lib/migrations/converters/discourse/steps/topics.rb
index e8c0bceb2e3..8ed78bf5b23 100644
--- a/migrations/converters/lib/migrations/converters/discourse/steps/topics.rb
+++ b/migrations/converters/lib/migrations/converters/discourse/steps/topics.rb
@@ -3,7 +3,7 @@
module Migrations
module Converters
module Discourse
- class Topics < Conversion::ProgressStep
+ class Topics < Conversion::Step
source do
def max_progress
@source_db.count <<~SQL
@@ -20,6 +20,7 @@ module Migrations
up.user_id AS og_image_user_id
FROM topics t
LEFT JOIN uploads up ON t.og_image_upload_id = up.id
+ ORDER BY t.id
SQL
end
end
diff --git a/migrations/converters/lib/migrations/converters/discourse/steps/user_associated_accounts.rb b/migrations/converters/lib/migrations/converters/discourse/steps/user_associated_accounts.rb
index 9233f441dac..125be371802 100644
--- a/migrations/converters/lib/migrations/converters/discourse/steps/user_associated_accounts.rb
+++ b/migrations/converters/lib/migrations/converters/discourse/steps/user_associated_accounts.rb
@@ -3,7 +3,7 @@
module Migrations
module Converters
module Discourse
- class UserAssociatedAccounts < Conversion::ProgressStep
+ class UserAssociatedAccounts < Conversion::Step
source { reads_table "user_associated_accounts", where: "user_id > 0" }
processor do
diff --git a/migrations/converters/lib/migrations/converters/discourse/steps/user_custom_fields.rb b/migrations/converters/lib/migrations/converters/discourse/steps/user_custom_fields.rb
index 21b92e73cfe..4f9602837a9 100644
--- a/migrations/converters/lib/migrations/converters/discourse/steps/user_custom_fields.rb
+++ b/migrations/converters/lib/migrations/converters/discourse/steps/user_custom_fields.rb
@@ -3,7 +3,7 @@
module Migrations
module Converters
module Discourse
- class UserCustomFields < Conversion::ProgressStep
+ class UserCustomFields < Conversion::Step
source do
def max_progress
@source_db.count <<~SQL
diff --git a/migrations/converters/lib/migrations/converters/discourse/steps/user_emails.rb b/migrations/converters/lib/migrations/converters/discourse/steps/user_emails.rb
index 50c68ec3fbc..d766510c9c7 100644
--- a/migrations/converters/lib/migrations/converters/discourse/steps/user_emails.rb
+++ b/migrations/converters/lib/migrations/converters/discourse/steps/user_emails.rb
@@ -3,7 +3,7 @@
module Migrations
module Converters
module Discourse
- class UserEmails < Conversion::ProgressStep
+ class UserEmails < Conversion::Step
source { reads_table "user_emails", where: "user_id > 0" }
processor do
diff --git a/migrations/converters/lib/migrations/converters/discourse/steps/user_field_options.rb b/migrations/converters/lib/migrations/converters/discourse/steps/user_field_options.rb
index d9d7f81392e..643a9e75af5 100644
--- a/migrations/converters/lib/migrations/converters/discourse/steps/user_field_options.rb
+++ b/migrations/converters/lib/migrations/converters/discourse/steps/user_field_options.rb
@@ -3,7 +3,7 @@
module Migrations
module Converters
module Discourse
- class UserFieldOptions < Conversion::ProgressStep
+ class UserFieldOptions < Conversion::Step
source { reads_table "user_field_options" }
processor do
diff --git a/migrations/converters/lib/migrations/converters/discourse/steps/user_field_values.rb b/migrations/converters/lib/migrations/converters/discourse/steps/user_field_values.rb
index 81c24ddb8c9..b6ba0d841a1 100644
--- a/migrations/converters/lib/migrations/converters/discourse/steps/user_field_values.rb
+++ b/migrations/converters/lib/migrations/converters/discourse/steps/user_field_values.rb
@@ -3,7 +3,7 @@
module Migrations
module Converters
module Discourse
- class UserFieldValues < Conversion::ProgressStep
+ class UserFieldValues < Conversion::Step
USER_FIELD_PREFIX = "user_field_"
# `USER_FIELD_PREFIX` as a `LIKE` pattern: the `_` characters are literal
diff --git a/migrations/converters/lib/migrations/converters/discourse/steps/user_fields.rb b/migrations/converters/lib/migrations/converters/discourse/steps/user_fields.rb
index ace334af6e1..bbb40e3a79b 100644
--- a/migrations/converters/lib/migrations/converters/discourse/steps/user_fields.rb
+++ b/migrations/converters/lib/migrations/converters/discourse/steps/user_fields.rb
@@ -3,7 +3,7 @@
module Migrations
module Converters
module Discourse
- class UserFields < Conversion::ProgressStep
+ class UserFields < Conversion::Step
source { reads_table "user_fields" }
processor do
diff --git a/migrations/converters/lib/migrations/converters/discourse/steps/user_options.rb b/migrations/converters/lib/migrations/converters/discourse/steps/user_options.rb
index 4e3bb9bc028..3a1fb033b61 100644
--- a/migrations/converters/lib/migrations/converters/discourse/steps/user_options.rb
+++ b/migrations/converters/lib/migrations/converters/discourse/steps/user_options.rb
@@ -3,7 +3,7 @@
module Migrations
module Converters
module Discourse
- class UserOptions < Conversion::ProgressStep
+ class UserOptions < Conversion::Step
source { reads_table "user_options", where: "user_id > 0" }
processor do
diff --git a/migrations/converters/lib/migrations/converters/discourse/steps/user_suspensions.rb b/migrations/converters/lib/migrations/converters/discourse/steps/user_suspensions.rb
index c4b015837c7..ea805f562ae 100644
--- a/migrations/converters/lib/migrations/converters/discourse/steps/user_suspensions.rb
+++ b/migrations/converters/lib/migrations/converters/discourse/steps/user_suspensions.rb
@@ -3,7 +3,7 @@
module Migrations
module Converters
module Discourse
- class UserSuspensions < Conversion::ProgressStep
+ class UserSuspensions < Conversion::Step
source do
def max_progress
@source_db.count <<~SQL
diff --git a/migrations/converters/lib/migrations/converters/discourse/steps/users.rb b/migrations/converters/lib/migrations/converters/discourse/steps/users.rb
index 206e2d2f14b..c1203eaf40f 100644
--- a/migrations/converters/lib/migrations/converters/discourse/steps/users.rb
+++ b/migrations/converters/lib/migrations/converters/discourse/steps/users.rb
@@ -3,7 +3,7 @@
module Migrations
module Converters
module Discourse
- class Users < Conversion::ProgressStep
+ class Users < Conversion::Step
source do
def max_progress
@source_db.count <<~SQL
diff --git a/migrations/converters/spec/lib/migrations/converters/adapter/postgres_spec.rb b/migrations/converters/spec/lib/migrations/converters/adapter/postgres_spec.rb
index 158728cc420..2a2e8705550 100644
--- a/migrations/converters/spec/lib/migrations/converters/adapter/postgres_spec.rb
+++ b/migrations/converters/spec/lib/migrations/converters/adapter/postgres_spec.rb
@@ -24,6 +24,8 @@ RSpec.describe Migrations::Converters::Adapter::Postgres do
allow(connection).to receive(:socket_io).and_return(socket_io)
allow(connection).to receive(:finished?).and_return(false)
allow(connection).to receive(:finish)
+ allow(connection).to receive(:quote_ident) { |id| %("#{id}") }
+ allow(connection).to receive(:escape_string) { |str| str }
allow(socket_io).to receive(:reopen)
end
@@ -37,9 +39,106 @@ RSpec.describe Migrations::Converters::Adapter::Postgres do
end
end
- describe "#select_all and #count_all" do
- before { allow(connection).to receive(:quote_ident) { |id| %("#{id}") } }
+ it "implements the whole PartitionSource interface" do
+ create_adapter do |adapter|
+ expect(Migrations::Conversion::PartitionSource.missing_from(adapter)).to be_empty
+ end
+ end
+
+ describe "#partition_bounds" do
+ it "returns the key's min and max" do
+ allow(connection).to receive(:exec).with(/MIN\(topic_id\)/).and_return(
+ [{ min: 0, max: 99 }],
+ )
+ create_adapter do |adapter|
+ expect(adapter.partition_bounds(:topic_id, "topic_users", "user_id > 0")).to eq([0, 99])
+ end
+ end
+ end
+
+ describe "#estimated_row_count" do
+ it "reads the planner's row estimate" do
+ allow(connection).to receive(:exec).with(/reltuples/).and_return([{ reltuples: 1234 }])
+ create_adapter { |adapter| expect(adapter.estimated_row_count("things")).to eq(1234) }
+ end
+ end
+
+ describe "#boundaries_by_scan" do
+ it "returns each bucket's first value" do
+ create_adapter do |adapter|
+ allow(adapter).to receive(:query).and_return([{ id: 0 }, { id: 50 }])
+ expect(adapter.boundaries_by_scan(:id, "things", nil, 2)).to eq([0, 50])
+ end
+ end
+ it "returns a tuple per bucket for a composite key" do
+ create_adapter do |adapter|
+ allow(adapter).to receive(:query).and_return(
+ [{ topic_id: 1, user_id: 10 }, { topic_id: 2, user_id: 5 }],
+ )
+ expect(adapter.boundaries_by_scan(%i[topic_id user_id], "topic_users", nil, 2)).to eq(
+ [[1, 10], [2, 5]],
+ )
+ end
+ end
+
+ it "drops duplicate boundaries from buckets that share a value" do
+ create_adapter do |adapter|
+ allow(adapter).to receive(:query).and_return([{ id: 5 }, { id: 5 }])
+ expect(adapter.boundaries_by_scan(:id, "things", nil, 2)).to eq([5])
+ end
+ end
+ end
+
+ describe "#chunk_filter" do
+ def chunk(lower, upper, base: "user_id > 0")
+ create_adapter { |adapter| adapter.chunk_filter(:topic_id, lower, upper, base:) }
+ end
+
+ it "limits to a half-open numeric range, AND-ed with the base" do
+ expect(chunk(25, 50)).to eq("user_id > 0 AND topic_id >= 25 AND topic_id < 50")
+ end
+
+ it "drops the upper bound for an open-ended chunk" do
+ expect(chunk(75, nil)).to eq("user_id > 0 AND topic_id >= 75")
+ end
+
+ it "is just the base filter when there's no chunk" do
+ expect(chunk(nil, nil)).to eq("user_id > 0")
+ end
+
+ it "is just the chunk conditions when there's no base" do
+ create_adapter do |adapter|
+ expect(adapter.chunk_filter(:topic_id, 25, 50)).to eq("topic_id >= 25 AND topic_id < 50")
+ end
+ end
+
+ it "quotes a text key (UUID, etc.)" do
+ create_adapter do |adapter|
+ expect(adapter.chunk_filter(:id, "uuid-a", "uuid-b")).to eq(
+ "id >= 'uuid-a' AND id < 'uuid-b'",
+ )
+ end
+ end
+
+ it "compares a composite key as a row value" do
+ create_adapter do |adapter|
+ expect(
+ adapter.chunk_filter(%i[topic_id user_id], [1, 10], [2, 5], base: "user_id > 0"),
+ ).to eq("user_id > 0 AND (topic_id, user_id) >= (1, 10) AND (topic_id, user_id) < (2, 5)")
+ end
+ end
+
+ it "leaves the upper off for an open-ended composite chunk" do
+ create_adapter do |adapter|
+ expect(adapter.chunk_filter(%i[topic_id user_id], [2, 5], nil)).to eq(
+ "(topic_id, user_id) >= (2, 5)",
+ )
+ end
+ end
+ end
+
+ describe "#select_all and #count_all" do
it "selects all rows from a quoted table" do
create_adapter do |adapter|
allow(adapter).to receive(:query).with('SELECT * FROM "things" WHERE active').and_return(
@@ -64,6 +163,24 @@ RSpec.describe Migrations::Converters::Adapter::Postgres do
expect(adapter.select_all("things")).to eq([:row])
end
end
+
+ it "orders by the given key so the rows arrive sorted" do
+ create_adapter do |adapter|
+ allow(adapter).to receive(:query).with(
+ 'SELECT * FROM "things" WHERE active ORDER BY id',
+ ).and_return([:row])
+ expect(adapter.select_all("things", where: "active", order: :id)).to eq([:row])
+ end
+ end
+
+ it "orders by every column of a composite key" do
+ create_adapter do |adapter|
+ allow(adapter).to receive(:query).with(
+ 'SELECT * FROM "things" ORDER BY topic_id, user_id',
+ ).and_return([:row])
+ expect(adapter.select_all("things", order: %i[topic_id user_id])).to eq([:row])
+ end
+ end
end
describe "#discard!" do
@@ -83,7 +200,6 @@ RSpec.describe Migrations::Converters::Adapter::Postgres do
expect { adapter.exec("SELECT 1") }.to raise_error(described_class::DiscardedError)
expect { adapter.query("SELECT 1") }.to raise_error(described_class::DiscardedError)
expect { adapter.query_value("SELECT 1") }.to raise_error(described_class::DiscardedError)
- expect { adapter.reset }.to raise_error(described_class::DiscardedError)
end
end
diff --git a/migrations/converters/spec/lib/migrations/converters/discourse/converter_spec.rb b/migrations/converters/spec/lib/migrations/converters/discourse/converter_spec.rb
new file mode 100644
index 00000000000..c715616c198
--- /dev/null
+++ b/migrations/converters/spec/lib/migrations/converters/discourse/converter_spec.rb
@@ -0,0 +1,24 @@
+# frozen_string_literal: true
+
+RSpec.describe Migrations::Converters::Discourse::Converter do
+ describe "#step_args" do
+ it "builds a fresh source adapter per step so concurrent steps don't share a connection" do
+ adapters = [
+ instance_double(Migrations::Converters::Adapter::Postgres),
+ instance_double(Migrations::Converters::Adapter::Postgres),
+ ]
+ allow(Migrations::Converters::Adapter::Postgres).to receive(:new).and_return(*adapters)
+
+ converter = described_class.new(source_db: { host: "localhost" })
+
+ first = converter.step_args(:first_step)[:source_db]
+ second = converter.step_args(:second_step)[:source_db]
+
+ expect(first).to be(adapters[0])
+ expect(second).to be(adapters[1])
+ expect(Migrations::Converters::Adapter::Postgres).to have_received(:new).with(
+ { host: "localhost" },
+ ).twice
+ end
+ end
+end
diff --git a/migrations/core/config/locales/migrations.en.yml b/migrations/core/config/locales/migrations.en.yml
index 407eebf0773..1d6a82486b3 100644
--- a/migrations/core/config/locales/migrations.en.yml
+++ b/migrations/core/config/locales/migrations.en.yml
@@ -15,6 +15,13 @@ en:
errors:
one: "%{number} error"
other: "%{number} errors"
+ finishing_up: "Finishing up…"
+ total: "Total"
+ steps:
+ one: "%{number} step"
+ other: "%{number} steps"
+ steps_failed: "%{number} failed"
+ steps_skipped: "%{number} skipped"
cli:
aborted: "Aborted"
@@ -22,3 +29,8 @@ en:
converter:
default_step_title: "Converting %{type}"
max_progress_calculation: "Calculating items took %{duration}"
+ errors:
+ header: "The conversion finished with errors:"
+ step_failed: "%{title} failed: %{error_class}: %{error_message}"
+ step_skipped: "%{title} was skipped because a dependency failed."
+ merge_failed: "Merging a step's shard into the run database failed: %{error_class}: %{error_message}"
diff --git a/migrations/core/lib/migrations/common/fork_manager.rb b/migrations/core/lib/migrations/common/fork_manager.rb
index 168bf27d382..3ea25fd4e13 100644
--- a/migrations/core/lib/migrations/common/fork_manager.rb
+++ b/migrations/core/lib/migrations/common/fork_manager.rb
@@ -1,78 +1,104 @@
# frozen_string_literal: true
module Migrations
+ # The fork hooks for a run. `before_fork` and `after_fork_parent` run around a
+ # fork, so a connection can close before it and reopen after. `after_fork_child`
+ # runs in the new child, e.g. to drop a connection it inherited.
+ #
+ # Steps add and remove hooks from several threads at once, so a mutex guards the
+ # hook lists. A fork copies the child hooks under that mutex, so the child always
+ # runs a consistent list.
module ForkManager
@before_fork_hooks = []
@after_fork_parent_hooks = []
@after_fork_child_hooks = []
@execute_parent_forks = true
+ @mutex = Mutex.new
class << self
def with_batched_forks
@execute_parent_forks = false
- run_before_fork_hooks
-
- yield
-
- run_after_fork_parent_hooks
- @execute_parent_forks = true
+ @mutex.synchronize { run_before_fork_hooks }
+
+ # Always run the after-fork hooks and reset the flag, even if forking
+ # raises (e.g. `Errno::EAGAIN`/`ENOMEM` under fork pressure). Otherwise the
+ # before-fork hooks' effects — a locked writer mutex, a closed run
+ # connection — would never be undone and the run would hang instead of
+ # failing the step.
+ begin
+ yield
+ ensure
+ @mutex.synchronize { run_after_fork_parent_hooks }
+ @execute_parent_forks = true
+ end
end
def before_fork(&block)
- if block
- @before_fork_hooks << block
- block
- end
+ return unless block
+ @mutex.synchronize { @before_fork_hooks << block }
+ block
end
def remove_before_fork(block)
- @before_fork_hooks.delete(block)
+ @mutex.synchronize { @before_fork_hooks.delete(block) }
end
def after_fork_parent(&block)
- if block
- @after_fork_parent_hooks << block
- block
- end
+ return unless block
+ @mutex.synchronize { @after_fork_parent_hooks << block }
+ block
end
def remove_after_fork_parent(block)
- @after_fork_parent_hooks.delete(block)
+ @mutex.synchronize { @after_fork_parent_hooks.delete(block) }
end
def after_fork_child(&block)
- if block
- @after_fork_child_hooks << block
- block
- end
+ return unless block
+ @mutex.synchronize { @after_fork_child_hooks << block }
+ block
end
def remove_after_fork_child(block)
- @after_fork_child_hooks.delete(block)
+ @mutex.synchronize { @after_fork_child_hooks.delete(block) }
end
def fork
- run_before_fork_hooks if @execute_parent_forks
+ # Snapshot the child hooks under the lock for a consistent list, but fork
+ # outside it so the child can't inherit the mutex held.
+ child_hooks =
+ if @execute_parent_forks
+ @mutex.synchronize do
+ run_before_fork_hooks
+ @after_fork_child_hooks.dup
+ end
+ else
+ @mutex.synchronize { @after_fork_child_hooks.dup }
+ end
pid =
Process.fork do
- run_after_fork_child_hooks
+ child_hooks.each(&:call)
yield
end
- run_after_fork_parent_hooks if @execute_parent_forks
+ @mutex.synchronize { run_after_fork_parent_hooks } if @execute_parent_forks
pid
end
def hook_count
- @before_fork_hooks.size + @after_fork_parent_hooks.size + @after_fork_child_hooks.size
+ @mutex.synchronize do
+ @before_fork_hooks.size + @after_fork_parent_hooks.size + @after_fork_child_hooks.size
+ end
end
def clear!
- @before_fork_hooks.clear
- @after_fork_parent_hooks.clear
- @after_fork_child_hooks.clear
+ @mutex.synchronize do
+ @before_fork_hooks.clear
+ @after_fork_parent_hooks.clear
+ @after_fork_child_hooks.clear
+ end
end
private
@@ -84,10 +110,6 @@ module Migrations
def run_after_fork_parent_hooks
@after_fork_parent_hooks.each(&:call)
end
-
- def run_after_fork_child_hooks
- @after_fork_child_hooks.each(&:call)
- end
end
end
end
diff --git a/migrations/core/lib/migrations/common/reporting/plain.rb b/migrations/core/lib/migrations/common/reporting/plain.rb
index b2a1600d043..6e417aa8564 100644
--- a/migrations/core/lib/migrations/common/reporting/plain.rb
+++ b/migrations/core/lib/migrations/common/reporting/plain.rb
@@ -66,6 +66,10 @@ module Migrations
@mutex.synchronize { @steps[id].total = max_progress if @steps[id] }
end
+ def report_concurrency(_id, _count)
+ # No-op: the plain log skips the live fork/CPU count.
+ end
+
def report_progress(id, current, skip_count, warning_count, error_count)
@mutex.synchronize do
step = @steps[id]
@@ -99,6 +103,21 @@ module Migrations
end
end
+ def report_finalizing_begin
+ @mutex.synchronize { @output.puts(I18n.t("progressbar.finishing_up")) }
+ end
+
+ def report_summary(runtime:, total:, failed:, skipped:)
+ tally = [I18n.t("progressbar.steps", count: total, number: total)]
+ tally << I18n.t("progressbar.steps_failed", number: failed) if failed > 0
+ tally << I18n.t("progressbar.steps_skipped", number: skipped) if skipped > 0
+ @mutex.synchronize do
+ @output.puts(
+ "#{I18n.t("progressbar.total")}: #{tally.join(", ")} (#{format_duration(runtime)})",
+ )
+ end
+ end
+
def close
# nothing to free — we write plain lines and keep nothing open
end
diff --git a/migrations/core/lib/migrations/common/reporting/reporter.rb b/migrations/core/lib/migrations/common/reporting/reporter.rb
index 1c6a8804459..fa4da5ac49a 100644
--- a/migrations/core/lib/migrations/common/reporting/reporter.rb
+++ b/migrations/core/lib/migrations/common/reporting/reporter.rb
@@ -26,6 +26,26 @@ module Migrations
StepHandle.new(self, id)
end
+ # Shows a transient "finishing up" status while the block runs, then clears
+ # it. Used for the tail of a run where every step is done but background work
+ # (merging shards) is still finishing, so the display doesn't just sit at
+ # 100%. Returns the block's value.
+ def finalizing
+ report_finalizing_begin
+ yield
+ ensure
+ report_finalizing_end
+ end
+
+ # Prints the end-of-run summary line: the total runtime and how many steps
+ # ran (with any that failed or were skipped).
+ # @param runtime [Float] the run's wall-clock seconds
+ # @param total [Integer] how many steps the run had
+ # @param failed [Integer] how many failed
+ # @param skipped [Integer] how many were skipped because a dependency failed
+ def report_summary(runtime:, total:, failed:, skipped:)
+ end
+
# The run is over and nothing else will be reported. Free anything held
# here. Called once per run, also when the run fails.
def close
@@ -47,6 +67,10 @@ module Migrations
raise NotImplementedError
end
+ def report_concurrency(_id, _count)
+ raise NotImplementedError
+ end
+
def report_progress(_id, _current, _skip_count, _warning_count, _error_count)
raise NotImplementedError
end
@@ -56,6 +80,13 @@ module Migrations
raise NotImplementedError
end
+ # Show and clear the transient "finishing up" status; `finalizing` wraps them.
+ def report_finalizing_begin
+ end
+
+ def report_finalizing_end
+ end
+
# A handle for one step. It holds the step's id, so everything it reports
# goes to the right step, no matter which thread calls it.
class StepHandle
@@ -68,11 +99,19 @@ module Migrations
@reporter.report_notice(@id, message)
end
+ def report_concurrency(count)
+ @reporter.report_concurrency(@id, count)
+ end
+
+ def begin_progress(max_progress:)
+ @reporter.report_progress_begin(@id, max_progress)
+ Progress.new(@reporter, @id)
+ end
+
# Yields a {Progress}. You may call its `update` from any thread. Pass
# `max_progress: nil` when the total is not known.
def with_progress(max_progress:)
- @reporter.report_progress_begin(@id, max_progress)
- yield Progress.new(@reporter, @id)
+ yield begin_progress(max_progress:)
end
# Ends the step. It runs from an `ensure`, so `$!` (the exception in
diff --git a/migrations/core/lib/migrations/common/reporting/tui.rb b/migrations/core/lib/migrations/common/reporting/tui.rb
index 218db299059..61394fb2b1a 100644
--- a/migrations/core/lib/migrations/common/reporting/tui.rb
+++ b/migrations/core/lib/migrations/common/reporting/tui.rb
@@ -8,10 +8,10 @@ module Migrations
# terminal history.
#
# The code that reports progress never draws anything itself. This class is a
- # thread-safe front end: each call only adds an event to a queue. A separate
- # thread reads the queue, updates the model, and redraws at a fixed rate. That
- # is what lets one thread report progress while another one started the step,
- # without the output getting mixed up.
+ # thread-safe front end: callers only hand off events. A separate thread
+ # applies them, updates the model, and redraws at a fixed rate. That lets one
+ # thread report progress while another started the step, without the output
+ # getting mixed up.
#
# The drawing itself is in {Renderer}. It runs on one thread only and has no
# side effects apart from its output and clock (both can be injected), so it
@@ -20,6 +20,8 @@ module Migrations
def initialize(fps: 10, output: $stdout, titles: [])
super()
@queue = Thread::Queue.new
+ @progress = {}
+ @progress_mutex = Mutex.new
@renderer = Renderer.new(output:, titles:)
@frame = 1.0 / fps
@closed = false
@@ -43,14 +45,34 @@ module Migrations
enqueue([:progress_begin, id, max_progress])
end
+ def report_concurrency(id, count)
+ enqueue([:concurrency, id, count])
+ end
+
def report_progress(id, current, skip_count, warning_count, error_count)
- enqueue([:progress, id, current, skip_count, warning_count, error_count])
+ # Progress is coalesced, not queued: one event per row across many steps
+ # would keep the queue full and starve the repaint. Keep the latest only.
+ @progress_mutex.synchronize do
+ @progress[id] = [current, skip_count, warning_count, error_count]
+ end
end
def report_finish(id, outcome)
enqueue([:finish, id, outcome])
end
+ def report_finalizing_begin
+ enqueue([:finalizing_begin])
+ end
+
+ def report_finalizing_end
+ enqueue([:finalizing_end])
+ end
+
+ def report_summary(runtime:, total:, failed:, skipped:)
+ enqueue([:summary, runtime, total, failed, skipped])
+ end
+
def close
return if @closed
@closed = true
@@ -94,12 +116,23 @@ module Migrations
loop do
frame_started_at = monotonic
break if drain_until_close
+ apply_coalesced_progress
@renderer.consume_resize
@renderer.repaint
sleep_rest_of_frame(frame_started_at)
end
end
+ def apply_coalesced_progress
+ snapshot = @progress_mutex.synchronize { @progress.dup }
+ snapshot.each { |id, values| @renderer.apply([:progress, id, *values]) }
+ end
+
+ def flush_progress(id)
+ values = @progress_mutex.synchronize { @progress.delete(id) }
+ @renderer.apply([:progress, id, *values]) if values
+ end
+
# Sleeps for whatever is left of this frame's time, so repaints happen at a
# steady rate instead of as fast as the loop can spin.
def sleep_rest_of_frame(frame_started_at)
@@ -108,9 +141,13 @@ module Migrations
end
def drain_until_close
- until @queue.empty?
+ # Drain only what's already queued; events arriving while we drain wait
+ # for the next frame, so a burst can't keep the loop here and starve the
+ # repaint.
+ @queue.size.times do
event = @queue.pop(true)
return true if event[0] == :close
+ flush_progress(event[1]) if event[0] == :finish
@renderer.apply(event)
end
false
diff --git a/migrations/core/lib/migrations/common/reporting/tui/ansi.rb b/migrations/core/lib/migrations/common/reporting/tui/ansi.rb
index 84c2f1a7b3f..b2f2cde834d 100644
--- a/migrations/core/lib/migrations/common/reporting/tui/ansi.rb
+++ b/migrations/core/lib/migrations/common/reporting/tui/ansi.rb
@@ -15,6 +15,7 @@ module Migrations
YELLOW = "\e[33m"
RED = "\e[31m"
CYAN = "\e[36m"
+ MAGENTA = "\e[35m"
DIM = "\e[90m"
ERASE_LINE = "\e[2K" # erase the whole line (also clears the terminal's wrap flag)
ERASE_BELOW = "\e[0J" # erase from the cursor to the end of the screen
diff --git a/migrations/core/lib/migrations/common/reporting/tui/renderer.rb b/migrations/core/lib/migrations/common/reporting/tui/renderer.rb
index 6bc1516d59b..e7f4f562bae 100644
--- a/migrations/core/lib/migrations/common/reporting/tui/renderer.rb
+++ b/migrations/core/lib/migrations/common/reporting/tui/renderer.rb
@@ -26,8 +26,9 @@ module Migrations
# Columns of a live row. ETA is left-aligned (it has its own "ETA " label);
# the rest are right-aligned. A bold percent takes the place of a progress
# bar: it lines up in columns, works for several steps at once, and avoids
- # block characters that don't look the same on every terminal.
- COLUMNS = %i[percent count elapsed eta rate].freeze
+ # block characters that don't look the same on every terminal. Concurrency
+ # sits last so it can't shift the columns the collapsed (finished) rows share.
+ COLUMNS = %i[percent count elapsed eta rate concurrency].freeze
LEFT_ALIGNED = %i[eta].freeze
Step =
@@ -44,6 +45,7 @@ module Migrations
:rate,
:rate_sampled_at,
:rate_sampled_current,
+ :concurrency,
keyword_init: true,
)
@@ -61,6 +63,7 @@ module Migrations
@live_count = 0
@last_live = nil
@resize_pending = false
+ @finalizing = false
@frame_buffer = +"" # reused on each repaint instead of making a new one
# Reserve the column widths up front: a finished row scrolls into the
# history and can't be realigned, so every column has to start at the
@@ -76,6 +79,7 @@ module Migrations
elapsed: TIME_WIDTH,
eta: 0,
rate: 0,
+ concurrency: 0,
}
end
@@ -149,7 +153,12 @@ module Migrations
rate: nil,
rate_sampled_at: nil,
rate_sampled_current: 0,
+ concurrency: 1,
)
+ when :concurrency
+ _, id, count = event
+ step = @steps[id]
+ step.concurrency = count if step
when :progress_begin
_, id, max_progress = event
step = @steps[id]
@@ -178,6 +187,13 @@ module Migrations
step.finished_at = now
step.state = outcome # :done, :interrupted, or :failed
@pending_permanent << collapsed_line(step)
+ when :finalizing_begin
+ @finalizing = true
+ when :finalizing_end
+ @finalizing = false
+ when :summary
+ _, runtime, total, failed, skipped = event
+ @pending_permanent << summary_line(runtime, total, failed, skipped)
end
end
@@ -193,6 +209,7 @@ module Migrations
permanent = @pending_permanent
@pending_permanent = []
live = final ? [] : format_live(running_steps)
+ live += [finalizing_line] if @finalizing && !final
return if permanent.empty? && live.empty? && @live_count == 0
return if !final && permanent.empty? && live == @last_live
@@ -293,6 +310,7 @@ module Migrations
elapsed: "",
eta: "",
rate: "",
+ concurrency: concurrency_cell(step.concurrency),
annot: "",
}
@@ -368,6 +386,36 @@ module Migrations
line
end
+ # The transient live line shown while the run finishes background work
+ # (merging shards) after every step is done. The spinner animates it.
+ def finalizing_line
+ "#{spinner} #{Ansi::DIM}#{I18n.t("progressbar.finishing_up")}#{Ansi::RESET}"
+ end
+
+ # The permanent end-of-run line: a Σ, the total runtime in the elapsed
+ # column (so it lines up under the per-step durations), and the step tally.
+ def summary_line(runtime, total, failed, skipped)
+ columns = [
+ Ansi.pad("", @column_widths[:percent]),
+ Ansi.pad("", @column_widths[:count]),
+ Ansi.pad(format_duration(runtime), @column_widths[:elapsed], :right),
+ ].join(" ")
+ line = +"#{Ansi::BOLD}Σ#{Ansi::RESET} #{title_field(I18n.t("progressbar.total"))} "
+ line << columns << summary_tally(total, failed, skipped)
+ line
+ end
+
+ def summary_tally(total, failed, skipped)
+ note = +" #{I18n.t("progressbar.steps", count: total, number: total)}"
+ if failed > 0
+ note << " #{Ansi::RED}#{I18n.t("progressbar.steps_failed", number: failed)}#{Ansi::RESET}"
+ end
+ if skipped > 0
+ note << " #{Ansi::YELLOW}#{I18n.t("progressbar.steps_skipped", number: skipped)}#{Ansi::RESET}"
+ end
+ note
+ end
+
def status_glyph(step)
case step.state
when :done
@@ -458,6 +506,14 @@ module Migrations
SPINNER[(now * SPINNER_RATE).to_i % SPINNER.size]
end
+ def concurrency_cell(count)
+ return "" if count <= 1
+
+ label = "#{count}×"
+ return label unless @color
+ "#{Ansi::MAGENTA}#{label}#{Ansi::RESET}"
+ end
+
def terminal_columns
@forced_width || @output.winsize[1]
rescue StandardError
diff --git a/migrations/core/lib/migrations/common/step_dependencies.rb b/migrations/core/lib/migrations/common/step_dependencies.rb
index 88dee1874fc..0233ef53d65 100644
--- a/migrations/core/lib/migrations/common/step_dependencies.rb
+++ b/migrations/core/lib/migrations/common/step_dependencies.rb
@@ -3,7 +3,7 @@
module Migrations
# Class-level dependency metadata for migration steps. Meant to be `extend`ed
# by a step base class (`Migrations::Importer::Step`,
- # `Migrations::Conversion::StepBase`); subclasses inherit the macros, and the
+ # `Migrations::Conversion::Step`); subclasses inherit the macros, and the
# resulting `dependencies` and `priority` are queryable without instantiating
# a step. `Migrations::TopologicalSorter` consumes both to order steps.
#
diff --git a/migrations/core/lib/migrations/conversion/base.rb b/migrations/core/lib/migrations/conversion/base.rb
index b1bcbdb3bd2..d489083660d 100644
--- a/migrations/core/lib/migrations/conversion/base.rb
+++ b/migrations/core/lib/migrations/conversion/base.rb
@@ -9,7 +9,7 @@ module Migrations
@settings = settings
end
- def run(only_steps: [], skip_steps: [])
+ def run(only_steps: [], skip_steps: [], max_parallel_steps: nil, no_fork: false)
if respond_to?(:setup)
puts "Initializing..."
setup
@@ -17,26 +17,29 @@ module Migrations
create_database
- @worker_pool = WorkerPool.new
-
# Titles are known from the step classes (no instantiation, no queries),
# so the reporter can reserve its title column up front. The steps
- # themselves are built and sized lazily in the loop — each step's count
- # is computed only when it runs.
+ # themselves are built and sized lazily. Each step's source is opened
+ # and counted in its own fork when it runs.
step_classes = filter_steps(steps, only_steps, skip_steps)
@reporter = Reporting::Factory.build(titles: step_classes.map(&:title))
- step_classes.each do |step_class|
- step = create_step(step_class)
- before_step_execution(step)
- execute_step(step)
- after_step_execution(step)
- end
+ StepScheduler.new(
+ step_classes:,
+ reporter: @reporter,
+ step_factory: method(:create_step),
+ shard_manager: @shard_manager,
+ writer: @writer,
+ budget: WorkerBudget.available, # usable CPUs minus one for the parent + merges
+ max_parallel_steps:,
+ no_fork:,
+ ).run
rescue SignalException
@aborted = true
exit(130)
ensure
Database::IntermediateDB.close
+ @shard_manager&.cleanup
# Restore the terminal (and flush the final frame) before printing the
# run-level abort line, so it lands cleanly below the reporter output
# instead of fighting the live region.
@@ -45,7 +48,7 @@ module Migrations
end
def steps
- step_class = StepBase
+ step_class = Step
current_module = self.class.name.deconstantize.constantize
classes =
@@ -61,25 +64,6 @@ module Migrations
TopologicalSorter.sort(classes)
end
- def before_step_execution(step)
- # do nothing
- end
-
- def execute_step(step)
- executor =
- if step.is_a?(ProgressStep)
- ProgressStepExecutor.new(step, pool: @worker_pool, reporter: @reporter)
- else
- StepExecutor.new(step, reporter: @reporter)
- end
-
- executor.execute
- end
-
- def after_step_execution(step)
- # do nothing
- end
-
def step_args(step_class)
{}
end
@@ -90,8 +74,13 @@ module Migrations
db_path = File.expand_path(settings[:intermediate_db][:path], Migrations.root_path)
Database.migrate(db_path, migrations_path: Database::INTERMEDIATE_DB_SCHEMA_PATH)
- db = Database.connect(db_path)
- Database::IntermediateDB.setup(db)
+ @shard_manager =
+ ShardManager.new(
+ canonical_path: db_path,
+ migrations_path: Database::INTERMEDIATE_DB_SCHEMA_PATH,
+ )
+ @writer = Database::Connection.new(path: db_path)
+ Database::IntermediateDB.setup(@writer)
end
def create_step(step_class)
diff --git a/migrations/core/lib/migrations/conversion/consolidator.rb b/migrations/core/lib/migrations/conversion/consolidator.rb
new file mode 100644
index 00000000000..d0a82cf9bcd
--- /dev/null
+++ b/migrations/core/lib/migrations/conversion/consolidator.rb
@@ -0,0 +1,70 @@
+# frozen_string_literal: true
+
+module Migrations
+ module Conversion
+ # Merges finished steps' shards into the run database on a background thread,
+ # off the steps' critical path. A coordinator hands its shards over the
+ # moment its workers exit and finishes the step right away, so the step no
+ # longer lingers at 100% while its (possibly large) merge runs, and the merge
+ # overlaps the steps that come after it.
+ #
+ # It writes straight to the run DB `Connection` — the only thing that touches
+ # it, since steps write to their own shards. That connection closes and
+ # reopens around every worker fork (Extralite isn't thread-safe, and a child
+ # must not inherit an open handle), so a merge takes `fork_mutex` to serialize
+ # against forking: a merge can't run while the connection is closed for a fork,
+ # and a fork waits for an in-flight merge.
+ class Consolidator
+ # @param shard_manager [ShardManager] used to discard each shard once merged
+ # @param connection [Database::Connection] the run DB connection merges write to
+ # @param fork_mutex [Mutex] serializes merges against worker forks (see above)
+ def initialize(shard_manager, connection, fork_mutex)
+ @shard_manager = shard_manager
+ @connection = connection
+ @fork_mutex = fork_mutex
+ @queue = Thread::Queue.new
+ @errors = []
+ @thread = Thread.new { run }
+ @thread.name = "consolidator"
+ end
+
+ # Hands a finished step's shards over to be merged in the background.
+ # @param shards [Array<String>] the shard database paths to merge, then discard
+ # @return [void]
+ def enqueue(shards)
+ @queue << shards
+ end
+
+ # Waits for every enqueued shard to be merged, then returns the merge errors.
+ # @return [Array<StandardError>] the errors from any merge that failed (empty
+ # when all merged cleanly)
+ def drain
+ @queue.close
+ @thread.join
+ @errors
+ end
+
+ private
+
+ def run
+ while (shards = @queue.pop)
+ shards.each { |shard_path| merge(shard_path) }
+ end
+ end
+
+ def merge(shard_path)
+ @fork_mutex.synchronize { @connection.merge_database(shard_path, tables: mergeable_tables) }
+ rescue StandardError => e
+ @errors << e
+ ensure
+ @shard_manager.discard(shard_path)
+ end
+
+ # `config` and `schema_migrations` are run-level and identical in every
+ # shard, so they're left out of the merge.
+ def mergeable_tables
+ @mergeable_tables ||= @connection.tables - %w[config schema_migrations]
+ end
+ end
+ end
+end
diff --git a/migrations/core/lib/migrations/conversion/convert_error.rb b/migrations/core/lib/migrations/conversion/convert_error.rb
new file mode 100644
index 00000000000..f96e669e24a
--- /dev/null
+++ b/migrations/core/lib/migrations/conversion/convert_error.rb
@@ -0,0 +1,60 @@
+# frozen_string_literal: true
+
+module Migrations
+ module Conversion
+ # Raised at the end of a run when one or more steps failed or were skipped, or
+ # a shard merge failed. The converter logs per-item errors and keeps going, so
+ # this is the single signal the CLI turns into a non-zero exit; its message is
+ # the run summary shown to the user (no backtrace).
+ #
+ # The scheduler hands it the raw outcomes; the human-facing summary — and its
+ # translations — are built here, where the message belongs.
+ class ConvertError < StandardError
+ include CLI::PresentableError
+
+ # @param failures [Hash{Class<Step> => StandardError, nil}] failed steps and
+ # the error each raised
+ # @param skipped [Array<Class<Step>>] steps skipped because a dependency failed
+ # @param merge_errors [Array<StandardError>] errors from merging shards
+ def initialize(failures: {}, skipped: [], merge_errors: [])
+ super(summary(failures, skipped, merge_errors))
+ end
+
+ private
+
+ def summary(failures, skipped, merge_errors)
+ lines = [I18n.t("converter.errors.header")]
+
+ failures.each do |step_class, error|
+ lines << " • #{failure_line(step_class, error)}"
+ Array(error&.backtrace).first(5).each { |frame| lines << " #{frame}" }
+ end
+
+ skipped.each do |step_class|
+ lines << " • #{I18n.t("converter.errors.step_skipped", title: step_class.title)}"
+ end
+
+ merge_errors.each { |error| lines << " • #{merge_line(error)}" }
+
+ lines.join("\n")
+ end
+
+ def failure_line(step_class, error)
+ I18n.t(
+ "converter.errors.step_failed",
+ title: step_class.title,
+ error_class: error&.class,
+ error_message: error&.message,
+ )
+ end
+
+ def merge_line(error)
+ I18n.t(
+ "converter.errors.merge_failed",
+ error_class: error.class,
+ error_message: error.message,
+ )
+ end
+ end
+ end
+end
diff --git a/migrations/core/lib/migrations/conversion/inline_progress_sink.rb b/migrations/core/lib/migrations/conversion/inline_progress_sink.rb
new file mode 100644
index 00000000000..2a482bcedba
--- /dev/null
+++ b/migrations/core/lib/migrations/conversion/inline_progress_sink.rb
@@ -0,0 +1,21 @@
+# frozen_string_literal: true
+
+module Migrations
+ module Conversion
+ # The no-fork counterpart to {PipeProgressSink}: drives the step's progress
+ # handle directly, in-process, instead of writing down a pipe.
+ class InlineProgressSink
+ def initialize(step_handle)
+ @step_handle = step_handle
+ end
+
+ def report_max_progress(value)
+ @progress = @step_handle.begin_progress(max_progress: value)
+ end
+
+ def report_progress(progress:, warnings:, errors:)
+ @progress.update(increment_by: progress, warning_count: warnings, error_count: errors)
+ end
+ end
+ end
+end
diff --git a/migrations/core/lib/migrations/conversion/parallel_job.rb b/migrations/core/lib/migrations/conversion/parallel_job.rb
deleted file mode 100644
index d7392613e2d..00000000000
--- a/migrations/core/lib/migrations/conversion/parallel_job.rb
+++ /dev/null
@@ -1,36 +0,0 @@
-# frozen_string_literal: true
-
-module Migrations
- module Conversion
- class ParallelJob
- def initialize(processor)
- @processor = processor
- @tracker = processor.tracker
-
- @offline_connection = Database::OfflineConnection.new
- end
-
- # Runs in the worker process, before the first item is processed.
- def setup
- Database::IntermediateDB.setup(@offline_connection)
- SetupGuard.run(@processor)
- end
-
- def run(item)
- @tracker.reset_stats!
- @offline_connection.clear!
-
- begin
- @processor.process(item)
- rescue StandardError => e
- @tracker.log_error("Failed to process item", exception: e, details: item)
- end
-
- [@offline_connection.parametrized_insert_statements, @tracker.stats]
- end
-
- def cleanup
- end
- end
- end
-end
diff --git a/migrations/core/lib/migrations/conversion/partition_source.rb b/migrations/core/lib/migrations/conversion/partition_source.rb
new file mode 100644
index 00000000000..e4630f5e8bd
--- /dev/null
+++ b/migrations/core/lib/migrations/conversion/partition_source.rb
@@ -0,0 +1,37 @@
+# frozen_string_literal: true
+
+module Migrations
+ module Conversion
+ # The interface a source DB adapter implements so a step can `partition_by`.
+ # An adapter includes this module to declare it, and must provide:
+ #
+ # partition_bounds(key, from, base) -> [min, max]
+ # estimated_row_count(from) -> Integer (planner row estimate)
+ # count_all(from, where:) -> Integer (exact row count)
+ # chunk_filter(key, lower, upper, base:) -> the SQL WHERE body for one chunk
+ #
+ # plus one way to read the sorted key — either the fast, in-SQL
+ #
+ # boundaries_by_scan(key, from, base, count) -> the chunk lower bounds
+ #
+ # or the streaming fallback the {Partitioner} samples itself:
+ #
+ # each_partition_key(key, from, base) -> yields each key value, in order
+ #
+ # The Partitioner checks a source against this in the parent before forking, so
+ # a source that can't partition fails fast with a clear error instead of
+ # crashing a worker with a bare NoMethodError.
+ module PartitionSource
+ REQUIRED = %i[partition_bounds estimated_row_count count_all chunk_filter].freeze
+ SCAN = %i[boundaries_by_scan each_partition_key].freeze # at least one
+
+ # The interface methods `source` doesn't provide (empty when it satisfies the
+ # contract).
+ def self.missing_from(source)
+ missing = REQUIRED.reject { |method| source.respond_to?(method) }
+ missing << SCAN.join(" or ") if SCAN.none? { |method| source.respond_to?(method) }
+ missing
+ end
+ end
+ end
+end
diff --git a/migrations/core/lib/migrations/conversion/partitioner.rb b/migrations/core/lib/migrations/conversion/partitioner.rb
new file mode 100644
index 00000000000..fb83e656e7b
--- /dev/null
+++ b/migrations/core/lib/migrations/conversion/partitioner.rb
@@ -0,0 +1,115 @@
+# frozen_string_literal: true
+
+module Migrations
+ module Conversion
+ # Works out how to split a partitioned step's source into `count` chunks. All
+ # the logic here is dialect-free; the dialect-specific SQL lives in the source
+ # DB adapter, which this calls through a few primitives:
+ #
+ # partition_bounds(key, from, base) -> [min, max]
+ # estimated_row_count(from) -> Integer (the planner's estimate)
+ # count_all(from, where:) -> Integer (exact)
+ # each_partition_key(key, from, base) -> yields each key value, sorted
+ #
+ # A dense numeric key strides evenly by value, which is cheap. A sparse one
+ # (big gaps from deletes, or ids from a shared sequence), or any non-numeric or
+ # composite key, is sampled from the sorted key instead, so the chunks hold
+ # even row counts whatever the distribution.
+ #
+ # The sampling streams the whole key column here and picks every Nth value. An
+ # adapter that can do better in SQL (e.g. Postgres with `NTILE`) may add an
+ # optional primitive and skip the streaming:
+ #
+ # boundaries_by_scan(key, from, base, count) -> the chunk lower bounds
+ #
+ # When it's there we use it; otherwise we fall back to streaming, so a new
+ # adapter gets a working partitioner from the four primitives alone.
+ class Partitioner
+ # Raised when a step declares `partition_by` but its source can't compute
+ # boundaries — the adapter is missing partition primitives (or there is no
+ # source DB at all, e.g. a file source).
+ class UnsupportedSourceError < StandardError
+ end
+
+ # How much wider a numeric key's value range may run than its row count
+ # before even value-sized chunks stop holding even row counts. Past this, the
+ # gaps leave some chunks nearly empty, so we sample the sorted key instead.
+ DENSE_RANGE_FACTOR = 4
+
+ # @param source_db [Object] the source DB adapter providing the primitives above
+ # @param key [Symbol, String, Array<Symbol>] the partition key: one column, or
+ # an array of columns for a composite key
+ # @param from [String] the table (or other source name) to split
+ # @param base [String, nil] a SQL filter that always applies, or nil for none
+ def initialize(source_db, key:, from:, base:)
+ @source_db = source_db
+ @key = key
+ @from = from
+ @base = base
+ end
+
+ # @param count [Integer] how many chunks to split the source into
+ # @return [Array] the `count` chunk lower bounds; chunk i covers
+ # `[bounds[i], bounds[i + 1])`, the last open-ended. Empty for an empty source.
+ # @raise [UnsupportedSourceError] if the source can't compute boundaries
+ def boundaries(count)
+ ensure_source_can_partition!
+ return scan_boundaries(count) if @key.is_a?(Array)
+
+ min, max = @source_db.partition_bounds(@key, @from, @base)
+ return [] if min.nil?
+ return numeric_boundaries(min, max, count) if min.is_a?(Numeric) && dense?(min, max)
+
+ scan_boundaries(count)
+ end
+
+ private
+
+ def ensure_source_can_partition!
+ missing = PartitionSource.missing_from(@source_db)
+ return if missing.empty?
+
+ raise UnsupportedSourceError,
+ "Can't partition `#{@from}`: its source doesn't implement the " \
+ "PartitionSource interface (missing #{missing.join(", ")}). " \
+ "Implement it, or remove `partition_by` from the step."
+ end
+
+ def dense?(min, max)
+ rows = @source_db.estimated_row_count(@from)
+ # No usable estimate (an adapter returns 0 or less when it can't estimate
+ # the row count): don't pay for a full key scan on a guess — default to the
+ # cheap value stride.
+ return true if rows <= 0
+ (max - min + 1) <= rows * DENSE_RANGE_FACTOR
+ end
+
+ def numeric_boundaries(min, max, count)
+ chunk_size = ((max - min + 1).to_f / count).ceil
+ Array.new(count) { |i| min + i * chunk_size }.uniq
+ end
+
+ def scan_boundaries(count)
+ if @source_db.respond_to?(:boundaries_by_scan)
+ @source_db.boundaries_by_scan(@key, @from, @base, count)
+ else
+ sampled_boundaries(count)
+ end
+ end
+
+ def sampled_boundaries(count)
+ total = @source_db.count_all(@from, where: @base)
+ return [] if total.zero?
+
+ sample_every = [(total.to_f / count).ceil, 1].max
+ boundaries = []
+ index = 0
+ @source_db.each_partition_key(@key, @from, @base) do |value|
+ boundaries << value if (index % sample_every).zero? && boundaries.size < count
+ index += 1
+ end
+ boundaries
+ end
+ end
+ end
+end
diff --git a/migrations/core/lib/migrations/conversion/pipe_progress_sink.rb b/migrations/core/lib/migrations/conversion/pipe_progress_sink.rb
new file mode 100644
index 00000000000..60de7671fd4
--- /dev/null
+++ b/migrations/core/lib/migrations/conversion/pipe_progress_sink.rb
@@ -0,0 +1,22 @@
+# frozen_string_literal: true
+
+module Migrations
+ module Conversion
+ # The worker's end of the progress channel. Writes one line per call down the
+ # pipe to the coordinator, which parses them (see
+ # {StepCoordinator#consume_progress}).
+ class PipeProgressSink
+ def initialize(io)
+ @io = io
+ end
+
+ def report_max_progress(value)
+ @io.write("m #{value}\n")
+ end
+
+ def report_progress(progress:, warnings:, errors:)
+ @io.write("p #{progress} #{warnings} #{errors}\n")
+ end
+ end
+ end
+end
diff --git a/migrations/core/lib/migrations/conversion/progress_step.rb b/migrations/core/lib/migrations/conversion/progress_step.rb
deleted file mode 100644
index 2224c023123..00000000000
--- a/migrations/core/lib/migrations/conversion/progress_step.rb
+++ /dev/null
@@ -1,184 +0,0 @@
-# frozen_string_literal: true
-
-module Migrations
- module Conversion
- # A step that processes many items and reports progress. It is split into
- # two roles with a hard boundary between them:
- #
- # * `source` enumerates the items (`items`, `max_progress`). It always runs
- # in the main process.
- # * `processor` handles one item at a time (`process`). It runs in a worker
- # (a forked process when the step runs in parallel), one instance per
- # worker. Per-worker state is built in its `setup` hook which runs after
- # the worker has started, never in the constructor. `setup` must not
- # create IntermediateDB records — only `process` writes; `setup` runs
- # under `SetupGuard`, so a write raises `SetupGuard::SetupError` in
- # serial and parallel mode alike.
- #
- # The roles are separate objects, so state can't leak across the
- # process boundary: a processor method that tries to read source-side
- # state fails with a `NoMethodError` instead of silently seeing stale or
- # missing data. `helpers` is mixed into both roles and is meant for pure
- # functions only (arguments in, value out).
- #
- # class Users < Conversion::ProgressStep
- # source do
- # attr_accessor :source_db
- #
- # def items
- # @source_db.query("SELECT ...")
- # end
- # end
- #
- # processor do
- # def setup
- # @upload_creator = UploadCreator.new
- # end
- #
- # def process(item)
- # IntermediateDB::User.create(...)
- # end
- # end
- # end
- #
- # The blocks are evaluated with `class_eval`, so use plain `def` inside
- # them. Avoid `define_method`: its block closes over the surrounding scope
- # and would smuggle step-level state across the role boundary.
- class ProgressStep < StepBase
- class Source
- include AttributeAssignment
-
- # `source_db` is the step's connection to the source data, when it has
- # one. Steps that read from somewhere else (a file, fixed data) leave it
- # nil. The converter injects it through `step_args`.
- attr_accessor :settings, :source_db
-
- class << self
- # Read a whole table: defines `items` (`SELECT * FROM <name>`) and
- # `max_progress` (its row count), both filtered by `where`. Override
- # `items`/`max_progress` for anything custom.
- def reads_table(name, where: nil)
- @table = { name:, where: }
- end
-
- attr_reader :table
-
- def reads_table?
- !table.nil?
- end
- end
-
- def initialize(args = {})
- assign_attributes(args)
- end
-
- def max_progress
- return unless self.class.reads_table?
- @source_db.count_all(self.class.table[:name], where: self.class.table[:where])
- end
-
- def items
- raise NotImplementedError unless self.class.reads_table?
- @source_db.select_all(self.class.table[:name], where: self.class.table[:where])
- end
- end
-
- class Processor
- include AttributeAssignment
-
- attr_accessor :settings
- attr_reader :tracker
-
- def initialize(args = {})
- @tracker = StepTracker.new
- assign_attributes(args)
- end
-
- def setup
- # do nothing
- end
-
- def process(item)
- raise NotImplementedError
- end
- end
-
- # The step object is only a coordinator: it builds the source (in the
- # main process) and one processor per worker. All per-item state —
- # including the trackers — lives on the roles, and `settings` are routed
- # to them; that's why this class is a sibling of `Step` rather than a
- # subclass — it has no `settings`, `tracker` or `execute` of its own.
- attr_reader :source
-
- def initialize(args = {})
- @args = args
- @source = self.class.source_class.new(args)
- end
-
- def create_processor
- self.class.processor_class.new(@args)
- end
-
- class << self
- def source(&block)
- raise ArgumentError, "`source` is already defined" if @source_block
- @source_block = block
- end
-
- def processor(&block)
- raise ArgumentError, "`processor` is already defined" if @processor_block
- @processor_block = block
- end
-
- def helpers(&block)
- raise ArgumentError, "`helpers` is already defined" if @helpers_block
- @helpers_block = block
- end
-
- def source_class
- @source_class ||=
- build_role_class(superclass_role(:source_class, Source), :Source, @source_block)
- end
-
- def processor_class
- @processor_class ||=
- build_role_class(
- superclass_role(:processor_class, Processor),
- :Processor,
- @processor_block,
- )
- end
-
- def helpers_module
- @helpers_module ||=
- Module.new.tap { |mod| mod.module_eval(&@helpers_block) if @helpers_block }
- end
-
- def run_in_parallel(value)
- @run_in_parallel = !!value
- end
-
- def run_in_parallel?
- @run_in_parallel == true
- end
-
- private
-
- # Steps usually inherit from `ProgressStep` directly and their roles
- # from the `Source`/`Processor` base classes. If a step subclasses
- # another step, its roles subclass the parent step's roles instead.
- def superclass_role(role_method, base_class)
- superclass < ProgressStep ? superclass.public_send(role_method) : base_class
- end
-
- def build_role_class(base_class, name, block)
- klass = Class.new(base_class)
- const_set(name, klass)
- klass.include(helpers_module)
- klass.class_eval(&block) if block
- klass
- end
- end
- end
- end
-end
diff --git a/migrations/core/lib/migrations/conversion/progress_step_executor.rb b/migrations/core/lib/migrations/conversion/progress_step_executor.rb
deleted file mode 100644
index 3c91ed565d8..00000000000
--- a/migrations/core/lib/migrations/conversion/progress_step_executor.rb
+++ /dev/null
@@ -1,120 +0,0 @@
-# frozen_string_literal: true
-
-module Migrations
- module Conversion
- class ProgressStepExecutor
- PRINT_RUNTIME_AFTER_SECONDS = 5
-
- def initialize(step, pool:, reporter:)
- @step = step
- @source = step.source
- @pool = pool
- @reporter = reporter
- end
-
- def execute
- @step_report = @reporter.start_step(@step.class.title)
- @max_progress = calculate_max_progress
-
- if execute_in_parallel?
- execute_parallel
- else
- execute_serially
- end
- ensure
- @step_report.finish
- end
-
- private
-
- # Parallelism only pays off above a minimum number of items, and the
- # queues need bounds for backpressure. Both are executor policy, scaled
- # by the size of the injected pool.
- def min_parallel_items
- @pool.size * 10
- end
-
- def max_queue_size
- @pool.size * 100
- end
-
- def execute_in_parallel?
- @step.class.run_in_parallel? && (@max_progress.nil? || @max_progress > min_parallel_items)
- end
-
- def execute_serially
- job = SerialJob.new(@step.create_processor)
- job.setup
-
- @step_report.with_progress(max_progress: @max_progress) do |progress|
- @source.items.each do |item|
- stats = job.run(item)
- progress.update(
- increment_by: stats.progress,
- warning_count: stats.warning_count,
- error_count: stats.error_count,
- )
- end
- end
- end
-
- def execute_parallel
- worker_output_queue = SizedQueue.new(max_queue_size)
- work_queue = SizedQueue.new(max_queue_size)
-
- batch =
- @pool.start(work_queue:, output_queue: worker_output_queue) do
- ParallelJob.new(@step.create_processor)
- end
- writer_thread = start_db_writer(worker_output_queue)
- push_work(work_queue)
-
- batch.wait
- worker_output_queue.close
- writer_thread.join
- end
-
- def calculate_max_progress
- start_time = Time.now
- max_progress = @source.max_progress
- duration = Time.now - start_time
-
- if duration > PRINT_RUNTIME_AFTER_SECONDS
- @step_report.notice(
- I18n.t(
- "converter.max_progress_calculation",
- duration: DateHelper.human_readable_time(duration),
- ),
- )
- end
-
- max_progress
- end
-
- def start_db_writer(worker_output_queue)
- Thread.new do
- Thread.current.name = "writer_thread"
-
- @step_report.with_progress(max_progress: @max_progress) do |progress|
- while (parametrized_insert_statements, stats = worker_output_queue.pop)
- parametrized_insert_statements.each do |sql, parameters|
- Database::IntermediateDB.insert(sql, *parameters)
- end
-
- progress.update(
- increment_by: stats.progress,
- warning_count: stats.warning_count,
- error_count: stats.error_count,
- )
- end
- end
- end
- end
-
- def push_work(work_queue)
- @source.items.each { |item| work_queue.push(item) }
- work_queue.close
- end
- end
- end
-end
diff --git a/migrations/core/lib/migrations/conversion/serial_job.rb b/migrations/core/lib/migrations/conversion/serial_job.rb
deleted file mode 100644
index 957558291f4..00000000000
--- a/migrations/core/lib/migrations/conversion/serial_job.rb
+++ /dev/null
@@ -1,31 +0,0 @@
-# frozen_string_literal: true
-
-module Migrations
- module Conversion
- class SerialJob
- def initialize(processor)
- @processor = processor
- @tracker = processor.tracker
- end
-
- def setup
- SetupGuard.run(@processor)
- end
-
- def run(item)
- @tracker.reset_stats!
-
- begin
- @processor.process(item)
- rescue StandardError => e
- @tracker.log_error("Failed to process item", exception: e, details: item)
- end
-
- @tracker.stats
- end
-
- def cleanup
- end
- end
- end
-end
diff --git a/migrations/core/lib/migrations/conversion/setup_guard.rb b/migrations/core/lib/migrations/conversion/setup_guard.rb
index f40b11552ae..0e9e4165479 100644
--- a/migrations/core/lib/migrations/conversion/setup_guard.rb
+++ b/migrations/core/lib/migrations/conversion/setup_guard.rb
@@ -2,13 +2,9 @@
module Migrations
module Conversion
- # Wraps a processor's `setup` so it can't create IntermediateDB records:
- # for the duration of the call the IntermediateDB connection is replaced
- # by one that raises on insert. Without the guard such writes would be
- # silently discarded in parallel mode (each worker's `OfflineConnection`
- # is cleared before the first item) but persisted in serial mode — the
- # guard makes both modes fail loudly and identically, with a backtrace
- # pointing at the offending write.
+ # Wraps a processor's `setup` so a write to the IntermediateDB raises instead
+ # of slipping through: `setup` builds per-worker state, not rows, so a stray
+ # write there would land outside the per-item error handling.
module SetupGuard
class SetupError < StandardError
end
diff --git a/migrations/core/lib/migrations/conversion/shard_manager.rb b/migrations/core/lib/migrations/conversion/shard_manager.rb
new file mode 100644
index 00000000000..785390de076
--- /dev/null
+++ b/migrations/core/lib/migrations/conversion/shard_manager.rb
@@ -0,0 +1,57 @@
+# frozen_string_literal: true
+
+require "fileutils"
+
+module Migrations
+ module Conversion
+ # Hands out per-worker shard databases for the sharded write path. A shard is
+ # a private copy of an empty database with the run's schema, so a worker opens
+ # it and runs the exact same inserts it would against the run database, just
+ # without touching the single run-level connection that is the throughput
+ # ceiling. Once a step's workers are done the coordinator merges their shards
+ # into the run database (see {Consolidator}) and discards
+ # them.
+ #
+ # The template is a fresh, empty database migrated from the same schema as the
+ # run DB — not a copy of the run DB. That keeps a shard cheap to copy, and —
+ # importantly — means a run against an existing IntermediateDB doesn't drag the
+ # existing rows into every shard; the merge just adds each shard's new rows
+ # with `INSERT OR IGNORE`.
+ class ShardManager
+ # @param canonical_path [String] the run's IntermediateDB; the shards and
+ # their template are placed next to it
+ # @param migrations_path [String] the schema migrations, used to build the
+ # empty template
+ def initialize(canonical_path:, migrations_path:)
+ @dir = File.join(File.dirname(canonical_path), "shards")
+ FileUtils.mkdir_p(@dir)
+
+ @template_path = File.join(@dir, "template.db")
+ Database.delete_database(@template_path)
+ Database.migrate(@template_path, migrations_path:)
+
+ @counter = 0
+ @mutex = Mutex.new
+ end
+
+ # @return [String] the path of a fresh shard, copied from the template
+ def create_shard
+ index = @mutex.synchronize { @counter += 1 }
+ path = File.join(@dir, "shard-#{index}.db")
+ FileUtils.cp(@template_path, path)
+ path
+ end
+
+ # Removes a shard and its WAL/SHM sidecar files.
+ # @param path [String] the shard database path returned by {#create_shard}
+ # @return [void]
+ def discard(path)
+ Database.delete_database(path)
+ end
+
+ def cleanup
+ FileUtils.rm_rf(@dir)
+ end
+ end
+ end
+end
diff --git a/migrations/core/lib/migrations/conversion/step.rb b/migrations/core/lib/migrations/conversion/step.rb
index 5fbaba5bf3b..20ed2b53bc2 100644
--- a/migrations/core/lib/migrations/conversion/step.rb
+++ b/migrations/core/lib/migrations/conversion/step.rb
@@ -2,22 +2,133 @@
module Migrations
module Conversion
- class Step < StepBase
- include AttributeAssignment
+ # A step that processes many items and reports progress. It is split into
+ # two roles:
+ #
+ # * `source` enumerates the items (`items`, `max_progress`).
+ # * `processor` handles one item at a time (`process`). Per-worker state is
+ # built in its `setup` hook which runs after the worker has started, never
+ # in the constructor. `setup` must not create IntermediateDB records, only
+ # `process` writes; `setup` runs under `SetupGuard`, so a write raises
+ # `SetupGuard::SetupError` whether the step runs inline or in a fork.
+ #
+ # Both roles run in the worker that processes the step (a forked process, or
+ # the main process under `--no-fork`): the worker builds the source, reads its
+ # slice of the rows, and feeds each one to a processor. For a partitioned step
+ # the parent also builds a source of its own, before forking, only to work out
+ # the chunk boundaries.
+ #
+ # The roles are separate objects, so a processor can't read source-side
+ # state: such a call fails with a `NoMethodError`. That keeps `process`
+ # from depending on the source's iteration state, which would otherwise be
+ # an easy way to write a step that breaks once it's split across forks.
+ # `helpers` is mixed into both roles and is meant for pure functions only
+ # (arguments in, value out).
+ #
+ # class Users < Conversion::Step
+ # source do
+ # def items
+ # @source_db.query("SELECT ...")
+ # end
+ # end
+ #
+ # processor do
+ # def setup
+ # @upload_creator = UploadCreator.new
+ # end
+ #
+ # def process(item)
+ # IntermediateDB::User.create(...)
+ # end
+ # end
+ # end
+ #
+ # The blocks are evaluated with `class_eval`, so use plain `def` inside them.
+ # Avoid `define_method`: its block closes over the surrounding scope and would
+ # leak step-level state across the role boundary.
+ class Step
+ extend StepDependencies
- attr_accessor :settings
- attr_reader :tracker
+ IntermediateDB = Database::IntermediateDB
+ Enums = Database::IntermediateDB::Enums
- # inside of Step it might make more sense to access it as `step` instead of `tracker`
- alias step tracker
+ attr_reader :source
def initialize(args = {})
- @tracker = StepTracker.new
- assign_attributes(args)
+ @args = args
+ @source = self.class.source_class.new(args)
end
- def execute
- # do nothing
+ def create_processor
+ self.class.processor_class.new(@args)
+ end
+
+ class << self
+ def title(
+ value = (
+ getter = true
+ nil
+ )
+ )
+ @title = value unless getter
+ @title.presence ||
+ I18n.t(
+ "converter.default_step_title",
+ type: name&.demodulize&.underscore&.humanize(capitalize: false),
+ )
+ end
+
+ def source(&block)
+ raise ArgumentError, "`source` is already defined" if @source_block
+ @source_block = block
+ end
+
+ def processor(&block)
+ raise ArgumentError, "`processor` is already defined" if @processor_block
+ @processor_block = block
+ end
+
+ def helpers(&block)
+ raise ArgumentError, "`helpers` is already defined" if @helpers_block
+ @helpers_block = block
+ end
+
+ def source_class
+ @source_class ||=
+ build_role_class(superclass_role(:source_class, Source), :Source, @source_block)
+ end
+
+ def partitionable?
+ source_class.partitionable?
+ end
+
+ def processor_class
+ @processor_class ||=
+ build_role_class(
+ superclass_role(:processor_class, Processor),
+ :Processor,
+ @processor_block,
+ )
+ end
+
+ def helpers_module
+ @helpers_module ||=
+ Module.new.tap { |mod| mod.module_eval(&@helpers_block) if @helpers_block }
+ end
+
+ private
+
+ def superclass_role(role_method, base_class)
+ superclass < Step ? superclass.public_send(role_method) : base_class
+ end
+
+ def build_role_class(base_class, name, block)
+ klass = Class.new(base_class)
+ const_set(name, klass)
+ klass.include(helpers_module)
+ klass.class_eval(&block) if block
+ klass
+ end
end
end
end
diff --git a/migrations/core/lib/migrations/conversion/step/processor.rb b/migrations/core/lib/migrations/conversion/step/processor.rb
new file mode 100644
index 00000000000..eb094b552cb
--- /dev/null
+++ b/migrations/core/lib/migrations/conversion/step/processor.rb
@@ -0,0 +1,26 @@
+# frozen_string_literal: true
+
+module Migrations
+ module Conversion
+ class Step
+ class Processor
+ include AttributeAssignment
+
+ attr_accessor :settings
+ attr_reader :tracker
+
+ def initialize(args = {})
+ @tracker = StepTracker.new
+ assign_attributes(args)
+ end
+
+ def setup
+ end
+
+ def process(item)
+ raise NotImplementedError
+ end
+ end
+ end
+ end
+end
diff --git a/migrations/core/lib/migrations/conversion/step/source.rb b/migrations/core/lib/migrations/conversion/step/source.rb
new file mode 100644
index 00000000000..84d64bbe2f5
--- /dev/null
+++ b/migrations/core/lib/migrations/conversion/step/source.rb
@@ -0,0 +1,121 @@
+# frozen_string_literal: true
+
+module Migrations
+ module Conversion
+ class Step
+ class Source
+ include AttributeAssignment
+
+ # `source_db` is the step's own connection to the source data, when it has
+ # one. Steps that read from somewhere else (a file, fixed data) leave it
+ # nil. The converter injects it through `step_args`.
+ attr_accessor :settings, :source_db
+
+ # The `[lower, upper]` key range this worker reads (`upper` nil means
+ # open-ended), or nil for an unpartitioned step. The runner sets it before
+ # `items`; the generated queries (and `partition_slice`) apply it.
+ attr_accessor :chunk
+
+ class << self
+ # Read a whole table: defines `items` (`SELECT * FROM <name>`) and
+ # `max_progress` (its row count), both filtered by `where`. Works with
+ # or without `partition_by` (a partitioned step also limits each query
+ # to its chunk). Override `items`/`max_progress` for anything custom.
+ #
+ # `order` (a column or array of them) sorts the read — worth setting for
+ # a large table whose rows insert faster in that order (it should match
+ # the target's key, and be indexed at the source so it doesn't add a
+ # sort). A partitioned step ignores it and orders by its key instead.
+ def reads_table(name, where: nil, order: nil)
+ @table = { name:, where:, order: }
+ end
+
+ attr_reader :table
+
+ def reads_table?
+ !table.nil?
+ end
+
+ # Split the step across forks on `key` (one column, or an array of
+ # columns for a composite key). `from`/`base` say which table and filter
+ # to scan for the boundaries; they default to the `reads_table` ones, so
+ # a table-reading step only needs the key.
+ def partition_by(key, from: nil, base: nil)
+ @partition = { key:, from:, base: }
+ end
+
+ attr_reader :partition
+
+ def partitionable?
+ !partition.nil?
+ end
+ end
+
+ def initialize(args = {})
+ assign_attributes(args)
+ end
+
+ # The chunk boundaries for splitting the source `count` ways, worked out
+ # in the parent before forking.
+ def partition_boundaries(count)
+ Partitioner.new(
+ @source_db,
+ key: partition_key,
+ from: partition_from,
+ base: partition_base,
+ ).boundaries(count)
+ end
+
+ # The WHERE body that limits a query to this worker's chunk. Add it to a
+ # custom query (`WHERE #{partition_slice}`); the generated queries already
+ # use it.
+ def partition_slice
+ lower, upper = chunk
+ @source_db.chunk_filter(partition_key, lower, upper, base: partition_base)
+ end
+
+ def max_progress
+ return unless self.class.reads_table?
+ @source_db.count_all(self.class.table[:name], where: read_filter)
+ end
+
+ def items
+ raise NotImplementedError unless self.class.reads_table?
+ @source_db.select_all(self.class.table[:name], where: read_filter, order: read_order)
+ end
+
+ def cleanup
+ @source_db&.close
+ end
+
+ private
+
+ # The rows the current read should see: this worker's chunk when
+ # partitioned, otherwise the whole table's filter.
+ def read_filter
+ self.class.partitionable? ? partition_slice : self.class.table[:where]
+ end
+
+ # A partitioned read is ordered by its key so each worker writes its shard
+ # in key order (sequential index inserts, in the shard and in the merge).
+ # An unpartitioned read uses `reads_table`'s `order`, if any.
+ def read_order
+ return partition_key if self.class.partitionable?
+ self.class.table[:order]
+ end
+
+ def partition_key
+ self.class.partition[:key]
+ end
+
+ def partition_from
+ self.class.partition[:from] || self.class.table&.fetch(:name)
+ end
+
+ def partition_base
+ self.class.partition[:base] || self.class.table&.fetch(:where)
+ end
+ end
+ end
+ end
+end
diff --git a/migrations/core/lib/migrations/conversion/step_base.rb b/migrations/core/lib/migrations/conversion/step_base.rb
deleted file mode 100644
index 1661476f12f..00000000000
--- a/migrations/core/lib/migrations/conversion/step_base.rb
+++ /dev/null
@@ -1,37 +0,0 @@
-# frozen_string_literal: true
-
-module Migrations
- module Conversion
- # Abstract base of all conversion steps; `Base#steps` discovers its
- # subclasses. It carries only what every step kind shares — `Step` adds
- # the imperative single-run contract, `ProgressStep` the per-item
- # source/processor roles.
- class StepBase
- extend StepDependencies
-
- # These constants also make bare `IntermediateDB::...` / `Enums::...`
- # references work inside `ProgressStep`'s `source` / `processor` blocks:
- # the blocks are written in step class bodies, and constants in methods
- # defined via `class_eval(&block)` resolve through the block's lexical
- # scope — the step class and its ancestors — not the role class.
- IntermediateDB = Database::IntermediateDB
- Enums = Database::IntermediateDB::Enums
-
- class << self
- def title(
- value = (
- getter = true
- nil
- )
- )
- @title = value unless getter
- @title.presence ||
- I18n.t(
- "converter.default_step_title",
- type: name&.demodulize&.underscore&.humanize(capitalize: false),
- )
- end
- end
- end
- end
-end
diff --git a/migrations/core/lib/migrations/conversion/step_coordinator.rb b/migrations/core/lib/migrations/conversion/step_coordinator.rb
new file mode 100644
index 00000000000..0e6e1c0e1c4
--- /dev/null
+++ b/migrations/core/lib/migrations/conversion/step_coordinator.rb
@@ -0,0 +1,222 @@
+# frozen_string_literal: true
+
+module Migrations
+ module Conversion
+ # Runs one step end to end on its own thread for the {StepScheduler}. It forks
+ # the step's workers — one for a normal step, several for a partitioned one (or
+ # none under `--no-fork`, where the single worker runs inline). Each worker
+ # builds the step itself, opens its own source, reads its slice of the rows,
+ # and writes them to its own shard.
+ #
+ # The coordinator relays the workers' combined progress to the reporter. Once
+ # they all exit cleanly it hands the finished shards to the {Consolidator},
+ # which merges them into the run database on a background thread, so the step
+ # finishes without waiting for its own merge.
+ #
+ # The step is built inside each worker, not here, so its source connection is
+ # opened there. The parent never holds a source connection, and nothing but
+ # progress crosses the pipes.
+ class StepCoordinator
+ class WorkerCrashedError < StandardError
+ end
+
+ attr_reader :step_class
+
+ # @param step_class [Class<Step>] the step this coordinator runs
+ # @param step_factory [#call] builds a step from its class, `->(step_class) { step }`;
+ # called inside each worker so the source connection opens there
+ # @param reporter [Reporting::Reporter] receives the step's progress and notices
+ # @param fork_mutex [Mutex] serialises forking across the run's coordinator threads
+ # @param scheduler [StepScheduler] told about a step-level failure for the run summary
+ # @param shard_manager [ShardManager] hands out and discards the per-worker shard DBs
+ # @param consolidator [Consolidator] merges the finished shards in the background
+ # @param fork_count [Integer] how many workers to fork (1 for an unpartitioned step)
+ # @param no_fork [Boolean] run the single worker inline instead of forking (`--no-fork`)
+ def initialize(
+ step_class:,
+ step_factory:,
+ reporter:,
+ fork_mutex:,
+ scheduler:,
+ shard_manager:,
+ consolidator:,
+ fork_count: 1,
+ no_fork: false
+ )
+ @step_class = step_class
+ @step_factory = step_factory
+ @reporter = reporter
+ @fork_mutex = fork_mutex
+ @scheduler = scheduler
+ @shard_manager = shard_manager
+ @consolidator = consolidator
+ @fork_count = fork_count
+ @no_fork = no_fork
+ end
+
+ # @return [Symbol] the step's outcome, `:done` or `:failed`
+ def run
+ @step_handle = @reporter.start_step(@step_class.title)
+ @step_handle.report_concurrency(@fork_count)
+ outcome = :done
+ shards = []
+ handed_off = false
+
+ begin
+ # `finish` reads `$!` in its ensure, so the work must raise inside this
+ # inner begin, before the rescue below clears it.
+ begin
+ run_workers(shards)
+ @consolidator.enqueue(shards)
+ handed_off = true
+ ensure
+ error = $!
+ @step_handle.notice(failure_notice(error)) if error.is_a?(StandardError)
+ @step_handle.finish
+ end
+ rescue StandardError => e
+ outcome = :failed
+ @scheduler.record_failure(@step_class, e)
+ ensure
+ shards.each { |shard_path| @shard_manager.discard(shard_path) } unless handed_off
+ end
+
+ outcome
+ end
+
+ private
+
+ def run_workers(shards)
+ return run_inline(shards) if @no_fork
+
+ boundaries = compute_boundaries
+ worker_count = boundaries.nil? ? 1 : [boundaries.size, 1].max
+
+ pipes = Array.new(worker_count) { IO.pipe }
+ worker_count.times { shards << @shard_manager.create_shard }
+
+ pids = nil
+ @fork_mutex.synchronize do
+ ForkManager.with_batched_forks do
+ pids =
+ worker_count.times.map do |index|
+ chunk = chunk_for(boundaries, index)
+ ForkManager.fork { run_in_fork(pipes, index, shards[index], chunk) }
+ end
+ end
+ end
+
+ pipes.each { |(_reader, writer)| writer.close }
+ readers = pipes.map(&:first)
+
+ begin
+ consume_progress(readers)
+ ensure
+ readers.each(&:close)
+ end
+
+ await(pids)
+ end
+
+ def run_inline(shards)
+ shard_path = @shard_manager.create_shard
+ shards << shard_path
+
+ StepRunner.new(
+ step: @step_factory.call(@step_class),
+ shard_path:,
+ reporter: InlineProgressSink.new(@step_handle),
+ chunk: nil,
+ ).run
+ end
+
+ def compute_boundaries
+ return nil if @fork_count == 1
+
+ step = @step_factory.call(@step_class)
+ begin
+ step.source.partition_boundaries(@fork_count)
+ ensure
+ step.source.cleanup
+ end
+ end
+
+ def chunk_for(boundaries, index)
+ return nil if boundaries.nil? || boundaries.empty?
+ [boundaries[index], boundaries[index + 1]]
+ end
+
+ def run_in_fork(pipes, mine, shard_path, chunk)
+ pipes.each_with_index do |(reader, writer), index|
+ reader.close
+ writer.close unless index == mine
+ end
+
+ step = @step_factory.call(@step_class)
+ StepRunner.new(
+ step:,
+ shard_path:,
+ reporter: PipeProgressSink.new(pipes[mine][1]),
+ chunk:,
+ ).run
+ end
+
+ def await(pids)
+ pids.each do |pid|
+ _, status = Process.waitpid2(pid)
+ next if status.success?
+
+ raise WorkerCrashedError,
+ "A worker for #{@step_class.title} exited unexpectedly (#{status}). " \
+ "Check the error output above for the cause."
+ end
+ end
+
+ def consume_progress(readers)
+ @step_handle.with_progress(max_progress: total_max_progress(readers)) do |progress|
+ drain_progress(readers, progress)
+ end
+ end
+
+ def total_max_progress(readers)
+ maxes = readers.map { |reader| read_max_progress(reader) }
+ return if maxes.any?(&:nil?)
+ maxes.sum
+ end
+
+ def drain_progress(readers, progress)
+ pending = readers.dup
+
+ until pending.empty?
+ ready, = IO.select(pending)
+ ready.each do |reader|
+ line = reader.gets
+ if line.nil?
+ pending.delete(reader)
+ next
+ end
+
+ _, increment, warnings, errors = line.split
+ progress.update(
+ increment_by: increment.to_i,
+ warning_count: warnings.to_i,
+ error_count: errors.to_i,
+ )
+ end
+ end
+ end
+
+ def read_max_progress(reader)
+ line = reader.gets
+ return if line.nil?
+
+ value = line.split[1]
+ value&.to_i
+ end
+
+ def failure_notice(error)
+ "#{error.class}: #{error.message}"
+ end
+ end
+ end
+end
diff --git a/migrations/core/lib/migrations/conversion/step_executor.rb b/migrations/core/lib/migrations/conversion/step_executor.rb
deleted file mode 100644
index 67b69c3df8c..00000000000
--- a/migrations/core/lib/migrations/conversion/step_executor.rb
+++ /dev/null
@@ -1,19 +0,0 @@
-# frozen_string_literal: true
-
-module Migrations
- module Conversion
- class StepExecutor
- def initialize(step, reporter:)
- @step = step
- @reporter = reporter
- end
-
- def execute
- step_report = @reporter.start_step(@step.class.title)
- @step.execute
- ensure
- step_report.finish
- end
- end
- end
-end
diff --git a/migrations/core/lib/migrations/conversion/step_runner.rb b/migrations/core/lib/migrations/conversion/step_runner.rb
new file mode 100644
index 00000000000..7370fd48c4f
--- /dev/null
+++ b/migrations/core/lib/migrations/conversion/step_runner.rb
@@ -0,0 +1,85 @@
+# frozen_string_literal: true
+
+module Migrations
+ module Conversion
+ # Runs one step's items to completion inside a single process. The worker
+ # opens its own source, so the read happens here in the fork, not in the
+ # parent, then reads the rows it is responsible for, processes each, and
+ # writes them to its shard. Nothing is sent in but the step to run, and
+ # nothing goes back but progress.
+ #
+ # The insert runs here against a real connection, so a bad row (a NULL in a
+ # NOT NULL column, say) raises right where it's processed, to be logged and
+ # skipped per item rather than failing the whole step.
+ class StepRunner
+ # How many processed items to accumulate before reporting progress.
+ REPORT_INTERVAL = 1_000
+
+ # @param step [Step] the step to run; its source and processor are built here
+ # @param shard_path [String] the SQLite shard this worker writes its rows to
+ # @param reporter [#report_max_progress, #report_progress] the worker's end of
+ # the progress channel ({PipeProgressSink} in a fork, {InlineProgressSink} inline)
+ # @param chunk [Array(Object, Object), nil] the `[lower, upper]` key range to read
+ # (upper nil = open-ended), or nil to read the whole source
+ def initialize(step:, shard_path:, reporter:, chunk: nil)
+ @step = step
+ @shard_path = shard_path
+ @reporter = reporter
+ @chunk = chunk
+ end
+
+ def run
+ source = @step.source
+ source.chunk = @chunk
+ connection = Database::Connection.new(path: @shard_path)
+
+ begin
+ # Point IntermediateDB at this worker's shard for the block.
+ # `with_connection` restores the previous connection afterwards without
+ # closing it — unlike `setup`, which closes it; under `--no-fork` that
+ # previous connection is the live run DB connection the rest of the
+ # run still needs.
+ Database::IntermediateDB.with_connection(connection) do
+ processor = @step.create_processor
+ SetupGuard.run(processor)
+
+ @reporter.report_max_progress(source.max_progress)
+ process_items(source, processor)
+ end
+ ensure
+ connection.close
+ source.cleanup
+ end
+ end
+
+ private
+
+ def process_items(source, processor)
+ tracker = processor.tracker
+ progress = warnings = errors = 0
+
+ source.items.each do |item|
+ tracker.reset_stats!
+
+ begin
+ processor.process(item)
+ rescue StandardError => e
+ tracker.log_error("Failed to process item", exception: e, details: item)
+ end
+
+ stats = tracker.stats
+ progress += stats.progress
+ warnings += stats.warning_count
+ errors += stats.error_count
+
+ next if progress < REPORT_INTERVAL
+ @reporter.report_progress(progress:, warnings:, errors:)
+ progress = warnings = errors = 0
+ end
+
+ return if progress.zero? && warnings.zero? && errors.zero?
+ @reporter.report_progress(progress:, warnings:, errors:)
+ end
+ end
+ end
+end
diff --git a/migrations/core/lib/migrations/conversion/step_scheduler.rb b/migrations/core/lib/migrations/conversion/step_scheduler.rb
new file mode 100644
index 00000000000..511cced12bf
--- /dev/null
+++ b/migrations/core/lib/migrations/conversion/step_scheduler.rb
@@ -0,0 +1,244 @@
+# frozen_string_literal: true
+
+module Migrations
+ module Conversion
+ # Runs the conversion steps concurrently and follows their dependency graph.
+ # Each running step gets its own {StepCoordinator} on its own thread, which
+ # forks one worker that reads the step's source and writes its shard. This
+ # class only decides what may start: up to `budget` steps run at once (the
+ # smaller of the fork budget and `--max-parallel-steps`), and a step starts
+ # only once its dependencies are done.
+ #
+ # We don't do slot accounting or full-pool exclusivity here. Every step is
+ # one fork, so "how many run at once" is the whole scheduling decision. When
+ # a heavy step is split across several forks, those count as several entries
+ # in the same budget.
+ class StepScheduler
+ FINISHED_STATES = %i[done failed skipped].freeze
+
+ # @param step_classes [Array<Class<Step>>] every step in the run, in any order
+ # @param reporter [Reporting::Reporter] shared by all steps to report progress
+ # @param step_factory [#call] builds a step from its class, `->(step_class) { step }`
+ # @param shard_manager [ShardManager] hands out the per-worker shard databases
+ # @param writer [Database::Connection] the run DB connection the consolidator
+ # merges shards into (held directly, so an inline step swapping the
+ # IntermediateDB connection can't divert a background merge)
+ # @param budget [Integer] the most steps (forks) to run at once, usually cores - 1
+ # @param max_parallel_steps [Integer, nil] a lower cap from `--max-parallel-steps`
+ # @param no_fork [Boolean] run each step inline, one at a time (`--no-fork`)
+ def initialize(
+ step_classes:,
+ reporter:,
+ step_factory:,
+ shard_manager:,
+ writer:,
+ budget:,
+ max_parallel_steps: nil,
+ no_fork: false
+ )
+ @step_classes = step_classes
+ @reporter = reporter
+ @step_factory = step_factory
+ @shard_manager = shard_manager
+ @writer = writer
+ @no_fork = no_fork
+
+ @budget = [budget, max_parallel_steps].compact.min
+ @budget = 1 if @budget < 1
+ # Inline runs share one process and one IntermediateDB connection, so
+ # only one step may run at a time.
+ @budget = 1 if @no_fork
+
+ @mutex = Mutex.new
+ @condition = ConditionVariable.new
+ @fork_mutex = Mutex.new
+
+ @states = step_classes.to_h { |step_class| [step_class, :pending] }
+ @available_forks = @budget
+ # A running step_class => the forks it reserved from the budget, returned
+ # to @available_forks when it finishes.
+ @reserved_forks = {}
+ @threads = []
+ @failures = {}
+ @merge_errors = []
+ end
+
+ attr_reader :budget
+
+ def run
+ started_at = monotonic_time
+ # Compact the heap before any fork so children share its pages copy-on-write.
+ Process.warmup
+ @consolidator = Consolidator.new(@shard_manager, @writer, @fork_mutex)
+
+ @mutex.synchronize do
+ loop do
+ schedule
+ break if all_finished?
+ @condition.wait(@mutex)
+ end
+ end
+
+ @threads.each(&:join)
+ # Every step is done, but the background merges may still be catching up.
+ # Show a "finishing up" status until they drain, so the display doesn't sit
+ # silently at 100%.
+ @merge_errors = @reporter.finalizing { @consolidator.drain }
+ report_summary(monotonic_time - started_at)
+ raise_summary_if_unsuccessful
+ end
+
+ # Records a step-level error for the end-of-run summary; a coordinator calls
+ # this when its step fails.
+ # @param step_class [Class<Step>] the step that failed
+ # @param error [StandardError] what it failed with
+ # @return [void]
+ def record_failure(step_class, error)
+ @mutex.synchronize { @failures[step_class] = error }
+ end
+
+ private
+
+ def schedule
+ loop do
+ step_class = next_ready_step
+ break unless step_class
+
+ forks = forks_for(step_class)
+ # Stop (don't skip past) when the next step can't get its forks yet, so
+ # a partitioned step can gather the budget instead of single-fork steps
+ # nibbling it away.
+ break if forks > @available_forks
+
+ @states[step_class] = :running
+ @available_forks -= forks
+ @reserved_forks[step_class] = forks
+ spawn_coordinator(step_class, forks)
+ end
+ end
+
+ # A partitioned step takes all but one fork, leaving one free so single-fork
+ # steps can still trickle through and overlap it.
+ def forks_for(step_class)
+ return 1 unless step_class.partitionable?
+ [@budget - 1, 1].max
+ end
+
+ def spawn_coordinator(step_class, fork_count)
+ coordinator = build_coordinator(step_class, fork_count)
+
+ @threads << Thread.new do
+ Thread.current.name = "step_#{step_class.name.demodulize}"
+
+ outcome = :failed
+ begin
+ outcome = coordinator.run
+ rescue SignalException
+ outcome = :failed
+ rescue StandardError => e
+ # Must be recorded even if `run` should have handled it, or the step
+ # never reaches a terminal state and the scheduler waits on it forever.
+ record_failure(step_class, e)
+ outcome = :failed
+ ensure
+ step_finished(step_class, outcome)
+ end
+ end
+ end
+
+ def build_coordinator(step_class, fork_count)
+ StepCoordinator.new(
+ step_class:,
+ step_factory: @step_factory,
+ reporter: @reporter,
+ fork_mutex: @fork_mutex,
+ scheduler: self,
+ shard_manager: @shard_manager,
+ consolidator: @consolidator,
+ fork_count:,
+ no_fork: @no_fork,
+ )
+ end
+
+ def step_finished(step_class, outcome)
+ @mutex.synchronize do
+ @available_forks += @reserved_forks.delete(step_class)
+ @states[step_class] = outcome
+ skip_dependents(step_class) if outcome == :failed
+ @condition.broadcast
+ end
+ end
+
+ def skip_dependents(failed_step_class)
+ loop do
+ newly_skipped =
+ @step_classes.select do |step_class|
+ @states[step_class] == :pending &&
+ dependencies_of(step_class).any? do |dependency|
+ %i[failed skipped].include?(@states[dependency])
+ end
+ end
+
+ break if newly_skipped.empty?
+ newly_skipped.each { |step_class| @states[step_class] = :skipped }
+ end
+ end
+
+ def next_ready_step
+ ready = @step_classes.select { |step_class| ready?(step_class) }
+ ready.min_by { |step_class| admission_key(step_class) }
+ end
+
+ def ready?(step_class)
+ @states[step_class] == :pending &&
+ dependencies_of(step_class).all? { |dependency| @states[dependency] == :done }
+ end
+
+ def dependencies_of(step_class)
+ step_class.dependencies & @step_classes
+ end
+
+ # Partitioned steps first (so they reach the front and gather their forks),
+ # then by priority (lower first, unset last), then by class name.
+ def admission_key(step_class)
+ priority = step_class.priority
+ [
+ step_class.partitionable? ? 0 : 1,
+ priority.nil? ? 1 : 0,
+ priority || 0,
+ step_class.name.to_s,
+ ]
+ end
+
+ def all_finished?
+ @states.each_value.all? { |state| FINISHED_STATES.include?(state) }
+ end
+
+ def report_summary(runtime)
+ outcomes = @states.values
+ @reporter.report_summary(
+ runtime:,
+ total: outcomes.size,
+ failed: outcomes.count(:failed),
+ skipped: outcomes.count(:skipped),
+ )
+ end
+
+ def monotonic_time
+ Process.clock_gettime(Process::CLOCK_MONOTONIC)
+ end
+
+ def raise_summary_if_unsuccessful
+ failed = @states.select { |_, state| state == :failed }.keys
+ skipped = @states.select { |_, state| state == :skipped }.keys
+ return if failed.empty? && skipped.empty? && @merge_errors.empty?
+
+ raise ConvertError.new(
+ failures: failed.to_h { |step_class| [step_class, @failures[step_class]] },
+ skipped:,
+ merge_errors: @merge_errors,
+ )
+ end
+ end
+ end
+end
diff --git a/migrations/core/lib/migrations/conversion/worker.rb b/migrations/core/lib/migrations/conversion/worker.rb
deleted file mode 100644
index b17c11bf993..00000000000
--- a/migrations/core/lib/migrations/conversion/worker.rb
+++ /dev/null
@@ -1,139 +0,0 @@
-# frozen_string_literal: true
-
-require "oj"
-
-module Migrations
- module Conversion
- class Worker
- class CrashedError < StandardError
- end
-
- OJ_SETTINGS = { mode: :object, class_cache: true, symbol_keys: true }
-
- def initialize(index, input_queue, output_queue, job)
- @index = index
- @input_queue = input_queue
- @output_queue = output_queue
- @job = job
-
- @threads = []
- @mutex = Mutex.new
- @data_processed = ConditionVariable.new
- @sent_count = 0
- @processed_count = 0
- @output_closed = false
- end
-
- def start
- parent_input_stream, parent_output_stream = IO.pipe
- fork_input_stream, fork_output_stream = IO.pipe
-
- worker_pid =
- start_fork(
- parent_input_stream,
- parent_output_stream,
- fork_input_stream,
- fork_output_stream,
- )
-
- fork_output_stream.close
- parent_input_stream.close
-
- start_input_thread(parent_output_stream, worker_pid)
- start_output_thread(fork_input_stream)
-
- self
- end
-
- def wait
- @threads.each(&:join)
- end
-
- private
-
- def start_fork(
- parent_input_stream,
- parent_output_stream,
- fork_input_stream,
- fork_output_stream
- )
- ForkManager.fork do
- Process.setproctitle("worker_process#{@index}")
-
- parent_output_stream.close
- fork_input_stream.close
-
- @job.setup
-
- Oj.load(parent_input_stream, OJ_SETTINGS) do |data|
- result = @job.run(data)
- Oj.to_stream(fork_output_stream, result, OJ_SETTINGS)
- end
- rescue SignalException
- exit(1)
- ensure
- @job.cleanup
- end
- end
-
- def start_input_thread(output_stream, worker_pid)
- @threads << Thread.new do
- Thread.current.name = "worker_#{@index}_input"
- # A `CrashedError` surfaces through `Worker#wait` (`Thread#join`);
- # without this, an interrupted run would additionally report the
- # error for every worker.
- Thread.current.report_on_exception = false
-
- begin
- while (data = @input_queue.pop)
- Oj.to_stream(output_stream, data, OJ_SETTINGS)
- @sent_count += 1
-
- # waiting on the condition variable alone would lose the wakeup
- # when the result arrives before this thread reaches `wait`, so
- # the counters act as the wait predicate
- @mutex.synchronize do
- @data_processed.wait(@mutex) while @processed_count < @sent_count && !@output_closed
- end
- end
- rescue Errno::EPIPE
- # The worker process died; the status check below raises the error.
- ensure
- output_stream.close
- _, status = Process.waitpid2(worker_pid)
-
- if !status.success?
- raise CrashedError,
- "Worker process #{@index} exited unexpectedly (#{status}). " \
- "Check the error output above for the cause."
- end
- end
- end
- end
-
- def start_output_thread(input_stream)
- @threads << Thread.new do
- Thread.current.name = "worker_#{@index}_output"
-
- begin
- Oj.load(input_stream, OJ_SETTINGS) do |data|
- @output_queue.push(data)
-
- @mutex.synchronize do
- @processed_count += 1
- @data_processed.signal
- end
- end
- ensure
- input_stream.close
-
- @mutex.synchronize do
- @output_closed = true
- @data_processed.signal
- end
- end
- end
- end
- end
- end
-end
diff --git a/migrations/core/lib/migrations/conversion/worker_budget.rb b/migrations/core/lib/migrations/conversion/worker_budget.rb
new file mode 100644
index 00000000000..6bd1bbec228
--- /dev/null
+++ b/migrations/core/lib/migrations/conversion/worker_budget.rb
@@ -0,0 +1,38 @@
+# frozen_string_literal: true
+
+require "etc"
+require "concurrent/utility/processor_counter" # just the CPU counter, not all of concurrent-ruby
+
+module Migrations
+ module Conversion
+ # Works out how many worker forks a run may use: the CPUs it can actually run
+ # on, minus a core kept for the parent process and the background shard merges.
+ #
+ # "CPUs it can run on" is not just the host core count. In a container the run
+ # is usually limited by a CFS quota (`docker --cpus`, k8s `limits.cpu`), which
+ # `Etc.nprocessors` ignores. So we take the tighter of two numbers:
+ #
+ # * `Etc.nprocessors` — respects CPU affinity (`taskset`, cpuset)
+ # * `Concurrent.available_processor_count` — respects the CFS quota
+ #
+ # Their minimum is the effective budget, so a 4-CPU pod on a 64-core node uses
+ # 4, not 64.
+ module WorkerBudget
+ # Cores held back for the scheduler/coordinator threads (mostly idle, waiting
+ # on IO) and the single consolidator merge thread. This is fixed, not scaled
+ # with core count: the background work is about one busy thread, so one core
+ # is enough on any machine — reserving more would just waste worker slots.
+ RESERVED_CORES = 1
+
+ def self.available(reserved: RESERVED_CORES)
+ [usable_cpus - reserved, 1].max
+ end
+
+ # The CFS quota can be fractional (e.g. `--cpus=1.5`), so floor it; never
+ # report fewer than one usable CPU.
+ def self.usable_cpus
+ [[Etc.nprocessors, Concurrent.available_processor_count.floor].min, 1].max
+ end
+ end
+ end
+end
diff --git a/migrations/core/lib/migrations/conversion/worker_pool.rb b/migrations/core/lib/migrations/conversion/worker_pool.rb
deleted file mode 100644
index e8f02c57008..00000000000
--- a/migrations/core/lib/migrations/conversion/worker_pool.rb
+++ /dev/null
@@ -1,48 +0,0 @@
-# frozen_string_literal: true
-
-require "etc"
-
-module Migrations
- module Conversion
- # Owns worker sizing and the fork machinery for a conversion run. It knows
- # nothing about steps, progress reporting, or scheduling — callers bring
- # the queues and a job factory, the pool brings the processes.
- class WorkerPool
- DEFAULT_SIZE = Etc.nprocessors - 1 # leave 1 CPU free to do other work
-
- attr_reader :size
-
- def initialize(size: DEFAULT_SIZE)
- @size = size
- end
-
- # Forks worker processes (default: all `size` of them), each running the
- # job built by `job_factory`. The factory is called once per worker, so
- # workers never share job state. Workers exit when `work_queue` is
- # closed and drained. Returns a handle for the started batch.
- def start(work_queue:, output_queue:, size: @size, &job_factory)
- worker_count = size.clamp(1, @size)
- Process.warmup
-
- workers = []
- ForkManager.with_batched_forks do
- worker_count.times do |index|
- workers << Worker.new(index, work_queue, output_queue, job_factory.call).start
- end
- end
-
- Batch.new(workers)
- end
-
- class Batch
- def initialize(workers)
- @workers = workers
- end
-
- def wait
- @workers.each(&:wait)
- end
- end
- end
- end
-end
diff --git a/migrations/core/lib/migrations/database.rb b/migrations/core/lib/migrations/database.rb
index 1d82dc6838a..f7f0049468d 100644
--- a/migrations/core/lib/migrations/database.rb
+++ b/migrations/core/lib/migrations/database.rb
@@ -2,8 +2,9 @@
require "date"
require "extralite"
+require "fileutils"
require "ipaddr"
-require "oj"
+require "json"
module Migrations
module Database
@@ -15,8 +16,15 @@ module Migrations
Migrator.new(db_path).migrate(migrations_path)
end
- def self.reset!(db_path)
- Migrator.new(db_path).reset!
+ # A WAL-mode database is three files: the main file plus its `-wal` and `-shm`
+ # sidecars.
+ def self.database_files(db_path)
+ [db_path, "#{db_path}-wal", "#{db_path}-shm"]
+ end
+
+ # Deletes the database and its sidecar files; no error if any are missing.
+ def self.delete_database(db_path)
+ FileUtils.rm_f(database_files(db_path))
end
def self.connect(path)
@@ -74,7 +82,7 @@ module Migrations
def self.to_json(value)
return nil if value.nil?
- Oj.dump(value, mode: :compat)
+ JSON.generate(value)
end
def self.to_date(text)
diff --git a/migrations/core/lib/migrations/database/connection.rb b/migrations/core/lib/migrations/database/connection.rb
index 67362ddddec..b8c2af15ad8 100644
--- a/migrations/core/lib/migrations/database/connection.rb
+++ b/migrations/core/lib/migrations/database/connection.rb
@@ -1,6 +1,7 @@
# frozen_string_literal: true
require "extralite"
+require "fileutils"
module Migrations
module Database
@@ -77,10 +78,30 @@ module Migrations
query_value(sql, *parameters)
end
+ def tables
+ @db.tables
+ end
+
def execute(sql, *parameters)
@db.execute(sql, *parameters)
end
+ # `ATTACH` can't run inside a transaction, so commit any open batch first.
+ def merge_database(other_path, tables:)
+ commit_transaction
+ @db.execute("ATTACH DATABASE ? AS merge_source", other_path)
+ begin
+ tables.each do |table|
+ quoted = quote_identifier(table)
+ @db.execute("INSERT OR IGNORE INTO main.#{quoted} SELECT * FROM merge_source.#{quoted}")
+ end
+ ensure
+ @db.execute("DETACH DATABASE merge_source")
+ end
+
+ nil
+ end
+
def begin_transaction
@db.execute("BEGIN DEFERRED TRANSACTION") unless @db.transaction_active?
end
@@ -94,6 +115,10 @@ module Migrations
private
+ def quote_identifier(name)
+ %("#{name.gsub('"', '""')}")
+ end
+
def close_connection(keep_path:)
return if @db.nil?
diff --git a/migrations/core/lib/migrations/database/migrator.rb b/migrations/core/lib/migrations/database/migrator.rb
index 8627dd53b3d..6db247d5e02 100644
--- a/migrations/core/lib/migrations/database/migrator.rb
+++ b/migrations/core/lib/migrations/database/migrator.rb
@@ -25,21 +25,10 @@ module Migrations
nil
end
- def reset!
- [@db_path, "#{@db_path}-wal", "#{@db_path}-shm"].each do |path|
- FileUtils.remove_file(path, force: true) if File.exist?(path)
- end
- nil
- end
-
private
def new_database?
- @db.query_single_splat(<<~SQL) == 0
- SELECT COUNT(*)
- FROM sqlite_schema
- WHERE type = 'table' AND name = 'schema_migrations'
- SQL
+ [email protected]?("schema_migrations")
end
def find_performed_migrations
diff --git a/migrations/core/lib/migrations/database/offline_connection.rb b/migrations/core/lib/migrations/database/offline_connection.rb
deleted file mode 100644
index 1d4e35c6cb6..00000000000
--- a/migrations/core/lib/migrations/database/offline_connection.rb
+++ /dev/null
@@ -1,31 +0,0 @@
-# frozen_string_literal: true
-
-module Migrations
- module Database
- class OfflineConnection
- def initialize
- @parametrized_insert_statements = []
- end
-
- def close
- @parametrized_insert_statements = nil
- end
-
- def closed?
- @parametrized_insert_statements.nil?
- end
-
- def insert(sql, parameters = [])
- @parametrized_insert_statements << [sql, parameters]
- end
-
- def parametrized_insert_statements
- @parametrized_insert_statements
- end
-
- def clear!
- @parametrized_insert_statements.clear if @parametrized_insert_statements
- end
- end
- end
-end
diff --git a/migrations/core/migrations-core.gemspec b/migrations/core/migrations-core.gemspec
index 7d465129057..98d2b080f04 100644
--- a/migrations/core/migrations-core.gemspec
+++ b/migrations/core/migrations-core.gemspec
@@ -13,11 +13,12 @@ Gem::Specification.new do |s|
s.add_dependency "activesupport"
s.add_dependency "colored2"
+ s.add_dependency "concurrent-ruby"
s.add_dependency "digest-xxhash"
s.add_dependency "extralite-bundle"
s.add_dependency "i18n"
+ s.add_dependency "json"
s.add_dependency "lru_redux"
- s.add_dependency "oj"
s.add_dependency "samovar"
s.add_dependency "unicode-display_width"
s.add_dependency "zeitwerk"
diff --git a/migrations/core/spec/lib/common/fork_manager_spec.rb b/migrations/core/spec/lib/common/fork_manager_spec.rb
index 54969acc5cb..25196b24202 100644
--- a/migrations/core/spec/lib/common/fork_manager_spec.rb
+++ b/migrations/core/spec/lib/common/fork_manager_spec.rb
@@ -55,4 +55,16 @@ RSpec.describe Migrations::ForkManager do
read_io.close
end
end
+
+ describe ".with_batched_forks" do
+ it "runs the after-fork parent hooks even when the block raises" do
+ ran = false
+ described_class.after_fork_parent { ran = true }
+
+ expect { described_class.with_batched_forks { raise "fork exploded" } }.to raise_error(
+ "fork exploded",
+ )
+ expect(ran).to be(true)
+ end
+ end
end
diff --git a/migrations/core/spec/lib/common/reporting/plain_spec.rb b/migrations/core/spec/lib/common/reporting/plain_spec.rb
index 3e00dc1dd83..76a72f99beb 100644
--- a/migrations/core/spec/lib/common/reporting/plain_spec.rb
+++ b/migrations/core/spec/lib/common/reporting/plain_spec.rb
@@ -106,4 +106,12 @@ RSpec.describe Migrations::Reporting::Plain do
expect(lines).to include(" Users 100% (800/800)")
end
+
+ it "prints a finishing-up line and a run summary" do
+ reporter.finalizing { nil }
+ reporter.report_summary(runtime: 138.0, total: 24, failed: 2, skipped: 1)
+
+ expect(lines).to include("Finishing up…")
+ expect(lines.last).to eq("Total: 24 steps, 2 failed, 1 skipped (2:18)")
+ end
end
diff --git a/migrations/core/spec/lib/common/reporting/tui_spec.rb b/migrations/core/spec/lib/common/reporting/tui_spec.rb
index bb205a688a2..30cce3f4551 100644
--- a/migrations/core/spec/lib/common/reporting/tui_spec.rb
+++ b/migrations/core/spec/lib/common/reporting/tui_spec.rb
@@ -237,6 +237,91 @@ RSpec.describe Migrations::Reporting::Tui do
end
end
+ describe "the concurrency indicator" do
+ let(:count_indicator) { /\d+×/ }
+
+ def running_step(title, concurrency)
+ renderer.apply([:start, title, title])
+ renderer.apply([:progress_begin, title, 1000])
+ renderer.apply([:concurrency, title, concurrency]) if concurrency
+ renderer.apply([:progress, title, 500, 0, 0, 0])
+ end
+
+ it "renders the fork count, e.g. 4×, for a step using more than one fork" do
+ running_step("Posts", 4)
+ renderer.repaint
+ expect(screen.content_rows.first).to include("4×")
+ end
+
+ it "shows nothing in the column for a single-fork step" do
+ running_step("Posts", 1)
+ renderer.repaint
+ expect(screen.content_rows.first).not_to match(count_indicator)
+ end
+
+ it "shows nothing when the step never reports a count" do
+ running_step("Posts", nil)
+ renderer.repaint
+ expect(screen.content_rows.first).not_to match(count_indicator)
+ end
+
+ it "updates the count live when it's re-reported" do
+ running_step("Posts", 8)
+ renderer.repaint
+ expect(screen.content_rows.first).to include("8×")
+
+ renderer.apply([:concurrency, "Posts", 2])
+ renderer.repaint
+ row = screen.content_rows.first
+ expect(row).to include("2×")
+ expect(row).not_to include("8×")
+ end
+
+ it "colors the count when color is on" do
+ running_step("Posts", 8)
+ renderer.repaint
+ expect(io.string).to include("#{Migrations::Reporting::Tui::Ansi::MAGENTA}8×")
+ end
+
+ it "does not affect the count column shared with finished rows" do
+ renderer.apply([:start, "Done", "Done"])
+ renderer.apply([:progress_begin, "Done", 1_000_000])
+ renderer.apply([:progress, "Done", 1_000_000, 0, 0, 0])
+ renderer.apply([:finish, "Done", :done])
+ running_step("Busy", 8)
+ renderer.repaint
+
+ rows = screen.content_rows
+ done = rows.find { |r| r.include?("Done") }
+ busy = rows.find { |r| r.include?("Busy") }
+ # the count sits after the shared columns, so the totals still line up
+ expect(busy.index("1,000") + "1,000".length).to eq(
+ done.index("1,000,000") + "1,000,000".length,
+ )
+ end
+
+ describe "without color" do
+ around do |example|
+ original = ENV["NO_COLOR"]
+ ENV["NO_COLOR"] = "1"
+ example.run
+ original.nil? ? ENV.delete("NO_COLOR") : ENV["NO_COLOR"] = original
+ end
+
+ it "renders the count as plain text, with no color codes" do
+ no_color = described_class.new(output: io, width: 120, clock: -> { time[0] })
+ no_color.apply([:start, "Posts", "Posts"])
+ no_color.apply([:progress_begin, "Posts", 1000])
+ no_color.apply([:concurrency, "Posts", 8])
+ no_color.apply([:progress, "Posts", 500, 0, 0, 0])
+ no_color.repaint
+
+ expect(screen.content_rows.first).to include("8×")
+ expect(io.string).not_to include(Migrations::Reporting::Tui::Ansi::MAGENTA)
+ end
+ end
+ end
+
describe "notices" do
it "renders a permanent line prefixed with the step title" do
renderer.apply([:start, "Posts", "Posts"])
@@ -395,6 +480,47 @@ RSpec.describe Migrations::Reporting::Tui do
end
end
end
+
+ describe "the finishing-up status and run summary" do
+ it "shows a transient finishing-up line while draining, replaced by the summary" do
+ renderer.apply([:finalizing_begin])
+ renderer.repaint
+ expect(screen.content_rows).to include(a_string_matching(/Finishing up/))
+
+ # the real flow ends `finalizing` and prints the summary in the same frame
+ renderer.apply([:finalizing_end])
+ renderer.apply([:summary, 5.0, 3, 0, 0])
+ renderer.repaint
+ expect(screen.content_rows).not_to include(a_string_matching(/Finishing up/))
+ expect(screen.content_rows).to include(a_string_matching(/Total.*3 steps/))
+ end
+
+ it "prints a summary with the runtime under the step-duration column" do
+ renderer.apply([:start, "Users", "Users"])
+ renderer.apply([:progress_begin, "Users", 1000])
+ renderer.apply([:progress, "Users", 1000, 0, 0, 0])
+ at(3.0)
+ renderer.apply([:finish, "Users", :done])
+ renderer.apply([:summary, 138.0, 24, 0, 0])
+ renderer.repaint
+
+ summary = screen.content_rows.find { |r| r.include?("Total") }
+ step = screen.content_rows.find { |r| r.include?("Users") }
+ expect(summary).to include("24 steps")
+ expect(summary).to include("2:18") # 138 seconds runtime
+ # the runtime lines up under the per-step durations
+ expect(summary.index(/\d+:\d\d/)).to eq(step.index(/\d+:\d\d/))
+ end
+
+ it "notes failed and skipped steps in the summary" do
+ renderer.apply([:summary, 60.0, 24, 2, 1])
+ renderer.repaint
+
+ row = screen.content_rows.find { |r| r.include?("Total") }
+ expect(row).to include("2 failed")
+ expect(row).to include("1 skipped")
+ end
+ end
end
# Facade-level behaviour exercises the real queue + render thread. Each test
@@ -426,6 +552,23 @@ RSpec.describe Migrations::Reporting::Tui do
expect(row).to include("800 warnings")
end
+ it "coalesces progress instead of queuing one event per update" do
+ step = reporter.start_step("Users")
+ queue = reporter.instance_variable_get(:@queue)
+
+ step.with_progress(max_progress: 10_000) do |progress|
+ 10_000.times { progress.update(increment_by: 1) }
+ # the 10k updates collapse into one coalesced slot, so the queue never
+ # fills and the render thread keeps repainting
+ expect(queue.size).to be < 50
+ end
+ step.finish
+ reporter.close
+
+ row = final_screen.content_rows.find { |r| r.start_with?("✓ Users") }
+ expect(row).to include("10,000")
+ end
+
it "renders an interrupted line when finish runs during a SignalException" do
step = reporter.start_step("Posts")
step.with_progress(max_progress: 1000) { |progress| progress.update(increment_by: 410) }
diff --git a/migrations/core/spec/lib/conversion/base_spec.rb b/migrations/core/spec/lib/conversion/base_spec.rb
index ba728de0f23..7002baca171 100644
--- a/migrations/core/spec/lib/conversion/base_spec.rb
+++ b/migrations/core/spec/lib/conversion/base_spec.rb
@@ -2,7 +2,6 @@
RSpec.describe Migrations::Conversion::Base do
describe "#run" do
- let(:offline_connection) { Migrations::Database::OfflineConnection.new }
let(:converter) do
TemporaryConverterModule::Converter.new({ intermediate_db: { path: "intermediate.db" } })
end
@@ -12,55 +11,12 @@ RSpec.describe Migrations::Conversion::Base do
"TemporaryConverterModule",
Module.new do
const_set("Converter", Class.new(Migrations::Conversion::Base))
- const_set(
- "Topics",
- Class.new(Migrations::Conversion::ProgressStep) do
- title "Converting topics"
- # forces `execute_in_parallel?` to consult the pool's size; with
- # `max_progress` below the parallel threshold the step still runs
- # serially, so a missing pool fails fast without forking workers
- run_in_parallel true
-
- source do
- def max_progress
- 5
- end
-
- def items
- Array.new(5) { |index| { id: index } }
- end
- end
-
- processor do
- def process(item)
- Migrations::Database::IntermediateDB.insert(
- "INSERT INTO topics (original_id) VALUES (?)",
- item[:id],
- )
- end
- end
- end,
- )
- const_set(
- "Users",
- Class.new(Migrations::Conversion::Step) do
- title "Converting users"
-
- def execute
- Migrations::Database::IntermediateDB.insert(
- "INSERT INTO users (original_id) VALUES (?)",
- 1,
- )
- end
- end,
- )
+ const_set("Topics", Class.new(Migrations::Conversion::Step) { title "Converting topics" })
+ const_set("Users", Class.new(Migrations::Conversion::Step) { title "Converting users" })
end,
)
- Migrations::Database::IntermediateDB.setup(offline_connection)
- # `run` closes the IntermediateDB in its `ensure`, which would discard
- # the recorded insert statements before they can be verified
- allow(offline_connection).to receive(:close)
+ Migrations::Database::IntermediateDB.setup(nil)
allow(converter).to receive(:create_database)
end
@@ -69,9 +25,15 @@ RSpec.describe Migrations::Conversion::Base do
Object.send(:remove_const, "TemporaryConverterModule")
end
- it "creates one pool and one reporter per run and wires them through both executor kinds" do
+ it "builds one reporter, then runs the scheduler over the filtered steps" do
+ scheduler = instance_double(Migrations::Conversion::StepScheduler, run: nil)
+ scheduler_args = nil
+ allow(Migrations::Conversion::StepScheduler).to receive(:new) do |**kwargs|
+ scheduler_args = kwargs
+ scheduler
+ end
+
reporter = nil
- allow(Migrations::Conversion::WorkerPool).to receive(:new).and_call_original
allow(Migrations::Reporting::Factory).to receive(
:build,
).and_wrap_original do |original, **kwargs|
@@ -80,17 +42,17 @@ RSpec.describe Migrations::Conversion::Base do
reporter
end
- expect { converter.run }.to output(/Converting topics.*Converting users/m).to_stdout
+ converter.run(skip_steps: ["users"], max_parallel_steps: 3)
- expect(Migrations::Conversion::WorkerPool).to have_received(:new).once
expect(Migrations::Reporting::Factory).to have_received(:build).once
+ expect(scheduler).to have_received(:run).once
expect(reporter).to have_received(:close).once
- expect(offline_connection.parametrized_insert_statements).to eq(
- [
- *Array.new(5) { |index| ["INSERT INTO topics (original_id) VALUES (?)", [index]] },
- ["INSERT INTO users (original_id) VALUES (?)", [1]],
- ],
- )
+
+ expect(scheduler_args[:step_classes]).to eq([TemporaryConverterModule::Topics])
+ expect(scheduler_args[:max_parallel_steps]).to eq(3)
+ expect(scheduler_args[:budget]).to be > 0
+ expect(scheduler_args[:reporter]).to be(reporter)
+ expect(scheduler_args[:step_factory]).to respond_to(:call)
end
end
@@ -103,7 +65,7 @@ RSpec.describe Migrations::Conversion::Base do
Module.new do
const_set("Converter", Class.new(Migrations::Conversion::Base))
const_set("Categories", Class.new(Migrations::Conversion::Step))
- const_set("Topics", Class.new(Migrations::Conversion::ProgressStep))
+ const_set("Topics", Class.new(Migrations::Conversion::Step))
const_set("Users", Class.new(Migrations::Conversion::Step))
const_set("SomeHelper", Class.new)
end,
@@ -112,7 +74,7 @@ RSpec.describe Migrations::Conversion::Base do
after { Object.send(:remove_const, "TemporaryConverterModule") }
- it "discovers both `Step` and `ProgressStep` subclasses" do
+ it "discovers `Step` subclasses" do
expect(converter.steps).to contain_exactly(
TemporaryConverterModule::Categories,
TemporaryConverterModule::Topics,
diff --git a/migrations/core/spec/lib/conversion/consolidator_spec.rb b/migrations/core/spec/lib/conversion/consolidator_spec.rb
new file mode 100644
index 00000000000..548e539c8f6
--- /dev/null
+++ b/migrations/core/spec/lib/conversion/consolidator_spec.rb
@@ -0,0 +1,40 @@
+# frozen_string_literal: true
+
+RSpec.describe Migrations::Conversion::Consolidator do
+ let(:shard_manager) { instance_double(Migrations::Conversion::ShardManager, discard: nil) }
+ let(:connection) do
+ instance_double(
+ Migrations::Database::Connection,
+ merge_database: nil,
+ tables: %w[topics config schema_migrations],
+ )
+ end
+ let(:fork_mutex) { Mutex.new }
+
+ it "merges every enqueued shard into the run connection and discards each" do
+ consolidator = described_class.new(shard_manager, connection, fork_mutex)
+ consolidator.enqueue(%w[a.db b.db])
+ consolidator.enqueue(%w[c.db])
+ errors = consolidator.drain
+
+ # `config` and `schema_migrations` are excluded from the merge
+ expect(connection).to have_received(:merge_database).with("a.db", tables: %w[topics])
+ expect(connection).to have_received(:merge_database).with("b.db", tables: %w[topics])
+ expect(connection).to have_received(:merge_database).with("c.db", tables: %w[topics])
+ expect(shard_manager).to have_received(:discard).with("a.db")
+ expect(shard_manager).to have_received(:discard).with("c.db")
+ expect(errors).to be_empty
+ end
+
+ it "collects a merge error and still discards the shard" do
+ boom = StandardError.new("merge boom")
+ allow(connection).to receive(:merge_database).with("bad.db", tables: anything).and_raise(boom)
+
+ consolidator = described_class.new(shard_manager, connection, fork_mutex)
+ consolidator.enqueue(%w[bad.db])
+ errors = consolidator.drain
+
+ expect(errors).to eq([boom])
+ expect(shard_manager).to have_received(:discard).with("bad.db")
+ end
+end
diff --git a/migrations/core/spec/lib/conversion/parallel_job_spec.rb b/migrations/core/spec/lib/conversion/parallel_job_spec.rb
deleted file mode 100644
index cdf1bca6c99..00000000000
--- a/migrations/core/spec/lib/conversion/parallel_job_spec.rb
+++ /dev/null
@@ -1,95 +0,0 @@
-# frozen_string_literal: true
-
-RSpec.describe Migrations::Conversion::ParallelJob do
- subject(:job) { described_class.new(processor) }
-
- let(:processor) { instance_double(Migrations::Conversion::ProgressStep::Processor) }
- let(:item) { { key: "value" } }
- let(:tracker) { instance_double(Migrations::Conversion::StepTracker) }
- let(:stats) { Migrations::Conversion::StepStats.new }
- let(:intermediate_db) { class_double(Migrations::Database::IntermediateDB).as_stubbed_const }
-
- before do
- allow(processor).to receive(:tracker).and_return(tracker)
- allow(processor).to receive(:setup)
-
- allow(tracker).to receive(:reset_stats!)
- allow(tracker).to receive(:log_error)
- allow(tracker).to receive(:stats).and_return(stats)
-
- allow(intermediate_db).to receive(:setup)
- allow(intermediate_db).to receive(:with_connection).and_yield
- allow(intermediate_db).to receive(:close)
- end
-
- describe "#setup" do
- it "sets up `OfflineConnection` as `IntermediateDB` connection" do
- job.setup
-
- expect(intermediate_db).to have_received(:setup).with(
- an_instance_of(Migrations::Database::OfflineConnection),
- )
- end
-
- it "runs the framework setup before the processor's `setup`" do
- job.setup
-
- expect(intermediate_db).to have_received(:setup).ordered
- expect(processor).to have_received(:setup).ordered
- end
-
- it "runs the processor's `setup` through the setup guard" do
- job.setup
-
- expect(intermediate_db).to have_received(:with_connection).with(
- an_instance_of(Migrations::Conversion::SetupGuard::NoWriteConnection),
- )
- expect(processor).to have_received(:setup)
- end
- end
-
- describe "#run" do
- let(:offline_connection) { instance_double(Migrations::Database::OfflineConnection) }
-
- before do
- allow(Migrations::Database::OfflineConnection).to receive(:new).and_return(offline_connection)
- allow(offline_connection).to receive(:clear!)
-
- allow(processor).to receive(:process)
- allow(offline_connection).to receive(:parametrized_insert_statements).and_return(
- [["SQL", [1, 2]], ["SQL", [2, 3]]],
- )
- end
-
- it "resets stats and clears the offline connection" do
- job.run(item)
-
- expect(tracker).to have_received(:reset_stats!)
- expect(offline_connection).to have_received(:clear!)
- end
-
- it "processes an item and logs errors if exceptions occur" do
- allow(processor).to receive(:process).and_raise(StandardError.new("error"))
-
- job.run(item)
-
- expect(tracker).to have_received(:log_error).with(
- "Failed to process item",
- exception: an_instance_of(StandardError),
- details: item,
- )
- end
-
- it "returns the parametrized insert statements and stats" do
- result = job.run(item)
-
- expect(result).to eq([[["SQL", [1, 2]], ["SQL", [2, 3]]], stats])
- end
- end
-
- describe "#cleanup" do
- it "can be called without errors" do
- expect { job.cleanup }.not_to raise_error
- end
- end
-end
diff --git a/migrations/core/spec/lib/conversion/partitioner_spec.rb b/migrations/core/spec/lib/conversion/partitioner_spec.rb
new file mode 100644
index 00000000000..4369f7766f4
--- /dev/null
+++ b/migrations/core/spec/lib/conversion/partitioner_spec.rb
@@ -0,0 +1,104 @@
+# frozen_string_literal: true
+
+RSpec.describe Migrations::Conversion::Partitioner do
+ # A stand-in for the source DB adapter, returning canned answers for the
+ # primitives the partitioner calls. It has no `boundaries_by_scan`, so its scan
+ # path goes through the streaming fallback.
+ def build_adapter(min: nil, max: nil, estimate: 0, total: 0, keys: [])
+ Class
+ .new do
+ define_method(:partition_bounds) { |_key, _from, _base| [min, max] }
+ define_method(:estimated_row_count) { |_from| estimate }
+ define_method(:count_all) { |_from, where:| total }
+ define_method(:chunk_filter) { |*| "" }
+ define_method(:each_partition_key) { |_key, _from, _base, &block| keys.each(&block) }
+ end
+ .new
+ end
+
+ def partitioner(adapter, key)
+ described_class.new(adapter, key:, from: "things", base: nil)
+ end
+
+ it "strides a dense numeric key evenly by value, without a scan" do
+ adapter = build_adapter(min: 0, max: 99, estimate: 100)
+ expect(partitioner(adapter, :id).boundaries(4)).to eq([0, 25, 50, 75])
+ end
+
+ it "samples the sorted key when a numeric key is too sparse for even chunks" do
+ # 4 rows scattered across a million-wide range: value-sized chunks would
+ # leave most forks empty, so it samples the actual ids instead
+ adapter =
+ build_adapter(
+ min: 0,
+ max: 1_000_000,
+ estimate: 4,
+ total: 4,
+ keys: [0, 10, 999_999, 1_000_000],
+ )
+ expect(partitioner(adapter, :id).boundaries(2)).to eq([0, 999_999])
+ end
+
+ it "strides a numeric key when the row estimate is unavailable (unanalysed table)" do
+ # estimate 0 stands in for reltuples 0/-1: don't fall into the scan on a guess
+ adapter = build_adapter(min: 0, max: 99, estimate: 0)
+ expect(partitioner(adapter, :id).boundaries(4)).to eq([0, 25, 50, 75])
+ end
+
+ it "samples the sorted key for a non-numeric key" do
+ adapter = build_adapter(min: "a", max: "d", total: 4, keys: %w[a b c d])
+ expect(partitioner(adapter, :id).boundaries(2)).to eq(%w[a c])
+ end
+
+ it "samples a composite key without asking for numeric bounds" do
+ adapter = build_adapter(total: 4, keys: [[1, 10], [1, 20], [2, 5], [3, 7]])
+ expect(partitioner(adapter, %i[topic_id user_id]).boundaries(2)).to eq([[1, 10], [2, 5]])
+ end
+
+ it "is empty for an empty source" do
+ expect(partitioner(build_adapter(min: nil, max: nil), :id).boundaries(4)).to eq([])
+ end
+
+ it "prefers the adapter's fast scan path when it offers one" do
+ # A valid PartitionSource with `boundaries_by_scan` but no `each_partition_key`,
+ # so the test fails (NoMethodError) unless the partitioner takes the fast path.
+ adapter =
+ Class
+ .new do
+ define_method(:partition_bounds) { |*| [nil, nil] }
+ define_method(:estimated_row_count) { |*| 0 }
+ define_method(:count_all) { |*, **| 0 }
+ define_method(:chunk_filter) { |*| "" }
+ define_method(:boundaries_by_scan) do |_key, _from, _base, count|
+ [10, 20, 30].first(count)
+ end
+ end
+ .new
+
+ expect(partitioner(adapter, %i[a b]).boundaries(2)).to eq([10, 20])
+ end
+
+ it "raises when there is no source to compute boundaries from" do
+ expect { partitioner(nil, :id).boundaries(4) }.to raise_error(
+ described_class::UnsupportedSourceError,
+ /doesn't implement the PartitionSource interface/,
+ )
+ end
+
+ it "raises, naming the gap, when the adapter is missing a primitive" do
+ adapter =
+ Class
+ .new do
+ def partition_bounds(*)
+ [0, 99]
+ end
+ # no estimated_row_count / count_all / each_partition_key / boundaries_by_scan
+ end
+ .new
+
+ expect { partitioner(adapter, :id).boundaries(4) }.to raise_error(
+ described_class::UnsupportedSourceError,
+ /estimated_row_count/,
+ )
+ end
+end
diff --git a/migrations/core/spec/lib/conversion/progress_step_executor_spec.rb b/migrations/core/spec/lib/conversion/progress_step_executor_spec.rb
deleted file mode 100644
index a10ad1569e0..00000000000
--- a/migrations/core/spec/lib/conversion/progress_step_executor_spec.rb
+++ /dev/null
@@ -1,144 +0,0 @@
-# frozen_string_literal: true
-
-RSpec.describe Migrations::Conversion::ProgressStepExecutor do
- subject(:executor) { described_class.new(step, pool:, reporter:) }
-
- let(:pool_size) { 2 }
- let(:pool) { Migrations::Conversion::WorkerPool.new(size: pool_size) }
- let(:reporter) { Migrations::Reporting::Plain.new(output: StringIO.new) }
- let(:item_count) { 30 }
- let(:offline_connection) { Migrations::Database::OfflineConnection.new }
- let(:step) { step_class.new(settings: { item_count: }) }
-
- before { Migrations::Database::IntermediateDB.setup(offline_connection) }
- after { Migrations::Database::IntermediateDB.setup(nil) }
-
- def define_fixture_step(run_parallel)
- Class.new(Migrations::Conversion::ProgressStep) do
- title "Fixture step"
- run_in_parallel run_parallel
-
- source do
- def max_progress
- settings[:item_count]
- end
-
- def items
- Array.new(settings[:item_count]) { |index| { id: index } }
- end
- end
-
- processor do
- def setup
- @prefix = "item"
- end
-
- def process(item)
- Migrations::Database::IntermediateDB.insert(
- "INSERT INTO items (name) VALUES (?)",
- "#{@prefix}-#{item[:id]}",
- )
- end
- end
- end
- end
-
- def expected_insert_statements
- Array.new(item_count) { |index| ["INSERT INTO items (name) VALUES (?)", ["item-#{index}"]] }
- end
-
- def execute_quietly
- original_stdout = $stdout
- $stdout = StringIO.new
- executor.execute
- ensure
- $stdout = original_stdout
- end
-
- describe "#execute" do
- context "when the step runs serially" do
- let(:step_class) { define_fixture_step(false) }
-
- it "runs the processor's `setup` and processes all items in order" do
- execute_quietly
-
- expect(offline_connection.parametrized_insert_statements).to eq(expected_insert_statements)
- end
- end
-
- context "when the step runs in parallel" do
- let(:step_class) { define_fixture_step(true) }
-
- it "forks workers and produces the same inserts as a serial run" do
- allow(Migrations::ForkManager).to receive(:fork).and_call_original
-
- execute_quietly
-
- expect(Migrations::ForkManager).to have_received(:fork).at_least(:once)
- expect(offline_connection.parametrized_insert_statements).to match_array(
- expected_insert_statements,
- )
- end
- end
-
- context "when calculating the max progress is slow" do
- let(:step_class) { define_fixture_step(false) }
- let(:events) { [] }
- let(:reporter) do
- handle = instance_double(Migrations::Reporting::Reporter::StepHandle)
- allow(handle).to receive(:notice) { events << :notice }
- allow(handle).to receive(:with_progress) { events << :with_progress }
- allow(handle).to receive(:finish) { events << :finish }
-
- instance_double(Migrations::Reporting::Plain).tap do |reporter|
- allow(reporter).to receive(:start_step) do
- events << :start_step
- handle
- end
- end
- end
-
- it "announces the step before the runtime notice and reports the step end last" do
- allow(Time).to receive(:now).and_return(Time.at(0), Time.at(10))
-
- executor.execute
-
- expect(events).to eq(%i[start_step notice with_progress finish])
- end
- end
-
- context "when the parallel threshold scales with the pool size" do
- let(:step_class) { define_fixture_step(true) }
-
- context "with no more items than `pool.size * 10`" do
- let(:item_count) { pool_size * 10 }
-
- it "runs serially despite `run_in_parallel`" do
- allow(Migrations::ForkManager).to receive(:fork).and_call_original
-
- execute_quietly
-
- expect(Migrations::ForkManager).not_to have_received(:fork)
- expect(offline_connection.parametrized_insert_statements).to eq(
- expected_insert_statements,
- )
- end
- end
-
- context "with one item more than `pool.size * 10`" do
- let(:item_count) { pool_size * 10 + 1 }
-
- it "runs in parallel" do
- allow(Migrations::ForkManager).to receive(:fork).and_call_original
-
- execute_quietly
-
- expect(Migrations::ForkManager).to have_received(:fork).exactly(pool_size).times
- expect(offline_connection.parametrized_insert_statements).to match_array(
- expected_insert_statements,
- )
- end
- end
- end
- end
-end
diff --git a/migrations/core/spec/lib/conversion/progress_step_spec.rb b/migrations/core/spec/lib/conversion/progress_step_spec.rb
deleted file mode 100644
index 2c5070a19f8..00000000000
--- a/migrations/core/spec/lib/conversion/progress_step_spec.rb
+++ /dev/null
@@ -1,419 +0,0 @@
-# frozen_string_literal: true
-
-RSpec.describe Migrations::Conversion::ProgressStep do
- def define_step(&block)
- Class.new(described_class, &block)
- end
-
- describe ".source" do
- it "defines methods on the source role only" do
- step_class =
- define_step do
- source do
- def items
- [1, 2, 3]
- end
- end
- end
-
- source = step_class.source_class.new
- expect(source.items).to eq([1, 2, 3])
-
- expect(step_class.new).not_to respond_to(:items)
- expect(step_class.processor_class.new).not_to respond_to(:items)
- end
-
- it "defines private methods on the source role" do
- step_class =
- define_step do
- source do
- def items
- [build_item]
- end
-
- private
-
- def build_item
- :item
- end
- end
- end
-
- source = step_class.source_class.new
- expect(source.items).to eq([:item])
- expect(source.respond_to?(:build_item)).to be(false)
- expect(source.respond_to?(:build_item, true)).to be(true)
- end
-
- it "raises an error when called twice" do
- expect do
- define_step do
- source {}
- source {}
- end
- end.to raise_error(ArgumentError, /`source` is already defined/)
- end
- end
-
- describe ".processor" do
- it "defines methods on the processor role only" do
- step_class =
- define_step do
- processor do
- def process(item)
- item * 2
- end
- end
- end
-
- processor = step_class.processor_class.new
- expect(processor.process(21)).to eq(42)
-
- expect(step_class.new).not_to respond_to(:process)
- expect(step_class.source_class.new).not_to respond_to(:process)
- end
-
- it "defines private methods on the processor role" do
- step_class =
- define_step do
- processor do
- def process(item)
- transform(item)
- end
-
- private
-
- def transform(item)
- item.to_s
- end
- end
- end
-
- processor = step_class.processor_class.new
- expect(processor.process(1)).to eq("1")
- expect(processor.respond_to?(:transform)).to be(false)
- expect(processor.respond_to?(:transform, true)).to be(true)
- end
-
- it "raises an error when called twice" do
- expect do
- define_step do
- processor {}
- processor {}
- end
- end.to raise_error(ArgumentError, /`processor` is already defined/)
- end
- end
-
- describe ".helpers" do
- it "mixes the helpers into both roles" do
- step_class =
- define_step do
- helpers do
- def double_it(value)
- value * 2
- end
- end
-
- source do
- def items
- [double_it(1)]
- end
- end
-
- processor do
- def process(item)
- double_it(item)
- end
- end
- end
-
- expect(step_class.source_class.new.items).to eq([2])
- expect(step_class.processor_class.new.process(2)).to eq(4)
- end
-
- it "works independently of the order of the DSL calls" do
- step_class =
- define_step do
- source do
- def items
- [double_it(1)]
- end
- end
-
- helpers do
- def double_it(value)
- value * 2
- end
- end
- end
-
- expect(step_class.source_class.new.items).to eq([2])
- end
-
- it "raises an error when called twice" do
- expect do
- define_step do
- helpers {}
- helpers {}
- end
- end.to raise_error(ArgumentError, /`helpers` is already defined/)
- end
- end
-
- describe "role isolation" do
- it "raises a NameError when the processor calls a source method" do
- step_class =
- define_step do
- source do
- def items
- []
- end
-
- private
-
- def source_helper
- "source-side state"
- end
- end
-
- processor do
- def process(item)
- source_helper
- end
- end
- end
-
- processor = step_class.processor_class.new
- expect { processor.process(1) }.to raise_error(NameError, /source_helper/)
- end
-
- it "does not expose instance variables set in `items` to the processor" do
- step_class =
- define_step do
- source do
- def items
- @leaked_state = "set in items"
- [1]
- end
- end
-
- processor do
- def process(item)
- instance_variable_defined?(:@leaked_state)
- end
- end
- end
-
- step = step_class.new
- step.source.items
-
- expect(step.create_processor.process(1)).to be(false)
- end
-
- it "does not capture local variables in methods defined with `def`" do
- local_state = "step-level local"
-
- step_class =
- define_step do
- source do
- def items
- local_state
- end
- end
- end
-
- expect { step_class.source_class.new.items }.to raise_error(NameError, /local_state/)
- expect(local_state).to eq("step-level local") # silence the unused variable warning
- end
- end
-
- describe "role defaults" do
- let(:step_class) { define_step }
-
- it "uses `nil` as default `max_progress`" do
- expect(step_class.source_class.new.max_progress).to be_nil
- end
-
- it "raises `NotImplementedError` when `items` is not defined" do
- expect { step_class.source_class.new.items }.to raise_error(NotImplementedError)
- end
-
- it "raises `NotImplementedError` when `process` is not defined" do
- expect { step_class.processor_class.new.process(1) }.to raise_error(NotImplementedError)
- end
-
- it "uses a no-op as default `setup`" do
- expect { step_class.processor_class.new.setup }.not_to raise_error
- end
- end
-
- describe "reads_table defaults" do
- # Stands in for the source DB adapter, recording the calls it receives (the
- # adapter builds the actual SQL; that's covered in its own spec).
- let(:source_db) do
- Class
- .new do
- attr_reader :select_args, :count_args
-
- def select_all(table, where:)
- @select_args = [table, where]
- [:row]
- end
-
- def count_all(table, where:)
- @count_args = [table, where]
- 4
- end
- end
- .new
- end
-
- it "reads the whole table through the adapter" do
- step_class = define_step { source { reads_table "things", where: "active" } }
- source = step_class.source_class.new(source_db:)
-
- expect(source.items).to eq([:row])
- expect(source_db.select_args).to eq(%w[things active])
-
- expect(source.max_progress).to eq(4)
- expect(source_db.count_args).to eq(%w[things active])
- end
-
- it "passes no filter when `reads_table` has no `where`" do
- step_class = define_step { source { reads_table "things" } }
- source = step_class.source_class.new(source_db:)
-
- source.items
- expect(source_db.select_args).to eq(["things", nil])
- end
- end
-
- describe "constant resolution" do
- # The role blocks of real steps are written inside a step class body, so
- # constants in their methods resolve through the step's lexical scope and
- # ancestry (`Step` defines `IntermediateDB` and `Enums`). A `Class.new`
- # block in this spec file would have the wrong lexical scope, hence the
- # string eval.
- before { Object.class_eval <<~RUBY }
- class ProgressStepConstantsFixture < Migrations::Conversion::ProgressStep
- source do
- def items
- [IntermediateDB, Enums]
- end
- end
-
- processor do
- def process(item)
- [IntermediateDB, Enums]
- end
- end
- end
- RUBY
-
- after { Object.send(:remove_const, :ProgressStepConstantsFixture) }
-
- it "resolves `IntermediateDB` and `Enums` inside role blocks via the step's ancestry" do
- expected = [Migrations::Database::IntermediateDB, Migrations::Database::IntermediateDB::Enums]
-
- expect(ProgressStepConstantsFixture.source_class.new.items).to eq(expected)
- expect(ProgressStepConstantsFixture.processor_class.new.process(nil)).to eq(expected)
- end
-
- it "does not define the constants on the role classes themselves" do
- expect(ProgressStepConstantsFixture.source_class).not_to have_constant(:IntermediateDB)
- expect(ProgressStepConstantsFixture.processor_class).not_to have_constant(:IntermediateDB)
- end
- end
-
- describe "#initialize" do
- it "routes args to the role that declares a matching setter" do
- step_class = define_step { source { attr_accessor :source_db } }
-
- settings = { a: 1 }
- step = step_class.new(settings:, source_db: "source db", unknown_arg: "dropped")
-
- expect(step.source.settings).to eq(settings)
- expect(step.source.source_db).to eq("source db")
-
- processor = step.create_processor
- expect(processor.settings).to eq(settings)
- expect(processor).not_to respond_to(:source_db)
- expect(processor).not_to respond_to(:unknown_arg)
- end
-
- it "keeps no per-step state on the coordinator" do
- step = define_step.new(settings: { a: 1 })
-
- expect(step).not_to be_a(Migrations::Conversion::Step)
- expect(step).not_to respond_to(:tracker)
- expect(step).not_to respond_to(:step)
- expect(step).not_to respond_to(:settings)
- expect(step).not_to respond_to(:execute)
- end
- end
-
- describe "#source" do
- it "returns the same source instance every time" do
- step = define_step.new
- expect(step.source).to be(step.source)
- end
- end
-
- describe "#create_processor" do
- it "creates a new processor with its own tracker on every call" do
- step = define_step.new
-
- processor1 = step.create_processor
- processor2 = step.create_processor
-
- expect(processor1).not_to be(processor2)
- expect(processor1.tracker).to be_a(Migrations::Conversion::StepTracker)
- expect(processor1.tracker).not_to be(processor2.tracker)
- end
- end
-
- describe "step subclasses" do
- it "inherits the roles of the parent step" do
- parent_class =
- define_step do
- source do
- def items
- [1]
- end
- end
-
- processor do
- def process(item)
- item + 1
- end
- end
- end
-
- step_class =
- Class.new(parent_class) do
- source do
- def max_progress
- 1
- end
- end
- end
-
- source = step_class.source_class.new
- expect(source.items).to eq([1])
- expect(source.max_progress).to eq(1)
- expect(step_class.processor_class.new.process(1)).to eq(2)
- end
- end
-
- describe ".run_in_parallel?" do
- it "defaults to false" do
- expect(define_step.run_in_parallel?).to be(false)
- end
-
- it "returns the configured value" do
- expect(define_step { run_in_parallel true }.run_in_parallel?).to be(true)
- expect(define_step { run_in_parallel false }.run_in_parallel?).to be(false)
- end
- end
-end
diff --git a/migrations/core/spec/lib/conversion/scheduler_integration_spec.rb b/migrations/core/spec/lib/conversion/scheduler_integration_spec.rb
new file mode 100644
index 00000000000..29b70bb198e
--- /dev/null
+++ b/migrations/core/spec/lib/conversion/scheduler_integration_spec.rb
@@ -0,0 +1,591 @@
+# frozen_string_literal: true
+
+require "extralite"
+require "timeout"
+
+RSpec.describe Migrations::Conversion::StepScheduler, :integration do
+ around do |example|
+ Dir.mktmpdir do |storage_path|
+ @db_path = File.join(storage_path, "intermediate.db")
+ migrations_path = File.join(storage_path, "migrations")
+ FileUtils.mkdir_p(migrations_path)
+ File.write(File.join(migrations_path, "001-schema.sql"), <<~SQL)
+ CREATE TABLE topics (id INTEGER);
+ CREATE TABLE users (id INTEGER);
+ CREATE TABLE notes (id INTEGER, body TEXT NOT NULL);
+ CREATE TABLE keyed (id INTEGER PRIMARY KEY, label TEXT);
+ CREATE TABLE log_entries (created_at TEXT, type INTEGER, message TEXT, exception TEXT, details TEXT);
+ SQL
+ Migrations::Database.migrate(@db_path, migrations_path:)
+
+ # The shard template is migrated fresh from the same schema (no data), so
+ # shards stay empty even when the run DB already has rows.
+ @shard_manager =
+ Migrations::Conversion::ShardManager.new(canonical_path: @db_path, migrations_path:)
+ @writer = Migrations::Database::Connection.new(path: @db_path)
+ Migrations::Database::IntermediateDB.setup(@writer)
+ begin
+ example.run
+ ensure
+ Migrations::Database::IntermediateDB.close
+ @shard_manager.cleanup
+ end
+ end
+ end
+
+ def rows(table)
+ db = Extralite::Database.new(@db_path)
+ db.query_splat("SELECT id FROM #{table} ORDER BY id")
+ ensure
+ db.close if db
+ end
+
+ def log_entry_count
+ db = Extralite::Database.new(@db_path)
+ db.query_single_splat("SELECT COUNT(*) FROM log_entries")
+ ensure
+ db.close if db
+ end
+
+ it "writes every row from two steps running concurrently" do
+ Object.const_set(
+ "TempIntegrationSteps",
+ Module.new do
+ const_set(
+ "Topics",
+ Class.new(Migrations::Conversion::Step) do
+ source do
+ def max_progress
+ 50
+ end
+
+ def items
+ Array.new(50) { |index| { id: index } }
+ end
+ end
+
+ processor do
+ def process(item)
+ Migrations::Database::IntermediateDB.insert(
+ "INSERT INTO topics (id) VALUES (?)",
+ item[:id],
+ )
+ end
+ end
+ end,
+ )
+ const_set(
+ "Users",
+ Class.new(Migrations::Conversion::Step) do
+ source do
+ def max_progress
+ 5
+ end
+
+ def items
+ Array.new(5) { |index| { id: index } }
+ end
+ end
+
+ processor do
+ def process(item)
+ Migrations::Database::IntermediateDB.insert(
+ "INSERT INTO users (id) VALUES (?)",
+ item[:id],
+ )
+ end
+ end
+ end,
+ )
+ end,
+ )
+
+ step_classes = [TempIntegrationSteps::Topics, TempIntegrationSteps::Users]
+ reporter = Migrations::Reporting::Factory.build(titles: step_classes.map(&:title))
+
+ expect do
+ Migrations::Conversion::StepScheduler.new(
+ step_classes:,
+ budget: 2,
+ reporter:,
+ step_factory: ->(step_class) { step_class.new },
+ shard_manager: @shard_manager,
+ writer: @writer,
+ ).run
+ ensure
+ reporter&.close
+ end.to output(/Converting topics/).to_stdout_from_any_process
+
+ # IntermediateDB writes are batched in a transaction; commit before reading
+ Migrations::Database::IntermediateDB.close
+
+ expect(rows("topics")).to eq((0..49).to_a)
+ expect(rows("users")).to eq((0..4).to_a)
+ ensure
+ Object.send(:remove_const, "TempIntegrationSteps")
+ end
+
+ it "splits a partitioned step across forks and merges every slice" do
+ Object.const_set(
+ "TempIntegrationSteps",
+ Module.new do
+ const_set(
+ "Topics",
+ Class.new(Migrations::Conversion::Step) do
+ source do
+ partition_by :id, from: "topics"
+
+ def all
+ Array.new(60) { |index| { id: index } }
+ end
+
+ # Give the coordinator `count` chunk lower bounds (the adapter does
+ # this for a real source); each fork reads only its own `[lower, upper)`.
+ def partition_boundaries(count)
+ size = (all.size.to_f / count).ceil
+ (0...count).map { |i| i * size }
+ end
+
+ def items
+ return all unless chunk
+
+ lower, upper = chunk
+ all.select { |it| it[:id] >= lower && (upper.nil? || it[:id] < upper) }
+ end
+
+ def max_progress
+ items.size
+ end
+ end
+
+ processor do
+ def process(item)
+ Migrations::Database::IntermediateDB.insert(
+ "INSERT INTO topics (id) VALUES (?)",
+ item[:id],
+ )
+ end
+ end
+ end,
+ )
+ end,
+ )
+
+ step_classes = [TempIntegrationSteps::Topics]
+ reporter = Migrations::Reporting::Factory.build(titles: step_classes.map(&:title))
+
+ Migrations::Conversion::StepScheduler.new(
+ step_classes:,
+ budget: 4,
+ reporter:,
+ step_factory: ->(step_class) { step_class.new },
+ shard_manager: @shard_manager,
+ writer: @writer,
+ ).run
+ reporter.close
+
+ Migrations::Database::IntermediateDB.close
+
+ expect(rows("topics")).to eq((0..59).to_a)
+ ensure
+ Object.send(:remove_const, "TempIntegrationSteps")
+ end
+
+ # The coordinator sizes the fork count from the boundaries, not the budget:
+ # `worker_count = [boundaries.size, 1].max`. These two cases exercise the
+ # collapse to a single worker.
+ it "collapses to one worker when a partitioned source yields fewer chunks than forks" do
+ Object.const_set(
+ "TempIntegrationSteps",
+ Module.new do
+ const_set(
+ "Topics",
+ Class.new(Migrations::Conversion::Step) do
+ source do
+ partition_by :id, from: "topics"
+
+ def all
+ Array.new(3) { |index| { id: index } }
+ end
+
+ # Only one boundary, though more forks are free.
+ def partition_boundaries(_count)
+ [0]
+ end
+
+ def items
+ return all unless chunk
+
+ lower, upper = chunk
+ all.select { |it| it[:id] >= lower && (upper.nil? || it[:id] < upper) }
+ end
+
+ def max_progress
+ items.size
+ end
+ end
+
+ processor do
+ def process(item)
+ Migrations::Database::IntermediateDB.insert(
+ "INSERT INTO topics (id) VALUES (?)",
+ item[:id],
+ )
+ end
+ end
+ end,
+ )
+ end,
+ )
+
+ step_classes = [TempIntegrationSteps::Topics]
+ reporter = Migrations::Reporting::Factory.build(titles: step_classes.map(&:title))
+
+ Migrations::Conversion::StepScheduler.new(
+ step_classes:,
+ budget: 4,
+ reporter:,
+ step_factory: ->(step_class) { step_class.new },
+ shard_manager: @shard_manager,
+ writer: @writer,
+ ).run
+ reporter.close
+
+ Migrations::Database::IntermediateDB.close
+
+ expect(rows("topics")).to eq([0, 1, 2]) # every row merged exactly once
+ ensure
+ Object.send(:remove_const, "TempIntegrationSteps")
+ end
+
+ it "runs a single empty worker for a partitioned step over an empty source" do
+ Object.const_set(
+ "TempIntegrationSteps",
+ Module.new do
+ const_set(
+ "Topics",
+ Class.new(Migrations::Conversion::Step) do
+ source do
+ partition_by :id, from: "topics"
+
+ # No rows, so no boundaries — the coordinator still runs one worker.
+ def partition_boundaries(_count)
+ []
+ end
+
+ def items
+ []
+ end
+
+ def max_progress
+ 0
+ end
+ end
+
+ processor do
+ def process(item)
+ Migrations::Database::IntermediateDB.insert(
+ "INSERT INTO topics (id) VALUES (?)",
+ item[:id],
+ )
+ end
+ end
+ end,
+ )
+ end,
+ )
+
+ step_classes = [TempIntegrationSteps::Topics]
+ reporter = Migrations::Reporting::Factory.build(titles: step_classes.map(&:title))
+
+ Migrations::Conversion::StepScheduler.new(
+ step_classes:,
+ budget: 4,
+ reporter:,
+ step_factory: ->(step_class) { step_class.new },
+ shard_manager: @shard_manager,
+ writer: @writer,
+ ).run
+ reporter.close
+
+ Migrations::Database::IntermediateDB.close
+
+ expect(rows("topics")).to eq([])
+ ensure
+ Object.send(:remove_const, "TempIntegrationSteps")
+ end
+
+ it "runs the step inline, without forking, when no_fork is set" do
+ Object.const_set(
+ "TempIntegrationSteps",
+ Module.new do
+ const_set(
+ "Topics",
+ Class.new(Migrations::Conversion::Step) do
+ source do
+ partition_by :id, from: "topics"
+
+ def all
+ Array.new(20) { |index| { id: index } }
+ end
+
+ def partition_boundaries(count)
+ size = (all.size.to_f / count).ceil
+ (0...count).map { |i| i * size }
+ end
+
+ def items
+ return all unless chunk
+
+ lower, upper = chunk
+ all.select { |it| it[:id] >= lower && (upper.nil? || it[:id] < upper) }
+ end
+
+ def max_progress
+ items.size
+ end
+ end
+
+ processor do
+ def process(item)
+ Migrations::Database::IntermediateDB.insert(
+ "INSERT INTO topics (id) VALUES (?)",
+ item[:id],
+ )
+ end
+ end
+ end,
+ )
+ end,
+ )
+
+ step_classes = [TempIntegrationSteps::Topics]
+ reporter = Migrations::Reporting::Factory.build(titles: step_classes.map(&:title))
+
+ allow(Migrations::ForkManager).to receive(:fork).and_call_original
+
+ Migrations::Conversion::StepScheduler.new(
+ step_classes:,
+ budget: 4,
+ reporter:,
+ step_factory: ->(step_class) { step_class.new },
+ shard_manager: @shard_manager,
+ writer: @writer,
+ no_fork: true,
+ ).run
+ reporter.close
+
+ Migrations::Database::IntermediateDB.close
+
+ expect(Migrations::ForkManager).not_to have_received(:fork)
+ expect(rows("topics")).to eq((0..19).to_a)
+ ensure
+ Object.send(:remove_const, "TempIntegrationSteps")
+ end
+
+ it "logs and skips a bad row instead of failing the step" do
+ Object.const_set(
+ "TempIntegrationSteps",
+ Module.new do
+ # The middle row's NULL `body` breaks the NOT NULL constraint on insert.
+ const_set(
+ "Notes",
+ Class.new(Migrations::Conversion::Step) do
+ source do
+ def items
+ [{ id: 1, body: "ok" }, { id: 2, body: nil }, { id: 3, body: "ok" }]
+ end
+ end
+
+ processor do
+ def process(item)
+ Migrations::Database::IntermediateDB.insert(
+ "INSERT INTO notes (id, body) VALUES (?, ?)",
+ item[:id],
+ item[:body],
+ )
+ end
+ end
+ end,
+ )
+ const_set(
+ "Users",
+ Class.new(Migrations::Conversion::Step) do
+ source do
+ def items
+ Array.new(5) { |index| { id: index } }
+ end
+ end
+
+ processor do
+ def process(item)
+ Migrations::Database::IntermediateDB.insert(
+ "INSERT INTO users (id) VALUES (?)",
+ item[:id],
+ )
+ end
+ end
+ end,
+ )
+ end,
+ )
+
+ step_classes = [TempIntegrationSteps::Notes, TempIntegrationSteps::Users]
+ reporter = Migrations::Reporting::Factory.build(titles: step_classes.map(&:title))
+
+ run =
+ lambda do
+ Migrations::Conversion::StepScheduler.new(
+ step_classes:,
+ budget: 2,
+ reporter:,
+ step_factory: ->(step_class) { step_class.new },
+ shard_manager: @shard_manager,
+ writer: @writer,
+ ).run
+ ensure
+ reporter&.close
+ end
+
+ expect { run.call }.not_to raise_error
+
+ Migrations::Database::IntermediateDB.close
+
+ expect(rows("notes")).to eq([1, 3])
+ expect(rows("users")).to eq((0..4).to_a)
+ expect(log_entry_count).to eq(1)
+ ensure
+ Object.send(:remove_const, "TempIntegrationSteps")
+ end
+
+ it "fails the step and finishes the run when a worker dies mid-stream" do
+ Object.const_set(
+ "TempIntegrationSteps",
+ Module.new do
+ # A worker that exits part-way closes its progress pipe, so the
+ # coordinator sees the non-zero exit and discards the half-written shard.
+ const_set(
+ "Crashing",
+ Class.new(Migrations::Conversion::Step) do
+ source do
+ def items
+ Array.new(300) { |index| { id: index } }
+ end
+ end
+
+ processor do
+ def process(item)
+ exit!(1) if item[:id] == 150
+ Migrations::Database::IntermediateDB.insert(
+ "INSERT INTO topics (id) VALUES (?)",
+ item[:id],
+ )
+ end
+ end
+ end,
+ )
+ const_set(
+ "Users",
+ Class.new(Migrations::Conversion::Step) do
+ source do
+ def items
+ Array.new(300) { |index| { id: index } }
+ end
+ end
+
+ processor do
+ def process(item)
+ Migrations::Database::IntermediateDB.insert(
+ "INSERT INTO users (id) VALUES (?)",
+ item[:id],
+ )
+ end
+ end
+ end,
+ )
+ end,
+ )
+
+ step_classes = [TempIntegrationSteps::Crashing, TempIntegrationSteps::Users]
+ reporter = Migrations::Reporting::Factory.build(titles: step_classes.map(&:title))
+
+ run =
+ lambda do
+ Migrations::Conversion::StepScheduler.new(
+ step_classes:,
+ budget: 2,
+ reporter:,
+ step_factory: ->(step_class) { step_class.new },
+ shard_manager: @shard_manager,
+ writer: @writer,
+ ).run
+ ensure
+ reporter&.close
+ end
+
+ # `Timeout` turns a hanging run into a failed example instead of a stuck suite.
+ expect { Timeout.timeout(30) { run.call } }.to raise_error(Migrations::Conversion::ConvertError)
+
+ Migrations::Database::IntermediateDB.close
+
+ expect(rows("users")).to eq((0..299).to_a)
+ # the crashed worker's half-written shard is discarded, not merged
+ expect(rows("topics")).to eq([])
+ ensure
+ Object.send(:remove_const, "TempIntegrationSteps")
+ end
+
+ it "adds new rows to an existing IntermediateDB without dropping or overwriting" do
+ # A row from a previous run already lives in the DB.
+ Migrations::Database::IntermediateDB.insert("INSERT INTO keyed VALUES (?, ?)", 1, "existing")
+
+ Object.const_set(
+ "TempIntegrationSteps",
+ Module.new do
+ const_set(
+ "Keyed",
+ Class.new(Migrations::Conversion::Step) do
+ source do
+ def items
+ # id 1 collides with the existing row; id 2 is new
+ [{ id: 1, label: "new" }, { id: 2, label: "added" }]
+ end
+ end
+
+ processor do
+ def process(item)
+ Migrations::Database::IntermediateDB.insert(
+ "INSERT INTO keyed VALUES (?, ?)",
+ item[:id],
+ item[:label],
+ )
+ end
+ end
+ end,
+ )
+ end,
+ )
+
+ step_classes = [TempIntegrationSteps::Keyed]
+ reporter = Migrations::Reporting::Factory.build(titles: step_classes.map(&:title))
+ Migrations::Conversion::StepScheduler.new(
+ step_classes:,
+ budget: 2,
+ reporter:,
+ step_factory: ->(step_class) { step_class.new },
+ shard_manager: @shard_manager,
+ writer: @writer,
+ ).run
+ reporter.close
+
+ Migrations::Database::IntermediateDB.close
+
+ db = Extralite::Database.new(@db_path)
+ keyed = db.query_array("SELECT id, label FROM keyed ORDER BY id")
+ db.close
+ # the existing row is kept as-is (not overwritten), and the new row is added
+ expect(keyed).to eq([[1, "existing"], [2, "added"]])
+ ensure
+ Object.send(:remove_const, "TempIntegrationSteps")
+ end
+end
diff --git a/migrations/core/spec/lib/conversion/serial_job_spec.rb b/migrations/core/spec/lib/conversion/serial_job_spec.rb
deleted file mode 100644
index 895a2444c77..00000000000
--- a/migrations/core/spec/lib/conversion/serial_job_spec.rb
+++ /dev/null
@@ -1,64 +0,0 @@
-# frozen_string_literal: true
-
-RSpec.describe Migrations::Conversion::SerialJob do
- subject(:job) { described_class.new(processor) }
-
- let(:processor) { instance_double(Migrations::Conversion::ProgressStep::Processor) }
- let(:item) { "Item" }
- let(:tracker) { instance_double(Migrations::Conversion::StepTracker) }
- let(:stats) { Migrations::Conversion::StepStats.new }
-
- before do
- allow(processor).to receive(:tracker).and_return(tracker)
- allow(processor).to receive(:setup)
-
- allow(tracker).to receive(:reset_stats!)
- allow(tracker).to receive(:log_error)
- allow(tracker).to receive(:stats).and_return(stats)
- end
-
- describe "#setup" do
- it "runs the processor's `setup`" do
- job.setup
- expect(processor).to have_received(:setup)
- end
-
- it "raises an error when the processor writes to the IntermediateDB during `setup`" do
- allow(processor).to receive(:setup) do
- Migrations::Database::IntermediateDB.insert("INSERT INTO foo (id) VALUES (?)", 1)
- end
-
- expect { job.setup }.to raise_error(Migrations::Conversion::SetupGuard::SetupError)
- end
- end
-
- describe "#run" do
- it "resets stats and processes item" do
- allow(processor).to receive(:process).and_return(stats)
-
- result = job.run(item)
- expect(result).to eq(stats)
-
- expect(tracker).to have_received(:reset_stats!)
- expect(processor).to have_received(:process).with(item)
- end
-
- it "logs error if processing item raises an exception" do
- allow(processor).to receive(:process).and_raise(StandardError)
-
- job.run(item)
-
- expect(tracker).to have_received(:log_error).with(
- "Failed to process item",
- exception: an_instance_of(StandardError),
- details: item,
- )
- end
- end
-
- describe "#cleanup" do
- it "can be called without errors" do
- expect { job.cleanup }.not_to raise_error
- end
- end
-end
diff --git a/migrations/core/spec/lib/conversion/setup_guard_spec.rb b/migrations/core/spec/lib/conversion/setup_guard_spec.rb
index 5b1e8a30791..0f643c6dec2 100644
--- a/migrations/core/spec/lib/conversion/setup_guard_spec.rb
+++ b/migrations/core/spec/lib/conversion/setup_guard_spec.rb
@@ -1,7 +1,7 @@
# frozen_string_literal: true
RSpec.describe Migrations::Conversion::SetupGuard do
- let(:processor) { instance_double(Migrations::Conversion::ProgressStep::Processor) }
+ let(:processor) { instance_double(Migrations::Conversion::Step::Processor) }
let(:intermediate_db) { Migrations::Database::IntermediateDB }
before { reset_memoization(intermediate_db, :@db) }
@@ -28,7 +28,9 @@ RSpec.describe Migrations::Conversion::SetupGuard do
end
it "restores the previous connection, even when `setup` writes" do
- connection = Migrations::Database::OfflineConnection.new
+ connection = instance_double(Migrations::Database::Connection)
+ allow(connection).to receive(:insert)
+ allow(connection).to receive(:close)
intermediate_db.setup(connection)
allow(processor).to receive(:setup) do
@@ -37,10 +39,11 @@ RSpec.describe Migrations::Conversion::SetupGuard do
expect { described_class.run(processor) }.to raise_error(described_class::SetupError)
+ # the guard blocked the write from `setup`; only the write made after it
+ # restored the connection gets through
intermediate_db.insert("INSERT INTO foo (id) VALUES (?)", 2)
- expect(connection.parametrized_insert_statements).to eq(
- [["INSERT INTO foo (id) VALUES (?)", [2]]],
- )
+ expect(connection).to have_received(:insert).with("INSERT INTO foo (id) VALUES (?)", [2])
+ expect(connection).not_to have_received(:insert).with("INSERT INTO foo (id) VALUES (?)", [1])
end
end
end
diff --git a/migrations/core/spec/lib/conversion/shard_manager_spec.rb b/migrations/core/spec/lib/conversion/shard_manager_spec.rb
new file mode 100644
index 00000000000..ca535d08708
--- /dev/null
+++ b/migrations/core/spec/lib/conversion/shard_manager_spec.rb
@@ -0,0 +1,54 @@
+# frozen_string_literal: true
+
+require "extralite"
+
+RSpec.describe Migrations::Conversion::ShardManager do
+ around do |example|
+ Dir.mktmpdir do |dir|
+ @canonical = File.join(dir, "intermediate.db")
+ @migrations = File.join(dir, "migrations")
+ FileUtils.mkdir_p(@migrations)
+ File.write(File.join(@migrations, "001-schema.sql"), <<~SQL)
+ CREATE TABLE widgets (id INTEGER PRIMARY KEY, name TEXT);
+ CREATE INDEX widgets_name ON widgets (name);
+ SQL
+ Migrations::Database.migrate(@canonical, migrations_path: @migrations)
+
+ # a row from a previous run already lives in the DB
+ db = Extralite::Database.new(@canonical)
+ db.execute("INSERT INTO widgets (id, name) VALUES (1, 'existing')")
+ db.close
+
+ example.run
+ end
+ end
+
+ def build_manager
+ described_class.new(canonical_path: @canonical, migrations_path: @migrations)
+ end
+
+ it "gives each shard the run's schema but none of its data" do
+ manager = build_manager
+ db = Extralite::Database.new(manager.create_shard)
+
+ tables =
+ db.query_array(
+ "SELECT name FROM sqlite_master WHERE type = 'table' AND name NOT LIKE 'sqlite_%'",
+ ).flatten
+ indexes = db.query_array("SELECT name FROM sqlite_master WHERE type = 'index'").flatten
+
+ expect(tables).to include("widgets") # schema is there
+ expect(indexes).to include("widgets_name") # indexes too
+ expect(db.query_single_splat("SELECT COUNT(*) FROM widgets")).to eq(0) # but no rows
+ ensure
+ db&.close
+ manager&.cleanup
+ end
+
+ it "hands out a distinct shard path each time" do
+ manager = build_manager
+ expect(manager.create_shard).not_to eq(manager.create_shard)
+ ensure
+ manager&.cleanup
+ end
+end
diff --git a/migrations/core/spec/lib/conversion/step_base_spec.rb b/migrations/core/spec/lib/conversion/step_base_spec.rb
deleted file mode 100644
index 0fa13656d5c..00000000000
--- a/migrations/core/spec/lib/conversion/step_base_spec.rb
+++ /dev/null
@@ -1,39 +0,0 @@
-# frozen_string_literal: true
-
-RSpec.describe Migrations::Conversion::StepBase do
- before do
- Object.const_set(
- "TemporaryStepsModule",
- Module.new do
- const_set("Topics", Class.new(Migrations::Conversion::ProgressStep))
- const_set("TopicUsers", Class.new(Migrations::Conversion::Step))
- const_set("Users", Class.new(Migrations::Conversion::Step))
- end,
- )
- end
-
- after { Object.send(:remove_const, "TemporaryStepsModule") }
-
- describe "dependency metadata" do
- it "exposes the StepDependencies macros at class level" do
- expect(described_class).to be_a(Migrations::StepDependencies)
- expect(described_class).to respond_to(:depends_on, :dependencies, :priority)
- end
-
- it "lets steps depend on both `Step` and `ProgressStep` subclasses" do
- TemporaryStepsModule::TopicUsers.depends_on(:topics, :users)
-
- expect(TemporaryStepsModule::TopicUsers.dependencies).to eq(
- [TemporaryStepsModule::Topics, TemporaryStepsModule::Users],
- )
- end
-
- it "doesn't bleed metadata between sibling step classes" do
- TemporaryStepsModule::TopicUsers.depends_on(:users)
- TemporaryStepsModule::TopicUsers.priority 1
-
- expect(TemporaryStepsModule::Users.dependencies).to eq([])
- expect(TemporaryStepsModule::Users.priority).to be_nil
- end
- end
-end
diff --git a/migrations/core/spec/lib/conversion/step_executor_spec.rb b/migrations/core/spec/lib/conversion/step_executor_spec.rb
deleted file mode 100644
index d0654446675..00000000000
--- a/migrations/core/spec/lib/conversion/step_executor_spec.rb
+++ /dev/null
@@ -1,28 +0,0 @@
-# frozen_string_literal: true
-
-RSpec.describe Migrations::Conversion::StepExecutor do
- subject(:executor) { described_class.new(step, reporter:) }
-
- let(:handle) { instance_double(Migrations::Reporting::Reporter::StepHandle, finish: nil) }
- let(:reporter) { instance_double(Migrations::Reporting::Plain, start_step: handle) }
- let(:step_class) { Class.new(Migrations::Conversion::Step) { title "Fixture step" } }
- let(:step) { step_class.new(settings: {}) }
-
- describe "#execute" do
- it "announces the step through the reporter and executes it" do
- allow(step).to receive(:execute)
-
- executor.execute
-
- expect(reporter).to have_received(:start_step).with("Fixture step")
- expect(step).to have_received(:execute)
- end
-
- it "finishes the step's handle even when the step fails" do
- allow(step).to receive(:execute).and_raise("boom")
-
- expect { executor.execute }.to raise_error("boom")
- expect(handle).to have_received(:finish)
- end
- end
-end
diff --git a/migrations/core/spec/lib/conversion/step_runner_spec.rb b/migrations/core/spec/lib/conversion/step_runner_spec.rb
new file mode 100644
index 00000000000..3faff1c2d6a
--- /dev/null
+++ b/migrations/core/spec/lib/conversion/step_runner_spec.rb
@@ -0,0 +1,151 @@
+# frozen_string_literal: true
+
+require "extralite"
+
+RSpec.describe Migrations::Conversion::StepRunner do
+ around do |example|
+ Dir.mktmpdir do |dir|
+ @shard_path = File.join(dir, "shard.db")
+ db = Extralite::Database.new(@shard_path)
+ db.execute("CREATE TABLE notes (id INTEGER, body TEXT NOT NULL)")
+ db.execute(
+ "CREATE TABLE log_entries (created_at TEXT, type INTEGER, message TEXT, exception TEXT, details TEXT)",
+ )
+ db.close
+
+ example.run
+ ensure
+ Migrations::Database::IntermediateDB.setup(nil)
+ end
+ end
+
+ class RecordingReporter
+ attr_reader :max_progress, :progress, :warnings, :errors
+
+ def initialize
+ @progress = @warnings = @errors = 0
+ end
+
+ def report_max_progress(value)
+ @max_progress = value
+ end
+
+ def report_progress(progress:, warnings:, errors:)
+ @progress += progress
+ @warnings += warnings
+ @errors += errors
+ end
+ end
+
+ let(:step_class) do
+ Class.new(Migrations::Conversion::Step) do
+ source do
+ def max_progress
+ 3
+ end
+
+ def items
+ [{ id: 1, body: "ok" }, { id: 2, body: nil }, { id: 3, body: "ok" }]
+ end
+ end
+
+ processor do
+ def process(item)
+ Migrations::Database::IntermediateDB.insert(
+ "INSERT INTO notes (id, body) VALUES (?, ?)",
+ item[:id],
+ item[:body],
+ )
+ end
+ end
+ end
+ end
+
+ let(:reporter) { RecordingReporter.new }
+
+ def shard_rows(table)
+ db = Extralite::Database.new(@shard_path)
+ db.query_splat("SELECT id FROM #{table} ORDER BY id")
+ ensure
+ db&.close
+ end
+
+ def shard_count(table)
+ db = Extralite::Database.new(@shard_path)
+ db.query_single_splat("SELECT COUNT(*) FROM #{table}")
+ ensure
+ db&.close
+ end
+
+ it "reads the source, writes every good row to the shard, and reports progress" do
+ described_class.new(step: step_class.new, shard_path: @shard_path, reporter:).run
+
+ expect(shard_rows("notes")).to eq([1, 3])
+ expect(shard_count("log_entries")).to eq(1)
+
+ # the failed row still counts toward progress
+ expect(reporter.max_progress).to eq(3)
+ expect(reporter.progress).to eq(3)
+ expect(reporter.errors).to eq(1)
+ end
+
+ it "closes the source when it's done" do
+ step = step_class.new
+ allow(step.source).to receive(:cleanup).and_call_original
+
+ described_class.new(step:, shard_path: @shard_path, reporter:).run
+
+ expect(step.source).to have_received(:cleanup)
+ end
+
+ context "when given a chunk" do
+ let(:partitioned_step_class) do
+ Class.new(Migrations::Conversion::Step) do
+ source do
+ def all
+ (1..10).map { |id| { id:, body: "ok" } }
+ end
+
+ def items
+ return all unless chunk
+
+ lower, upper = chunk
+ all.select { |it| it[:id] >= lower && (upper.nil? || it[:id] < upper) }
+ end
+ end
+
+ processor do
+ def process(item)
+ Migrations::Database::IntermediateDB.insert(
+ "INSERT INTO notes (id, body) VALUES (?, ?)",
+ item[:id],
+ item[:body],
+ )
+ end
+ end
+ end
+ end
+
+ it "reads only its own chunk of the source" do
+ described_class.new(
+ step: partitioned_step_class.new,
+ shard_path: @shard_path,
+ reporter:,
+ chunk: [1, 6],
+ ).run
+
+ expect(shard_rows("notes")).to eq([1, 2, 3, 4, 5])
+ end
+
+ it "reads an open-ended last chunk" do
+ described_class.new(
+ step: partitioned_step_class.new,
+ shard_path: @shard_path,
+ reporter:,
+ chunk: [6, nil],
+ ).run
+
+ expect(shard_rows("notes")).to eq([6, 7, 8, 9, 10])
+ end
+ end
+end
diff --git a/migrations/core/spec/lib/conversion/step_scheduler_spec.rb b/migrations/core/spec/lib/conversion/step_scheduler_spec.rb
new file mode 100644
index 00000000000..f0e6d77af6d
--- /dev/null
+++ b/migrations/core/spec/lib/conversion/step_scheduler_spec.rb
@@ -0,0 +1,247 @@
+# frozen_string_literal: true
+
+RSpec.describe Migrations::Conversion::StepScheduler do
+ let(:events) { Queue.new }
+
+ # Fake StepCoordinator: announces its step, then blocks until the example
+ # releases it, so we can drive ordering and concurrency without forking.
+ class FakeCoordinator
+ attr_reader :step_class
+
+ def initialize(step_class, scheduler, events:, outcome: :done, raise_in_run: false)
+ @step_class = step_class
+ @scheduler = scheduler
+ @events = events
+ @outcome = outcome
+ @raise_in_run = raise_in_run
+ @finish = Queue.new
+ end
+
+ def run
+ @events << @step_class.name.demodulize
+ @finish.pop
+
+ raise "unhandled error in #{@step_class.name.demodulize}" if @raise_in_run
+
+ if @outcome == :failed
+ @scheduler.record_failure(
+ @step_class,
+ RuntimeError.new("boom in #{@step_class.name.demodulize}"),
+ )
+ end
+
+ @outcome
+ end
+
+ def finish!
+ @finish << true
+ end
+ end
+
+ # Builds named step classes in a throwaway namespace so `depends_on` (which
+ # resolves names lexically) and the class-name tie-break have real classes to
+ # work with. `stub_const` removes them after the example.
+ def define_steps(*names, partitionable: [])
+ namespace = Module.new
+ names.each do |name|
+ step_class =
+ Class.new(Migrations::Conversion::Step) do
+ if partitionable.include?(name)
+ source do
+ partition_by :id, from: "things"
+
+ def items
+ []
+ end
+ end
+ end
+ end
+ namespace.const_set(name, step_class)
+ end
+ stub_const("TempSteps", namespace)
+ names.map { |name| namespace.const_get(name) }
+ end
+
+ def build_scheduler(step_classes, budget: 4, max_parallel_steps: nil, config: {})
+ reporter = instance_double(Migrations::Reporting::Reporter)
+ allow(reporter).to receive(:finalizing).and_yield
+ allow(reporter).to receive(:report_summary)
+
+ scheduler =
+ described_class.new(
+ step_classes:,
+ reporter:,
+ step_factory: ->(_step_class) {},
+ shard_manager: instance_double(Migrations::Conversion::ShardManager),
+ writer: instance_double(Migrations::Database::Connection),
+ budget:,
+ max_parallel_steps:,
+ )
+
+ coordinators =
+ step_classes.to_h do |step_class|
+ options = config[step_class] || {}
+ coordinator =
+ FakeCoordinator.new(
+ step_class,
+ scheduler,
+ events:,
+ outcome: options.fetch(:outcome, :done),
+ raise_in_run: options.fetch(:raise_in_run, false),
+ )
+ [step_class, coordinator]
+ end
+
+ allow(scheduler).to receive(:build_coordinator) { |step_class| coordinators.fetch(step_class) }
+
+ by_name = coordinators.transform_keys { |step_class| step_class.name.demodulize }
+ [scheduler, by_name]
+ end
+
+ def run_in_background(scheduler)
+ Thread.new do
+ Thread.current.report_on_exception = false
+ scheduler.run
+ end
+ end
+
+ it "starts a step only once its dependencies are done" do
+ a, b = define_steps(:A, :B)
+ b.depends_on(:a)
+ scheduler, by_name = build_scheduler([a, b])
+
+ runner = run_in_background(scheduler)
+
+ expect(events.pop).to eq("A")
+ expect(events).to be_empty
+ by_name["A"].finish!
+
+ expect(events.pop).to eq("B")
+ by_name["B"].finish!
+
+ runner.join
+ end
+
+ it "admits ready steps in a deterministic order on an empty graph" do
+ steps = define_steps(:C, :A, :B)
+ scheduler, by_name = build_scheduler(steps, budget: 1)
+
+ runner = run_in_background(scheduler)
+
+ %w[A B C].each do |name|
+ expect(events.pop).to eq(name)
+ by_name[name].finish!
+ end
+
+ runner.join
+ end
+
+ it "never runs more steps at once than the budget" do
+ steps = define_steps(:A, :B, :C, :D)
+ scheduler, by_name = build_scheduler(steps, budget: 2)
+
+ runner = run_in_background(scheduler)
+
+ expect([events.pop, events.pop].sort).to eq(%w[A B])
+ expect(events).to be_empty
+
+ by_name["A"].finish!
+ expect(events.pop).to eq("C")
+ expect(events).to be_empty
+
+ by_name["B"].finish!
+ expect(events.pop).to eq("D")
+
+ by_name["C"].finish!
+ by_name["D"].finish!
+ runner.join
+ end
+
+ it "leaves a fork free for a single-fork step to overlap a partitioned step" do
+ big, x, y = define_steps(:Big, :X, :Y, partitionable: %i[Big])
+ scheduler, by_name = build_scheduler([big, x, y], budget: 4)
+
+ runner = run_in_background(scheduler)
+
+ # Big takes 3 of the 4 forks, so one single-fork step fills the last fork
+ # and overlaps Big instead of queueing behind it.
+ expect([events.pop, events.pop].sort).to eq(%w[Big X])
+ expect(events).to be_empty
+ by_name["X"].finish!
+
+ expect(events.pop).to eq("Y")
+ by_name["Big"].finish!
+ by_name["Y"].finish!
+
+ runner.join
+ end
+
+ it "caps the budget with --max-parallel-steps" do
+ steps = define_steps(:A, :B, :C)
+ scheduler, by_name = build_scheduler(steps, budget: 8, max_parallel_steps: 1)
+
+ runner = run_in_background(scheduler)
+
+ %w[A B C].each do |name|
+ expect(events.pop).to eq(name)
+ expect(events).to be_empty
+ by_name[name].finish!
+ end
+
+ runner.join
+ end
+
+ it "skips a failed step's dependents, runs independent steps, and raises a summary" do
+ a, b, c = define_steps(:A, :B, :C)
+ b.depends_on(:a)
+
+ scheduler, by_name = build_scheduler([a, b, c], config: { a => { outcome: :failed } })
+
+ runner = run_in_background(scheduler)
+
+ expect([events.pop, events.pop].sort).to eq(%w[A C])
+ by_name["A"].finish!
+ by_name["C"].finish!
+
+ expect { runner.join }.to raise_error(Migrations::Conversion::ConvertError) do |error|
+ expect(error.message).to match(/boom in A/)
+ expect(error.message).to match(/skipped/)
+ end
+ expect(events).to be_empty
+ end
+
+ it "still finishes when a coordinator raises instead of reporting back" do
+ a, b = define_steps(:A, :B)
+ scheduler, by_name = build_scheduler([a, b], config: { a => { raise_in_run: true } })
+
+ runner = run_in_background(scheduler)
+
+ expect([events.pop, events.pop].sort).to eq(%w[A B])
+ by_name["A"].finish!
+ by_name["B"].finish!
+
+ expect { runner.join }.to raise_error(
+ Migrations::Conversion::ConvertError,
+ /unhandled error in A/,
+ )
+ end
+
+ it "fails the run when a shard merge errors, even though every step succeeded" do
+ merge_error = StandardError.new("merge boom")
+ allow(Migrations::Conversion::Consolidator).to receive(:new).and_return(
+ instance_double(Migrations::Conversion::Consolidator, enqueue: nil, drain: [merge_error]),
+ )
+
+ a, b = define_steps(:A, :B)
+ scheduler, by_name = build_scheduler([a, b])
+
+ runner = run_in_background(scheduler)
+ expect([events.pop, events.pop].sort).to eq(%w[A B])
+ by_name["A"].finish!
+ by_name["B"].finish!
+
+ expect { runner.join }.to raise_error(Migrations::Conversion::ConvertError) do |error|
+ expect(error.message).to include("merge boom")
+ end
+ end
+end
diff --git a/migrations/core/spec/lib/conversion/step_spec.rb b/migrations/core/spec/lib/conversion/step_spec.rb
index f34f0d5397f..9d1340f4453 100644
--- a/migrations/core/spec/lib/conversion/step_spec.rb
+++ b/migrations/core/spec/lib/conversion/step_spec.rb
@@ -1,76 +1,512 @@
# frozen_string_literal: true
RSpec.describe Migrations::Conversion::Step do
- before do
- Object.const_set(
- "TemporaryModule",
- Module.new do
- const_set("TopicUsers", Class.new(Migrations::Conversion::Step) {})
- const_set("Users", Class.new(Migrations::Conversion::Step) {})
- end,
- )
+ def define_step(&block)
+ Class.new(described_class, &block)
end
- after do
- TemporaryModule.send(:remove_const, "TopicUsers")
- TemporaryModule.send(:remove_const, "Users")
- Object.send(:remove_const, "TemporaryModule")
+ describe ".source" do
+ it "defines methods on the source role only" do
+ step_class =
+ define_step do
+ source do
+ def items
+ [1, 2, 3]
+ end
+ end
+ end
+
+ source = step_class.source_class.new
+ expect(source.items).to eq([1, 2, 3])
+
+ expect(step_class.new).not_to respond_to(:items)
+ expect(step_class.processor_class.new).not_to respond_to(:items)
+ end
+
+ it "defines private methods on the source role" do
+ step_class =
+ define_step do
+ source do
+ def items
+ [build_item]
+ end
+
+ private
+
+ def build_item
+ :item
+ end
+ end
+ end
+
+ source = step_class.source_class.new
+ expect(source.items).to eq([:item])
+ expect(source.respond_to?(:build_item)).to be(false)
+ expect(source.respond_to?(:build_item, true)).to be(true)
+ end
+
+ it "raises an error when called twice" do
+ expect do
+ define_step do
+ source {}
+ source {}
+ end
+ end.to raise_error(ArgumentError, /`source` is already defined/)
+ end
end
- describe ".title" do
- it "uses the classname within title" do
- expect(TemporaryModule::TopicUsers.title).to eq("Converting topic users")
- expect(TemporaryModule::Users.title).to eq("Converting users")
+ describe ".processor" do
+ it "defines methods on the processor role only" do
+ step_class =
+ define_step do
+ processor do
+ def process(item)
+ item * 2
+ end
+ end
+ end
+
+ processor = step_class.processor_class.new
+ expect(processor.process(21)).to eq(42)
+
+ expect(step_class.new).not_to respond_to(:process)
+ expect(step_class.source_class.new).not_to respond_to(:process)
end
- it "uses the `title` attribute if it has been set" do
- TemporaryModule::Users.title "Foo bar"
- expect(TemporaryModule::Users.title).to eq("Foo bar")
+ it "defines private methods on the processor role" do
+ step_class =
+ define_step do
+ processor do
+ def process(item)
+ transform(item)
+ end
+
+ private
+
+ def transform(item)
+ item.to_s
+ end
+ end
+ end
+
+ processor = step_class.processor_class.new
+ expect(processor.process(1)).to eq("1")
+ expect(processor.respond_to?(:transform)).to be(false)
+ expect(processor.respond_to?(:transform, true)).to be(true)
+ end
+
+ it "raises an error when called twice" do
+ expect do
+ define_step do
+ processor {}
+ processor {}
+ end
+ end.to raise_error(ArgumentError, /`processor` is already defined/)
+ end
+ end
+
+ describe ".helpers" do
+ it "mixes the helpers into both roles" do
+ step_class =
+ define_step do
+ helpers do
+ def double_it(value)
+ value * 2
+ end
+ end
+
+ source do
+ def items
+ [double_it(1)]
+ end
+ end
+
+ processor do
+ def process(item)
+ double_it(item)
+ end
+ end
+ end
+
+ expect(step_class.source_class.new.items).to eq([2])
+ expect(step_class.processor_class.new.process(2)).to eq(4)
+ end
+
+ it "works independently of the order of the DSL calls" do
+ step_class =
+ define_step do
+ source do
+ def items
+ [double_it(1)]
+ end
+ end
+
+ helpers do
+ def double_it(value)
+ value * 2
+ end
+ end
+ end
+
+ expect(step_class.source_class.new.items).to eq([2])
+ end
+
+ it "raises an error when called twice" do
+ expect do
+ define_step do
+ helpers {}
+ helpers {}
+ end
+ end.to raise_error(ArgumentError, /`helpers` is already defined/)
+ end
+ end
+
+ describe "role isolation" do
+ it "raises a NameError when the processor calls a source method" do
+ step_class =
+ define_step do
+ source do
+ def items
+ []
+ end
+
+ private
+
+ def source_helper
+ "source-side state"
+ end
+ end
+
+ processor do
+ def process(item)
+ source_helper
+ end
+ end
+ end
+
+ processor = step_class.processor_class.new
+ expect { processor.process(1) }.to raise_error(NameError, /source_helper/)
+ end
+
+ it "does not expose instance variables set in `items` to the processor" do
+ step_class =
+ define_step do
+ source do
+ def items
+ @leaked_state = "set in items"
+ [1]
+ end
+ end
+
+ processor do
+ def process(item)
+ instance_variable_defined?(:@leaked_state)
+ end
+ end
+ end
+
+ step = step_class.new
+ step.source.items
+
+ expect(step.create_processor.process(1)).to be(false)
+ end
+
+ it "does not capture local variables in methods defined with `def`" do
+ local_state = "step-level local"
+
+ step_class =
+ define_step do
+ source do
+ def items
+ local_state
+ end
+ end
+ end
+
+ expect { step_class.source_class.new.items }.to raise_error(NameError, /local_state/)
+ expect(local_state).to eq("step-level local") # silence the unused variable warning
+ end
+ end
+
+ describe "role defaults" do
+ let(:step_class) { define_step }
+
+ it "uses `nil` as default `max_progress`" do
+ expect(step_class.source_class.new.max_progress).to be_nil
+ end
+
+ it "raises `NotImplementedError` when `items` is not defined" do
+ expect { step_class.source_class.new.items }.to raise_error(NotImplementedError)
+ end
+
+ it "raises `NotImplementedError` when `process` is not defined" do
+ expect { step_class.processor_class.new.process(1) }.to raise_error(NotImplementedError)
+ end
+
+ it "uses a no-op as default `setup`" do
+ expect { step_class.processor_class.new.setup }.not_to raise_error
+ end
+
+ it "uses a no-op `cleanup` when the source has no connection" do
+ expect { step_class.source_class.new.cleanup }.not_to raise_error
+ end
+ end
+
+ describe "reads_table defaults" do
+ # Stands in for the source DB adapter, recording the calls it receives (the
+ # adapter builds the actual SQL; that's covered in its own spec).
+ let(:source_db) do
+ Class
+ .new do
+ attr_reader :select_args, :count_args
+
+ def chunk_filter(column, lower, upper, base:)
+ "#{base} AND #{column} >= #{lower} AND #{column} < #{upper}"
+ end
+
+ def select_all(table, where:, order: nil)
+ @select_args = [table, where, order]
+ [:row]
+ end
+
+ def count_all(table, where:)
+ @count_args = [table, where]
+ 4
+ end
+ end
+ .new
+ end
+
+ it "reads the whole table through the adapter" do
+ step_class = define_step { source { reads_table "things", where: "active" } }
+ source = step_class.source_class.new(source_db:)
+
+ expect(source.items).to eq([:row])
+ expect(source_db.select_args).to eq(["things", "active", nil]) # unordered when unpartitioned
+
+ expect(source.max_progress).to eq(4)
+ expect(source_db.count_args).to eq(%w[things active])
+ end
+
+ it "narrows to the chunk and orders by the key when also partitioned" do
+ step_class =
+ define_step do
+ source do
+ reads_table "things", where: "active"
+ partition_by :id
+ end
+ end
+ source = step_class.source_class.new(source_db:, chunk: [1, 5])
+
+ source.items
+ # ordered by the key so the worker writes its shard in key order
+ expect(source_db.select_args).to eq(["things", "active AND id >= 1 AND id < 5", :id])
+ end
+
+ it "passes no filter when `reads_table` has no `where`" do
+ step_class = define_step { source { reads_table "things" } }
+ source = step_class.source_class.new(source_db:)
+
+ source.items
+ expect(source_db.select_args).to eq(["things", nil, nil])
+ end
+
+ it "orders an unpartitioned read by the declared `order`" do
+ step_class = define_step { source { reads_table "things", where: "active", order: :id } }
+ source = step_class.source_class.new(source_db:)
+
+ source.items
+ expect(source_db.select_args).to eq(["things", "active", :id])
+ end
+ end
+
+ describe "#cleanup" do
+ it "closes the per-step `source_db` connection" do
+ step_class = define_step { source { attr_accessor :source_db } }
+ source_db = instance_double(Migrations::Database::Connection, close: nil)
+ source = step_class.source_class.new(source_db:)
+
+ source.cleanup
+
+ expect(source_db).to have_received(:close)
+ end
+ end
+
+ describe "constant resolution" do
+ # Real steps write their role blocks inside a step class body, so constants
+ # in their methods resolve through the step's lexical scope and ancestry
+ # (`Step` defines `IntermediateDB` and `Enums`). A `Class.new` block in this
+ # spec file would have the wrong lexical scope, so we use a string eval.
+ before { Object.class_eval <<~RUBY }
+ class StepConstantsFixture < Migrations::Conversion::Step
+ source do
+ def items
+ [IntermediateDB, Enums]
+ end
+ end
+
+ processor do
+ def process(item)
+ [IntermediateDB, Enums]
+ end
+ end
+ end
+ RUBY
+
+ after { Object.send(:remove_const, :StepConstantsFixture) }
+
+ it "resolves `IntermediateDB` and `Enums` inside role blocks via the step's ancestry" do
+ expected = [Migrations::Database::IntermediateDB, Migrations::Database::IntermediateDB::Enums]
+
+ expect(StepConstantsFixture.source_class.new.items).to eq(expected)
+ expect(StepConstantsFixture.processor_class.new.process(nil)).to eq(expected)
+ end
+
+ it "does not define the constants on the role classes themselves" do
+ expect(StepConstantsFixture.source_class).not_to have_constant(:IntermediateDB)
+ expect(StepConstantsFixture.processor_class).not_to have_constant(:IntermediateDB)
end
end
describe "#initialize" do
- it "works when no arguments are supplied" do
- step = nil
- expect { step = TemporaryModule::Users.new }.not_to raise_error
- expect(step.settings).to be_nil
+ it "routes args to the role that declares a matching setter" do
+ step_class = define_step { source { attr_accessor :source_db } }
+
+ settings = { a: 1 }
+ step = step_class.new(settings:, source_db: "source db", unknown_arg: "dropped")
+
+ expect(step.source.settings).to eq(settings)
+ expect(step.source.source_db).to eq("source db")
+
+ processor = step.create_processor
+ expect(processor.settings).to eq(settings)
+ expect(processor).not_to respond_to(:source_db)
+ expect(processor).not_to respond_to(:unknown_arg)
+ end
+
+ it "keeps no per-step state on the coordinator" do
+ step = define_step.new(settings: { a: 1 })
+
+ expect(step).not_to respond_to(:tracker)
+ expect(step).not_to respond_to(:step)
+ expect(step).not_to respond_to(:settings)
+ expect(step).not_to respond_to(:execute)
+ end
+ end
+
+ describe "#source" do
+ it "returns the same source instance every time" do
+ step = define_step.new
+ expect(step.source).to be(step.source)
+ end
+ end
+
+ describe "#create_processor" do
+ it "creates a new processor with its own tracker on every call" do
+ step = define_step.new
+
+ processor1 = step.create_processor
+ processor2 = step.create_processor
+
+ expect(processor1).not_to be(processor2)
+ expect(processor1.tracker).to be_a(Migrations::Conversion::StepTracker)
+ expect(processor1.tracker).not_to be(processor2.tracker)
+ end
+ end
+
+ describe "step subclasses" do
+ it "inherits the roles of the parent step" do
+ parent_class =
+ define_step do
+ source do
+ def items
+ [1]
+ end
+ end
+
+ processor do
+ def process(item)
+ item + 1
+ end
+ end
+ end
+
+ step_class =
+ Class.new(parent_class) do
+ source do
+ def max_progress
+ 1
+ end
+ end
+ end
+
+ source = step_class.source_class.new
+ expect(source.items).to eq([1])
+ expect(source.max_progress).to eq(1)
+ expect(step_class.processor_class.new.process(1)).to eq(2)
end
+ end
- it "creates its own tracker" do
- step = TemporaryModule::Users.new
- expect(step.tracker).to be_a(Migrations::Conversion::StepTracker)
- expect(step.tracker).not_to be(TemporaryModule::Users.new.tracker)
+ describe ".partitionable?" do
+ it "defaults to false" do
+ expect(define_step.partitionable?).to be(false)
end
- it "initializes the `settings` attribute if given" do
- settings = { a: 1, b: 2 }
- step = TemporaryModule::Users.new(settings:)
- expect(step.settings).to eq(settings)
+ it "is true when the source declares a partition key" do
+ step =
+ define_step do
+ source do
+ partition_by :id, from: "things", base: "active"
+
+ def items
+ []
+ end
+ end
+ end
+
+ expect(step.partitionable?).to be(true)
end
- it "initializes additional attributes if they exist" do
- TemporaryModule::Users.class_eval { attr_accessor :foo, :bar }
+ it "is true for a composite partition key" do
+ step = define_step { source { partition_by %i[topic_id user_id], from: "topic_users" } }
+
+ expect(step.partitionable?).to be(true)
+ end
+ end
+
+ describe "dependency metadata" do
+ before do
+ Object.const_set(
+ "TemporaryStepsModule",
+ Module.new do
+ const_set("Topics", Class.new(Migrations::Conversion::Step))
+ const_set("TopicUsers", Class.new(Migrations::Conversion::Step))
+ const_set("Users", Class.new(Migrations::Conversion::Step))
+ end,
+ )
+ end
+
+ after { Object.send(:remove_const, "TemporaryStepsModule") }
+
+ it "exposes the StepDependencies macros at class level" do
+ expect(described_class).to be_a(Migrations::StepDependencies)
+ expect(described_class).to respond_to(:depends_on, :dependencies, :priority)
+ end
- settings = { a: 1, b: 2 }
- foo = "a string"
- bar = false
+ it "lets steps depend on other steps" do
+ TemporaryStepsModule::TopicUsers.depends_on(:topics, :users)
- step = TemporaryModule::Users.new(settings:, foo:, bar:, non_existent: 123)
- expect(step.settings).to eq(settings)
- expect(step.foo).to eq(foo)
- expect(step.bar).to eq(bar)
- expect(step).to_not respond_to(:non_existent)
+ expect(TemporaryStepsModule::TopicUsers.dependencies).to eq(
+ [TemporaryStepsModule::Topics, TemporaryStepsModule::Users],
+ )
end
- it "skips attributes with a private setter" do
- TemporaryModule::Users.class_eval do
- attr_writer :secret
- private :secret=
- end
+ it "doesn't bleed metadata between sibling step classes" do
+ TemporaryStepsModule::TopicUsers.depends_on(:users)
+ TemporaryStepsModule::TopicUsers.priority 1
- step = nil
- expect { step = TemporaryModule::Users.new(secret: 123) }.not_to raise_error
- expect(step.instance_variable_defined?(:@secret)).to be(false)
+ expect(TemporaryStepsModule::Users.dependencies).to eq([])
+ expect(TemporaryStepsModule::Users.priority).to be_nil
end
end
end
diff --git a/migrations/core/spec/lib/conversion/worker_budget_spec.rb b/migrations/core/spec/lib/conversion/worker_budget_spec.rb
new file mode 100644
index 00000000000..9f91be50ef0
--- /dev/null
+++ b/migrations/core/spec/lib/conversion/worker_budget_spec.rb
@@ -0,0 +1,33 @@
+# frozen_string_literal: true
+
+RSpec.describe Migrations::Conversion::WorkerBudget do
+ describe ".available" do
+ it "keeps the reserved cores back from the usable CPUs" do
+ allow(described_class).to receive(:usable_cpus).and_return(20)
+
+ expect(described_class.available).to eq(19) # reserve 1 by default
+ expect(described_class.available(reserved: 4)).to eq(16)
+ end
+
+ it "never drops below a single worker" do
+ allow(described_class).to receive(:usable_cpus).and_return(1)
+ expect(described_class.available).to eq(1)
+ end
+ end
+
+ describe ".usable_cpus" do
+ it "takes the tighter of the affinity- and quota-aware counts" do
+ allow(Etc).to receive(:nprocessors).and_return(64) # host affinity
+ allow(Concurrent).to receive(:available_processor_count).and_return(4.0) # CFS quota
+
+ expect(described_class.usable_cpus).to eq(4)
+ end
+
+ it "floors a fractional quota and never reports fewer than one" do
+ allow(Etc).to receive(:nprocessors).and_return(8)
+ allow(Concurrent).to receive(:available_processor_count).and_return(0.5)
+
+ expect(described_class.usable_cpus).to eq(1)
+ end
+ end
+end
diff --git a/migrations/core/spec/lib/conversion/worker_pool_spec.rb b/migrations/core/spec/lib/conversion/worker_pool_spec.rb
deleted file mode 100644
index fb9f4afc3c3..00000000000
--- a/migrations/core/spec/lib/conversion/worker_pool_spec.rb
+++ /dev/null
@@ -1,124 +0,0 @@
-# frozen_string_literal: true
-
-RSpec.describe Migrations::Conversion::WorkerPool do
- subject(:pool) { described_class.new(size: pool_size) }
-
- let(:pool_size) { 2 }
- let(:work_queue) { Queue.new }
- let(:output_queue) { Queue.new }
-
- after do
- work_queue.close if !work_queue.closed?
- output_queue.close if !output_queue.closed?
- end
-
- def create_echo_job
- job = instance_double(Migrations::Conversion::ParallelJob, setup: nil, cleanup: nil)
- allow(job).to receive(:run) { |item| { echoed: item[:id] } }
- job
- end
-
- def drained_output
- results = []
- results << output_queue.pop until output_queue.empty?
- results
- end
-
- describe "#start" do
- it "forks one worker per pool slot by default and processes all queued items" do
- allow(Migrations::ForkManager).to receive(:fork).and_call_original
-
- batch = pool.start(work_queue:, output_queue:) { create_echo_job }
- 5.times { |id| work_queue << { id: } }
- work_queue.close
- batch.wait
- output_queue.close
-
- expect(Migrations::ForkManager).to have_received(:fork).exactly(pool_size).times
- expect(drained_output).to match_array(Array.new(5) { |id| { echoed: id } })
- end
-
- def start_empty_batch(**kwargs)
- batch = pool.start(work_queue:, output_queue:, **kwargs) { create_echo_job }
- work_queue.close
- batch.wait
- output_queue.close
- end
-
- it "forks a single worker when started with `size: 1`" do
- allow(Migrations::ForkManager).to receive(:fork).and_call_original
-
- start_empty_batch(size: 1)
-
- expect(Migrations::ForkManager).to have_received(:fork).once
- end
-
- it "clamps the requested size to the pool size" do
- allow(Migrations::ForkManager).to receive(:fork).and_call_original
-
- start_empty_batch(size: pool_size + 10)
-
- expect(Migrations::ForkManager).to have_received(:fork).exactly(pool_size).times
- end
-
- it "clamps the requested size to a minimum of one worker" do
- allow(Migrations::ForkManager).to receive(:fork).and_call_original
-
- start_empty_batch(size: 0)
-
- expect(Migrations::ForkManager).to have_received(:fork).once
- end
-
- it "runs the fork hooks once per `start` call, not once per worker" do
- before_fork_count = 0
- after_fork_count = 0
- before_fork_hook = Migrations::ForkManager.before_fork { before_fork_count += 1 }
- after_fork_hook = Migrations::ForkManager.after_fork_parent { after_fork_count += 1 }
-
- start_empty_batch
-
- expect(before_fork_count).to eq(1)
- expect(after_fork_count).to eq(1)
- ensure
- Migrations::ForkManager.remove_before_fork(before_fork_hook)
- Migrations::ForkManager.remove_after_fork_parent(after_fork_hook)
- end
-
- it "builds a fresh job for every worker" do
- jobs = []
-
- batch =
- pool.start(work_queue:, output_queue:) do
- job = create_echo_job
- jobs << job
- job
- end
- work_queue.close
- batch.wait
- output_queue.close
-
- expect(jobs.size).to eq(pool_size)
- expect(jobs.uniq.size).to eq(pool_size)
- end
- end
-
- describe "Batch#wait" do
- it "has run every worker's `cleanup` by the time `wait` returns" do
- Dir.mktmpdir do |dir|
- batch =
- pool.start(work_queue:, output_queue:) do
- job = instance_double(Migrations::Conversion::ParallelJob, setup: nil)
- allow(job).to receive(:cleanup) do
- File.write(File.join(dir, "worker_#{Process.pid}"), "exited")
- end
- job
- end
- work_queue.close
- batch.wait
- output_queue.close
-
- expect(Dir.children(dir).size).to eq(pool_size)
- end
- end
- end
-end
diff --git a/migrations/core/spec/lib/conversion/worker_spec.rb b/migrations/core/spec/lib/conversion/worker_spec.rb
deleted file mode 100644
index 74d35d142ab..00000000000
--- a/migrations/core/spec/lib/conversion/worker_spec.rb
+++ /dev/null
@@ -1,302 +0,0 @@
-# frozen_string_literal: true
-
-RSpec.describe Migrations::Conversion::Worker do
- subject(:worker) { described_class.new(index, input_queue, output_queue, job) }
-
- let(:index) { 1 }
- let(:input_queue) { Queue.new }
- let(:output_queue) { Queue.new }
- let(:job) do
- instance_double(Migrations::Conversion::ParallelJob, setup: nil, run: "result", cleanup: nil)
- end
-
- after do
- input_queue.close if !input_queue.closed?
- output_queue.close if !output_queue.closed?
- end
-
- describe "#start" do
- it "works when `input_queue` is empty" do
- expect do
- worker.start
- input_queue.close
- worker.wait
- output_queue.close
- end.not_to raise_error
- end
-
- it "uses `ForkManager.fork`" do
- allow(Migrations::ForkManager).to receive(:fork).and_call_original
-
- worker.start
- input_queue.close
- worker.wait
- output_queue.close
-
- expect(Migrations::ForkManager).to have_received(:fork)
- end
-
- it "writes the output of `job.run` into `output_queue`" do
- allow(job).to receive(:run) { |data| "run: #{data[:text]}" }
-
- worker.start
- input_queue << { text: "Item 1" } << { text: "Item 2" } << { text: "Item 3" }
- input_queue.close
- worker.wait
- output_queue.close
-
- expect(output_queue).to have_queue_contents("run: Item 1", "run: Item 2", "run: Item 3")
- end
-
- def create_progress_stats(progress: 1, warning_count: 0, error_count: 0)
- Migrations::Conversion::StepStats.new(progress:, warning_count:, error_count:)
- end
-
- it "writes objects to the `output_queue`" do
- all_stats = [
- create_progress_stats,
- create_progress_stats(warning_count: 1),
- create_progress_stats(warning_count: 1, error_count: 1),
- create_progress_stats(warning_count: 2, error_count: 1),
- ]
-
- allow(job).to receive(:run) do |data|
- index = data[:index]
- [index, all_stats[index]]
- end
-
- worker.start
- input_queue << { index: 0 } << { index: 1 } << { index: 2 } << { index: 3 }
- input_queue.close
- worker.wait
- output_queue.close
-
- expect(output_queue).to have_queue_contents(
- [0, all_stats[0]],
- [1, all_stats[1]],
- [2, all_stats[2]],
- [3, all_stats[3]],
- )
- end
-
- # Characterization specs: they pin the wire contract of the IPC pipeline
- # (currently Oj in `:object` mode) with payload shapes that real steps
- # produce. Items are echoed back by the job, so each spec exercises both
- # pipe directions with the same payload.
- describe "wire format" do
- def round_trip(*items)
- allow(job).to receive(:run) { |data| data }
-
- worker.start
- items.each { |item| input_queue << item }
- input_queue.close
- worker.wait
- output_queue.close
-
- results = []
- results << output_queue.pop until output_queue.empty?
- results
- end
-
- it "round-trips a site-settings-shaped item with nested hashes and a Time value" do
- item = {
- name: "logo_url",
- value: "/uploads/default/original/1X/logo.png",
- data_type: 18,
- updated_at: Time.utc(2023, 5, 17, 12, 34, 56, 789_123),
- uploads: [
- { id: 1, url: "/uploads/default/original/1X/logo.png", filename: "logo.png" },
- { id: 2, url: "/uploads/default/original/1X/logo2.png", filename: "logo2.png" },
- ],
- }
-
- result = round_trip(item).first
-
- expect(result).to eq(item)
- expect(result.keys).to all(be_a(Symbol))
- expect(result[:uploads].flat_map(&:keys)).to all(be_a(Symbol))
- expect(result[:updated_at]).to be_a(Time)
- expect(result[:updated_at]).to eq(item[:updated_at])
- end
-
- it "round-trips a wide users-shaped item with full fidelity" do
- item = {
- id: 42,
- username: "alice",
- name: "Alice Άλφα 𝒜",
- active: true,
- admin: false,
- moderator: false,
- staged: false,
- approved: true,
- approved_at: Time.utc(2020, 1, 2, 3, 4, 5),
- approved_by_id: 1,
- created_at: Time.utc(2019, 12, 31, 23, 59, 59, 999_999),
- first_seen_at: Time.utc(2020, 1, 2, 3, 4, 6),
- last_seen_at: Time.utc(2024, 11, 5, 6, 7, 8),
- silenced_till: nil,
- suspended_at: nil,
- suspended_till: nil,
- date_of_birth: Date.new(1990, 4, 1),
- ip_address: IPAddr.new("192.168.0.1"),
- registration_ip_address: IPAddr.new("2001:db8::1"),
- locale: nil,
- title: nil,
- trust_level: 2,
- group_locked_trust_level: nil,
- manual_locked_trust_level: nil,
- primary_group_id: nil,
- flair_group_id: nil,
- uploaded_avatar_id: 99,
- views: 1234,
- avatar_url: "/uploads/default/original/1X/avatar.png",
- avatar_filename: "avatar.png",
- avatar_origin: nil,
- avatar_user_id: 42,
- }
-
- result = round_trip(item).first
-
- expect(result).to eq(item)
- expect(result[:created_at]).to be_a(Time)
- expect(result[:date_of_birth]).to be_a(Date)
- expect(result[:ip_address]).to be_a(IPAddr)
- end
-
- it "round-trips string edge cases" do
- item = {
- emoji: "héllo wörld 👋🏼 🇦🇹 文字",
- newlines: "line1\nline2\r\nline3",
- quotes_and_backslashes: %q(she said "hi\there" \\ and left),
- control_chars: "tab\there\u0000null",
- large: "Ω" * 1_000_000,
- }
-
- result = round_trip(item).first
-
- expect(result).to eq(item)
- expect(result[:large].bytesize).to be >= 1_000_000
- end
-
- it "round-trips a realistic statements batch with stats on the return path" do
- statements = [
- [
- "INSERT INTO users (original_id, username, created_at, active) VALUES (?, ?, ?, ?)",
- [42, "alice", "2019-12-31T23:59:59Z", 1],
- ],
- [
- "INSERT INTO user_emails (user_id, email, \"primary\", created_at) VALUES (?, ?, ?, ?)",
- [42, "[email protected]", 1, "2019-12-31T23:59:59Z"],
- ],
- [
- "INSERT INTO user_options (user_id, timezone, dark_scheme_id) VALUES (?, ?, ?)",
- [42, nil, nil],
- ],
- ]
- stats = create_progress_stats(progress: 1, warning_count: 2, error_count: 3)
-
- allow(job).to receive(:run).and_return([statements, stats])
-
- worker.start
- input_queue << { id: 42 }
- input_queue.close
- worker.wait
- output_queue.close
-
- result_statements, result_stats = output_queue.pop
- expect(result_statements).to eq(statements)
- expect(result_stats).to eq(stats)
- end
-
- it "preserves FIFO ordering under volume" do
- item_count = 2000
-
- results = round_trip(*(0...item_count).map { |i| { index: i, payload: "item #{i}" } })
-
- expect(results.map { |item| item[:index] }).to eq((0...item_count).to_a)
- end
- end
-
- it "runs `job.setup` once in the worker process before processing items" do
- temp_file = Tempfile.new("setup_check")
- temp_file_path = temp_file.path
- parent_pid = Process.pid
-
- allow(job).to receive(:setup) do
- process = Process.pid == parent_pid ? "parent" : "worker"
- File.write(temp_file_path, "setup in #{process}\n", mode: "a+")
- end
- allow(job).to receive(:run) do |data|
- File.write(temp_file_path, "run: #{data[:text]}\n", mode: "a+")
- data[:text]
- end
-
- worker.start
- input_queue << { text: "Item 1" } << { text: "Item 2" }
- input_queue.close
- worker.wait
- output_queue.close
-
- expect(File.read(temp_file_path)).to eq <<~LOG
- setup in worker
- run: Item 1
- run: Item 2
- LOG
- ensure
- temp_file.unlink
- end
-
- it "raises an error in the parent process when `job.setup` fails in the worker process" do
- allow(job).to receive(:setup) do
- $stderr.reopen(File::NULL) # silence the worker's backtrace in the test output
- raise Migrations::Conversion::SetupGuard::SetupError, "setup failed"
- end
-
- worker.start
- input_queue << { text: "Item 1" }
- input_queue.close
-
- expect { worker.wait }.to raise_error(
- described_class::CrashedError,
- /Worker process #{index} exited unexpectedly/,
- )
- end
-
- it "raises an error when the worker process dies even though no pipe write failed" do
- allow(job).to receive(:setup) { exit!(1) }
-
- worker.start
- input_queue.close
-
- expect { worker.wait }.to raise_error(described_class::CrashedError)
- end
-
- it "runs `job.cleanup` at the end" do
- temp_file = Tempfile.new("method_call_check")
- temp_file_path = temp_file.path
-
- allow(job).to receive(:run) do |data|
- File.write(temp_file_path, "run: #{data[:text]}\n", mode: "a+")
- data[:text]
- end
- allow(job).to receive(:cleanup) do
- File.write(temp_file_path, "cleanup\n", mode: "a+")
- end
-
- worker.start
- input_queue << { text: "Item 1" } << { text: "Item 2" } << { text: "Item 3" }
- input_queue.close
- worker.wait
- output_queue.close
-
- expect(File.read(temp_file_path)).to eq <<~LOG
- run: Item 1
- run: Item 2
- run: Item 3
- cleanup
- LOG
- ensure
- temp_file.unlink
- end
- end
-end
diff --git a/migrations/core/spec/lib/database/intermediate_db_spec.rb b/migrations/core/spec/lib/database/intermediate_db_spec.rb
index 87f2c01bbef..8ac93a4efc3 100644
--- a/migrations/core/spec/lib/database/intermediate_db_spec.rb
+++ b/migrations/core/spec/lib/database/intermediate_db_spec.rb
@@ -29,23 +29,6 @@ RSpec.describe Migrations::Database::IntermediateDB do
end
end
- it "works with `Migrations::Database::OfflineConnection`" do
- connection = Migrations::Database::OfflineConnection.new
-
- described_class.setup(connection)
- described_class.insert("INSERT INTO foo (id, name) VALUES (?, ?)", 1, "Alice")
- described_class.insert("INSERT INTO foo (id, name) VALUES (?, ?)", 2, "Bob")
-
- expect(connection.parametrized_insert_statements).to eq(
- [
- ["INSERT INTO foo (id, name) VALUES (?, ?)", [1, "Alice"]],
- ["INSERT INTO foo (id, name) VALUES (?, ?)", [2, "Bob"]],
- ],
- )
-
- connection.close
- end
-
it "switches the connection" do
old_connection = create_connection_double
new_connection = create_connection_double
diff --git a/migrations/core/spec/lib/database/migrator_spec.rb b/migrations/core/spec/lib/database/migrator_spec.rb
index b0a1f675769..55cebc4fe91 100644
--- a/migrations/core/spec/lib/database/migrator_spec.rb
+++ b/migrations/core/spec/lib/database/migrator_spec.rb
@@ -75,22 +75,4 @@ RSpec.describe Migrations::Database::Migrator do
end
end
end
-
- describe "#reset!" do
- it "deletes all DB related files" do
- migrate(migrations_directory: "invalid", ignore_errors: true) do |db_path, storage_path|
- File.write(File.join(storage_path, "hello_world.txt"), "Hello World!")
-
- expect(Dir.children(storage_path)).to contain_exactly(
- "intermediate.db",
- "intermediate.db-shm",
- "intermediate.db-wal",
- "hello_world.txt",
- )
-
- described_class.new(db_path).reset!
- expect(Dir.children(storage_path)).to contain_exactly("hello_world.txt")
- end
- end
- end
end
diff --git a/migrations/core/spec/lib/database/offline_connection_spec.rb b/migrations/core/spec/lib/database/offline_connection_spec.rb
deleted file mode 100644
index d36b5ed89b0..00000000000
--- a/migrations/core/spec/lib/database/offline_connection_spec.rb
+++ /dev/null
@@ -1,67 +0,0 @@
-# frozen_string_literal: true
-
-RSpec.describe Migrations::Database::OfflineConnection do
- subject(:connection) { described_class.new }
-
- let!(:sql) { "INSERT INTO foo (id, name) VALUES (?, ?)" }
-
- it_behaves_like "a database connection"
-
- describe "#close" do
- it "removes the cached statements" do
- connection.insert(sql, [1, "Alice"])
- connection.insert(sql, [2, "Bob"])
-
- expect(connection.parametrized_insert_statements).to_not be_empty
-
- connection.close
- expect(connection.parametrized_insert_statements).to be_nil
- end
- end
-
- describe "#closed?" do
- it "correctly reports if connection is closed" do
- expect(connection.closed?).to be false
- connection.close
- expect(connection.closed?).to be true
- end
- end
-
- describe "#insert" do
- it "can be called without errors" do
- expect { connection.insert(sql, [1, "Alice"]) }.not_to raise_error
- end
- end
-
- describe "#parametrized_insert_statements" do
- it "returns an empty array if nothing has been cached" do
- expect(connection.parametrized_insert_statements).to eq([])
- end
-
- it "returns the cached INSERT statements and parameters in original order" do
- connection.insert(sql, [1, "Alice"])
- connection.insert(sql, [2, "Bob"])
- connection.insert(sql, [3, "Carol"])
-
- expected_data = [[sql, [1, "Alice"]], [sql, [2, "Bob"]], [sql, [3, "Carol"]]]
- expect(connection.parametrized_insert_statements).to eq(expected_data)
-
- # multiple calls return the same data
- expect(connection.parametrized_insert_statements).to eq(expected_data)
- expect(connection.parametrized_insert_statements).to eq(expected_data)
- end
- end
-
- describe "#clear!" do
- it "clears all cached data" do
- connection.insert(sql, [1, "Alice"])
- connection.insert(sql, [2, "Bob"])
- connection.insert(sql, [3, "Carol"])
-
- expect(connection.parametrized_insert_statements).to_not be_empty
-
- connection.clear!
- expect(connection.parametrized_insert_statements).to eq([])
- end
- end
-end
diff --git a/migrations/core/spec/lib/database_spec.rb b/migrations/core/spec/lib/database_spec.rb
index 08f4b48173e..7e78cf3e13e 100644
--- a/migrations/core/spec/lib/database_spec.rb
+++ b/migrations/core/spec/lib/database_spec.rb
@@ -26,15 +26,27 @@ RSpec.describe Migrations::Database do
expect(migrator_instance).to have_received(:migrate).with(migrations_path)
end
end
+ end
- describe ".reset!" do
- it "resets the database" do
- allow(migrator_instance).to receive(:reset!)
+ describe ".database_files" do
+ it "lists the database file and its WAL/SHM sidecars" do
+ expect(described_class.database_files("path/to/db")).to eq(
+ %w[path/to/db path/to/db-wal path/to/db-shm],
+ )
+ end
+ end
- described_class.reset!(db_path)
+ describe ".delete_database" do
+ it "removes the database and its sidecars, leaves other files, ignores missing ones" do
+ Dir.mktmpdir do |storage_path|
+ db_path = File.join(storage_path, "test.db")
+ FileUtils.touch(db_path)
+ FileUtils.touch("#{db_path}-wal")
+ # no -shm file: deletion must not raise on the missing one
+ FileUtils.touch(File.join(storage_path, "keep.txt"))
- expect(Migrations::Database::Migrator).to have_received(:new).with(db_path)
- expect(migrator_instance).to have_received(:reset!)
+ expect { described_class.delete_database(db_path) }.not_to raise_error
+ expect(Dir.children(storage_path)).to contain_exactly("keep.txt")
end
end
end
@@ -161,7 +173,7 @@ RSpec.describe Migrations::Database do
number: 123,
date: DateTime.new(2023, 10, 5, 17, 30, 0),
),
- ).to eq(%q|{"text":"foo","number":123,"date":"2023-10-05T17:30:00.000+00:00"}|)
+ ).to eq(%q|{"text":"foo","number":123,"date":"2023-10-05T17:30:00+00:00"}|)
end
it "returns nil for nil input" do
diff --git a/migrations/importer/lib/migrations/importer/executor.rb b/migrations/importer/lib/migrations/importer/executor.rb
index ee9c6333f6e..b8eafec71a1 100644
--- a/migrations/importer/lib/migrations/importer/executor.rb
+++ b/migrations/importer/lib/migrations/importer/executor.rb
@@ -36,7 +36,7 @@ module Migrations
private
def attach_mappings_db(db_path, reset)
- Database.reset!(db_path) if reset
+ Database.delete_database(db_path) if reset
migrate_and_attach(db_path, Database::MAPPINGS_DB_SCHEMA_PATH, "mapped")
end
diff --git a/migrations/importer/lib/migrations/importer/uploads/tasks/base.rb b/migrations/importer/lib/migrations/importer/uploads/tasks/base.rb
index bd152da436c..2d4676272f2 100644
--- a/migrations/importer/lib/migrations/importer/uploads/tasks/base.rb
+++ b/migrations/importer/lib/migrations/importer/uploads/tasks/base.rb
@@ -8,9 +8,6 @@ module Migrations
module Uploads
module Tasks
class Base
- class NotImplementedError < StandardError
- end
-
TRANSACTION_SIZE = 1000
QUEUE_SIZE = 1000
DEFAULT_THREAD_FACTOR = 1.5
diff --git a/migrations/tooling/config/schema/intermediate_db/ignored.rb b/migrations/tooling/config/schema/intermediate_db/ignored.rb
index f8ae0657803..f316a4d3dde 100644
--- a/migrations/tooling/config/schema/intermediate_db/ignored.rb
+++ b/migrations/tooling/config/schema/intermediate_db/ignored.rb
@@ -168,6 +168,7 @@ Migrations::Tooling::Schema.ignored do
:bookmarks,
:browser_pageview_event_scores,
:browser_pageview_events,
+ :browser_pageview_session_engagement_daily_rollups,
:browser_pageview_session_engagements,
:custom_emojis,
:developers,
diff --git a/migrations/tooling/config/schema/plugin_manifest.yml b/migrations/tooling/config/schema/plugin_manifest.yml
index 95faf61c1d8..546f77285c1 100644
--- a/migrations/tooling/config/schema/plugin_manifest.yml
+++ b/migrations/tooling/config/schema/plugin_manifest.yml
@@ -167,6 +167,7 @@ plugins:
- discourse_rewind_share_publicly
discourse-rss-polling:
tables:
+ - discourse_rss_polling_poll_attempts
- discourse_rss_polling_rss_feeds
columns: {}
discourse-solved:
@@ -218,10 +219,10 @@ plugins:
columns: {}
plugin_checksums:
automation: e69972690991bc04d92b2d161aac8921
- chat: 93de6fd994b8fbd1a3921c3deacd1d7a
+ chat: 43ee047298d56af4fad013d534155812
discourse-adplugin: 9e52419decf1099c5bc301a90e6cc91a
discourse-affiliate: '08c6e1a851441a082e9be9ddddc63910'
- discourse-ai: 3063169bf8303c6c3f944391e3e42142
+ discourse-ai: 166cf3770ac2b5524b22a0e33cbfa7ed
discourse-assign: 7a7a36e8ac0214e8a6aca1a1ed9899aa
discourse-cakeday: a5bb91366c8dc8d267088eb0c2679187
discourse-calendar: 07aa9951f5fde7daf03be9edfbda4dec
@@ -241,7 +242,7 @@ plugin_checksums:
discourse-post-voting: ece13b87f2fbac3e64f8aa3abe07fe3d
discourse-reactions: e9de82d2ebb48eb434be2b646c7fec26
discourse-rewind: 1e0f2acf6e8409f2d2f011e14405fa06
- discourse-rss-polling: b7cb3eeb38beff3ce10b333f24128d2f
+ discourse-rss-polling: 526db84c3d2ade0fa03497031fea2333
discourse-solved: 67743e14f4c5c2fae4487193c374e429
discourse-subscriptions: 05eb12877a6a842a290a98fa0e70d462
discourse-templates: a131c89a3e2b765f1eb6ec811c0fecea
diff --git a/migrations/tooling/scripts/compare_intermediate_dbs.rb b/migrations/tooling/scripts/compare_intermediate_dbs.rb
new file mode 100644
index 00000000000..3a3a7e7e67e
--- /dev/null
+++ b/migrations/tooling/scripts/compare_intermediate_dbs.rb
@@ -0,0 +1,69 @@
+# frozen_string_literal: true
+
+# Compares two IntermediateDB files for order-insensitive equality. The
+# concurrent converter writes rows in no fixed order, so we sort each table's
+# rows and compare a digest instead of diffing the files.
+#
+# Usage (from the repo root, under the migrations bundle):
+#
+# cd migrations/core
+# bundle exec ruby ../tooling/scripts/compare_intermediate_dbs.rb \
+# /path/to/sequential.db /path/to/concurrent.db
+
+require "extralite"
+require "digest"
+
+# `schema_migrations.applied_at` is a wall-clock time and always differs.
+IGNORED_TABLES = %w[schema_migrations].freeze
+
+def tables(db)
+ (db.tables - IGNORED_TABLES).sort
+end
+
+def quote(identifier)
+ %("#{identifier.gsub('"', '""')}")
+end
+
+def table_digest(db, table)
+ rows = []
+ count = 0
+ db.query_array("SELECT * FROM #{quote(table)}") do |row|
+ rows << row.map { |value| value.is_a?(String) ? value.b : value.inspect }.join("")
+ count += 1
+ end
+ rows.sort!
+ [count, Digest::SHA256.hexdigest(rows.join(""))]
+end
+
+seq_path, conc_path = ARGV
+abort "usage: compare_intermediate_dbs.rb SEQUENTIAL_DB CONCURRENT_DB" unless seq_path && conc_path
+
+seq = Extralite::Database.new(seq_path)
+conc = Extralite::Database.new(conc_path)
+
+seq_tables = tables(seq)
+conc_tables = tables(conc)
+
+ok = true
+
+if seq_tables != conc_tables
+ ok = false
+ warn "Table sets differ:"
+ warn " only in #{seq_path}: #{(seq_tables - conc_tables).join(", ")}"
+ warn " only in #{conc_path}: #{(conc_tables - seq_tables).join(", ")}"
+end
+
+(seq_tables & conc_tables).each do |table|
+ seq_count, seq_digest = table_digest(seq, table)
+ conc_count, conc_digest = table_digest(conc, table)
+
+ if seq_digest == conc_digest
+ puts " ok #{table.ljust(32)} #{seq_count} rows"
+ else
+ ok = false
+ puts " DIFF #{table.ljust(32)} sequential=#{seq_count} rows, concurrent=#{conc_count} rows"
+ end
+end
+
+puts(ok ? "\nIdentical (order-insensitive)." : "\nDifferences found.")
+exit(ok ? 0 : 1)