A platform for community discussion. Free, open, simple.
MT: Run conversion steps concurrently and split the heavy ones across cores
Gerhard Schlager12a065662765
diff --git a/Gemfile.lock b/Gemfile.lockindex 9d084599b39..389c38a25a7 100644--- a/Gemfile.lock+++ b/Gemfile.lock@@ -15,11 +15,12 @@ PATHmigrations-core (0.0.1)activesupportcolored2+ concurrent-rubydigest-xxhashextralite-bundlei18n+ jsonlru_redux- ojsamovarunicode-display_widthzeitwerkdiff --git a/migrations/README.md b/migrations/README.mdindex 18730b6751d..8db0d6b2465 100644--- a/migrations/README.md+++ b/migrations/README.md@@ -45,6 +45,54 @@ the same pattern. The discard mechanism itself is library-specific — mysql2 haneeds one at all: connect, fork an empty child that exits normally, wait for it, and queryagain from the parent (see the fork-safety specs in `postgres_spec.rb`).+### Partitioning large steps++Most steps run in a single worker. A handful are large enough that it's worth+splitting them across CPU cores, so the framework can run one worker per chunk of+the source. A step opts in from its `source` block:++```ruby+source do+ reads_table "topic_users", where: "user_id > 0"+ partition_by :topic_id+end+```++`reads_table` is the part that reads a whole table: it defines `items`+(`SELECT * FROM topic_users WHERE …`) and `max_progress` (the row count), filtered+by `where`. It works on its own, without partitioning — a plain table-copy step+declares just `reads_table` and writes neither method. `partition_by` adds the+split: it takes the key (normally a single indexed column, so each chunk is an+index range scan; pass an array for a composite key) and reuses the table and+filter from `reads_table`, so it only needs the column. When both are present the+generated queries add the chunk to their `WHERE` automatically.++Override `items` when you need specific columns, a join, or a particular order —+and then add `partition_slice` to its `WHERE` yourself:++```ruby+def items+ @source_db.query("SELECT id, name FROM topic_users WHERE #{partition_slice} AND name IS NOT NULL")+end+```++The framework does the rest. Before forking, it asks the adapter for the chunk+boundaries — evenly sized chunks over a numeric key, or a sorted-key scan for a+text/UUID/composite key. It then forks one worker per chunk; each worker reads+its `[lower, upper)` slice (that's what `partition_slice` expands to), writes its+own SQLite shard, and the shards are merged back into the run database.++Two things to get right:++- **In a custom query, add `partition_slice` to the `WHERE`.** Miss it and each+ worker reads the whole source instead of its slice — duplicated work and wrong+ counts.+- **Only partition order-independent steps.** Workers run concurrently and their+ output is merged, so there is no global order across the step. A running total+ or a sequence number across all rows can't be partitioned. Deduplication can,+ but do it in the source query (`DISTINCT ON`, a window function, a view) and+ partition on the dedup key, rather than keeping state in `process`.+## Schema DSLThe schema DSL lives in `migrations/tooling/lib/migrations/tooling/schema/dsl/`. Config sources@@ -100,11 +148,3 @@ bin/lint --fix path/to/file```Uses both rubocop and syntax_tree. Always lint changed files.--## Known issues--- **Parallel step items must be hashes (or other non-scalar JSON values).** Worker processes- receive their items as an Oj stream over a pipe, and Oj's stream parser can only detect the- end of a bare scalar (a number or string) once the next byte arrives — so a step whose- `items` yields scalars stalls the worker pipeline. Wrap scalar items in a hash- (e.g. `{ id: ... }`). This should go away if we switch the worker pipes from Oj to JSON.diff --git a/migrations/converters/lib/migrations-converters.rb b/migrations/converters/lib/migrations-converters.rbindex 15bb421d3c9..467d8a67ae3 100644--- a/migrations/converters/lib/migrations-converters.rb+++ b/migrations/converters/lib/migrations-converters.rb@@ -1,4 +1,12 @@# frozen_string_literal: truerequire "migrations-core"++# `pg` isn't safe to `require` from several threads at once: doing so races+# pg.rb's own body and it re-initializes its constants with a warning. The+# Postgres adapter that pulls it in is autoloaded lazily and first referenced+# inside the scheduler's concurrent coordinator threads and forks, so load it+# here on the main thread, before any of that starts.+require "pg"+require_relative "migrations/converters"diff --git a/migrations/converters/lib/migrations/converters/adapter/postgres.rb b/migrations/converters/lib/migrations/converters/adapter/postgres.rbindex 274076d6c8a..0401ad8b51f 100644--- a/migrations/converters/lib/migrations/converters/adapter/postgres.rb+++ b/migrations/converters/lib/migrations/converters/adapter/postgres.rb@@ -6,6 +6,8 @@ module Migrationsmodule Convertersmodule Adapterclass Postgres+ include Conversion::PartitionSource+class DiscardedError < StandardErrorend@@ -34,27 +36,24 @@ module Migrationsendend- def query_first_row(sql)- connection.exec(sql).first- end-- def query_value(sql, column = nil)- if (row = query_first_row(sql))- column ? row[column.to_sym] : row.values.first- else- nil- end+ def query_value(sql)+ query_first_row(sql)&.values&.firstenddef count(sql)query_value(sql).to_iend- # Reads every row of `table`, optionally narrowed by the `where` body- # (nil reads the whole table). Backs the `reads_table` default in a step's- # source; the table name is quoted here so the SQL stays in the adapter.- def select_all(table, where: nil)- query("SELECT * FROM #{connection.quote_ident(table)}#{where_clause(where)}")+ # Reads every row of `table`, optionally narrowed by the `where` body (nil+ # reads the whole table) and sorted by `order` (a column or array of them).+ # Backs the `reads_table` default in a step's source; a partitioned step+ # orders by its key so each worker writes its shard in key order — sequential+ # SQLite index inserts in the shard and, later, in the merge.+ def select_all(table, where: nil, order: nil)+ query(+ "SELECT * FROM #{connection.quote_ident(table)}" \+ "#{where_clause(where)}#{order_clause(order)}",+ )end# The row count of `table`, optionally narrowed by `where`. Backs@@ -63,6 +62,59 @@ module Migrationscount("SELECT COUNT(*) FROM #{connection.quote_ident(table)}#{where_clause(where)}")end+ # The following back {Conversion::Partitioner}; see its primitive list.++ # The key's min and max. Answered from the key's index, so this stays cheap+ # and doesn't scan the table.+ def partition_bounds(column, from, base)+ row =+ query_first_row(+ "SELECT MIN(#{column}) AS min, MAX(#{column}) AS max " \+ "FROM #{connection.quote_ident(from)}#{where_clause(base)}",+ )+ [row[:min], row[:max]]+ end++ # The planner's row estimate (`pg_class.reltuples`), so the partitioner can+ # tell a dense key from a sparse one without a COUNT(*) scan. It's the whole+ # table's estimate and ignores `base`, which is fine for a magnitude check;+ # it is -1 before the table is analysed.+ def estimated_row_count(from)+ query_value(+ "SELECT reltuples::bigint FROM pg_class WHERE oid = #{quote(from)}::regclass",+ ).to_i+ end++ # The chunk lower bounds, computed in SQL: `NTILE` splits the sorted key+ # into `count` equal-sized buckets and `DISTINCT ON` takes each bucket's+ # first value, so only the boundaries cross the wire, not the whole key+ # column. Works the same for a scalar or a composite key. This is the+ # optional fast path the {Conversion::Partitioner} prefers over streaming.+ def boundaries_by_scan(key, from, base, count)+ select = Array(key).join(", ")+ rows = query(<<~SQL)+ SELECT DISTINCT ON (bucket) #{select}+ FROM (+ SELECT #{select}, NTILE(#{count}) OVER (ORDER BY #{select}) AS bucket+ FROM #{connection.quote_ident(from)}#{where_clause(base)}+ ) buckets+ ORDER BY bucket, #{select}+ SQL+ rows.map { |row| boundary_value(row, key) }.uniq+ end++ # The WHERE body limiting a query to the chunk `[lower, upper)` of `key`+ # (one column, or an array for a composite key), AND-ed with `base`. A+ # composite key compares as a row value: `(a, b) >= (a0, b0)`.+ def chunk_filter(key, lower, upper, base: nil)+ return base if lower.nil?++ expression = key_expression(key)+ conditions = ["#{expression} >= #{value_expression(key, lower)}"]+ conditions << "#{expression} < #{value_expression(key, upper)}" unless upper.nil?+ [base, *conditions].compact.join(" AND ")+ end+def close@connection.finish if @connection && [email protected]?@connection = nil@@ -92,15 +144,6 @@ module Migrations@discarded = trueend- def reset- connection.reset- configure_connection- end-- def escape_string(str)- connection.escape_string(str)- end-def encode_array(array)@array_encoder ||= PG::TextEncoder::Array.new@@ -109,6 +152,14 @@ module Migrationsprivate+ def query_first_row(sql)+ connection.exec(sql).first+ end++ def escape_string(str)+ connection.escape_string(str)+ end+# `" WHERE <filter>"`, or "" when there's no filter — so a missing filter# reads the whole table instead of leaning on a `WHERE TRUE` that not every# dialect accepts.@@ -116,6 +167,29 @@ module Migrationsfilter ? " WHERE #{filter}" : ""end+ # `" ORDER BY <cols>"`, or "" when unordered. Accepts a column or an array.+ def order_clause(order)+ order ? " ORDER BY #{Array(order).join(", ")}" : ""+ end++ def boundary_value(row, key)+ return row.values.first unless key.is_a?(Array)+ key.map { |part| row[part.to_sym] }+ end++ def key_expression(key)+ key.is_a?(Array) ? "(#{key.join(", ")})" : key.to_s+ end++ def value_expression(key, value)+ return quote(value) unless key.is_a?(Array)+ "(#{value.map { |part| quote(part) }.join(", ")})"+ end++ def quote(value)+ value.is_a?(Numeric) ? value.to_s : "'#{escape_string(value.to_s)}'"+ end+def connectionif @discardedraise DiscardedError,diff --git a/migrations/converters/lib/migrations/converters/cli/convert_command.rb b/migrations/converters/lib/migrations/converters/cli/convert_command.rbindex 448fdbf724d..da5874b4fe9 100644--- a/migrations/converters/lib/migrations/converters/cli/convert_command.rb+++ b/migrations/converters/lib/migrations/converters/cli/convert_command.rb@@ -22,6 +22,15 @@ module Migrations"Skip the specified steps (comma-separated).",default: [],type: STEP_LIST+ option "--max-parallel-steps <count>",+ "Maximum number of steps to run at the same time. " \+ "Caps the source DB connections a run opens; " \+ "defaults to the worker budget (usable CPUs minus one).",+ type: Integer+ option "--no-fork",+ "Run each step inline in one process instead of forking workers. " \+ "Forces serial execution; meant for debugging (a breakpoint in a " \+ "step's `process` stops in the main run, not an unreachable child)."endone :converter_type, "The converter to run (e.g. discourse)."@@ -39,10 +48,15 @@ module Migrationssettings = load_settings(type)- Database.reset!(settings[:intermediate_db][:path]) if @options[:reset]+ Database.delete_database(settings[:intermediate_db][:path]) if @options[:reset]converter = "migrations/converters/#{type}/converter".camelize.constantize- converter.new(settings).run(only_steps: @options[:only], skip_steps: @options[:skip])+ converter.new(settings).run(+ only_steps: @options[:only],+ skip_steps: @options[:skip],+ max_parallel_steps: @options[:max_parallel_steps],+ no_fork: @options[:no_fork] || false,+ )endprivatediff --git a/migrations/converters/lib/migrations/converters/discourse/converter.rb b/migrations/converters/lib/migrations/converters/discourse/converter.rbindex 1dcd91eb7f5..093f699f23c 100644--- a/migrations/converters/lib/migrations/converters/discourse/converter.rb+++ b/migrations/converters/lib/migrations/converters/discourse/converter.rb@@ -4,13 +4,10 @@ module Migrationsmodule Convertersmodule Discourseclass Converter < Conversion::Base- def initialize(settings)- super- @source_db = Adapter::Postgres.new(settings[:source_db])- end-+ # Steps run concurrently and a Postgres connection can't be shared, so each+ # step gets its own adapter; the step's source closes it in its `cleanup`.def step_args(step_class)- { source_db: @source_db }+ { source_db: Adapter::Postgres.new(settings[:source_db]) }endendenddiff --git a/migrations/converters/lib/migrations/converters/discourse/steps/badge_groupings.rb b/migrations/converters/lib/migrations/converters/discourse/steps/badge_groupings.rbindex cc0d47f1949..66d1a9b2cd8 100644--- a/migrations/converters/lib/migrations/converters/discourse/steps/badge_groupings.rb+++ b/migrations/converters/lib/migrations/converters/discourse/steps/badge_groupings.rb@@ -3,7 +3,7 @@module Migrationsmodule Convertersmodule Discourse- class BadgeGroupings < Conversion::ProgressStep+ class BadgeGroupings < Conversion::Stepsource do# Skip the system badge groupings.reads_table "badge_groupings", where: "id NOT IN (1, 2, 3, 4, 5)"diff --git a/migrations/converters/lib/migrations/converters/discourse/steps/badges.rb b/migrations/converters/lib/migrations/converters/discourse/steps/badges.rbindex 2724bb81420..bac2243edab 100644--- a/migrations/converters/lib/migrations/converters/discourse/steps/badges.rb+++ b/migrations/converters/lib/migrations/converters/discourse/steps/badges.rb@@ -3,7 +3,7 @@module Migrationsmodule Convertersmodule Discourse- class Badges < Conversion::ProgressStep+ class Badges < Conversion::Stepsource dodef max_progress@source_db.count <<~SQLdiff --git a/migrations/converters/lib/migrations/converters/discourse/steps/categories.rb b/migrations/converters/lib/migrations/converters/discourse/steps/categories.rbindex 1afa697cd02..d30f5ba3d6c 100644--- a/migrations/converters/lib/migrations/converters/discourse/steps/categories.rb+++ b/migrations/converters/lib/migrations/converters/discourse/steps/categories.rb@@ -3,7 +3,7 @@module Migrationsmodule Convertersmodule Discourse- class Categories < Conversion::ProgressStep+ class Categories < Conversion::StepSEEDED_CATEGORY_SETTINGS = %w[uncategorized_category_idmeta_category_iddiff --git a/migrations/converters/lib/migrations/converters/discourse/steps/category_custom_fields.rb b/migrations/converters/lib/migrations/converters/discourse/steps/category_custom_fields.rbindex 9f93e53adfd..34823f1aea6 100644--- a/migrations/converters/lib/migrations/converters/discourse/steps/category_custom_fields.rb+++ b/migrations/converters/lib/migrations/converters/discourse/steps/category_custom_fields.rb@@ -3,7 +3,7 @@module Migrationsmodule Convertersmodule Discourse- class CategoryCustomFields < Conversion::ProgressStep+ class CategoryCustomFields < Conversion::Stepsource { reads_table "category_custom_fields" }processor dodiff --git a/migrations/converters/lib/migrations/converters/discourse/steps/category_moderation_groups.rb b/migrations/converters/lib/migrations/converters/discourse/steps/category_moderation_groups.rbindex 2fca4046e9d..d5c35cc08f6 100644--- a/migrations/converters/lib/migrations/converters/discourse/steps/category_moderation_groups.rb+++ b/migrations/converters/lib/migrations/converters/discourse/steps/category_moderation_groups.rb@@ -3,7 +3,7 @@module Migrationsmodule Convertersmodule Discourse- class CategoryModerationGroups < Conversion::ProgressStep+ class CategoryModerationGroups < Conversion::Stepsource { reads_table "category_moderation_groups", where: "group_id > 0" }processor dodiff --git a/migrations/converters/lib/migrations/converters/discourse/steps/category_users.rb b/migrations/converters/lib/migrations/converters/discourse/steps/category_users.rbindex 06df06dfc86..858fafc8d0b 100644--- a/migrations/converters/lib/migrations/converters/discourse/steps/category_users.rb+++ b/migrations/converters/lib/migrations/converters/discourse/steps/category_users.rb@@ -3,7 +3,7 @@module Migrationsmodule Convertersmodule Discourse- class CategoryUsers < Conversion::ProgressStep+ class CategoryUsers < Conversion::Stepsource { reads_table "category_users", where: "user_id > 0" }processor dodiff --git a/migrations/converters/lib/migrations/converters/discourse/steps/group_users.rb b/migrations/converters/lib/migrations/converters/discourse/steps/group_users.rbindex 554a2db2b2e..97701b46794 100644--- a/migrations/converters/lib/migrations/converters/discourse/steps/group_users.rb+++ b/migrations/converters/lib/migrations/converters/discourse/steps/group_users.rb@@ -3,7 +3,7 @@module Migrationsmodule Convertersmodule Discourse- class GroupUsers < Conversion::ProgressStep+ class GroupUsers < Conversion::Stepsource do# Skip the automatic Trust Level groups.reads_table "group_users", where: "group_id NOT IN (10, 11, 12, 13, 14) AND user_id > 0"diff --git a/migrations/converters/lib/migrations/converters/discourse/steps/groups.rb b/migrations/converters/lib/migrations/converters/discourse/steps/groups.rbindex b1f6a596d3a..239e08dad06 100644--- a/migrations/converters/lib/migrations/converters/discourse/steps/groups.rb+++ b/migrations/converters/lib/migrations/converters/discourse/steps/groups.rb@@ -3,7 +3,7 @@module Migrationsmodule Convertersmodule Discourse- class Groups < Conversion::ProgressStep+ class Groups < Conversion::Stepsource dodef max_progress@source_db.count <<~SQLdiff --git a/migrations/converters/lib/migrations/converters/discourse/steps/muted_users.rb b/migrations/converters/lib/migrations/converters/discourse/steps/muted_users.rbindex 8af34ee4516..1a9d422b855 100644--- a/migrations/converters/lib/migrations/converters/discourse/steps/muted_users.rb+++ b/migrations/converters/lib/migrations/converters/discourse/steps/muted_users.rb@@ -3,7 +3,7 @@module Migrationsmodule Convertersmodule Discourse- class MutedUsers < Conversion::ProgressStep+ class MutedUsers < Conversion::Stepsource { reads_table "muted_users" }processor dodiff --git a/migrations/converters/lib/migrations/converters/discourse/steps/permalink_normalizations.rb b/migrations/converters/lib/migrations/converters/discourse/steps/permalink_normalizations.rbindex a181a451a4f..a351b4cc83d 100644--- a/migrations/converters/lib/migrations/converters/discourse/steps/permalink_normalizations.rb+++ b/migrations/converters/lib/migrations/converters/discourse/steps/permalink_normalizations.rb@@ -4,18 +4,35 @@ module Migrationsmodule Convertersmodule Discourseclass PermalinkNormalizations < Conversion::Step- def execute- normalizations = @source_db.query_value <<~SQL- SELECT value- FROM site_settings- WHERE name = 'permalink_normalizations'- SQL+ source do+ def max_progress+ normalizations.size+ end- return if normalizations.blank?+ def items+ normalizations+ end- normalizations- .split("|")- .each { |normalization| IntermediateDB::PermalinkNormalization.create(normalization:) }+ private++ def normalizations+ @normalizations ||=+ begin+ value = @source_db.query_value(<<~SQL)+ SELECT value+ FROM site_settings+ WHERE name = 'permalink_normalizations'+ SQL++ value.present? ? value.split("|") : []+ end+ end+ end++ processor do+ def process(item)+ IntermediateDB::PermalinkNormalization.create(normalization: item)+ endendendenddiff --git a/migrations/converters/lib/migrations/converters/discourse/steps/site_settings.rb b/migrations/converters/lib/migrations/converters/discourse/steps/site_settings.rbindex 963eabb6fd9..ec127beaa95 100644--- a/migrations/converters/lib/migrations/converters/discourse/steps/site_settings.rb+++ b/migrations/converters/lib/migrations/converters/discourse/steps/site_settings.rb@@ -3,7 +3,7 @@module Migrationsmodule Convertersmodule Discourse- class SiteSettings < Conversion::ProgressStep+ class SiteSettings < Conversion::Stepsource dodef max_progress@source_db.count <<~SQLdiff --git a/migrations/converters/lib/migrations/converters/discourse/steps/tag_group_memberships.rb b/migrations/converters/lib/migrations/converters/discourse/steps/tag_group_memberships.rbindex 11b4d223dee..5966218e6cd 100644--- a/migrations/converters/lib/migrations/converters/discourse/steps/tag_group_memberships.rb+++ b/migrations/converters/lib/migrations/converters/discourse/steps/tag_group_memberships.rb@@ -3,7 +3,7 @@module Migrationsmodule Convertersmodule Discourse- class TagGroupMemberships < Conversion::ProgressStep+ class TagGroupMemberships < Conversion::Stepsource { reads_table "tag_group_memberships" }processor dodiff --git a/migrations/converters/lib/migrations/converters/discourse/steps/tag_group_permissions.rb b/migrations/converters/lib/migrations/converters/discourse/steps/tag_group_permissions.rbindex 0d3b341b93e..6506da03102 100644--- a/migrations/converters/lib/migrations/converters/discourse/steps/tag_group_permissions.rb+++ b/migrations/converters/lib/migrations/converters/discourse/steps/tag_group_permissions.rb@@ -3,7 +3,7 @@module Migrationsmodule Convertersmodule Discourse- class TagGroupPermissions < Conversion::ProgressStep+ class TagGroupPermissions < Conversion::Stepsource { reads_table "tag_group_permissions" }processor dodiff --git a/migrations/converters/lib/migrations/converters/discourse/steps/tag_groups.rb b/migrations/converters/lib/migrations/converters/discourse/steps/tag_groups.rbindex 68bbdcd2925..db912b6f5cb 100644--- a/migrations/converters/lib/migrations/converters/discourse/steps/tag_groups.rb+++ b/migrations/converters/lib/migrations/converters/discourse/steps/tag_groups.rb@@ -3,7 +3,7 @@module Migrationsmodule Convertersmodule Discourse- class TagGroups < Conversion::ProgressStep+ class TagGroups < Conversion::Stepsource { reads_table "tag_groups" }processor dodiff --git a/migrations/converters/lib/migrations/converters/discourse/steps/tag_users.rb b/migrations/converters/lib/migrations/converters/discourse/steps/tag_users.rbindex f2fbe4507d1..3951db5eec7 100644--- a/migrations/converters/lib/migrations/converters/discourse/steps/tag_users.rb+++ b/migrations/converters/lib/migrations/converters/discourse/steps/tag_users.rb@@ -3,7 +3,7 @@module Migrationsmodule Convertersmodule Discourse- class TagUsers < Conversion::ProgressStep+ class TagUsers < Conversion::Stepsource { reads_table "tag_users", where: "user_id > 0" }processor dodiff --git a/migrations/converters/lib/migrations/converters/discourse/steps/tags.rb b/migrations/converters/lib/migrations/converters/discourse/steps/tags.rbindex a75f12d45c5..06aed00f817 100644--- a/migrations/converters/lib/migrations/converters/discourse/steps/tags.rb+++ b/migrations/converters/lib/migrations/converters/discourse/steps/tags.rb@@ -3,7 +3,7 @@module Migrationsmodule Convertersmodule Discourse- class Tags < Conversion::ProgressStep+ class Tags < Conversion::Stepsource { reads_table "tags" }processor dodiff --git a/migrations/converters/lib/migrations/converters/discourse/steps/topic_allowed_groups.rb b/migrations/converters/lib/migrations/converters/discourse/steps/topic_allowed_groups.rbindex 923170d3649..d28106a4abc 100644--- a/migrations/converters/lib/migrations/converters/discourse/steps/topic_allowed_groups.rb+++ b/migrations/converters/lib/migrations/converters/discourse/steps/topic_allowed_groups.rb@@ -3,7 +3,7 @@module Migrationsmodule Convertersmodule Discourse- class TopicAllowedGroups < Conversion::ProgressStep+ class TopicAllowedGroups < Conversion::Stepsource { reads_table "topic_allowed_groups" }processor dodiff --git a/migrations/converters/lib/migrations/converters/discourse/steps/topic_allowed_users.rb b/migrations/converters/lib/migrations/converters/discourse/steps/topic_allowed_users.rbindex dcc24f45eb4..7421904d440 100644--- a/migrations/converters/lib/migrations/converters/discourse/steps/topic_allowed_users.rb+++ b/migrations/converters/lib/migrations/converters/discourse/steps/topic_allowed_users.rb@@ -3,7 +3,7 @@module Migrationsmodule Convertersmodule Discourse- class TopicAllowedUsers < Conversion::ProgressStep+ class TopicAllowedUsers < Conversion::Stepsource { reads_table "topic_allowed_users", where: "user_id > 0" }processor dodiff --git a/migrations/converters/lib/migrations/converters/discourse/steps/topic_tags.rb b/migrations/converters/lib/migrations/converters/discourse/steps/topic_tags.rbindex 5d281c950c2..5f5e33161ce 100644--- a/migrations/converters/lib/migrations/converters/discourse/steps/topic_tags.rb+++ b/migrations/converters/lib/migrations/converters/discourse/steps/topic_tags.rb@@ -3,7 +3,7 @@module Migrationsmodule Convertersmodule Discourse- class TopicTags < Conversion::ProgressStep+ class TopicTags < Conversion::Stepsource { reads_table "topic_tags" }processor dodiff --git a/migrations/converters/lib/migrations/converters/discourse/steps/topic_users.rb b/migrations/converters/lib/migrations/converters/discourse/steps/topic_users.rbindex dfc4f1bcd6b..b7ee85cc9b8 100644--- a/migrations/converters/lib/migrations/converters/discourse/steps/topic_users.rb+++ b/migrations/converters/lib/migrations/converters/discourse/steps/topic_users.rb@@ -3,8 +3,11 @@module Migrationsmodule Convertersmodule Discourse- class TopicUsers < Conversion::ProgressStep- source { reads_table "topic_users", where: "user_id > 0" }+ class TopicUsers < Conversion::Step+ source do+ reads_table "topic_users", where: "user_id > 0"+ partition_by :topic_id+ endprocessor dodef process(item)diff --git a/migrations/converters/lib/migrations/converters/discourse/steps/topics.rb b/migrations/converters/lib/migrations/converters/discourse/steps/topics.rbindex e8c0bceb2e3..8ed78bf5b23 100644--- a/migrations/converters/lib/migrations/converters/discourse/steps/topics.rb+++ b/migrations/converters/lib/migrations/converters/discourse/steps/topics.rb@@ -3,7 +3,7 @@module Migrationsmodule Convertersmodule Discourse- class Topics < Conversion::ProgressStep+ class Topics < Conversion::Stepsource dodef max_progress@source_db.count <<~SQL@@ -20,6 +20,7 @@ module Migrationsup.user_id AS og_image_user_idFROM topics tLEFT JOIN uploads up ON t.og_image_upload_id = up.id+ ORDER BY t.idSQLendenddiff --git a/migrations/converters/lib/migrations/converters/discourse/steps/user_associated_accounts.rb b/migrations/converters/lib/migrations/converters/discourse/steps/user_associated_accounts.rbindex 9233f441dac..125be371802 100644--- a/migrations/converters/lib/migrations/converters/discourse/steps/user_associated_accounts.rb+++ b/migrations/converters/lib/migrations/converters/discourse/steps/user_associated_accounts.rb@@ -3,7 +3,7 @@module Migrationsmodule Convertersmodule Discourse- class UserAssociatedAccounts < Conversion::ProgressStep+ class UserAssociatedAccounts < Conversion::Stepsource { reads_table "user_associated_accounts", where: "user_id > 0" }processor dodiff --git a/migrations/converters/lib/migrations/converters/discourse/steps/user_custom_fields.rb b/migrations/converters/lib/migrations/converters/discourse/steps/user_custom_fields.rbindex 21b92e73cfe..4f9602837a9 100644--- a/migrations/converters/lib/migrations/converters/discourse/steps/user_custom_fields.rb+++ b/migrations/converters/lib/migrations/converters/discourse/steps/user_custom_fields.rb@@ -3,7 +3,7 @@module Migrationsmodule Convertersmodule Discourse- class UserCustomFields < Conversion::ProgressStep+ class UserCustomFields < Conversion::Stepsource dodef max_progress@source_db.count <<~SQLdiff --git a/migrations/converters/lib/migrations/converters/discourse/steps/user_emails.rb b/migrations/converters/lib/migrations/converters/discourse/steps/user_emails.rbindex 50c68ec3fbc..d766510c9c7 100644--- a/migrations/converters/lib/migrations/converters/discourse/steps/user_emails.rb+++ b/migrations/converters/lib/migrations/converters/discourse/steps/user_emails.rb@@ -3,7 +3,7 @@module Migrationsmodule Convertersmodule Discourse- class UserEmails < Conversion::ProgressStep+ class UserEmails < Conversion::Stepsource { reads_table "user_emails", where: "user_id > 0" }processor dodiff --git a/migrations/converters/lib/migrations/converters/discourse/steps/user_field_options.rb b/migrations/converters/lib/migrations/converters/discourse/steps/user_field_options.rbindex d9d7f81392e..643a9e75af5 100644--- a/migrations/converters/lib/migrations/converters/discourse/steps/user_field_options.rb+++ b/migrations/converters/lib/migrations/converters/discourse/steps/user_field_options.rb@@ -3,7 +3,7 @@module Migrationsmodule Convertersmodule Discourse- class UserFieldOptions < Conversion::ProgressStep+ class UserFieldOptions < Conversion::Stepsource { reads_table "user_field_options" }processor dodiff --git a/migrations/converters/lib/migrations/converters/discourse/steps/user_field_values.rb b/migrations/converters/lib/migrations/converters/discourse/steps/user_field_values.rbindex 81c24ddb8c9..b6ba0d841a1 100644--- a/migrations/converters/lib/migrations/converters/discourse/steps/user_field_values.rb+++ b/migrations/converters/lib/migrations/converters/discourse/steps/user_field_values.rb@@ -3,7 +3,7 @@module Migrationsmodule Convertersmodule Discourse- class UserFieldValues < Conversion::ProgressStep+ class UserFieldValues < Conversion::StepUSER_FIELD_PREFIX = "user_field_"# `USER_FIELD_PREFIX` as a `LIKE` pattern: the `_` characters are literaldiff --git a/migrations/converters/lib/migrations/converters/discourse/steps/user_fields.rb b/migrations/converters/lib/migrations/converters/discourse/steps/user_fields.rbindex ace334af6e1..bbb40e3a79b 100644--- a/migrations/converters/lib/migrations/converters/discourse/steps/user_fields.rb+++ b/migrations/converters/lib/migrations/converters/discourse/steps/user_fields.rb@@ -3,7 +3,7 @@module Migrationsmodule Convertersmodule Discourse- class UserFields < Conversion::ProgressStep+ class UserFields < Conversion::Stepsource { reads_table "user_fields" }processor dodiff --git a/migrations/converters/lib/migrations/converters/discourse/steps/user_options.rb b/migrations/converters/lib/migrations/converters/discourse/steps/user_options.rbindex 4e3bb9bc028..3a1fb033b61 100644--- a/migrations/converters/lib/migrations/converters/discourse/steps/user_options.rb+++ b/migrations/converters/lib/migrations/converters/discourse/steps/user_options.rb@@ -3,7 +3,7 @@module Migrationsmodule Convertersmodule Discourse- class UserOptions < Conversion::ProgressStep+ class UserOptions < Conversion::Stepsource { reads_table "user_options", where: "user_id > 0" }processor dodiff --git a/migrations/converters/lib/migrations/converters/discourse/steps/user_suspensions.rb b/migrations/converters/lib/migrations/converters/discourse/steps/user_suspensions.rbindex c4b015837c7..ea805f562ae 100644--- a/migrations/converters/lib/migrations/converters/discourse/steps/user_suspensions.rb+++ b/migrations/converters/lib/migrations/converters/discourse/steps/user_suspensions.rb@@ -3,7 +3,7 @@module Migrationsmodule Convertersmodule Discourse- class UserSuspensions < Conversion::ProgressStep+ class UserSuspensions < Conversion::Stepsource dodef max_progress@source_db.count <<~SQLdiff --git a/migrations/converters/lib/migrations/converters/discourse/steps/users.rb b/migrations/converters/lib/migrations/converters/discourse/steps/users.rbindex 206e2d2f14b..c1203eaf40f 100644--- a/migrations/converters/lib/migrations/converters/discourse/steps/users.rb+++ b/migrations/converters/lib/migrations/converters/discourse/steps/users.rb@@ -3,7 +3,7 @@module Migrationsmodule Convertersmodule Discourse- class Users < Conversion::ProgressStep+ class Users < Conversion::Stepsource dodef max_progress@source_db.count <<~SQLdiff --git a/migrations/converters/spec/lib/migrations/converters/adapter/postgres_spec.rb b/migrations/converters/spec/lib/migrations/converters/adapter/postgres_spec.rbindex 158728cc420..2a2e8705550 100644--- a/migrations/converters/spec/lib/migrations/converters/adapter/postgres_spec.rb+++ b/migrations/converters/spec/lib/migrations/converters/adapter/postgres_spec.rb@@ -24,6 +24,8 @@ RSpec.describe Migrations::Converters::Adapter::Postgres doallow(connection).to receive(:socket_io).and_return(socket_io)allow(connection).to receive(:finished?).and_return(false)allow(connection).to receive(:finish)+ allow(connection).to receive(:quote_ident) { |id| %("#{id}") }+ allow(connection).to receive(:escape_string) { |str| str }allow(socket_io).to receive(:reopen)end@@ -37,9 +39,106 @@ RSpec.describe Migrations::Converters::Adapter::Postgres doendend- describe "#select_all and #count_all" do- before { allow(connection).to receive(:quote_ident) { |id| %("#{id}") } }+ it "implements the whole PartitionSource interface" do+ create_adapter do |adapter|+ expect(Migrations::Conversion::PartitionSource.missing_from(adapter)).to be_empty+ end+ end++ describe "#partition_bounds" do+ it "returns the key's min and max" do+ allow(connection).to receive(:exec).with(/MIN\(topic_id\)/).and_return(+ [{ min: 0, max: 99 }],+ )+ create_adapter do |adapter|+ expect(adapter.partition_bounds(:topic_id, "topic_users", "user_id > 0")).to eq([0, 99])+ end+ end+ end++ describe "#estimated_row_count" do+ it "reads the planner's row estimate" do+ allow(connection).to receive(:exec).with(/reltuples/).and_return([{ reltuples: 1234 }])+ create_adapter { |adapter| expect(adapter.estimated_row_count("things")).to eq(1234) }+ end+ end++ describe "#boundaries_by_scan" do+ it "returns each bucket's first value" do+ create_adapter do |adapter|+ allow(adapter).to receive(:query).and_return([{ id: 0 }, { id: 50 }])+ expect(adapter.boundaries_by_scan(:id, "things", nil, 2)).to eq([0, 50])+ end+ end+ it "returns a tuple per bucket for a composite key" do+ create_adapter do |adapter|+ allow(adapter).to receive(:query).and_return(+ [{ topic_id: 1, user_id: 10 }, { topic_id: 2, user_id: 5 }],+ )+ expect(adapter.boundaries_by_scan(%i[topic_id user_id], "topic_users", nil, 2)).to eq(+ [[1, 10], [2, 5]],+ )+ end+ end++ it "drops duplicate boundaries from buckets that share a value" do+ create_adapter do |adapter|+ allow(adapter).to receive(:query).and_return([{ id: 5 }, { id: 5 }])+ expect(adapter.boundaries_by_scan(:id, "things", nil, 2)).to eq([5])+ end+ end+ end++ describe "#chunk_filter" do+ def chunk(lower, upper, base: "user_id > 0")+ create_adapter { |adapter| adapter.chunk_filter(:topic_id, lower, upper, base:) }+ end++ it "limits to a half-open numeric range, AND-ed with the base" do+ expect(chunk(25, 50)).to eq("user_id > 0 AND topic_id >= 25 AND topic_id < 50")+ end++ it "drops the upper bound for an open-ended chunk" do+ expect(chunk(75, nil)).to eq("user_id > 0 AND topic_id >= 75")+ end++ it "is just the base filter when there's no chunk" do+ expect(chunk(nil, nil)).to eq("user_id > 0")+ end++ it "is just the chunk conditions when there's no base" do+ create_adapter do |adapter|+ expect(adapter.chunk_filter(:topic_id, 25, 50)).to eq("topic_id >= 25 AND topic_id < 50")+ end+ end++ it "quotes a text key (UUID, etc.)" do+ create_adapter do |adapter|+ expect(adapter.chunk_filter(:id, "uuid-a", "uuid-b")).to eq(+ "id >= 'uuid-a' AND id < 'uuid-b'",+ )+ end+ end++ it "compares a composite key as a row value" do+ create_adapter do |adapter|+ expect(+ adapter.chunk_filter(%i[topic_id user_id], [1, 10], [2, 5], base: "user_id > 0"),+ ).to eq("user_id > 0 AND (topic_id, user_id) >= (1, 10) AND (topic_id, user_id) < (2, 5)")+ end+ end++ it "leaves the upper off for an open-ended composite chunk" do+ create_adapter do |adapter|+ expect(adapter.chunk_filter(%i[topic_id user_id], [2, 5], nil)).to eq(+ "(topic_id, user_id) >= (2, 5)",+ )+ end+ end+ end++ describe "#select_all and #count_all" doit "selects all rows from a quoted table" docreate_adapter do |adapter|allow(adapter).to receive(:query).with('SELECT * FROM "things" WHERE active').and_return(@@ -64,6 +163,24 @@ RSpec.describe Migrations::Converters::Adapter::Postgres doexpect(adapter.select_all("things")).to eq([:row])endend++ it "orders by the given key so the rows arrive sorted" do+ create_adapter do |adapter|+ allow(adapter).to receive(:query).with(+ 'SELECT * FROM "things" WHERE active ORDER BY id',+ ).and_return([:row])+ expect(adapter.select_all("things", where: "active", order: :id)).to eq([:row])+ end+ end++ it "orders by every column of a composite key" do+ create_adapter do |adapter|+ allow(adapter).to receive(:query).with(+ 'SELECT * FROM "things" ORDER BY topic_id, user_id',+ ).and_return([:row])+ expect(adapter.select_all("things", order: %i[topic_id user_id])).to eq([:row])+ end+ endenddescribe "#discard!" do@@ -83,7 +200,6 @@ RSpec.describe Migrations::Converters::Adapter::Postgres doexpect { adapter.exec("SELECT 1") }.to raise_error(described_class::DiscardedError)expect { adapter.query("SELECT 1") }.to raise_error(described_class::DiscardedError)expect { adapter.query_value("SELECT 1") }.to raise_error(described_class::DiscardedError)- expect { adapter.reset }.to raise_error(described_class::DiscardedError)endenddiff --git a/migrations/converters/spec/lib/migrations/converters/discourse/converter_spec.rb b/migrations/converters/spec/lib/migrations/converters/discourse/converter_spec.rbnew file mode 100644index 00000000000..c715616c198--- /dev/null+++ b/migrations/converters/spec/lib/migrations/converters/discourse/converter_spec.rb@@ -0,0 +1,24 @@+# frozen_string_literal: true++RSpec.describe Migrations::Converters::Discourse::Converter do+ describe "#step_args" do+ it "builds a fresh source adapter per step so concurrent steps don't share a connection" do+ adapters = [+ instance_double(Migrations::Converters::Adapter::Postgres),+ instance_double(Migrations::Converters::Adapter::Postgres),+ ]+ allow(Migrations::Converters::Adapter::Postgres).to receive(:new).and_return(*adapters)++ converter = described_class.new(source_db: { host: "localhost" })++ first = converter.step_args(:first_step)[:source_db]+ second = converter.step_args(:second_step)[:source_db]++ expect(first).to be(adapters[0])+ expect(second).to be(adapters[1])+ expect(Migrations::Converters::Adapter::Postgres).to have_received(:new).with(+ { host: "localhost" },+ ).twice+ end+ end+enddiff --git a/migrations/core/config/locales/migrations.en.yml b/migrations/core/config/locales/migrations.en.ymlindex 407eebf0773..1d6a82486b3 100644--- a/migrations/core/config/locales/migrations.en.yml+++ b/migrations/core/config/locales/migrations.en.yml@@ -15,6 +15,13 @@ en:errors:one: "%{number} error"other: "%{number} errors"+ finishing_up: "Finishing up…"+ total: "Total"+ steps:+ one: "%{number} step"+ other: "%{number} steps"+ steps_failed: "%{number} failed"+ steps_skipped: "%{number} skipped"cli:aborted: "Aborted"@@ -22,3 +29,8 @@ en:converter:default_step_title: "Converting %{type}"max_progress_calculation: "Calculating items took %{duration}"+ errors:+ header: "The conversion finished with errors:"+ step_failed: "%{title} failed: %{error_class}: %{error_message}"+ step_skipped: "%{title} was skipped because a dependency failed."+ merge_failed: "Merging a step's shard into the run database failed: %{error_class}: %{error_message}"diff --git a/migrations/core/lib/migrations/common/fork_manager.rb b/migrations/core/lib/migrations/common/fork_manager.rbindex 168bf27d382..3ea25fd4e13 100644--- a/migrations/core/lib/migrations/common/fork_manager.rb+++ b/migrations/core/lib/migrations/common/fork_manager.rb@@ -1,78 +1,104 @@# frozen_string_literal: truemodule Migrations+ # The fork hooks for a run. `before_fork` and `after_fork_parent` run around a+ # fork, so a connection can close before it and reopen after. `after_fork_child`+ # runs in the new child, e.g. to drop a connection it inherited.+ #+ # Steps add and remove hooks from several threads at once, so a mutex guards the+ # hook lists. A fork copies the child hooks under that mutex, so the child always+ # runs a consistent list.module ForkManager@before_fork_hooks = []@after_fork_parent_hooks = []@after_fork_child_hooks = []@execute_parent_forks = true+ @mutex = Mutex.newclass << selfdef with_batched_forks@execute_parent_forks = false- run_before_fork_hooks-- yield-- run_after_fork_parent_hooks- @execute_parent_forks = true+ @mutex.synchronize { run_before_fork_hooks }++ # Always run the after-fork hooks and reset the flag, even if forking+ # raises (e.g. `Errno::EAGAIN`/`ENOMEM` under fork pressure). Otherwise the+ # before-fork hooks' effects — a locked writer mutex, a closed run+ # connection — would never be undone and the run would hang instead of+ # failing the step.+ begin+ yield+ ensure+ @mutex.synchronize { run_after_fork_parent_hooks }+ @execute_parent_forks = true+ endenddef before_fork(&block)- if block- @before_fork_hooks << block- block- end+ return unless block+ @mutex.synchronize { @before_fork_hooks << block }+ blockenddef remove_before_fork(block)- @before_fork_hooks.delete(block)+ @mutex.synchronize { @before_fork_hooks.delete(block) }enddef after_fork_parent(&block)- if block- @after_fork_parent_hooks << block- block- end+ return unless block+ @mutex.synchronize { @after_fork_parent_hooks << block }+ blockenddef remove_after_fork_parent(block)- @after_fork_parent_hooks.delete(block)+ @mutex.synchronize { @after_fork_parent_hooks.delete(block) }enddef after_fork_child(&block)- if block- @after_fork_child_hooks << block- block- end+ return unless block+ @mutex.synchronize { @after_fork_child_hooks << block }+ blockenddef remove_after_fork_child(block)- @after_fork_child_hooks.delete(block)+ @mutex.synchronize { @after_fork_child_hooks.delete(block) }enddef fork- run_before_fork_hooks if @execute_parent_forks+ # Snapshot the child hooks under the lock for a consistent list, but fork+ # outside it so the child can't inherit the mutex held.+ child_hooks =+ if @execute_parent_forks+ @mutex.synchronize do+ run_before_fork_hooks+ @after_fork_child_hooks.dup+ end+ else+ @mutex.synchronize { @after_fork_child_hooks.dup }+ endpid =Process.fork do- run_after_fork_child_hooks+ child_hooks.each(&:call)yieldend- run_after_fork_parent_hooks if @execute_parent_forks+ @mutex.synchronize { run_after_fork_parent_hooks } if @execute_parent_forkspidenddef hook_count- @before_fork_hooks.size + @after_fork_parent_hooks.size + @after_fork_child_hooks.size+ @mutex.synchronize do+ @before_fork_hooks.size + @after_fork_parent_hooks.size + @after_fork_child_hooks.size+ endenddef clear!- @before_fork_hooks.clear- @after_fork_parent_hooks.clear- @after_fork_child_hooks.clear+ @mutex.synchronize do+ @before_fork_hooks.clear+ @after_fork_parent_hooks.clear+ @after_fork_child_hooks.clear+ endendprivate@@ -84,10 +110,6 @@ module Migrationsdef run_after_fork_parent_hooks@after_fork_parent_hooks.each(&:call)end-- def run_after_fork_child_hooks- @after_fork_child_hooks.each(&:call)- endendendenddiff --git a/migrations/core/lib/migrations/common/reporting/plain.rb b/migrations/core/lib/migrations/common/reporting/plain.rbindex b2a1600d043..6e417aa8564 100644--- a/migrations/core/lib/migrations/common/reporting/plain.rb+++ b/migrations/core/lib/migrations/common/reporting/plain.rb@@ -66,6 +66,10 @@ module Migrations@mutex.synchronize { @steps[id].total = max_progress if @steps[id] }end+ def report_concurrency(_id, _count)+ # No-op: the plain log skips the live fork/CPU count.+ end+def report_progress(id, current, skip_count, warning_count, error_count)@mutex.synchronize dostep = @steps[id]@@ -99,6 +103,21 @@ module Migrationsendend+ def report_finalizing_begin+ @mutex.synchronize { @output.puts(I18n.t("progressbar.finishing_up")) }+ end++ def report_summary(runtime:, total:, failed:, skipped:)+ tally = [I18n.t("progressbar.steps", count: total, number: total)]+ tally << I18n.t("progressbar.steps_failed", number: failed) if failed > 0+ tally << I18n.t("progressbar.steps_skipped", number: skipped) if skipped > 0+ @mutex.synchronize do+ @output.puts(+ "#{I18n.t("progressbar.total")}: #{tally.join(", ")} (#{format_duration(runtime)})",+ )+ end+ end+def close# nothing to free — we write plain lines and keep nothing openenddiff --git a/migrations/core/lib/migrations/common/reporting/reporter.rb b/migrations/core/lib/migrations/common/reporting/reporter.rbindex 1c6a8804459..fa4da5ac49a 100644--- a/migrations/core/lib/migrations/common/reporting/reporter.rb+++ b/migrations/core/lib/migrations/common/reporting/reporter.rb@@ -26,6 +26,26 @@ module MigrationsStepHandle.new(self, id)end+ # Shows a transient "finishing up" status while the block runs, then clears+ # it. Used for the tail of a run where every step is done but background work+ # (merging shards) is still finishing, so the display doesn't just sit at+ # 100%. Returns the block's value.+ def finalizing+ report_finalizing_begin+ yield+ ensure+ report_finalizing_end+ end++ # Prints the end-of-run summary line: the total runtime and how many steps+ # ran (with any that failed or were skipped).+ # @param runtime [Float] the run's wall-clock seconds+ # @param total [Integer] how many steps the run had+ # @param failed [Integer] how many failed+ # @param skipped [Integer] how many were skipped because a dependency failed+ def report_summary(runtime:, total:, failed:, skipped:)+ end+# The run is over and nothing else will be reported. Free anything held# here. Called once per run, also when the run fails.def close@@ -47,6 +67,10 @@ module Migrationsraise NotImplementedErrorend+ def report_concurrency(_id, _count)+ raise NotImplementedError+ end+def report_progress(_id, _current, _skip_count, _warning_count, _error_count)raise NotImplementedErrorend@@ -56,6 +80,13 @@ module Migrationsraise NotImplementedErrorend+ # Show and clear the transient "finishing up" status; `finalizing` wraps them.+ def report_finalizing_begin+ end++ def report_finalizing_end+ end+# A handle for one step. It holds the step's id, so everything it reports# goes to the right step, no matter which thread calls it.class StepHandle@@ -68,11 +99,19 @@ module Migrations@reporter.report_notice(@id, message)end+ def report_concurrency(count)+ @reporter.report_concurrency(@id, count)+ end++ def begin_progress(max_progress:)+ @reporter.report_progress_begin(@id, max_progress)+ Progress.new(@reporter, @id)+ end+# Yields a {Progress}. You may call its `update` from any thread. Pass# `max_progress: nil` when the total is not known.def with_progress(max_progress:)- @reporter.report_progress_begin(@id, max_progress)- yield Progress.new(@reporter, @id)+ yield begin_progress(max_progress:)end# Ends the step. It runs from an `ensure`, so `$!` (the exception indiff --git a/migrations/core/lib/migrations/common/reporting/tui.rb b/migrations/core/lib/migrations/common/reporting/tui.rbindex 218db299059..61394fb2b1a 100644--- a/migrations/core/lib/migrations/common/reporting/tui.rb+++ b/migrations/core/lib/migrations/common/reporting/tui.rb@@ -8,10 +8,10 @@ module Migrations# terminal history.## The code that reports progress never draws anything itself. This class is a- # thread-safe front end: each call only adds an event to a queue. A separate- # thread reads the queue, updates the model, and redraws at a fixed rate. That- # is what lets one thread report progress while another one started the step,- # without the output getting mixed up.+ # thread-safe front end: callers only hand off events. A separate thread+ # applies them, updates the model, and redraws at a fixed rate. That lets one+ # thread report progress while another started the step, without the output+ # getting mixed up.## The drawing itself is in {Renderer}. It runs on one thread only and has no# side effects apart from its output and clock (both can be injected), so it@@ -20,6 +20,8 @@ module Migrationsdef initialize(fps: 10, output: $stdout, titles: [])super()@queue = Thread::Queue.new+ @progress = {}+ @progress_mutex = Mutex.new@renderer = Renderer.new(output:, titles:)@frame = 1.0 / fps@closed = false@@ -43,14 +45,34 @@ module Migrationsenqueue([:progress_begin, id, max_progress])end+ def report_concurrency(id, count)+ enqueue([:concurrency, id, count])+ end+def report_progress(id, current, skip_count, warning_count, error_count)- enqueue([:progress, id, current, skip_count, warning_count, error_count])+ # Progress is coalesced, not queued: one event per row across many steps+ # would keep the queue full and starve the repaint. Keep the latest only.+ @progress_mutex.synchronize do+ @progress[id] = [current, skip_count, warning_count, error_count]+ endenddef report_finish(id, outcome)enqueue([:finish, id, outcome])end+ def report_finalizing_begin+ enqueue([:finalizing_begin])+ end++ def report_finalizing_end+ enqueue([:finalizing_end])+ end++ def report_summary(runtime:, total:, failed:, skipped:)+ enqueue([:summary, runtime, total, failed, skipped])+ end+def closereturn if @closed@closed = true@@ -94,12 +116,23 @@ module Migrationsloop doframe_started_at = monotonicbreak if drain_until_close+ apply_coalesced_progress@renderer.consume_resize@renderer.repaintsleep_rest_of_frame(frame_started_at)endend+ def apply_coalesced_progress+ snapshot = @progress_mutex.synchronize { @progress.dup }+ snapshot.each { |id, values| @renderer.apply([:progress, id, *values]) }+ end++ def flush_progress(id)+ values = @progress_mutex.synchronize { @progress.delete(id) }+ @renderer.apply([:progress, id, *values]) if values+ end+# Sleeps for whatever is left of this frame's time, so repaints happen at a# steady rate instead of as fast as the loop can spin.def sleep_rest_of_frame(frame_started_at)@@ -108,9 +141,13 @@ module Migrationsenddef drain_until_close- until @queue.empty?+ # Drain only what's already queued; events arriving while we drain wait+ # for the next frame, so a burst can't keep the loop here and starve the+ # repaint.+ @queue.size.times doevent = @queue.pop(true)return true if event[0] == :close+ flush_progress(event[1]) if event[0] == :finish@renderer.apply(event)endfalsediff --git a/migrations/core/lib/migrations/common/reporting/tui/ansi.rb b/migrations/core/lib/migrations/common/reporting/tui/ansi.rbindex 84c2f1a7b3f..b2f2cde834d 100644--- a/migrations/core/lib/migrations/common/reporting/tui/ansi.rb+++ b/migrations/core/lib/migrations/common/reporting/tui/ansi.rb@@ -15,6 +15,7 @@ module MigrationsYELLOW = "\e[33m"RED = "\e[31m"CYAN = "\e[36m"+ MAGENTA = "\e[35m"DIM = "\e[90m"ERASE_LINE = "\e[2K" # erase the whole line (also clears the terminal's wrap flag)ERASE_BELOW = "\e[0J" # erase from the cursor to the end of the screendiff --git a/migrations/core/lib/migrations/common/reporting/tui/renderer.rb b/migrations/core/lib/migrations/common/reporting/tui/renderer.rbindex 6bc1516d59b..e7f4f562bae 100644--- a/migrations/core/lib/migrations/common/reporting/tui/renderer.rb+++ b/migrations/core/lib/migrations/common/reporting/tui/renderer.rb@@ -26,8 +26,9 @@ module Migrations# Columns of a live row. ETA is left-aligned (it has its own "ETA " label);# the rest are right-aligned. A bold percent takes the place of a progress# bar: it lines up in columns, works for several steps at once, and avoids- # block characters that don't look the same on every terminal.- COLUMNS = %i[percent count elapsed eta rate].freeze+ # block characters that don't look the same on every terminal. Concurrency+ # sits last so it can't shift the columns the collapsed (finished) rows share.+ COLUMNS = %i[percent count elapsed eta rate concurrency].freezeLEFT_ALIGNED = %i[eta].freezeStep =@@ -44,6 +45,7 @@ module Migrations:rate,:rate_sampled_at,:rate_sampled_current,+ :concurrency,keyword_init: true,)@@ -61,6 +63,7 @@ module Migrations@live_count = 0@last_live = nil@resize_pending = false+ @finalizing = false@frame_buffer = +"" # reused on each repaint instead of making a new one# Reserve the column widths up front: a finished row scrolls into the# history and can't be realigned, so every column has to start at the@@ -76,6 +79,7 @@ module Migrationselapsed: TIME_WIDTH,eta: 0,rate: 0,+ concurrency: 0,}end@@ -149,7 +153,12 @@ module Migrationsrate: nil,rate_sampled_at: nil,rate_sampled_current: 0,+ concurrency: 1,)+ when :concurrency+ _, id, count = event+ step = @steps[id]+ step.concurrency = count if stepwhen :progress_begin_, id, max_progress = eventstep = @steps[id]@@ -178,6 +187,13 @@ module Migrationsstep.finished_at = nowstep.state = outcome # :done, :interrupted, or :failed@pending_permanent << collapsed_line(step)+ when :finalizing_begin+ @finalizing = true+ when :finalizing_end+ @finalizing = false+ when :summary+ _, runtime, total, failed, skipped = event+ @pending_permanent << summary_line(runtime, total, failed, skipped)endend@@ -193,6 +209,7 @@ module Migrationspermanent = @pending_permanent@pending_permanent = []live = final ? [] : format_live(running_steps)+ live += [finalizing_line] if @finalizing && !finalreturn if permanent.empty? && live.empty? && @live_count == 0return if !final && permanent.empty? && live == @last_live@@ -293,6 +310,7 @@ module Migrationselapsed: "",eta: "",rate: "",+ concurrency: concurrency_cell(step.concurrency),annot: "",}@@ -368,6 +386,36 @@ module Migrationslineend+ # The transient live line shown while the run finishes background work+ # (merging shards) after every step is done. The spinner animates it.+ def finalizing_line+ "#{spinner} #{Ansi::DIM}#{I18n.t("progressbar.finishing_up")}#{Ansi::RESET}"+ end++ # The permanent end-of-run line: a Σ, the total runtime in the elapsed+ # column (so it lines up under the per-step durations), and the step tally.+ def summary_line(runtime, total, failed, skipped)+ columns = [+ Ansi.pad("", @column_widths[:percent]),+ Ansi.pad("", @column_widths[:count]),+ Ansi.pad(format_duration(runtime), @column_widths[:elapsed], :right),+ ].join(" ")+ line = +"#{Ansi::BOLD}Σ#{Ansi::RESET} #{title_field(I18n.t("progressbar.total"))} "+ line << columns << summary_tally(total, failed, skipped)+ line+ end++ def summary_tally(total, failed, skipped)+ note = +" #{I18n.t("progressbar.steps", count: total, number: total)}"+ if failed > 0+ note << " #{Ansi::RED}#{I18n.t("progressbar.steps_failed", number: failed)}#{Ansi::RESET}"+ end+ if skipped > 0+ note << " #{Ansi::YELLOW}#{I18n.t("progressbar.steps_skipped", number: skipped)}#{Ansi::RESET}"+ end+ note+ end+def status_glyph(step)case step.statewhen :done@@ -458,6 +506,14 @@ module MigrationsSPINNER[(now * SPINNER_RATE).to_i % SPINNER.size]end+ def concurrency_cell(count)+ return "" if count <= 1++ label = "#{count}×"+ return label unless @color+ "#{Ansi::MAGENTA}#{label}#{Ansi::RESET}"+ end+def terminal_columns@forced_width || @output.winsize[1]rescue StandardErrordiff --git a/migrations/core/lib/migrations/common/step_dependencies.rb b/migrations/core/lib/migrations/common/step_dependencies.rbindex 88dee1874fc..0233ef53d65 100644--- a/migrations/core/lib/migrations/common/step_dependencies.rb+++ b/migrations/core/lib/migrations/common/step_dependencies.rb@@ -3,7 +3,7 @@module Migrations# Class-level dependency metadata for migration steps. Meant to be `extend`ed# by a step base class (`Migrations::Importer::Step`,- # `Migrations::Conversion::StepBase`); subclasses inherit the macros, and the+ # `Migrations::Conversion::Step`); subclasses inherit the macros, and the# resulting `dependencies` and `priority` are queryable without instantiating# a step. `Migrations::TopologicalSorter` consumes both to order steps.#diff --git a/migrations/core/lib/migrations/conversion/base.rb b/migrations/core/lib/migrations/conversion/base.rbindex b1bcbdb3bd2..d489083660d 100644--- a/migrations/core/lib/migrations/conversion/base.rb+++ b/migrations/core/lib/migrations/conversion/base.rb@@ -9,7 +9,7 @@ module Migrations@settings = settingsend- def run(only_steps: [], skip_steps: [])+ def run(only_steps: [], skip_steps: [], max_parallel_steps: nil, no_fork: false)if respond_to?(:setup)puts "Initializing..."setup@@ -17,26 +17,29 @@ module Migrationscreate_database- @worker_pool = WorkerPool.new-# Titles are known from the step classes (no instantiation, no queries),# so the reporter can reserve its title column up front. The steps- # themselves are built and sized lazily in the loop — each step's count- # is computed only when it runs.+ # themselves are built and sized lazily. Each step's source is opened+ # and counted in its own fork when it runs.step_classes = filter_steps(steps, only_steps, skip_steps)@reporter = Reporting::Factory.build(titles: step_classes.map(&:title))- step_classes.each do |step_class|- step = create_step(step_class)- before_step_execution(step)- execute_step(step)- after_step_execution(step)- end+ StepScheduler.new(+ step_classes:,+ reporter: @reporter,+ step_factory: method(:create_step),+ shard_manager: @shard_manager,+ writer: @writer,+ budget: WorkerBudget.available, # usable CPUs minus one for the parent + merges+ max_parallel_steps:,+ no_fork:,+ ).runrescue SignalException@aborted = trueexit(130)ensureDatabase::IntermediateDB.close+ @shard_manager&.cleanup# Restore the terminal (and flush the final frame) before printing the# run-level abort line, so it lands cleanly below the reporter output# instead of fighting the live region.@@ -45,7 +48,7 @@ module Migrationsenddef steps- step_class = StepBase+ step_class = Stepcurrent_module = self.class.name.deconstantize.constantizeclasses =@@ -61,25 +64,6 @@ module MigrationsTopologicalSorter.sort(classes)end- def before_step_execution(step)- # do nothing- end-- def execute_step(step)- executor =- if step.is_a?(ProgressStep)- ProgressStepExecutor.new(step, pool: @worker_pool, reporter: @reporter)- else- StepExecutor.new(step, reporter: @reporter)- end-- executor.execute- end-- def after_step_execution(step)- # do nothing- end-def step_args(step_class){}end@@ -90,8 +74,13 @@ module Migrationsdb_path = File.expand_path(settings[:intermediate_db][:path], Migrations.root_path)Database.migrate(db_path, migrations_path: Database::INTERMEDIATE_DB_SCHEMA_PATH)- db = Database.connect(db_path)- Database::IntermediateDB.setup(db)+ @shard_manager =+ ShardManager.new(+ canonical_path: db_path,+ migrations_path: Database::INTERMEDIATE_DB_SCHEMA_PATH,+ )+ @writer = Database::Connection.new(path: db_path)+ Database::IntermediateDB.setup(@writer)enddef create_step(step_class)diff --git a/migrations/core/lib/migrations/conversion/consolidator.rb b/migrations/core/lib/migrations/conversion/consolidator.rbnew file mode 100644index 00000000000..d0a82cf9bcd--- /dev/null+++ b/migrations/core/lib/migrations/conversion/consolidator.rb@@ -0,0 +1,70 @@+# frozen_string_literal: true++module Migrations+ module Conversion+ # Merges finished steps' shards into the run database on a background thread,+ # off the steps' critical path. A coordinator hands its shards over the+ # moment its workers exit and finishes the step right away, so the step no+ # longer lingers at 100% while its (possibly large) merge runs, and the merge+ # overlaps the steps that come after it.+ #+ # It writes straight to the run DB `Connection` — the only thing that touches+ # it, since steps write to their own shards. That connection closes and+ # reopens around every worker fork (Extralite isn't thread-safe, and a child+ # must not inherit an open handle), so a merge takes `fork_mutex` to serialize+ # against forking: a merge can't run while the connection is closed for a fork,+ # and a fork waits for an in-flight merge.+ class Consolidator+ # @param shard_manager [ShardManager] used to discard each shard once merged+ # @param connection [Database::Connection] the run DB connection merges write to+ # @param fork_mutex [Mutex] serializes merges against worker forks (see above)+ def initialize(shard_manager, connection, fork_mutex)+ @shard_manager = shard_manager+ @connection = connection+ @fork_mutex = fork_mutex+ @queue = Thread::Queue.new+ @errors = []+ @thread = Thread.new { run }+ @thread.name = "consolidator"+ end++ # Hands a finished step's shards over to be merged in the background.+ # @param shards [Array<String>] the shard database paths to merge, then discard+ # @return [void]+ def enqueue(shards)+ @queue << shards+ end++ # Waits for every enqueued shard to be merged, then returns the merge errors.+ # @return [Array<StandardError>] the errors from any merge that failed (empty+ # when all merged cleanly)+ def drain+ @queue.close+ @thread.join+ @errors+ end++ private++ def run+ while (shards = @queue.pop)+ shards.each { |shard_path| merge(shard_path) }+ end+ end++ def merge(shard_path)+ @fork_mutex.synchronize { @connection.merge_database(shard_path, tables: mergeable_tables) }+ rescue StandardError => e+ @errors << e+ ensure+ @shard_manager.discard(shard_path)+ end++ # `config` and `schema_migrations` are run-level and identical in every+ # shard, so they're left out of the merge.+ def mergeable_tables+ @mergeable_tables ||= @connection.tables - %w[config schema_migrations]+ end+ end+ end+enddiff --git a/migrations/core/lib/migrations/conversion/convert_error.rb b/migrations/core/lib/migrations/conversion/convert_error.rbnew file mode 100644index 00000000000..f96e669e24a--- /dev/null+++ b/migrations/core/lib/migrations/conversion/convert_error.rb@@ -0,0 +1,60 @@+# frozen_string_literal: true++module Migrations+ module Conversion+ # Raised at the end of a run when one or more steps failed or were skipped, or+ # a shard merge failed. The converter logs per-item errors and keeps going, so+ # this is the single signal the CLI turns into a non-zero exit; its message is+ # the run summary shown to the user (no backtrace).+ #+ # The scheduler hands it the raw outcomes; the human-facing summary — and its+ # translations — are built here, where the message belongs.+ class ConvertError < StandardError+ include CLI::PresentableError++ # @param failures [Hash{Class<Step> => StandardError, nil}] failed steps and+ # the error each raised+ # @param skipped [Array<Class<Step>>] steps skipped because a dependency failed+ # @param merge_errors [Array<StandardError>] errors from merging shards+ def initialize(failures: {}, skipped: [], merge_errors: [])+ super(summary(failures, skipped, merge_errors))+ end++ private++ def summary(failures, skipped, merge_errors)+ lines = [I18n.t("converter.errors.header")]++ failures.each do |step_class, error|+ lines << " • #{failure_line(step_class, error)}"+ Array(error&.backtrace).first(5).each { |frame| lines << " #{frame}" }+ end++ skipped.each do |step_class|+ lines << " • #{I18n.t("converter.errors.step_skipped", title: step_class.title)}"+ end++ merge_errors.each { |error| lines << " • #{merge_line(error)}" }++ lines.join("\n")+ end++ def failure_line(step_class, error)+ I18n.t(+ "converter.errors.step_failed",+ title: step_class.title,+ error_class: error&.class,+ error_message: error&.message,+ )+ end++ def merge_line(error)+ I18n.t(+ "converter.errors.merge_failed",+ error_class: error.class,+ error_message: error.message,+ )+ end+ end+ end+enddiff --git a/migrations/core/lib/migrations/conversion/inline_progress_sink.rb b/migrations/core/lib/migrations/conversion/inline_progress_sink.rbnew file mode 100644index 00000000000..2a482bcedba--- /dev/null+++ b/migrations/core/lib/migrations/conversion/inline_progress_sink.rb@@ -0,0 +1,21 @@+# frozen_string_literal: true++module Migrations+ module Conversion+ # The no-fork counterpart to {PipeProgressSink}: drives the step's progress+ # handle directly, in-process, instead of writing down a pipe.+ class InlineProgressSink+ def initialize(step_handle)+ @step_handle = step_handle+ end++ def report_max_progress(value)+ @progress = @step_handle.begin_progress(max_progress: value)+ end++ def report_progress(progress:, warnings:, errors:)+ @progress.update(increment_by: progress, warning_count: warnings, error_count: errors)+ end+ end+ end+enddiff --git a/migrations/core/lib/migrations/conversion/parallel_job.rb b/migrations/core/lib/migrations/conversion/parallel_job.rbdeleted file mode 100644index d7392613e2d..00000000000--- a/migrations/core/lib/migrations/conversion/parallel_job.rb+++ /dev/null@@ -1,36 +0,0 @@-# frozen_string_literal: true--module Migrations- module Conversion- class ParallelJob- def initialize(processor)- @processor = processor- @tracker = processor.tracker-- @offline_connection = Database::OfflineConnection.new- end-- # Runs in the worker process, before the first item is processed.- def setup- Database::IntermediateDB.setup(@offline_connection)- SetupGuard.run(@processor)- end-- def run(item)- @tracker.reset_stats!- @offline_connection.clear!-- begin- @processor.process(item)- rescue StandardError => e- @tracker.log_error("Failed to process item", exception: e, details: item)- end-- [@offline_connection.parametrized_insert_statements, @tracker.stats]- end-- def cleanup- end- end- end-enddiff --git a/migrations/core/lib/migrations/conversion/partition_source.rb b/migrations/core/lib/migrations/conversion/partition_source.rbnew file mode 100644index 00000000000..e4630f5e8bd--- /dev/null+++ b/migrations/core/lib/migrations/conversion/partition_source.rb@@ -0,0 +1,37 @@+# frozen_string_literal: true++module Migrations+ module Conversion+ # The interface a source DB adapter implements so a step can `partition_by`.+ # An adapter includes this module to declare it, and must provide:+ #+ # partition_bounds(key, from, base) -> [min, max]+ # estimated_row_count(from) -> Integer (planner row estimate)+ # count_all(from, where:) -> Integer (exact row count)+ # chunk_filter(key, lower, upper, base:) -> the SQL WHERE body for one chunk+ #+ # plus one way to read the sorted key — either the fast, in-SQL+ #+ # boundaries_by_scan(key, from, base, count) -> the chunk lower bounds+ #+ # or the streaming fallback the {Partitioner} samples itself:+ #+ # each_partition_key(key, from, base) -> yields each key value, in order+ #+ # The Partitioner checks a source against this in the parent before forking, so+ # a source that can't partition fails fast with a clear error instead of+ # crashing a worker with a bare NoMethodError.+ module PartitionSource+ REQUIRED = %i[partition_bounds estimated_row_count count_all chunk_filter].freeze+ SCAN = %i[boundaries_by_scan each_partition_key].freeze # at least one++ # The interface methods `source` doesn't provide (empty when it satisfies the+ # contract).+ def self.missing_from(source)+ missing = REQUIRED.reject { |method| source.respond_to?(method) }+ missing << SCAN.join(" or ") if SCAN.none? { |method| source.respond_to?(method) }+ missing+ end+ end+ end+enddiff --git a/migrations/core/lib/migrations/conversion/partitioner.rb b/migrations/core/lib/migrations/conversion/partitioner.rbnew file mode 100644index 00000000000..fb83e656e7b--- /dev/null+++ b/migrations/core/lib/migrations/conversion/partitioner.rb@@ -0,0 +1,115 @@+# frozen_string_literal: true++module Migrations+ module Conversion+ # Works out how to split a partitioned step's source into `count` chunks. All+ # the logic here is dialect-free; the dialect-specific SQL lives in the source+ # DB adapter, which this calls through a few primitives:+ #+ # partition_bounds(key, from, base) -> [min, max]+ # estimated_row_count(from) -> Integer (the planner's estimate)+ # count_all(from, where:) -> Integer (exact)+ # each_partition_key(key, from, base) -> yields each key value, sorted+ #+ # A dense numeric key strides evenly by value, which is cheap. A sparse one+ # (big gaps from deletes, or ids from a shared sequence), or any non-numeric or+ # composite key, is sampled from the sorted key instead, so the chunks hold+ # even row counts whatever the distribution.+ #+ # The sampling streams the whole key column here and picks every Nth value. An+ # adapter that can do better in SQL (e.g. Postgres with `NTILE`) may add an+ # optional primitive and skip the streaming:+ #+ # boundaries_by_scan(key, from, base, count) -> the chunk lower bounds+ #+ # When it's there we use it; otherwise we fall back to streaming, so a new+ # adapter gets a working partitioner from the four primitives alone.+ class Partitioner+ # Raised when a step declares `partition_by` but its source can't compute+ # boundaries — the adapter is missing partition primitives (or there is no+ # source DB at all, e.g. a file source).+ class UnsupportedSourceError < StandardError+ end++ # How much wider a numeric key's value range may run than its row count+ # before even value-sized chunks stop holding even row counts. Past this, the+ # gaps leave some chunks nearly empty, so we sample the sorted key instead.+ DENSE_RANGE_FACTOR = 4++ # @param source_db [Object] the source DB adapter providing the primitives above+ # @param key [Symbol, String, Array<Symbol>] the partition key: one column, or+ # an array of columns for a composite key+ # @param from [String] the table (or other source name) to split+ # @param base [String, nil] a SQL filter that always applies, or nil for none+ def initialize(source_db, key:, from:, base:)+ @source_db = source_db+ @key = key+ @from = from+ @base = base+ end++ # @param count [Integer] how many chunks to split the source into+ # @return [Array] the `count` chunk lower bounds; chunk i covers+ # `[bounds[i], bounds[i + 1])`, the last open-ended. Empty for an empty source.+ # @raise [UnsupportedSourceError] if the source can't compute boundaries+ def boundaries(count)+ ensure_source_can_partition!+ return scan_boundaries(count) if @key.is_a?(Array)++ min, max = @source_db.partition_bounds(@key, @from, @base)+ return [] if min.nil?+ return numeric_boundaries(min, max, count) if min.is_a?(Numeric) && dense?(min, max)++ scan_boundaries(count)+ end++ private++ def ensure_source_can_partition!+ missing = PartitionSource.missing_from(@source_db)+ return if missing.empty?++ raise UnsupportedSourceError,+ "Can't partition `#{@from}`: its source doesn't implement the " \+ "PartitionSource interface (missing #{missing.join(", ")}). " \+ "Implement it, or remove `partition_by` from the step."+ end++ def dense?(min, max)+ rows = @source_db.estimated_row_count(@from)+ # No usable estimate (an adapter returns 0 or less when it can't estimate+ # the row count): don't pay for a full key scan on a guess — default to the+ # cheap value stride.+ return true if rows <= 0+ (max - min + 1) <= rows * DENSE_RANGE_FACTOR+ end++ def numeric_boundaries(min, max, count)+ chunk_size = ((max - min + 1).to_f / count).ceil+ Array.new(count) { |i| min + i * chunk_size }.uniq+ end++ def scan_boundaries(count)+ if @source_db.respond_to?(:boundaries_by_scan)+ @source_db.boundaries_by_scan(@key, @from, @base, count)+ else+ sampled_boundaries(count)+ end+ end++ def sampled_boundaries(count)+ total = @source_db.count_all(@from, where: @base)+ return [] if total.zero?++ sample_every = [(total.to_f / count).ceil, 1].max+ boundaries = []+ index = 0+ @source_db.each_partition_key(@key, @from, @base) do |value|+ boundaries << value if (index % sample_every).zero? && boundaries.size < count+ index += 1+ end+ boundaries+ end+ end+ end+enddiff --git a/migrations/core/lib/migrations/conversion/pipe_progress_sink.rb b/migrations/core/lib/migrations/conversion/pipe_progress_sink.rbnew file mode 100644index 00000000000..60de7671fd4--- /dev/null+++ b/migrations/core/lib/migrations/conversion/pipe_progress_sink.rb@@ -0,0 +1,22 @@+# frozen_string_literal: true++module Migrations+ module Conversion+ # The worker's end of the progress channel. Writes one line per call down the+ # pipe to the coordinator, which parses them (see+ # {StepCoordinator#consume_progress}).+ class PipeProgressSink+ def initialize(io)+ @io = io+ end++ def report_max_progress(value)+ @io.write("m #{value}\n")+ end++ def report_progress(progress:, warnings:, errors:)+ @io.write("p #{progress} #{warnings} #{errors}\n")+ end+ end+ end+enddiff --git a/migrations/core/lib/migrations/conversion/progress_step.rb b/migrations/core/lib/migrations/conversion/progress_step.rbdeleted file mode 100644index 2224c023123..00000000000--- a/migrations/core/lib/migrations/conversion/progress_step.rb+++ /dev/null@@ -1,184 +0,0 @@-# frozen_string_literal: true--module Migrations- module Conversion- # A step that processes many items and reports progress. It is split into- # two roles with a hard boundary between them:- #- # * `source` enumerates the items (`items`, `max_progress`). It always runs- # in the main process.- # * `processor` handles one item at a time (`process`). It runs in a worker- # (a forked process when the step runs in parallel), one instance per- # worker. Per-worker state is built in its `setup` hook which runs after- # the worker has started, never in the constructor. `setup` must not- # create IntermediateDB records — only `process` writes; `setup` runs- # under `SetupGuard`, so a write raises `SetupGuard::SetupError` in- # serial and parallel mode alike.- #- # The roles are separate objects, so state can't leak across the- # process boundary: a processor method that tries to read source-side- # state fails with a `NoMethodError` instead of silently seeing stale or- # missing data. `helpers` is mixed into both roles and is meant for pure- # functions only (arguments in, value out).- #- # class Users < Conversion::ProgressStep- # source do- # attr_accessor :source_db- #- # def items- # @source_db.query("SELECT ...")- # end- # end- #- # processor do- # def setup- # @upload_creator = UploadCreator.new- # end- #- # def process(item)- # IntermediateDB::User.create(...)- # end- # end- # end- #- # The blocks are evaluated with `class_eval`, so use plain `def` inside- # them. Avoid `define_method`: its block closes over the surrounding scope- # and would smuggle step-level state across the role boundary.- class ProgressStep < StepBase- class Source- include AttributeAssignment-- # `source_db` is the step's connection to the source data, when it has- # one. Steps that read from somewhere else (a file, fixed data) leave it- # nil. The converter injects it through `step_args`.- attr_accessor :settings, :source_db-- class << self- # Read a whole table: defines `items` (`SELECT * FROM <name>`) and- # `max_progress` (its row count), both filtered by `where`. Override- # `items`/`max_progress` for anything custom.- def reads_table(name, where: nil)- @table = { name:, where: }- end-- attr_reader :table-- def reads_table?- !table.nil?- end- end-- def initialize(args = {})- assign_attributes(args)- end-- def max_progress- return unless self.class.reads_table?- @source_db.count_all(self.class.table[:name], where: self.class.table[:where])- end-- def items- raise NotImplementedError unless self.class.reads_table?- @source_db.select_all(self.class.table[:name], where: self.class.table[:where])- end- end-- class Processor- include AttributeAssignment-- attr_accessor :settings- attr_reader :tracker-- def initialize(args = {})- @tracker = StepTracker.new- assign_attributes(args)- end-- def setup- # do nothing- end-- def process(item)- raise NotImplementedError- end- end-- # The step object is only a coordinator: it builds the source (in the- # main process) and one processor per worker. All per-item state —- # including the trackers — lives on the roles, and `settings` are routed- # to them; that's why this class is a sibling of `Step` rather than a- # subclass — it has no `settings`, `tracker` or `execute` of its own.- attr_reader :source-- def initialize(args = {})- @args = args- @source = self.class.source_class.new(args)- end-- def create_processor- self.class.processor_class.new(@args)- end-- class << self- def source(&block)- raise ArgumentError, "`source` is already defined" if @source_block- @source_block = block- end-- def processor(&block)- raise ArgumentError, "`processor` is already defined" if @processor_block- @processor_block = block- end-- def helpers(&block)- raise ArgumentError, "`helpers` is already defined" if @helpers_block- @helpers_block = block- end-- def source_class- @source_class ||=- build_role_class(superclass_role(:source_class, Source), :Source, @source_block)- end-- def processor_class- @processor_class ||=- build_role_class(- superclass_role(:processor_class, Processor),- :Processor,- @processor_block,- )- end-- def helpers_module- @helpers_module ||=- Module.new.tap { |mod| mod.module_eval(&@helpers_block) if @helpers_block }- end-- def run_in_parallel(value)- @run_in_parallel = !!value- end-- def run_in_parallel?- @run_in_parallel == true- end-- private-- # Steps usually inherit from `ProgressStep` directly and their roles- # from the `Source`/`Processor` base classes. If a step subclasses- # another step, its roles subclass the parent step's roles instead.- def superclass_role(role_method, base_class)- superclass < ProgressStep ? superclass.public_send(role_method) : base_class- end-- def build_role_class(base_class, name, block)- klass = Class.new(base_class)- const_set(name, klass)- klass.include(helpers_module)- klass.class_eval(&block) if block- klass- end- end- end- end-enddiff --git a/migrations/core/lib/migrations/conversion/progress_step_executor.rb b/migrations/core/lib/migrations/conversion/progress_step_executor.rbdeleted file mode 100644index 3c91ed565d8..00000000000--- a/migrations/core/lib/migrations/conversion/progress_step_executor.rb+++ /dev/null@@ -1,120 +0,0 @@-# frozen_string_literal: true--module Migrations- module Conversion- class ProgressStepExecutor- PRINT_RUNTIME_AFTER_SECONDS = 5-- def initialize(step, pool:, reporter:)- @step = step- @source = step.source- @pool = pool- @reporter = reporter- end-- def execute- @step_report = @reporter.start_step(@step.class.title)- @max_progress = calculate_max_progress-- if execute_in_parallel?- execute_parallel- else- execute_serially- end- ensure- @step_report.finish- end-- private-- # Parallelism only pays off above a minimum number of items, and the- # queues need bounds for backpressure. Both are executor policy, scaled- # by the size of the injected pool.- def min_parallel_items- @pool.size * 10- end-- def max_queue_size- @pool.size * 100- end-- def execute_in_parallel?- @step.class.run_in_parallel? && (@max_progress.nil? || @max_progress > min_parallel_items)- end-- def execute_serially- job = SerialJob.new(@step.create_processor)- job.setup-- @step_report.with_progress(max_progress: @max_progress) do |progress|- @source.items.each do |item|- stats = job.run(item)- progress.update(- increment_by: stats.progress,- warning_count: stats.warning_count,- error_count: stats.error_count,- )- end- end- end-- def execute_parallel- worker_output_queue = SizedQueue.new(max_queue_size)- work_queue = SizedQueue.new(max_queue_size)-- batch =- @pool.start(work_queue:, output_queue: worker_output_queue) do- ParallelJob.new(@step.create_processor)- end- writer_thread = start_db_writer(worker_output_queue)- push_work(work_queue)-- batch.wait- worker_output_queue.close- writer_thread.join- end-- def calculate_max_progress- start_time = Time.now- max_progress = @source.max_progress- duration = Time.now - start_time-- if duration > PRINT_RUNTIME_AFTER_SECONDS- @step_report.notice(- I18n.t(- "converter.max_progress_calculation",- duration: DateHelper.human_readable_time(duration),- ),- )- end-- max_progress- end-- def start_db_writer(worker_output_queue)- Thread.new do- Thread.current.name = "writer_thread"-- @step_report.with_progress(max_progress: @max_progress) do |progress|- while (parametrized_insert_statements, stats = worker_output_queue.pop)- parametrized_insert_statements.each do |sql, parameters|- Database::IntermediateDB.insert(sql, *parameters)- end-- progress.update(- increment_by: stats.progress,- warning_count: stats.warning_count,- error_count: stats.error_count,- )- end- end- end- end-- def push_work(work_queue)- @source.items.each { |item| work_queue.push(item) }- work_queue.close- end- end- end-enddiff --git a/migrations/core/lib/migrations/conversion/serial_job.rb b/migrations/core/lib/migrations/conversion/serial_job.rbdeleted file mode 100644index 957558291f4..00000000000--- a/migrations/core/lib/migrations/conversion/serial_job.rb+++ /dev/null@@ -1,31 +0,0 @@-# frozen_string_literal: true--module Migrations- module Conversion- class SerialJob- def initialize(processor)- @processor = processor- @tracker = processor.tracker- end-- def setup- SetupGuard.run(@processor)- end-- def run(item)- @tracker.reset_stats!-- begin- @processor.process(item)- rescue StandardError => e- @tracker.log_error("Failed to process item", exception: e, details: item)- end-- @tracker.stats- end-- def cleanup- end- end- end-enddiff --git a/migrations/core/lib/migrations/conversion/setup_guard.rb b/migrations/core/lib/migrations/conversion/setup_guard.rbindex f40b11552ae..0e9e4165479 100644--- a/migrations/core/lib/migrations/conversion/setup_guard.rb+++ b/migrations/core/lib/migrations/conversion/setup_guard.rb@@ -2,13 +2,9 @@module Migrationsmodule Conversion- # Wraps a processor's `setup` so it can't create IntermediateDB records:- # for the duration of the call the IntermediateDB connection is replaced- # by one that raises on insert. Without the guard such writes would be- # silently discarded in parallel mode (each worker's `OfflineConnection`- # is cleared before the first item) but persisted in serial mode — the- # guard makes both modes fail loudly and identically, with a backtrace- # pointing at the offending write.+ # Wraps a processor's `setup` so a write to the IntermediateDB raises instead+ # of slipping through: `setup` builds per-worker state, not rows, so a stray+ # write there would land outside the per-item error handling.module SetupGuardclass SetupError < StandardErrorenddiff --git a/migrations/core/lib/migrations/conversion/shard_manager.rb b/migrations/core/lib/migrations/conversion/shard_manager.rbnew file mode 100644index 00000000000..785390de076--- /dev/null+++ b/migrations/core/lib/migrations/conversion/shard_manager.rb@@ -0,0 +1,57 @@+# frozen_string_literal: true++require "fileutils"++module Migrations+ module Conversion+ # Hands out per-worker shard databases for the sharded write path. A shard is+ # a private copy of an empty database with the run's schema, so a worker opens+ # it and runs the exact same inserts it would against the run database, just+ # without touching the single run-level connection that is the throughput+ # ceiling. Once a step's workers are done the coordinator merges their shards+ # into the run database (see {Consolidator}) and discards+ # them.+ #+ # The template is a fresh, empty database migrated from the same schema as the+ # run DB — not a copy of the run DB. That keeps a shard cheap to copy, and —+ # importantly — means a run against an existing IntermediateDB doesn't drag the+ # existing rows into every shard; the merge just adds each shard's new rows+ # with `INSERT OR IGNORE`.+ class ShardManager+ # @param canonical_path [String] the run's IntermediateDB; the shards and+ # their template are placed next to it+ # @param migrations_path [String] the schema migrations, used to build the+ # empty template+ def initialize(canonical_path:, migrations_path:)+ @dir = File.join(File.dirname(canonical_path), "shards")+ FileUtils.mkdir_p(@dir)++ @template_path = File.join(@dir, "template.db")+ Database.delete_database(@template_path)+ Database.migrate(@template_path, migrations_path:)++ @counter = 0+ @mutex = Mutex.new+ end++ # @return [String] the path of a fresh shard, copied from the template+ def create_shard+ index = @mutex.synchronize { @counter += 1 }+ path = File.join(@dir, "shard-#{index}.db")+ FileUtils.cp(@template_path, path)+ path+ end++ # Removes a shard and its WAL/SHM sidecar files.+ # @param path [String] the shard database path returned by {#create_shard}+ # @return [void]+ def discard(path)+ Database.delete_database(path)+ end++ def cleanup+ FileUtils.rm_rf(@dir)+ end+ end+ end+enddiff --git a/migrations/core/lib/migrations/conversion/step.rb b/migrations/core/lib/migrations/conversion/step.rbindex 5fbaba5bf3b..20ed2b53bc2 100644--- a/migrations/core/lib/migrations/conversion/step.rb+++ b/migrations/core/lib/migrations/conversion/step.rb@@ -2,22 +2,133 @@module Migrationsmodule Conversion- class Step < StepBase- include AttributeAssignment+ # A step that processes many items and reports progress. It is split into+ # two roles:+ #+ # * `source` enumerates the items (`items`, `max_progress`).+ # * `processor` handles one item at a time (`process`). Per-worker state is+ # built in its `setup` hook which runs after the worker has started, never+ # in the constructor. `setup` must not create IntermediateDB records, only+ # `process` writes; `setup` runs under `SetupGuard`, so a write raises+ # `SetupGuard::SetupError` whether the step runs inline or in a fork.+ #+ # Both roles run in the worker that processes the step (a forked process, or+ # the main process under `--no-fork`): the worker builds the source, reads its+ # slice of the rows, and feeds each one to a processor. For a partitioned step+ # the parent also builds a source of its own, before forking, only to work out+ # the chunk boundaries.+ #+ # The roles are separate objects, so a processor can't read source-side+ # state: such a call fails with a `NoMethodError`. That keeps `process`+ # from depending on the source's iteration state, which would otherwise be+ # an easy way to write a step that breaks once it's split across forks.+ # `helpers` is mixed into both roles and is meant for pure functions only+ # (arguments in, value out).+ #+ # class Users < Conversion::Step+ # source do+ # def items+ # @source_db.query("SELECT ...")+ # end+ # end+ #+ # processor do+ # def setup+ # @upload_creator = UploadCreator.new+ # end+ #+ # def process(item)+ # IntermediateDB::User.create(...)+ # end+ # end+ # end+ #+ # The blocks are evaluated with `class_eval`, so use plain `def` inside them.+ # Avoid `define_method`: its block closes over the surrounding scope and would+ # leak step-level state across the role boundary.+ class Step+ extend StepDependencies- attr_accessor :settings- attr_reader :tracker+ IntermediateDB = Database::IntermediateDB+ Enums = Database::IntermediateDB::Enums- # inside of Step it might make more sense to access it as `step` instead of `tracker`- alias step tracker+ attr_reader :sourcedef initialize(args = {})- @tracker = StepTracker.new- assign_attributes(args)+ @args = args+ @source = self.class.source_class.new(args)end- def execute- # do nothing+ def create_processor+ self.class.processor_class.new(@args)+ end++ class << self+ def title(+ value = (+ getter = true+ nil+ )+ )+ @title = value unless getter+ @title.presence ||+ I18n.t(+ "converter.default_step_title",+ type: name&.demodulize&.underscore&.humanize(capitalize: false),+ )+ end++ def source(&block)+ raise ArgumentError, "`source` is already defined" if @source_block+ @source_block = block+ end++ def processor(&block)+ raise ArgumentError, "`processor` is already defined" if @processor_block+ @processor_block = block+ end++ def helpers(&block)+ raise ArgumentError, "`helpers` is already defined" if @helpers_block+ @helpers_block = block+ end++ def source_class+ @source_class ||=+ build_role_class(superclass_role(:source_class, Source), :Source, @source_block)+ end++ def partitionable?+ source_class.partitionable?+ end++ def processor_class+ @processor_class ||=+ build_role_class(+ superclass_role(:processor_class, Processor),+ :Processor,+ @processor_block,+ )+ end++ def helpers_module+ @helpers_module ||=+ Module.new.tap { |mod| mod.module_eval(&@helpers_block) if @helpers_block }+ end++ private++ def superclass_role(role_method, base_class)+ superclass < Step ? superclass.public_send(role_method) : base_class+ end++ def build_role_class(base_class, name, block)+ klass = Class.new(base_class)+ const_set(name, klass)+ klass.include(helpers_module)+ klass.class_eval(&block) if block+ klass+ endendendenddiff --git a/migrations/core/lib/migrations/conversion/step/processor.rb b/migrations/core/lib/migrations/conversion/step/processor.rbnew file mode 100644index 00000000000..eb094b552cb--- /dev/null+++ b/migrations/core/lib/migrations/conversion/step/processor.rb@@ -0,0 +1,26 @@+# frozen_string_literal: true++module Migrations+ module Conversion+ class Step+ class Processor+ include AttributeAssignment++ attr_accessor :settings+ attr_reader :tracker++ def initialize(args = {})+ @tracker = StepTracker.new+ assign_attributes(args)+ end++ def setup+ end++ def process(item)+ raise NotImplementedError+ end+ end+ end+ end+enddiff --git a/migrations/core/lib/migrations/conversion/step/source.rb b/migrations/core/lib/migrations/conversion/step/source.rbnew file mode 100644index 00000000000..84d64bbe2f5--- /dev/null+++ b/migrations/core/lib/migrations/conversion/step/source.rb@@ -0,0 +1,121 @@+# frozen_string_literal: true++module Migrations+ module Conversion+ class Step+ class Source+ include AttributeAssignment++ # `source_db` is the step's own connection to the source data, when it has+ # one. Steps that read from somewhere else (a file, fixed data) leave it+ # nil. The converter injects it through `step_args`.+ attr_accessor :settings, :source_db++ # The `[lower, upper]` key range this worker reads (`upper` nil means+ # open-ended), or nil for an unpartitioned step. The runner sets it before+ # `items`; the generated queries (and `partition_slice`) apply it.+ attr_accessor :chunk++ class << self+ # Read a whole table: defines `items` (`SELECT * FROM <name>`) and+ # `max_progress` (its row count), both filtered by `where`. Works with+ # or without `partition_by` (a partitioned step also limits each query+ # to its chunk). Override `items`/`max_progress` for anything custom.+ #+ # `order` (a column or array of them) sorts the read — worth setting for+ # a large table whose rows insert faster in that order (it should match+ # the target's key, and be indexed at the source so it doesn't add a+ # sort). A partitioned step ignores it and orders by its key instead.+ def reads_table(name, where: nil, order: nil)+ @table = { name:, where:, order: }+ end++ attr_reader :table++ def reads_table?+ !table.nil?+ end++ # Split the step across forks on `key` (one column, or an array of+ # columns for a composite key). `from`/`base` say which table and filter+ # to scan for the boundaries; they default to the `reads_table` ones, so+ # a table-reading step only needs the key.+ def partition_by(key, from: nil, base: nil)+ @partition = { key:, from:, base: }+ end++ attr_reader :partition++ def partitionable?+ !partition.nil?+ end+ end++ def initialize(args = {})+ assign_attributes(args)+ end++ # The chunk boundaries for splitting the source `count` ways, worked out+ # in the parent before forking.+ def partition_boundaries(count)+ Partitioner.new(+ @source_db,+ key: partition_key,+ from: partition_from,+ base: partition_base,+ ).boundaries(count)+ end++ # The WHERE body that limits a query to this worker's chunk. Add it to a+ # custom query (`WHERE #{partition_slice}`); the generated queries already+ # use it.+ def partition_slice+ lower, upper = chunk+ @source_db.chunk_filter(partition_key, lower, upper, base: partition_base)+ end++ def max_progress+ return unless self.class.reads_table?+ @source_db.count_all(self.class.table[:name], where: read_filter)+ end++ def items+ raise NotImplementedError unless self.class.reads_table?+ @source_db.select_all(self.class.table[:name], where: read_filter, order: read_order)+ end++ def cleanup+ @source_db&.close+ end++ private++ # The rows the current read should see: this worker's chunk when+ # partitioned, otherwise the whole table's filter.+ def read_filter+ self.class.partitionable? ? partition_slice : self.class.table[:where]+ end++ # A partitioned read is ordered by its key so each worker writes its shard+ # in key order (sequential index inserts, in the shard and in the merge).+ # An unpartitioned read uses `reads_table`'s `order`, if any.+ def read_order+ return partition_key if self.class.partitionable?+ self.class.table[:order]+ end++ def partition_key+ self.class.partition[:key]+ end++ def partition_from+ self.class.partition[:from] || self.class.table&.fetch(:name)+ end++ def partition_base+ self.class.partition[:base] || self.class.table&.fetch(:where)+ end+ end+ end+ end+enddiff --git a/migrations/core/lib/migrations/conversion/step_base.rb b/migrations/core/lib/migrations/conversion/step_base.rbdeleted file mode 100644index 1661476f12f..00000000000--- a/migrations/core/lib/migrations/conversion/step_base.rb+++ /dev/null@@ -1,37 +0,0 @@-# frozen_string_literal: true--module Migrations- module Conversion- # Abstract base of all conversion steps; `Base#steps` discovers its- # subclasses. It carries only what every step kind shares — `Step` adds- # the imperative single-run contract, `ProgressStep` the per-item- # source/processor roles.- class StepBase- extend StepDependencies-- # These constants also make bare `IntermediateDB::...` / `Enums::...`- # references work inside `ProgressStep`'s `source` / `processor` blocks:- # the blocks are written in step class bodies, and constants in methods- # defined via `class_eval(&block)` resolve through the block's lexical- # scope — the step class and its ancestors — not the role class.- IntermediateDB = Database::IntermediateDB- Enums = Database::IntermediateDB::Enums-- class << self- def title(- value = (- getter = true- nil- )- )- @title = value unless getter- @title.presence ||- I18n.t(- "converter.default_step_title",- type: name&.demodulize&.underscore&.humanize(capitalize: false),- )- end- end- end- end-enddiff --git a/migrations/core/lib/migrations/conversion/step_coordinator.rb b/migrations/core/lib/migrations/conversion/step_coordinator.rbnew file mode 100644index 00000000000..0e6e1c0e1c4--- /dev/null+++ b/migrations/core/lib/migrations/conversion/step_coordinator.rb@@ -0,0 +1,222 @@+# frozen_string_literal: true++module Migrations+ module Conversion+ # Runs one step end to end on its own thread for the {StepScheduler}. It forks+ # the step's workers — one for a normal step, several for a partitioned one (or+ # none under `--no-fork`, where the single worker runs inline). Each worker+ # builds the step itself, opens its own source, reads its slice of the rows,+ # and writes them to its own shard.+ #+ # The coordinator relays the workers' combined progress to the reporter. Once+ # they all exit cleanly it hands the finished shards to the {Consolidator},+ # which merges them into the run database on a background thread, so the step+ # finishes without waiting for its own merge.+ #+ # The step is built inside each worker, not here, so its source connection is+ # opened there. The parent never holds a source connection, and nothing but+ # progress crosses the pipes.+ class StepCoordinator+ class WorkerCrashedError < StandardError+ end++ attr_reader :step_class++ # @param step_class [Class<Step>] the step this coordinator runs+ # @param step_factory [#call] builds a step from its class, `->(step_class) { step }`;+ # called inside each worker so the source connection opens there+ # @param reporter [Reporting::Reporter] receives the step's progress and notices+ # @param fork_mutex [Mutex] serialises forking across the run's coordinator threads+ # @param scheduler [StepScheduler] told about a step-level failure for the run summary+ # @param shard_manager [ShardManager] hands out and discards the per-worker shard DBs+ # @param consolidator [Consolidator] merges the finished shards in the background+ # @param fork_count [Integer] how many workers to fork (1 for an unpartitioned step)+ # @param no_fork [Boolean] run the single worker inline instead of forking (`--no-fork`)+ def initialize(+ step_class:,+ step_factory:,+ reporter:,+ fork_mutex:,+ scheduler:,+ shard_manager:,+ consolidator:,+ fork_count: 1,+ no_fork: false+ )+ @step_class = step_class+ @step_factory = step_factory+ @reporter = reporter+ @fork_mutex = fork_mutex+ @scheduler = scheduler+ @shard_manager = shard_manager+ @consolidator = consolidator+ @fork_count = fork_count+ @no_fork = no_fork+ end++ # @return [Symbol] the step's outcome, `:done` or `:failed`+ def run+ @step_handle = @reporter.start_step(@step_class.title)+ @step_handle.report_concurrency(@fork_count)+ outcome = :done+ shards = []+ handed_off = false++ begin+ # `finish` reads `$!` in its ensure, so the work must raise inside this+ # inner begin, before the rescue below clears it.+ begin+ run_workers(shards)+ @consolidator.enqueue(shards)+ handed_off = true+ ensure+ error = $!+ @step_handle.notice(failure_notice(error)) if error.is_a?(StandardError)+ @step_handle.finish+ end+ rescue StandardError => e+ outcome = :failed+ @scheduler.record_failure(@step_class, e)+ ensure+ shards.each { |shard_path| @shard_manager.discard(shard_path) } unless handed_off+ end++ outcome+ end++ private++ def run_workers(shards)+ return run_inline(shards) if @no_fork++ boundaries = compute_boundaries+ worker_count = boundaries.nil? ? 1 : [boundaries.size, 1].max++ pipes = Array.new(worker_count) { IO.pipe }+ worker_count.times { shards << @shard_manager.create_shard }++ pids = nil+ @fork_mutex.synchronize do+ ForkManager.with_batched_forks do+ pids =+ worker_count.times.map do |index|+ chunk = chunk_for(boundaries, index)+ ForkManager.fork { run_in_fork(pipes, index, shards[index], chunk) }+ end+ end+ end++ pipes.each { |(_reader, writer)| writer.close }+ readers = pipes.map(&:first)++ begin+ consume_progress(readers)+ ensure+ readers.each(&:close)+ end++ await(pids)+ end++ def run_inline(shards)+ shard_path = @shard_manager.create_shard+ shards << shard_path++ StepRunner.new(+ step: @step_factory.call(@step_class),+ shard_path:,+ reporter: InlineProgressSink.new(@step_handle),+ chunk: nil,+ ).run+ end++ def compute_boundaries+ return nil if @fork_count == 1++ step = @step_factory.call(@step_class)+ begin+ step.source.partition_boundaries(@fork_count)+ ensure+ step.source.cleanup+ end+ end++ def chunk_for(boundaries, index)+ return nil if boundaries.nil? || boundaries.empty?+ [boundaries[index], boundaries[index + 1]]+ end++ def run_in_fork(pipes, mine, shard_path, chunk)+ pipes.each_with_index do |(reader, writer), index|+ reader.close+ writer.close unless index == mine+ end++ step = @step_factory.call(@step_class)+ StepRunner.new(+ step:,+ shard_path:,+ reporter: PipeProgressSink.new(pipes[mine][1]),+ chunk:,+ ).run+ end++ def await(pids)+ pids.each do |pid|+ _, status = Process.waitpid2(pid)+ next if status.success?++ raise WorkerCrashedError,+ "A worker for #{@step_class.title} exited unexpectedly (#{status}). " \+ "Check the error output above for the cause."+ end+ end++ def consume_progress(readers)+ @step_handle.with_progress(max_progress: total_max_progress(readers)) do |progress|+ drain_progress(readers, progress)+ end+ end++ def total_max_progress(readers)+ maxes = readers.map { |reader| read_max_progress(reader) }+ return if maxes.any?(&:nil?)+ maxes.sum+ end++ def drain_progress(readers, progress)+ pending = readers.dup++ until pending.empty?+ ready, = IO.select(pending)+ ready.each do |reader|+ line = reader.gets+ if line.nil?+ pending.delete(reader)+ next+ end++ _, increment, warnings, errors = line.split+ progress.update(+ increment_by: increment.to_i,+ warning_count: warnings.to_i,+ error_count: errors.to_i,+ )+ end+ end+ end++ def read_max_progress(reader)+ line = reader.gets+ return if line.nil?++ value = line.split[1]+ value&.to_i+ end++ def failure_notice(error)+ "#{error.class}: #{error.message}"+ end+ end+ end+enddiff --git a/migrations/core/lib/migrations/conversion/step_executor.rb b/migrations/core/lib/migrations/conversion/step_executor.rbdeleted file mode 100644index 67b69c3df8c..00000000000--- a/migrations/core/lib/migrations/conversion/step_executor.rb+++ /dev/null@@ -1,19 +0,0 @@-# frozen_string_literal: true--module Migrations- module Conversion- class StepExecutor- def initialize(step, reporter:)- @step = step- @reporter = reporter- end-- def execute- step_report = @reporter.start_step(@step.class.title)- @step.execute- ensure- step_report.finish- end- end- end-enddiff --git a/migrations/core/lib/migrations/conversion/step_runner.rb b/migrations/core/lib/migrations/conversion/step_runner.rbnew file mode 100644index 00000000000..7370fd48c4f--- /dev/null+++ b/migrations/core/lib/migrations/conversion/step_runner.rb@@ -0,0 +1,85 @@+# frozen_string_literal: true++module Migrations+ module Conversion+ # Runs one step's items to completion inside a single process. The worker+ # opens its own source, so the read happens here in the fork, not in the+ # parent, then reads the rows it is responsible for, processes each, and+ # writes them to its shard. Nothing is sent in but the step to run, and+ # nothing goes back but progress.+ #+ # The insert runs here against a real connection, so a bad row (a NULL in a+ # NOT NULL column, say) raises right where it's processed, to be logged and+ # skipped per item rather than failing the whole step.+ class StepRunner+ # How many processed items to accumulate before reporting progress.+ REPORT_INTERVAL = 1_000++ # @param step [Step] the step to run; its source and processor are built here+ # @param shard_path [String] the SQLite shard this worker writes its rows to+ # @param reporter [#report_max_progress, #report_progress] the worker's end of+ # the progress channel ({PipeProgressSink} in a fork, {InlineProgressSink} inline)+ # @param chunk [Array(Object, Object), nil] the `[lower, upper]` key range to read+ # (upper nil = open-ended), or nil to read the whole source+ def initialize(step:, shard_path:, reporter:, chunk: nil)+ @step = step+ @shard_path = shard_path+ @reporter = reporter+ @chunk = chunk+ end++ def run+ source = @step.source+ source.chunk = @chunk+ connection = Database::Connection.new(path: @shard_path)++ begin+ # Point IntermediateDB at this worker's shard for the block.+ # `with_connection` restores the previous connection afterwards without+ # closing it — unlike `setup`, which closes it; under `--no-fork` that+ # previous connection is the live run DB connection the rest of the+ # run still needs.+ Database::IntermediateDB.with_connection(connection) do+ processor = @step.create_processor+ SetupGuard.run(processor)++ @reporter.report_max_progress(source.max_progress)+ process_items(source, processor)+ end+ ensure+ connection.close+ source.cleanup+ end+ end++ private++ def process_items(source, processor)+ tracker = processor.tracker+ progress = warnings = errors = 0++ source.items.each do |item|+ tracker.reset_stats!++ begin+ processor.process(item)+ rescue StandardError => e+ tracker.log_error("Failed to process item", exception: e, details: item)+ end++ stats = tracker.stats+ progress += stats.progress+ warnings += stats.warning_count+ errors += stats.error_count++ next if progress < REPORT_INTERVAL+ @reporter.report_progress(progress:, warnings:, errors:)+ progress = warnings = errors = 0+ end++ return if progress.zero? && warnings.zero? && errors.zero?+ @reporter.report_progress(progress:, warnings:, errors:)+ end+ end+ end+enddiff --git a/migrations/core/lib/migrations/conversion/step_scheduler.rb b/migrations/core/lib/migrations/conversion/step_scheduler.rbnew file mode 100644index 00000000000..511cced12bf--- /dev/null+++ b/migrations/core/lib/migrations/conversion/step_scheduler.rb@@ -0,0 +1,244 @@+# frozen_string_literal: true++module Migrations+ module Conversion+ # Runs the conversion steps concurrently and follows their dependency graph.+ # Each running step gets its own {StepCoordinator} on its own thread, which+ # forks one worker that reads the step's source and writes its shard. This+ # class only decides what may start: up to `budget` steps run at once (the+ # smaller of the fork budget and `--max-parallel-steps`), and a step starts+ # only once its dependencies are done.+ #+ # We don't do slot accounting or full-pool exclusivity here. Every step is+ # one fork, so "how many run at once" is the whole scheduling decision. When+ # a heavy step is split across several forks, those count as several entries+ # in the same budget.+ class StepScheduler+ FINISHED_STATES = %i[done failed skipped].freeze++ # @param step_classes [Array<Class<Step>>] every step in the run, in any order+ # @param reporter [Reporting::Reporter] shared by all steps to report progress+ # @param step_factory [#call] builds a step from its class, `->(step_class) { step }`+ # @param shard_manager [ShardManager] hands out the per-worker shard databases+ # @param writer [Database::Connection] the run DB connection the consolidator+ # merges shards into (held directly, so an inline step swapping the+ # IntermediateDB connection can't divert a background merge)+ # @param budget [Integer] the most steps (forks) to run at once, usually cores - 1+ # @param max_parallel_steps [Integer, nil] a lower cap from `--max-parallel-steps`+ # @param no_fork [Boolean] run each step inline, one at a time (`--no-fork`)+ def initialize(+ step_classes:,+ reporter:,+ step_factory:,+ shard_manager:,+ writer:,+ budget:,+ max_parallel_steps: nil,+ no_fork: false+ )+ @step_classes = step_classes+ @reporter = reporter+ @step_factory = step_factory+ @shard_manager = shard_manager+ @writer = writer+ @no_fork = no_fork++ @budget = [budget, max_parallel_steps].compact.min+ @budget = 1 if @budget < 1+ # Inline runs share one process and one IntermediateDB connection, so+ # only one step may run at a time.+ @budget = 1 if @no_fork++ @mutex = Mutex.new+ @condition = ConditionVariable.new+ @fork_mutex = Mutex.new++ @states = step_classes.to_h { |step_class| [step_class, :pending] }+ @available_forks = @budget+ # A running step_class => the forks it reserved from the budget, returned+ # to @available_forks when it finishes.+ @reserved_forks = {}+ @threads = []+ @failures = {}+ @merge_errors = []+ end++ attr_reader :budget++ def run+ started_at = monotonic_time+ # Compact the heap before any fork so children share its pages copy-on-write.+ Process.warmup+ @consolidator = Consolidator.new(@shard_manager, @writer, @fork_mutex)++ @mutex.synchronize do+ loop do+ schedule+ break if all_finished?+ @condition.wait(@mutex)+ end+ end++ @threads.each(&:join)+ # Every step is done, but the background merges may still be catching up.+ # Show a "finishing up" status until they drain, so the display doesn't sit+ # silently at 100%.+ @merge_errors = @reporter.finalizing { @consolidator.drain }+ report_summary(monotonic_time - started_at)+ raise_summary_if_unsuccessful+ end++ # Records a step-level error for the end-of-run summary; a coordinator calls+ # this when its step fails.+ # @param step_class [Class<Step>] the step that failed+ # @param error [StandardError] what it failed with+ # @return [void]+ def record_failure(step_class, error)+ @mutex.synchronize { @failures[step_class] = error }+ end++ private++ def schedule+ loop do+ step_class = next_ready_step+ break unless step_class++ forks = forks_for(step_class)+ # Stop (don't skip past) when the next step can't get its forks yet, so+ # a partitioned step can gather the budget instead of single-fork steps+ # nibbling it away.+ break if forks > @available_forks++ @states[step_class] = :running+ @available_forks -= forks+ @reserved_forks[step_class] = forks+ spawn_coordinator(step_class, forks)+ end+ end++ # A partitioned step takes all but one fork, leaving one free so single-fork+ # steps can still trickle through and overlap it.+ def forks_for(step_class)+ return 1 unless step_class.partitionable?+ [@budget - 1, 1].max+ end++ def spawn_coordinator(step_class, fork_count)+ coordinator = build_coordinator(step_class, fork_count)++ @threads << Thread.new do+ Thread.current.name = "step_#{step_class.name.demodulize}"++ outcome = :failed+ begin+ outcome = coordinator.run+ rescue SignalException+ outcome = :failed+ rescue StandardError => e+ # Must be recorded even if `run` should have handled it, or the step+ # never reaches a terminal state and the scheduler waits on it forever.+ record_failure(step_class, e)+ outcome = :failed+ ensure+ step_finished(step_class, outcome)+ end+ end+ end++ def build_coordinator(step_class, fork_count)+ StepCoordinator.new(+ step_class:,+ step_factory: @step_factory,+ reporter: @reporter,+ fork_mutex: @fork_mutex,+ scheduler: self,+ shard_manager: @shard_manager,+ consolidator: @consolidator,+ fork_count:,+ no_fork: @no_fork,+ )+ end++ def step_finished(step_class, outcome)+ @mutex.synchronize do+ @available_forks += @reserved_forks.delete(step_class)+ @states[step_class] = outcome+ skip_dependents(step_class) if outcome == :failed+ @condition.broadcast+ end+ end++ def skip_dependents(failed_step_class)+ loop do+ newly_skipped =+ @step_classes.select do |step_class|+ @states[step_class] == :pending &&+ dependencies_of(step_class).any? do |dependency|+ %i[failed skipped].include?(@states[dependency])+ end+ end++ break if newly_skipped.empty?+ newly_skipped.each { |step_class| @states[step_class] = :skipped }+ end+ end++ def next_ready_step+ ready = @step_classes.select { |step_class| ready?(step_class) }+ ready.min_by { |step_class| admission_key(step_class) }+ end++ def ready?(step_class)+ @states[step_class] == :pending &&+ dependencies_of(step_class).all? { |dependency| @states[dependency] == :done }+ end++ def dependencies_of(step_class)+ step_class.dependencies & @step_classes+ end++ # Partitioned steps first (so they reach the front and gather their forks),+ # then by priority (lower first, unset last), then by class name.+ def admission_key(step_class)+ priority = step_class.priority+ [+ step_class.partitionable? ? 0 : 1,+ priority.nil? ? 1 : 0,+ priority || 0,+ step_class.name.to_s,+ ]+ end++ def all_finished?+ @states.each_value.all? { |state| FINISHED_STATES.include?(state) }+ end++ def report_summary(runtime)+ outcomes = @states.values+ @reporter.report_summary(+ runtime:,+ total: outcomes.size,+ failed: outcomes.count(:failed),+ skipped: outcomes.count(:skipped),+ )+ end++ def monotonic_time+ Process.clock_gettime(Process::CLOCK_MONOTONIC)+ end++ def raise_summary_if_unsuccessful+ failed = @states.select { |_, state| state == :failed }.keys+ skipped = @states.select { |_, state| state == :skipped }.keys+ return if failed.empty? && skipped.empty? && @merge_errors.empty?++ raise ConvertError.new(+ failures: failed.to_h { |step_class| [step_class, @failures[step_class]] },+ skipped:,+ merge_errors: @merge_errors,+ )+ end+ end+ end+enddiff --git a/migrations/core/lib/migrations/conversion/worker.rb b/migrations/core/lib/migrations/conversion/worker.rbdeleted file mode 100644index b17c11bf993..00000000000--- a/migrations/core/lib/migrations/conversion/worker.rb+++ /dev/null@@ -1,139 +0,0 @@-# frozen_string_literal: true--require "oj"--module Migrations- module Conversion- class Worker- class CrashedError < StandardError- end-- OJ_SETTINGS = { mode: :object, class_cache: true, symbol_keys: true }-- def initialize(index, input_queue, output_queue, job)- @index = index- @input_queue = input_queue- @output_queue = output_queue- @job = job-- @threads = []- @mutex = Mutex.new- @data_processed = ConditionVariable.new- @sent_count = 0- @processed_count = 0- @output_closed = false- end-- def start- parent_input_stream, parent_output_stream = IO.pipe- fork_input_stream, fork_output_stream = IO.pipe-- worker_pid =- start_fork(- parent_input_stream,- parent_output_stream,- fork_input_stream,- fork_output_stream,- )-- fork_output_stream.close- parent_input_stream.close-- start_input_thread(parent_output_stream, worker_pid)- start_output_thread(fork_input_stream)-- self- end-- def wait- @threads.each(&:join)- end-- private-- def start_fork(- parent_input_stream,- parent_output_stream,- fork_input_stream,- fork_output_stream- )- ForkManager.fork do- Process.setproctitle("worker_process#{@index}")-- parent_output_stream.close- fork_input_stream.close-- @job.setup-- Oj.load(parent_input_stream, OJ_SETTINGS) do |data|- result = @job.run(data)- Oj.to_stream(fork_output_stream, result, OJ_SETTINGS)- end- rescue SignalException- exit(1)- ensure- @job.cleanup- end- end-- def start_input_thread(output_stream, worker_pid)- @threads << Thread.new do- Thread.current.name = "worker_#{@index}_input"- # A `CrashedError` surfaces through `Worker#wait` (`Thread#join`);- # without this, an interrupted run would additionally report the- # error for every worker.- Thread.current.report_on_exception = false-- begin- while (data = @input_queue.pop)- Oj.to_stream(output_stream, data, OJ_SETTINGS)- @sent_count += 1-- # waiting on the condition variable alone would lose the wakeup- # when the result arrives before this thread reaches `wait`, so- # the counters act as the wait predicate- @mutex.synchronize do- @data_processed.wait(@mutex) while @processed_count < @sent_count && !@output_closed- end- end- rescue Errno::EPIPE- # The worker process died; the status check below raises the error.- ensure- output_stream.close- _, status = Process.waitpid2(worker_pid)-- if !status.success?- raise CrashedError,- "Worker process #{@index} exited unexpectedly (#{status}). " \- "Check the error output above for the cause."- end- end- end- end-- def start_output_thread(input_stream)- @threads << Thread.new do- Thread.current.name = "worker_#{@index}_output"-- begin- Oj.load(input_stream, OJ_SETTINGS) do |data|- @output_queue.push(data)-- @mutex.synchronize do- @processed_count += 1- @data_processed.signal- end- end- ensure- input_stream.close-- @mutex.synchronize do- @output_closed = true- @data_processed.signal- end- end- end- end- end- end-enddiff --git a/migrations/core/lib/migrations/conversion/worker_budget.rb b/migrations/core/lib/migrations/conversion/worker_budget.rbnew file mode 100644index 00000000000..6bd1bbec228--- /dev/null+++ b/migrations/core/lib/migrations/conversion/worker_budget.rb@@ -0,0 +1,38 @@+# frozen_string_literal: true++require "etc"+require "concurrent/utility/processor_counter" # just the CPU counter, not all of concurrent-ruby++module Migrations+ module Conversion+ # Works out how many worker forks a run may use: the CPUs it can actually run+ # on, minus a core kept for the parent process and the background shard merges.+ #+ # "CPUs it can run on" is not just the host core count. In a container the run+ # is usually limited by a CFS quota (`docker --cpus`, k8s `limits.cpu`), which+ # `Etc.nprocessors` ignores. So we take the tighter of two numbers:+ #+ # * `Etc.nprocessors` — respects CPU affinity (`taskset`, cpuset)+ # * `Concurrent.available_processor_count` — respects the CFS quota+ #+ # Their minimum is the effective budget, so a 4-CPU pod on a 64-core node uses+ # 4, not 64.+ module WorkerBudget+ # Cores held back for the scheduler/coordinator threads (mostly idle, waiting+ # on IO) and the single consolidator merge thread. This is fixed, not scaled+ # with core count: the background work is about one busy thread, so one core+ # is enough on any machine — reserving more would just waste worker slots.+ RESERVED_CORES = 1++ def self.available(reserved: RESERVED_CORES)+ [usable_cpus - reserved, 1].max+ end++ # The CFS quota can be fractional (e.g. `--cpus=1.5`), so floor it; never+ # report fewer than one usable CPU.+ def self.usable_cpus+ [[Etc.nprocessors, Concurrent.available_processor_count.floor].min, 1].max+ end+ end+ end+enddiff --git a/migrations/core/lib/migrations/conversion/worker_pool.rb b/migrations/core/lib/migrations/conversion/worker_pool.rbdeleted file mode 100644index e8f02c57008..00000000000--- a/migrations/core/lib/migrations/conversion/worker_pool.rb+++ /dev/null@@ -1,48 +0,0 @@-# frozen_string_literal: true--require "etc"--module Migrations- module Conversion- # Owns worker sizing and the fork machinery for a conversion run. It knows- # nothing about steps, progress reporting, or scheduling — callers bring- # the queues and a job factory, the pool brings the processes.- class WorkerPool- DEFAULT_SIZE = Etc.nprocessors - 1 # leave 1 CPU free to do other work-- attr_reader :size-- def initialize(size: DEFAULT_SIZE)- @size = size- end-- # Forks worker processes (default: all `size` of them), each running the- # job built by `job_factory`. The factory is called once per worker, so- # workers never share job state. Workers exit when `work_queue` is- # closed and drained. Returns a handle for the started batch.- def start(work_queue:, output_queue:, size: @size, &job_factory)- worker_count = size.clamp(1, @size)- Process.warmup-- workers = []- ForkManager.with_batched_forks do- worker_count.times do |index|- workers << Worker.new(index, work_queue, output_queue, job_factory.call).start- end- end-- Batch.new(workers)- end-- class Batch- def initialize(workers)- @workers = workers- end-- def wait- @workers.each(&:wait)- end- end- end- end-enddiff --git a/migrations/core/lib/migrations/database.rb b/migrations/core/lib/migrations/database.rbindex 1d82dc6838a..f7f0049468d 100644--- a/migrations/core/lib/migrations/database.rb+++ b/migrations/core/lib/migrations/database.rb@@ -2,8 +2,9 @@require "date"require "extralite"+require "fileutils"require "ipaddr"-require "oj"+require "json"module Migrationsmodule Database@@ -15,8 +16,15 @@ module MigrationsMigrator.new(db_path).migrate(migrations_path)end- def self.reset!(db_path)- Migrator.new(db_path).reset!+ # A WAL-mode database is three files: the main file plus its `-wal` and `-shm`+ # sidecars.+ def self.database_files(db_path)+ [db_path, "#{db_path}-wal", "#{db_path}-shm"]+ end++ # Deletes the database and its sidecar files; no error if any are missing.+ def self.delete_database(db_path)+ FileUtils.rm_f(database_files(db_path))enddef self.connect(path)@@ -74,7 +82,7 @@ module Migrationsdef self.to_json(value)return nil if value.nil?- Oj.dump(value, mode: :compat)+ JSON.generate(value)enddef self.to_date(text)diff --git a/migrations/core/lib/migrations/database/connection.rb b/migrations/core/lib/migrations/database/connection.rbindex 67362ddddec..b8c2af15ad8 100644--- a/migrations/core/lib/migrations/database/connection.rb+++ b/migrations/core/lib/migrations/database/connection.rb@@ -1,6 +1,7 @@# frozen_string_literal: truerequire "extralite"+require "fileutils"module Migrationsmodule Database@@ -77,10 +78,30 @@ module Migrationsquery_value(sql, *parameters)end+ def tables+ @db.tables+ end+def execute(sql, *parameters)@db.execute(sql, *parameters)end+ # `ATTACH` can't run inside a transaction, so commit any open batch first.+ def merge_database(other_path, tables:)+ commit_transaction+ @db.execute("ATTACH DATABASE ? AS merge_source", other_path)+ begin+ tables.each do |table|+ quoted = quote_identifier(table)+ @db.execute("INSERT OR IGNORE INTO main.#{quoted} SELECT * FROM merge_source.#{quoted}")+ end+ ensure+ @db.execute("DETACH DATABASE merge_source")+ end++ nil+ end+def begin_transaction@db.execute("BEGIN DEFERRED TRANSACTION") unless @db.transaction_active?end@@ -94,6 +115,10 @@ module Migrationsprivate+ def quote_identifier(name)+ %("#{name.gsub('"', '""')}")+ end+def close_connection(keep_path:)return if @db.nil?diff --git a/migrations/core/lib/migrations/database/migrator.rb b/migrations/core/lib/migrations/database/migrator.rbindex 8627dd53b3d..6db247d5e02 100644--- a/migrations/core/lib/migrations/database/migrator.rb+++ b/migrations/core/lib/migrations/database/migrator.rb@@ -25,21 +25,10 @@ module Migrationsnilend- def reset!- [@db_path, "#{@db_path}-wal", "#{@db_path}-shm"].each do |path|- FileUtils.remove_file(path, force: true) if File.exist?(path)- end- nil- end-privatedef new_database?- @db.query_single_splat(<<~SQL) == 0- SELECT COUNT(*)- FROM sqlite_schema- WHERE type = 'table' AND name = 'schema_migrations'- SQL+ [email protected]?("schema_migrations")enddef find_performed_migrationsdiff --git a/migrations/core/lib/migrations/database/offline_connection.rb b/migrations/core/lib/migrations/database/offline_connection.rbdeleted file mode 100644index 1d4e35c6cb6..00000000000--- a/migrations/core/lib/migrations/database/offline_connection.rb+++ /dev/null@@ -1,31 +0,0 @@-# frozen_string_literal: true--module Migrations- module Database- class OfflineConnection- def initialize- @parametrized_insert_statements = []- end-- def close- @parametrized_insert_statements = nil- end-- def closed?- @parametrized_insert_statements.nil?- end-- def insert(sql, parameters = [])- @parametrized_insert_statements << [sql, parameters]- end-- def parametrized_insert_statements- @parametrized_insert_statements- end-- def clear!- @parametrized_insert_statements.clear if @parametrized_insert_statements- end- end- end-enddiff --git a/migrations/core/migrations-core.gemspec b/migrations/core/migrations-core.gemspecindex 7d465129057..98d2b080f04 100644--- a/migrations/core/migrations-core.gemspec+++ b/migrations/core/migrations-core.gemspec@@ -13,11 +13,12 @@ Gem::Specification.new do |s|s.add_dependency "activesupport"s.add_dependency "colored2"+ s.add_dependency "concurrent-ruby"s.add_dependency "digest-xxhash"s.add_dependency "extralite-bundle"s.add_dependency "i18n"+ s.add_dependency "json"s.add_dependency "lru_redux"- s.add_dependency "oj"s.add_dependency "samovar"s.add_dependency "unicode-display_width"s.add_dependency "zeitwerk"diff --git a/migrations/core/spec/lib/common/fork_manager_spec.rb b/migrations/core/spec/lib/common/fork_manager_spec.rbindex 54969acc5cb..25196b24202 100644--- a/migrations/core/spec/lib/common/fork_manager_spec.rb+++ b/migrations/core/spec/lib/common/fork_manager_spec.rb@@ -55,4 +55,16 @@ RSpec.describe Migrations::ForkManager doread_io.closeendend++ describe ".with_batched_forks" do+ it "runs the after-fork parent hooks even when the block raises" do+ ran = false+ described_class.after_fork_parent { ran = true }++ expect { described_class.with_batched_forks { raise "fork exploded" } }.to raise_error(+ "fork exploded",+ )+ expect(ran).to be(true)+ end+ endenddiff --git a/migrations/core/spec/lib/common/reporting/plain_spec.rb b/migrations/core/spec/lib/common/reporting/plain_spec.rbindex 3e00dc1dd83..76a72f99beb 100644--- a/migrations/core/spec/lib/common/reporting/plain_spec.rb+++ b/migrations/core/spec/lib/common/reporting/plain_spec.rb@@ -106,4 +106,12 @@ RSpec.describe Migrations::Reporting::Plain doexpect(lines).to include(" Users 100% (800/800)")end++ it "prints a finishing-up line and a run summary" do+ reporter.finalizing { nil }+ reporter.report_summary(runtime: 138.0, total: 24, failed: 2, skipped: 1)++ expect(lines).to include("Finishing up…")+ expect(lines.last).to eq("Total: 24 steps, 2 failed, 1 skipped (2:18)")+ endenddiff --git a/migrations/core/spec/lib/common/reporting/tui_spec.rb b/migrations/core/spec/lib/common/reporting/tui_spec.rbindex bb205a688a2..30cce3f4551 100644--- a/migrations/core/spec/lib/common/reporting/tui_spec.rb+++ b/migrations/core/spec/lib/common/reporting/tui_spec.rb@@ -237,6 +237,91 @@ RSpec.describe Migrations::Reporting::Tui doendend+ describe "the concurrency indicator" do+ let(:count_indicator) { /\d+×/ }++ def running_step(title, concurrency)+ renderer.apply([:start, title, title])+ renderer.apply([:progress_begin, title, 1000])+ renderer.apply([:concurrency, title, concurrency]) if concurrency+ renderer.apply([:progress, title, 500, 0, 0, 0])+ end++ it "renders the fork count, e.g. 4×, for a step using more than one fork" do+ running_step("Posts", 4)+ renderer.repaint+ expect(screen.content_rows.first).to include("4×")+ end++ it "shows nothing in the column for a single-fork step" do+ running_step("Posts", 1)+ renderer.repaint+ expect(screen.content_rows.first).not_to match(count_indicator)+ end++ it "shows nothing when the step never reports a count" do+ running_step("Posts", nil)+ renderer.repaint+ expect(screen.content_rows.first).not_to match(count_indicator)+ end++ it "updates the count live when it's re-reported" do+ running_step("Posts", 8)+ renderer.repaint+ expect(screen.content_rows.first).to include("8×")++ renderer.apply([:concurrency, "Posts", 2])+ renderer.repaint+ row = screen.content_rows.first+ expect(row).to include("2×")+ expect(row).not_to include("8×")+ end++ it "colors the count when color is on" do+ running_step("Posts", 8)+ renderer.repaint+ expect(io.string).to include("#{Migrations::Reporting::Tui::Ansi::MAGENTA}8×")+ end++ it "does not affect the count column shared with finished rows" do+ renderer.apply([:start, "Done", "Done"])+ renderer.apply([:progress_begin, "Done", 1_000_000])+ renderer.apply([:progress, "Done", 1_000_000, 0, 0, 0])+ renderer.apply([:finish, "Done", :done])+ running_step("Busy", 8)+ renderer.repaint++ rows = screen.content_rows+ done = rows.find { |r| r.include?("Done") }+ busy = rows.find { |r| r.include?("Busy") }+ # the count sits after the shared columns, so the totals still line up+ expect(busy.index("1,000") + "1,000".length).to eq(+ done.index("1,000,000") + "1,000,000".length,+ )+ end++ describe "without color" do+ around do |example|+ original = ENV["NO_COLOR"]+ ENV["NO_COLOR"] = "1"+ example.run+ original.nil? ? ENV.delete("NO_COLOR") : ENV["NO_COLOR"] = original+ end++ it "renders the count as plain text, with no color codes" do+ no_color = described_class.new(output: io, width: 120, clock: -> { time[0] })+ no_color.apply([:start, "Posts", "Posts"])+ no_color.apply([:progress_begin, "Posts", 1000])+ no_color.apply([:concurrency, "Posts", 8])+ no_color.apply([:progress, "Posts", 500, 0, 0, 0])+ no_color.repaint++ expect(screen.content_rows.first).to include("8×")+ expect(io.string).not_to include(Migrations::Reporting::Tui::Ansi::MAGENTA)+ end+ end+ end+describe "notices" doit "renders a permanent line prefixed with the step title" dorenderer.apply([:start, "Posts", "Posts"])@@ -395,6 +480,47 @@ RSpec.describe Migrations::Reporting::Tui doendendend++ describe "the finishing-up status and run summary" do+ it "shows a transient finishing-up line while draining, replaced by the summary" do+ renderer.apply([:finalizing_begin])+ renderer.repaint+ expect(screen.content_rows).to include(a_string_matching(/Finishing up/))++ # the real flow ends `finalizing` and prints the summary in the same frame+ renderer.apply([:finalizing_end])+ renderer.apply([:summary, 5.0, 3, 0, 0])+ renderer.repaint+ expect(screen.content_rows).not_to include(a_string_matching(/Finishing up/))+ expect(screen.content_rows).to include(a_string_matching(/Total.*3 steps/))+ end++ it "prints a summary with the runtime under the step-duration column" do+ renderer.apply([:start, "Users", "Users"])+ renderer.apply([:progress_begin, "Users", 1000])+ renderer.apply([:progress, "Users", 1000, 0, 0, 0])+ at(3.0)+ renderer.apply([:finish, "Users", :done])+ renderer.apply([:summary, 138.0, 24, 0, 0])+ renderer.repaint++ summary = screen.content_rows.find { |r| r.include?("Total") }+ step = screen.content_rows.find { |r| r.include?("Users") }+ expect(summary).to include("24 steps")+ expect(summary).to include("2:18") # 138 seconds runtime+ # the runtime lines up under the per-step durations+ expect(summary.index(/\d+:\d\d/)).to eq(step.index(/\d+:\d\d/))+ end++ it "notes failed and skipped steps in the summary" do+ renderer.apply([:summary, 60.0, 24, 2, 1])+ renderer.repaint++ row = screen.content_rows.find { |r| r.include?("Total") }+ expect(row).to include("2 failed")+ expect(row).to include("1 skipped")+ end+ endend# Facade-level behaviour exercises the real queue + render thread. Each test@@ -426,6 +552,23 @@ RSpec.describe Migrations::Reporting::Tui doexpect(row).to include("800 warnings")end+ it "coalesces progress instead of queuing one event per update" do+ step = reporter.start_step("Users")+ queue = reporter.instance_variable_get(:@queue)++ step.with_progress(max_progress: 10_000) do |progress|+ 10_000.times { progress.update(increment_by: 1) }+ # the 10k updates collapse into one coalesced slot, so the queue never+ # fills and the render thread keeps repainting+ expect(queue.size).to be < 50+ end+ step.finish+ reporter.close++ row = final_screen.content_rows.find { |r| r.start_with?("✓ Users") }+ expect(row).to include("10,000")+ end+it "renders an interrupted line when finish runs during a SignalException" dostep = reporter.start_step("Posts")step.with_progress(max_progress: 1000) { |progress| progress.update(increment_by: 410) }diff --git a/migrations/core/spec/lib/conversion/base_spec.rb b/migrations/core/spec/lib/conversion/base_spec.rbindex ba728de0f23..7002baca171 100644--- a/migrations/core/spec/lib/conversion/base_spec.rb+++ b/migrations/core/spec/lib/conversion/base_spec.rb@@ -2,7 +2,6 @@RSpec.describe Migrations::Conversion::Base dodescribe "#run" do- let(:offline_connection) { Migrations::Database::OfflineConnection.new }let(:converter) doTemporaryConverterModule::Converter.new({ intermediate_db: { path: "intermediate.db" } })end@@ -12,55 +11,12 @@ RSpec.describe Migrations::Conversion::Base do"TemporaryConverterModule",Module.new doconst_set("Converter", Class.new(Migrations::Conversion::Base))- const_set(- "Topics",- Class.new(Migrations::Conversion::ProgressStep) do- title "Converting topics"- # forces `execute_in_parallel?` to consult the pool's size; with- # `max_progress` below the parallel threshold the step still runs- # serially, so a missing pool fails fast without forking workers- run_in_parallel true-- source do- def max_progress- 5- end-- def items- Array.new(5) { |index| { id: index } }- end- end-- processor do- def process(item)- Migrations::Database::IntermediateDB.insert(- "INSERT INTO topics (original_id) VALUES (?)",- item[:id],- )- end- end- end,- )- const_set(- "Users",- Class.new(Migrations::Conversion::Step) do- title "Converting users"-- def execute- Migrations::Database::IntermediateDB.insert(- "INSERT INTO users (original_id) VALUES (?)",- 1,- )- end- end,- )+ const_set("Topics", Class.new(Migrations::Conversion::Step) { title "Converting topics" })+ const_set("Users", Class.new(Migrations::Conversion::Step) { title "Converting users" })end,)- Migrations::Database::IntermediateDB.setup(offline_connection)- # `run` closes the IntermediateDB in its `ensure`, which would discard- # the recorded insert statements before they can be verified- allow(offline_connection).to receive(:close)+ Migrations::Database::IntermediateDB.setup(nil)allow(converter).to receive(:create_database)end@@ -69,9 +25,15 @@ RSpec.describe Migrations::Conversion::Base doObject.send(:remove_const, "TemporaryConverterModule")end- it "creates one pool and one reporter per run and wires them through both executor kinds" do+ it "builds one reporter, then runs the scheduler over the filtered steps" do+ scheduler = instance_double(Migrations::Conversion::StepScheduler, run: nil)+ scheduler_args = nil+ allow(Migrations::Conversion::StepScheduler).to receive(:new) do |**kwargs|+ scheduler_args = kwargs+ scheduler+ end+reporter = nil- allow(Migrations::Conversion::WorkerPool).to receive(:new).and_call_originalallow(Migrations::Reporting::Factory).to receive(:build,).and_wrap_original do |original, **kwargs|@@ -80,17 +42,17 @@ RSpec.describe Migrations::Conversion::Base doreporterend- expect { converter.run }.to output(/Converting topics.*Converting users/m).to_stdout+ converter.run(skip_steps: ["users"], max_parallel_steps: 3)- expect(Migrations::Conversion::WorkerPool).to have_received(:new).onceexpect(Migrations::Reporting::Factory).to have_received(:build).once+ expect(scheduler).to have_received(:run).onceexpect(reporter).to have_received(:close).once- expect(offline_connection.parametrized_insert_statements).to eq(- [- *Array.new(5) { |index| ["INSERT INTO topics (original_id) VALUES (?)", [index]] },- ["INSERT INTO users (original_id) VALUES (?)", [1]],- ],- )++ expect(scheduler_args[:step_classes]).to eq([TemporaryConverterModule::Topics])+ expect(scheduler_args[:max_parallel_steps]).to eq(3)+ expect(scheduler_args[:budget]).to be > 0+ expect(scheduler_args[:reporter]).to be(reporter)+ expect(scheduler_args[:step_factory]).to respond_to(:call)endend@@ -103,7 +65,7 @@ RSpec.describe Migrations::Conversion::Base doModule.new doconst_set("Converter", Class.new(Migrations::Conversion::Base))const_set("Categories", Class.new(Migrations::Conversion::Step))- const_set("Topics", Class.new(Migrations::Conversion::ProgressStep))+ const_set("Topics", Class.new(Migrations::Conversion::Step))const_set("Users", Class.new(Migrations::Conversion::Step))const_set("SomeHelper", Class.new)end,@@ -112,7 +74,7 @@ RSpec.describe Migrations::Conversion::Base doafter { Object.send(:remove_const, "TemporaryConverterModule") }- it "discovers both `Step` and `ProgressStep` subclasses" do+ it "discovers `Step` subclasses" doexpect(converter.steps).to contain_exactly(TemporaryConverterModule::Categories,TemporaryConverterModule::Topics,diff --git a/migrations/core/spec/lib/conversion/consolidator_spec.rb b/migrations/core/spec/lib/conversion/consolidator_spec.rbnew file mode 100644index 00000000000..548e539c8f6--- /dev/null+++ b/migrations/core/spec/lib/conversion/consolidator_spec.rb@@ -0,0 +1,40 @@+# frozen_string_literal: true++RSpec.describe Migrations::Conversion::Consolidator do+ let(:shard_manager) { instance_double(Migrations::Conversion::ShardManager, discard: nil) }+ let(:connection) do+ instance_double(+ Migrations::Database::Connection,+ merge_database: nil,+ tables: %w[topics config schema_migrations],+ )+ end+ let(:fork_mutex) { Mutex.new }++ it "merges every enqueued shard into the run connection and discards each" do+ consolidator = described_class.new(shard_manager, connection, fork_mutex)+ consolidator.enqueue(%w[a.db b.db])+ consolidator.enqueue(%w[c.db])+ errors = consolidator.drain++ # `config` and `schema_migrations` are excluded from the merge+ expect(connection).to have_received(:merge_database).with("a.db", tables: %w[topics])+ expect(connection).to have_received(:merge_database).with("b.db", tables: %w[topics])+ expect(connection).to have_received(:merge_database).with("c.db", tables: %w[topics])+ expect(shard_manager).to have_received(:discard).with("a.db")+ expect(shard_manager).to have_received(:discard).with("c.db")+ expect(errors).to be_empty+ end++ it "collects a merge error and still discards the shard" do+ boom = StandardError.new("merge boom")+ allow(connection).to receive(:merge_database).with("bad.db", tables: anything).and_raise(boom)++ consolidator = described_class.new(shard_manager, connection, fork_mutex)+ consolidator.enqueue(%w[bad.db])+ errors = consolidator.drain++ expect(errors).to eq([boom])+ expect(shard_manager).to have_received(:discard).with("bad.db")+ end+enddiff --git a/migrations/core/spec/lib/conversion/parallel_job_spec.rb b/migrations/core/spec/lib/conversion/parallel_job_spec.rbdeleted file mode 100644index cdf1bca6c99..00000000000--- a/migrations/core/spec/lib/conversion/parallel_job_spec.rb+++ /dev/null@@ -1,95 +0,0 @@-# frozen_string_literal: true--RSpec.describe Migrations::Conversion::ParallelJob do- subject(:job) { described_class.new(processor) }-- let(:processor) { instance_double(Migrations::Conversion::ProgressStep::Processor) }- let(:item) { { key: "value" } }- let(:tracker) { instance_double(Migrations::Conversion::StepTracker) }- let(:stats) { Migrations::Conversion::StepStats.new }- let(:intermediate_db) { class_double(Migrations::Database::IntermediateDB).as_stubbed_const }-- before do- allow(processor).to receive(:tracker).and_return(tracker)- allow(processor).to receive(:setup)-- allow(tracker).to receive(:reset_stats!)- allow(tracker).to receive(:log_error)- allow(tracker).to receive(:stats).and_return(stats)-- allow(intermediate_db).to receive(:setup)- allow(intermediate_db).to receive(:with_connection).and_yield- allow(intermediate_db).to receive(:close)- end-- describe "#setup" do- it "sets up `OfflineConnection` as `IntermediateDB` connection" do- job.setup-- expect(intermediate_db).to have_received(:setup).with(- an_instance_of(Migrations::Database::OfflineConnection),- )- end-- it "runs the framework setup before the processor's `setup`" do- job.setup-- expect(intermediate_db).to have_received(:setup).ordered- expect(processor).to have_received(:setup).ordered- end-- it "runs the processor's `setup` through the setup guard" do- job.setup-- expect(intermediate_db).to have_received(:with_connection).with(- an_instance_of(Migrations::Conversion::SetupGuard::NoWriteConnection),- )- expect(processor).to have_received(:setup)- end- end-- describe "#run" do- let(:offline_connection) { instance_double(Migrations::Database::OfflineConnection) }-- before do- allow(Migrations::Database::OfflineConnection).to receive(:new).and_return(offline_connection)- allow(offline_connection).to receive(:clear!)-- allow(processor).to receive(:process)- allow(offline_connection).to receive(:parametrized_insert_statements).and_return(- [["SQL", [1, 2]], ["SQL", [2, 3]]],- )- end-- it "resets stats and clears the offline connection" do- job.run(item)-- expect(tracker).to have_received(:reset_stats!)- expect(offline_connection).to have_received(:clear!)- end-- it "processes an item and logs errors if exceptions occur" do- allow(processor).to receive(:process).and_raise(StandardError.new("error"))-- job.run(item)-- expect(tracker).to have_received(:log_error).with(- "Failed to process item",- exception: an_instance_of(StandardError),- details: item,- )- end-- it "returns the parametrized insert statements and stats" do- result = job.run(item)-- expect(result).to eq([[["SQL", [1, 2]], ["SQL", [2, 3]]], stats])- end- end-- describe "#cleanup" do- it "can be called without errors" do- expect { job.cleanup }.not_to raise_error- end- end-enddiff --git a/migrations/core/spec/lib/conversion/partitioner_spec.rb b/migrations/core/spec/lib/conversion/partitioner_spec.rbnew file mode 100644index 00000000000..4369f7766f4--- /dev/null+++ b/migrations/core/spec/lib/conversion/partitioner_spec.rb@@ -0,0 +1,104 @@+# frozen_string_literal: true++RSpec.describe Migrations::Conversion::Partitioner do+ # A stand-in for the source DB adapter, returning canned answers for the+ # primitives the partitioner calls. It has no `boundaries_by_scan`, so its scan+ # path goes through the streaming fallback.+ def build_adapter(min: nil, max: nil, estimate: 0, total: 0, keys: [])+ Class+ .new do+ define_method(:partition_bounds) { |_key, _from, _base| [min, max] }+ define_method(:estimated_row_count) { |_from| estimate }+ define_method(:count_all) { |_from, where:| total }+ define_method(:chunk_filter) { |*| "" }+ define_method(:each_partition_key) { |_key, _from, _base, &block| keys.each(&block) }+ end+ .new+ end++ def partitioner(adapter, key)+ described_class.new(adapter, key:, from: "things", base: nil)+ end++ it "strides a dense numeric key evenly by value, without a scan" do+ adapter = build_adapter(min: 0, max: 99, estimate: 100)+ expect(partitioner(adapter, :id).boundaries(4)).to eq([0, 25, 50, 75])+ end++ it "samples the sorted key when a numeric key is too sparse for even chunks" do+ # 4 rows scattered across a million-wide range: value-sized chunks would+ # leave most forks empty, so it samples the actual ids instead+ adapter =+ build_adapter(+ min: 0,+ max: 1_000_000,+ estimate: 4,+ total: 4,+ keys: [0, 10, 999_999, 1_000_000],+ )+ expect(partitioner(adapter, :id).boundaries(2)).to eq([0, 999_999])+ end++ it "strides a numeric key when the row estimate is unavailable (unanalysed table)" do+ # estimate 0 stands in for reltuples 0/-1: don't fall into the scan on a guess+ adapter = build_adapter(min: 0, max: 99, estimate: 0)+ expect(partitioner(adapter, :id).boundaries(4)).to eq([0, 25, 50, 75])+ end++ it "samples the sorted key for a non-numeric key" do+ adapter = build_adapter(min: "a", max: "d", total: 4, keys: %w[a b c d])+ expect(partitioner(adapter, :id).boundaries(2)).to eq(%w[a c])+ end++ it "samples a composite key without asking for numeric bounds" do+ adapter = build_adapter(total: 4, keys: [[1, 10], [1, 20], [2, 5], [3, 7]])+ expect(partitioner(adapter, %i[topic_id user_id]).boundaries(2)).to eq([[1, 10], [2, 5]])+ end++ it "is empty for an empty source" do+ expect(partitioner(build_adapter(min: nil, max: nil), :id).boundaries(4)).to eq([])+ end++ it "prefers the adapter's fast scan path when it offers one" do+ # A valid PartitionSource with `boundaries_by_scan` but no `each_partition_key`,+ # so the test fails (NoMethodError) unless the partitioner takes the fast path.+ adapter =+ Class+ .new do+ define_method(:partition_bounds) { |*| [nil, nil] }+ define_method(:estimated_row_count) { |*| 0 }+ define_method(:count_all) { |*, **| 0 }+ define_method(:chunk_filter) { |*| "" }+ define_method(:boundaries_by_scan) do |_key, _from, _base, count|+ [10, 20, 30].first(count)+ end+ end+ .new++ expect(partitioner(adapter, %i[a b]).boundaries(2)).to eq([10, 20])+ end++ it "raises when there is no source to compute boundaries from" do+ expect { partitioner(nil, :id).boundaries(4) }.to raise_error(+ described_class::UnsupportedSourceError,+ /doesn't implement the PartitionSource interface/,+ )+ end++ it "raises, naming the gap, when the adapter is missing a primitive" do+ adapter =+ Class+ .new do+ def partition_bounds(*)+ [0, 99]+ end+ # no estimated_row_count / count_all / each_partition_key / boundaries_by_scan+ end+ .new++ expect { partitioner(adapter, :id).boundaries(4) }.to raise_error(+ described_class::UnsupportedSourceError,+ /estimated_row_count/,+ )+ end+enddiff --git a/migrations/core/spec/lib/conversion/progress_step_executor_spec.rb b/migrations/core/spec/lib/conversion/progress_step_executor_spec.rbdeleted file mode 100644index a10ad1569e0..00000000000--- a/migrations/core/spec/lib/conversion/progress_step_executor_spec.rb+++ /dev/null@@ -1,144 +0,0 @@-# frozen_string_literal: true--RSpec.describe Migrations::Conversion::ProgressStepExecutor do- subject(:executor) { described_class.new(step, pool:, reporter:) }-- let(:pool_size) { 2 }- let(:pool) { Migrations::Conversion::WorkerPool.new(size: pool_size) }- let(:reporter) { Migrations::Reporting::Plain.new(output: StringIO.new) }- let(:item_count) { 30 }- let(:offline_connection) { Migrations::Database::OfflineConnection.new }- let(:step) { step_class.new(settings: { item_count: }) }-- before { Migrations::Database::IntermediateDB.setup(offline_connection) }- after { Migrations::Database::IntermediateDB.setup(nil) }-- def define_fixture_step(run_parallel)- Class.new(Migrations::Conversion::ProgressStep) do- title "Fixture step"- run_in_parallel run_parallel-- source do- def max_progress- settings[:item_count]- end-- def items- Array.new(settings[:item_count]) { |index| { id: index } }- end- end-- processor do- def setup- @prefix = "item"- end-- def process(item)- Migrations::Database::IntermediateDB.insert(- "INSERT INTO items (name) VALUES (?)",- "#{@prefix}-#{item[:id]}",- )- end- end- end- end-- def expected_insert_statements- Array.new(item_count) { |index| ["INSERT INTO items (name) VALUES (?)", ["item-#{index}"]] }- end-- def execute_quietly- original_stdout = $stdout- $stdout = StringIO.new- executor.execute- ensure- $stdout = original_stdout- end-- describe "#execute" do- context "when the step runs serially" do- let(:step_class) { define_fixture_step(false) }-- it "runs the processor's `setup` and processes all items in order" do- execute_quietly-- expect(offline_connection.parametrized_insert_statements).to eq(expected_insert_statements)- end- end-- context "when the step runs in parallel" do- let(:step_class) { define_fixture_step(true) }-- it "forks workers and produces the same inserts as a serial run" do- allow(Migrations::ForkManager).to receive(:fork).and_call_original-- execute_quietly-- expect(Migrations::ForkManager).to have_received(:fork).at_least(:once)- expect(offline_connection.parametrized_insert_statements).to match_array(- expected_insert_statements,- )- end- end-- context "when calculating the max progress is slow" do- let(:step_class) { define_fixture_step(false) }- let(:events) { [] }- let(:reporter) do- handle = instance_double(Migrations::Reporting::Reporter::StepHandle)- allow(handle).to receive(:notice) { events << :notice }- allow(handle).to receive(:with_progress) { events << :with_progress }- allow(handle).to receive(:finish) { events << :finish }-- instance_double(Migrations::Reporting::Plain).tap do |reporter|- allow(reporter).to receive(:start_step) do- events << :start_step- handle- end- end- end-- it "announces the step before the runtime notice and reports the step end last" do- allow(Time).to receive(:now).and_return(Time.at(0), Time.at(10))-- executor.execute-- expect(events).to eq(%i[start_step notice with_progress finish])- end- end-- context "when the parallel threshold scales with the pool size" do- let(:step_class) { define_fixture_step(true) }-- context "with no more items than `pool.size * 10`" do- let(:item_count) { pool_size * 10 }-- it "runs serially despite `run_in_parallel`" do- allow(Migrations::ForkManager).to receive(:fork).and_call_original-- execute_quietly-- expect(Migrations::ForkManager).not_to have_received(:fork)- expect(offline_connection.parametrized_insert_statements).to eq(- expected_insert_statements,- )- end- end-- context "with one item more than `pool.size * 10`" do- let(:item_count) { pool_size * 10 + 1 }-- it "runs in parallel" do- allow(Migrations::ForkManager).to receive(:fork).and_call_original-- execute_quietly-- expect(Migrations::ForkManager).to have_received(:fork).exactly(pool_size).times- expect(offline_connection.parametrized_insert_statements).to match_array(- expected_insert_statements,- )- end- end- end- end-enddiff --git a/migrations/core/spec/lib/conversion/progress_step_spec.rb b/migrations/core/spec/lib/conversion/progress_step_spec.rbdeleted file mode 100644index 2c5070a19f8..00000000000--- a/migrations/core/spec/lib/conversion/progress_step_spec.rb+++ /dev/null@@ -1,419 +0,0 @@-# frozen_string_literal: true--RSpec.describe Migrations::Conversion::ProgressStep do- def define_step(&block)- Class.new(described_class, &block)- end-- describe ".source" do- it "defines methods on the source role only" do- step_class =- define_step do- source do- def items- [1, 2, 3]- end- end- end-- source = step_class.source_class.new- expect(source.items).to eq([1, 2, 3])-- expect(step_class.new).not_to respond_to(:items)- expect(step_class.processor_class.new).not_to respond_to(:items)- end-- it "defines private methods on the source role" do- step_class =- define_step do- source do- def items- [build_item]- end-- private-- def build_item- :item- end- end- end-- source = step_class.source_class.new- expect(source.items).to eq([:item])- expect(source.respond_to?(:build_item)).to be(false)- expect(source.respond_to?(:build_item, true)).to be(true)- end-- it "raises an error when called twice" do- expect do- define_step do- source {}- source {}- end- end.to raise_error(ArgumentError, /`source` is already defined/)- end- end-- describe ".processor" do- it "defines methods on the processor role only" do- step_class =- define_step do- processor do- def process(item)- item * 2- end- end- end-- processor = step_class.processor_class.new- expect(processor.process(21)).to eq(42)-- expect(step_class.new).not_to respond_to(:process)- expect(step_class.source_class.new).not_to respond_to(:process)- end-- it "defines private methods on the processor role" do- step_class =- define_step do- processor do- def process(item)- transform(item)- end-- private-- def transform(item)- item.to_s- end- end- end-- processor = step_class.processor_class.new- expect(processor.process(1)).to eq("1")- expect(processor.respond_to?(:transform)).to be(false)- expect(processor.respond_to?(:transform, true)).to be(true)- end-- it "raises an error when called twice" do- expect do- define_step do- processor {}- processor {}- end- end.to raise_error(ArgumentError, /`processor` is already defined/)- end- end-- describe ".helpers" do- it "mixes the helpers into both roles" do- step_class =- define_step do- helpers do- def double_it(value)- value * 2- end- end-- source do- def items- [double_it(1)]- end- end-- processor do- def process(item)- double_it(item)- end- end- end-- expect(step_class.source_class.new.items).to eq([2])- expect(step_class.processor_class.new.process(2)).to eq(4)- end-- it "works independently of the order of the DSL calls" do- step_class =- define_step do- source do- def items- [double_it(1)]- end- end-- helpers do- def double_it(value)- value * 2- end- end- end-- expect(step_class.source_class.new.items).to eq([2])- end-- it "raises an error when called twice" do- expect do- define_step do- helpers {}- helpers {}- end- end.to raise_error(ArgumentError, /`helpers` is already defined/)- end- end-- describe "role isolation" do- it "raises a NameError when the processor calls a source method" do- step_class =- define_step do- source do- def items- []- end-- private-- def source_helper- "source-side state"- end- end-- processor do- def process(item)- source_helper- end- end- end-- processor = step_class.processor_class.new- expect { processor.process(1) }.to raise_error(NameError, /source_helper/)- end-- it "does not expose instance variables set in `items` to the processor" do- step_class =- define_step do- source do- def items- @leaked_state = "set in items"- [1]- end- end-- processor do- def process(item)- instance_variable_defined?(:@leaked_state)- end- end- end-- step = step_class.new- step.source.items-- expect(step.create_processor.process(1)).to be(false)- end-- it "does not capture local variables in methods defined with `def`" do- local_state = "step-level local"-- step_class =- define_step do- source do- def items- local_state- end- end- end-- expect { step_class.source_class.new.items }.to raise_error(NameError, /local_state/)- expect(local_state).to eq("step-level local") # silence the unused variable warning- end- end-- describe "role defaults" do- let(:step_class) { define_step }-- it "uses `nil` as default `max_progress`" do- expect(step_class.source_class.new.max_progress).to be_nil- end-- it "raises `NotImplementedError` when `items` is not defined" do- expect { step_class.source_class.new.items }.to raise_error(NotImplementedError)- end-- it "raises `NotImplementedError` when `process` is not defined" do- expect { step_class.processor_class.new.process(1) }.to raise_error(NotImplementedError)- end-- it "uses a no-op as default `setup`" do- expect { step_class.processor_class.new.setup }.not_to raise_error- end- end-- describe "reads_table defaults" do- # Stands in for the source DB adapter, recording the calls it receives (the- # adapter builds the actual SQL; that's covered in its own spec).- let(:source_db) do- Class- .new do- attr_reader :select_args, :count_args-- def select_all(table, where:)- @select_args = [table, where]- [:row]- end-- def count_all(table, where:)- @count_args = [table, where]- 4- end- end- .new- end-- it "reads the whole table through the adapter" do- step_class = define_step { source { reads_table "things", where: "active" } }- source = step_class.source_class.new(source_db:)-- expect(source.items).to eq([:row])- expect(source_db.select_args).to eq(%w[things active])-- expect(source.max_progress).to eq(4)- expect(source_db.count_args).to eq(%w[things active])- end-- it "passes no filter when `reads_table` has no `where`" do- step_class = define_step { source { reads_table "things" } }- source = step_class.source_class.new(source_db:)-- source.items- expect(source_db.select_args).to eq(["things", nil])- end- end-- describe "constant resolution" do- # The role blocks of real steps are written inside a step class body, so- # constants in their methods resolve through the step's lexical scope and- # ancestry (`Step` defines `IntermediateDB` and `Enums`). A `Class.new`- # block in this spec file would have the wrong lexical scope, hence the- # string eval.- before { Object.class_eval <<~RUBY }- class ProgressStepConstantsFixture < Migrations::Conversion::ProgressStep- source do- def items- [IntermediateDB, Enums]- end- end-- processor do- def process(item)- [IntermediateDB, Enums]- end- end- end- RUBY-- after { Object.send(:remove_const, :ProgressStepConstantsFixture) }-- it "resolves `IntermediateDB` and `Enums` inside role blocks via the step's ancestry" do- expected = [Migrations::Database::IntermediateDB, Migrations::Database::IntermediateDB::Enums]-- expect(ProgressStepConstantsFixture.source_class.new.items).to eq(expected)- expect(ProgressStepConstantsFixture.processor_class.new.process(nil)).to eq(expected)- end-- it "does not define the constants on the role classes themselves" do- expect(ProgressStepConstantsFixture.source_class).not_to have_constant(:IntermediateDB)- expect(ProgressStepConstantsFixture.processor_class).not_to have_constant(:IntermediateDB)- end- end-- describe "#initialize" do- it "routes args to the role that declares a matching setter" do- step_class = define_step { source { attr_accessor :source_db } }-- settings = { a: 1 }- step = step_class.new(settings:, source_db: "source db", unknown_arg: "dropped")-- expect(step.source.settings).to eq(settings)- expect(step.source.source_db).to eq("source db")-- processor = step.create_processor- expect(processor.settings).to eq(settings)- expect(processor).not_to respond_to(:source_db)- expect(processor).not_to respond_to(:unknown_arg)- end-- it "keeps no per-step state on the coordinator" do- step = define_step.new(settings: { a: 1 })-- expect(step).not_to be_a(Migrations::Conversion::Step)- expect(step).not_to respond_to(:tracker)- expect(step).not_to respond_to(:step)- expect(step).not_to respond_to(:settings)- expect(step).not_to respond_to(:execute)- end- end-- describe "#source" do- it "returns the same source instance every time" do- step = define_step.new- expect(step.source).to be(step.source)- end- end-- describe "#create_processor" do- it "creates a new processor with its own tracker on every call" do- step = define_step.new-- processor1 = step.create_processor- processor2 = step.create_processor-- expect(processor1).not_to be(processor2)- expect(processor1.tracker).to be_a(Migrations::Conversion::StepTracker)- expect(processor1.tracker).not_to be(processor2.tracker)- end- end-- describe "step subclasses" do- it "inherits the roles of the parent step" do- parent_class =- define_step do- source do- def items- [1]- end- end-- processor do- def process(item)- item + 1- end- end- end-- step_class =- Class.new(parent_class) do- source do- def max_progress- 1- end- end- end-- source = step_class.source_class.new- expect(source.items).to eq([1])- expect(source.max_progress).to eq(1)- expect(step_class.processor_class.new.process(1)).to eq(2)- end- end-- describe ".run_in_parallel?" do- it "defaults to false" do- expect(define_step.run_in_parallel?).to be(false)- end-- it "returns the configured value" do- expect(define_step { run_in_parallel true }.run_in_parallel?).to be(true)- expect(define_step { run_in_parallel false }.run_in_parallel?).to be(false)- end- end-enddiff --git a/migrations/core/spec/lib/conversion/scheduler_integration_spec.rb b/migrations/core/spec/lib/conversion/scheduler_integration_spec.rbnew file mode 100644index 00000000000..29b70bb198e--- /dev/null+++ b/migrations/core/spec/lib/conversion/scheduler_integration_spec.rb@@ -0,0 +1,591 @@+# frozen_string_literal: true++require "extralite"+require "timeout"++RSpec.describe Migrations::Conversion::StepScheduler, :integration do+ around do |example|+ Dir.mktmpdir do |storage_path|+ @db_path = File.join(storage_path, "intermediate.db")+ migrations_path = File.join(storage_path, "migrations")+ FileUtils.mkdir_p(migrations_path)+ File.write(File.join(migrations_path, "001-schema.sql"), <<~SQL)+ CREATE TABLE topics (id INTEGER);+ CREATE TABLE users (id INTEGER);+ CREATE TABLE notes (id INTEGER, body TEXT NOT NULL);+ CREATE TABLE keyed (id INTEGER PRIMARY KEY, label TEXT);+ CREATE TABLE log_entries (created_at TEXT, type INTEGER, message TEXT, exception TEXT, details TEXT);+ SQL+ Migrations::Database.migrate(@db_path, migrations_path:)++ # The shard template is migrated fresh from the same schema (no data), so+ # shards stay empty even when the run DB already has rows.+ @shard_manager =+ Migrations::Conversion::ShardManager.new(canonical_path: @db_path, migrations_path:)+ @writer = Migrations::Database::Connection.new(path: @db_path)+ Migrations::Database::IntermediateDB.setup(@writer)+ begin+ example.run+ ensure+ Migrations::Database::IntermediateDB.close+ @shard_manager.cleanup+ end+ end+ end++ def rows(table)+ db = Extralite::Database.new(@db_path)+ db.query_splat("SELECT id FROM #{table} ORDER BY id")+ ensure+ db.close if db+ end++ def log_entry_count+ db = Extralite::Database.new(@db_path)+ db.query_single_splat("SELECT COUNT(*) FROM log_entries")+ ensure+ db.close if db+ end++ it "writes every row from two steps running concurrently" do+ Object.const_set(+ "TempIntegrationSteps",+ Module.new do+ const_set(+ "Topics",+ Class.new(Migrations::Conversion::Step) do+ source do+ def max_progress+ 50+ end++ def items+ Array.new(50) { |index| { id: index } }+ end+ end++ processor do+ def process(item)+ Migrations::Database::IntermediateDB.insert(+ "INSERT INTO topics (id) VALUES (?)",+ item[:id],+ )+ end+ end+ end,+ )+ const_set(+ "Users",+ Class.new(Migrations::Conversion::Step) do+ source do+ def max_progress+ 5+ end++ def items+ Array.new(5) { |index| { id: index } }+ end+ end++ processor do+ def process(item)+ Migrations::Database::IntermediateDB.insert(+ "INSERT INTO users (id) VALUES (?)",+ item[:id],+ )+ end+ end+ end,+ )+ end,+ )++ step_classes = [TempIntegrationSteps::Topics, TempIntegrationSteps::Users]+ reporter = Migrations::Reporting::Factory.build(titles: step_classes.map(&:title))++ expect do+ Migrations::Conversion::StepScheduler.new(+ step_classes:,+ budget: 2,+ reporter:,+ step_factory: ->(step_class) { step_class.new },+ shard_manager: @shard_manager,+ writer: @writer,+ ).run+ ensure+ reporter&.close+ end.to output(/Converting topics/).to_stdout_from_any_process++ # IntermediateDB writes are batched in a transaction; commit before reading+ Migrations::Database::IntermediateDB.close++ expect(rows("topics")).to eq((0..49).to_a)+ expect(rows("users")).to eq((0..4).to_a)+ ensure+ Object.send(:remove_const, "TempIntegrationSteps")+ end++ it "splits a partitioned step across forks and merges every slice" do+ Object.const_set(+ "TempIntegrationSteps",+ Module.new do+ const_set(+ "Topics",+ Class.new(Migrations::Conversion::Step) do+ source do+ partition_by :id, from: "topics"++ def all+ Array.new(60) { |index| { id: index } }+ end++ # Give the coordinator `count` chunk lower bounds (the adapter does+ # this for a real source); each fork reads only its own `[lower, upper)`.+ def partition_boundaries(count)+ size = (all.size.to_f / count).ceil+ (0...count).map { |i| i * size }+ end++ def items+ return all unless chunk++ lower, upper = chunk+ all.select { |it| it[:id] >= lower && (upper.nil? || it[:id] < upper) }+ end++ def max_progress+ items.size+ end+ end++ processor do+ def process(item)+ Migrations::Database::IntermediateDB.insert(+ "INSERT INTO topics (id) VALUES (?)",+ item[:id],+ )+ end+ end+ end,+ )+ end,+ )++ step_classes = [TempIntegrationSteps::Topics]+ reporter = Migrations::Reporting::Factory.build(titles: step_classes.map(&:title))++ Migrations::Conversion::StepScheduler.new(+ step_classes:,+ budget: 4,+ reporter:,+ step_factory: ->(step_class) { step_class.new },+ shard_manager: @shard_manager,+ writer: @writer,+ ).run+ reporter.close++ Migrations::Database::IntermediateDB.close++ expect(rows("topics")).to eq((0..59).to_a)+ ensure+ Object.send(:remove_const, "TempIntegrationSteps")+ end++ # The coordinator sizes the fork count from the boundaries, not the budget:+ # `worker_count = [boundaries.size, 1].max`. These two cases exercise the+ # collapse to a single worker.+ it "collapses to one worker when a partitioned source yields fewer chunks than forks" do+ Object.const_set(+ "TempIntegrationSteps",+ Module.new do+ const_set(+ "Topics",+ Class.new(Migrations::Conversion::Step) do+ source do+ partition_by :id, from: "topics"++ def all+ Array.new(3) { |index| { id: index } }+ end++ # Only one boundary, though more forks are free.+ def partition_boundaries(_count)+ [0]+ end++ def items+ return all unless chunk++ lower, upper = chunk+ all.select { |it| it[:id] >= lower && (upper.nil? || it[:id] < upper) }+ end++ def max_progress+ items.size+ end+ end++ processor do+ def process(item)+ Migrations::Database::IntermediateDB.insert(+ "INSERT INTO topics (id) VALUES (?)",+ item[:id],+ )+ end+ end+ end,+ )+ end,+ )++ step_classes = [TempIntegrationSteps::Topics]+ reporter = Migrations::Reporting::Factory.build(titles: step_classes.map(&:title))++ Migrations::Conversion::StepScheduler.new(+ step_classes:,+ budget: 4,+ reporter:,+ step_factory: ->(step_class) { step_class.new },+ shard_manager: @shard_manager,+ writer: @writer,+ ).run+ reporter.close++ Migrations::Database::IntermediateDB.close++ expect(rows("topics")).to eq([0, 1, 2]) # every row merged exactly once+ ensure+ Object.send(:remove_const, "TempIntegrationSteps")+ end++ it "runs a single empty worker for a partitioned step over an empty source" do+ Object.const_set(+ "TempIntegrationSteps",+ Module.new do+ const_set(+ "Topics",+ Class.new(Migrations::Conversion::Step) do+ source do+ partition_by :id, from: "topics"++ # No rows, so no boundaries — the coordinator still runs one worker.+ def partition_boundaries(_count)+ []+ end++ def items+ []+ end++ def max_progress+ 0+ end+ end++ processor do+ def process(item)+ Migrations::Database::IntermediateDB.insert(+ "INSERT INTO topics (id) VALUES (?)",+ item[:id],+ )+ end+ end+ end,+ )+ end,+ )++ step_classes = [TempIntegrationSteps::Topics]+ reporter = Migrations::Reporting::Factory.build(titles: step_classes.map(&:title))++ Migrations::Conversion::StepScheduler.new(+ step_classes:,+ budget: 4,+ reporter:,+ step_factory: ->(step_class) { step_class.new },+ shard_manager: @shard_manager,+ writer: @writer,+ ).run+ reporter.close++ Migrations::Database::IntermediateDB.close++ expect(rows("topics")).to eq([])+ ensure+ Object.send(:remove_const, "TempIntegrationSteps")+ end++ it "runs the step inline, without forking, when no_fork is set" do+ Object.const_set(+ "TempIntegrationSteps",+ Module.new do+ const_set(+ "Topics",+ Class.new(Migrations::Conversion::Step) do+ source do+ partition_by :id, from: "topics"++ def all+ Array.new(20) { |index| { id: index } }+ end++ def partition_boundaries(count)+ size = (all.size.to_f / count).ceil+ (0...count).map { |i| i * size }+ end++ def items+ return all unless chunk++ lower, upper = chunk+ all.select { |it| it[:id] >= lower && (upper.nil? || it[:id] < upper) }+ end++ def max_progress+ items.size+ end+ end++ processor do+ def process(item)+ Migrations::Database::IntermediateDB.insert(+ "INSERT INTO topics (id) VALUES (?)",+ item[:id],+ )+ end+ end+ end,+ )+ end,+ )++ step_classes = [TempIntegrationSteps::Topics]+ reporter = Migrations::Reporting::Factory.build(titles: step_classes.map(&:title))++ allow(Migrations::ForkManager).to receive(:fork).and_call_original++ Migrations::Conversion::StepScheduler.new(+ step_classes:,+ budget: 4,+ reporter:,+ step_factory: ->(step_class) { step_class.new },+ shard_manager: @shard_manager,+ writer: @writer,+ no_fork: true,+ ).run+ reporter.close++ Migrations::Database::IntermediateDB.close++ expect(Migrations::ForkManager).not_to have_received(:fork)+ expect(rows("topics")).to eq((0..19).to_a)+ ensure+ Object.send(:remove_const, "TempIntegrationSteps")+ end++ it "logs and skips a bad row instead of failing the step" do+ Object.const_set(+ "TempIntegrationSteps",+ Module.new do+ # The middle row's NULL `body` breaks the NOT NULL constraint on insert.+ const_set(+ "Notes",+ Class.new(Migrations::Conversion::Step) do+ source do+ def items+ [{ id: 1, body: "ok" }, { id: 2, body: nil }, { id: 3, body: "ok" }]+ end+ end++ processor do+ def process(item)+ Migrations::Database::IntermediateDB.insert(+ "INSERT INTO notes (id, body) VALUES (?, ?)",+ item[:id],+ item[:body],+ )+ end+ end+ end,+ )+ const_set(+ "Users",+ Class.new(Migrations::Conversion::Step) do+ source do+ def items+ Array.new(5) { |index| { id: index } }+ end+ end++ processor do+ def process(item)+ Migrations::Database::IntermediateDB.insert(+ "INSERT INTO users (id) VALUES (?)",+ item[:id],+ )+ end+ end+ end,+ )+ end,+ )++ step_classes = [TempIntegrationSteps::Notes, TempIntegrationSteps::Users]+ reporter = Migrations::Reporting::Factory.build(titles: step_classes.map(&:title))++ run =+ lambda do+ Migrations::Conversion::StepScheduler.new(+ step_classes:,+ budget: 2,+ reporter:,+ step_factory: ->(step_class) { step_class.new },+ shard_manager: @shard_manager,+ writer: @writer,+ ).run+ ensure+ reporter&.close+ end++ expect { run.call }.not_to raise_error++ Migrations::Database::IntermediateDB.close++ expect(rows("notes")).to eq([1, 3])+ expect(rows("users")).to eq((0..4).to_a)+ expect(log_entry_count).to eq(1)+ ensure+ Object.send(:remove_const, "TempIntegrationSteps")+ end++ it "fails the step and finishes the run when a worker dies mid-stream" do+ Object.const_set(+ "TempIntegrationSteps",+ Module.new do+ # A worker that exits part-way closes its progress pipe, so the+ # coordinator sees the non-zero exit and discards the half-written shard.+ const_set(+ "Crashing",+ Class.new(Migrations::Conversion::Step) do+ source do+ def items+ Array.new(300) { |index| { id: index } }+ end+ end++ processor do+ def process(item)+ exit!(1) if item[:id] == 150+ Migrations::Database::IntermediateDB.insert(+ "INSERT INTO topics (id) VALUES (?)",+ item[:id],+ )+ end+ end+ end,+ )+ const_set(+ "Users",+ Class.new(Migrations::Conversion::Step) do+ source do+ def items+ Array.new(300) { |index| { id: index } }+ end+ end++ processor do+ def process(item)+ Migrations::Database::IntermediateDB.insert(+ "INSERT INTO users (id) VALUES (?)",+ item[:id],+ )+ end+ end+ end,+ )+ end,+ )++ step_classes = [TempIntegrationSteps::Crashing, TempIntegrationSteps::Users]+ reporter = Migrations::Reporting::Factory.build(titles: step_classes.map(&:title))++ run =+ lambda do+ Migrations::Conversion::StepScheduler.new(+ step_classes:,+ budget: 2,+ reporter:,+ step_factory: ->(step_class) { step_class.new },+ shard_manager: @shard_manager,+ writer: @writer,+ ).run+ ensure+ reporter&.close+ end++ # `Timeout` turns a hanging run into a failed example instead of a stuck suite.+ expect { Timeout.timeout(30) { run.call } }.to raise_error(Migrations::Conversion::ConvertError)++ Migrations::Database::IntermediateDB.close++ expect(rows("users")).to eq((0..299).to_a)+ # the crashed worker's half-written shard is discarded, not merged+ expect(rows("topics")).to eq([])+ ensure+ Object.send(:remove_const, "TempIntegrationSteps")+ end++ it "adds new rows to an existing IntermediateDB without dropping or overwriting" do+ # A row from a previous run already lives in the DB.+ Migrations::Database::IntermediateDB.insert("INSERT INTO keyed VALUES (?, ?)", 1, "existing")++ Object.const_set(+ "TempIntegrationSteps",+ Module.new do+ const_set(+ "Keyed",+ Class.new(Migrations::Conversion::Step) do+ source do+ def items+ # id 1 collides with the existing row; id 2 is new+ [{ id: 1, label: "new" }, { id: 2, label: "added" }]+ end+ end++ processor do+ def process(item)+ Migrations::Database::IntermediateDB.insert(+ "INSERT INTO keyed VALUES (?, ?)",+ item[:id],+ item[:label],+ )+ end+ end+ end,+ )+ end,+ )++ step_classes = [TempIntegrationSteps::Keyed]+ reporter = Migrations::Reporting::Factory.build(titles: step_classes.map(&:title))+ Migrations::Conversion::StepScheduler.new(+ step_classes:,+ budget: 2,+ reporter:,+ step_factory: ->(step_class) { step_class.new },+ shard_manager: @shard_manager,+ writer: @writer,+ ).run+ reporter.close++ Migrations::Database::IntermediateDB.close++ db = Extralite::Database.new(@db_path)+ keyed = db.query_array("SELECT id, label FROM keyed ORDER BY id")+ db.close+ # the existing row is kept as-is (not overwritten), and the new row is added+ expect(keyed).to eq([[1, "existing"], [2, "added"]])+ ensure+ Object.send(:remove_const, "TempIntegrationSteps")+ end+enddiff --git a/migrations/core/spec/lib/conversion/serial_job_spec.rb b/migrations/core/spec/lib/conversion/serial_job_spec.rbdeleted file mode 100644index 895a2444c77..00000000000--- a/migrations/core/spec/lib/conversion/serial_job_spec.rb+++ /dev/null@@ -1,64 +0,0 @@-# frozen_string_literal: true--RSpec.describe Migrations::Conversion::SerialJob do- subject(:job) { described_class.new(processor) }-- let(:processor) { instance_double(Migrations::Conversion::ProgressStep::Processor) }- let(:item) { "Item" }- let(:tracker) { instance_double(Migrations::Conversion::StepTracker) }- let(:stats) { Migrations::Conversion::StepStats.new }-- before do- allow(processor).to receive(:tracker).and_return(tracker)- allow(processor).to receive(:setup)-- allow(tracker).to receive(:reset_stats!)- allow(tracker).to receive(:log_error)- allow(tracker).to receive(:stats).and_return(stats)- end-- describe "#setup" do- it "runs the processor's `setup`" do- job.setup- expect(processor).to have_received(:setup)- end-- it "raises an error when the processor writes to the IntermediateDB during `setup`" do- allow(processor).to receive(:setup) do- Migrations::Database::IntermediateDB.insert("INSERT INTO foo (id) VALUES (?)", 1)- end-- expect { job.setup }.to raise_error(Migrations::Conversion::SetupGuard::SetupError)- end- end-- describe "#run" do- it "resets stats and processes item" do- allow(processor).to receive(:process).and_return(stats)-- result = job.run(item)- expect(result).to eq(stats)-- expect(tracker).to have_received(:reset_stats!)- expect(processor).to have_received(:process).with(item)- end-- it "logs error if processing item raises an exception" do- allow(processor).to receive(:process).and_raise(StandardError)-- job.run(item)-- expect(tracker).to have_received(:log_error).with(- "Failed to process item",- exception: an_instance_of(StandardError),- details: item,- )- end- end-- describe "#cleanup" do- it "can be called without errors" do- expect { job.cleanup }.not_to raise_error- end- end-enddiff --git a/migrations/core/spec/lib/conversion/setup_guard_spec.rb b/migrations/core/spec/lib/conversion/setup_guard_spec.rbindex 5b1e8a30791..0f643c6dec2 100644--- a/migrations/core/spec/lib/conversion/setup_guard_spec.rb+++ b/migrations/core/spec/lib/conversion/setup_guard_spec.rb@@ -1,7 +1,7 @@# frozen_string_literal: trueRSpec.describe Migrations::Conversion::SetupGuard do- let(:processor) { instance_double(Migrations::Conversion::ProgressStep::Processor) }+ let(:processor) { instance_double(Migrations::Conversion::Step::Processor) }let(:intermediate_db) { Migrations::Database::IntermediateDB }before { reset_memoization(intermediate_db, :@db) }@@ -28,7 +28,9 @@ RSpec.describe Migrations::Conversion::SetupGuard doendit "restores the previous connection, even when `setup` writes" do- connection = Migrations::Database::OfflineConnection.new+ connection = instance_double(Migrations::Database::Connection)+ allow(connection).to receive(:insert)+ allow(connection).to receive(:close)intermediate_db.setup(connection)allow(processor).to receive(:setup) do@@ -37,10 +39,11 @@ RSpec.describe Migrations::Conversion::SetupGuard doexpect { described_class.run(processor) }.to raise_error(described_class::SetupError)+ # the guard blocked the write from `setup`; only the write made after it+ # restored the connection gets throughintermediate_db.insert("INSERT INTO foo (id) VALUES (?)", 2)- expect(connection.parametrized_insert_statements).to eq(- [["INSERT INTO foo (id) VALUES (?)", [2]]],- )+ expect(connection).to have_received(:insert).with("INSERT INTO foo (id) VALUES (?)", [2])+ expect(connection).not_to have_received(:insert).with("INSERT INTO foo (id) VALUES (?)", [1])endendenddiff --git a/migrations/core/spec/lib/conversion/shard_manager_spec.rb b/migrations/core/spec/lib/conversion/shard_manager_spec.rbnew file mode 100644index 00000000000..ca535d08708--- /dev/null+++ b/migrations/core/spec/lib/conversion/shard_manager_spec.rb@@ -0,0 +1,54 @@+# frozen_string_literal: true++require "extralite"++RSpec.describe Migrations::Conversion::ShardManager do+ around do |example|+ Dir.mktmpdir do |dir|+ @canonical = File.join(dir, "intermediate.db")+ @migrations = File.join(dir, "migrations")+ FileUtils.mkdir_p(@migrations)+ File.write(File.join(@migrations, "001-schema.sql"), <<~SQL)+ CREATE TABLE widgets (id INTEGER PRIMARY KEY, name TEXT);+ CREATE INDEX widgets_name ON widgets (name);+ SQL+ Migrations::Database.migrate(@canonical, migrations_path: @migrations)++ # a row from a previous run already lives in the DB+ db = Extralite::Database.new(@canonical)+ db.execute("INSERT INTO widgets (id, name) VALUES (1, 'existing')")+ db.close++ example.run+ end+ end++ def build_manager+ described_class.new(canonical_path: @canonical, migrations_path: @migrations)+ end++ it "gives each shard the run's schema but none of its data" do+ manager = build_manager+ db = Extralite::Database.new(manager.create_shard)++ tables =+ db.query_array(+ "SELECT name FROM sqlite_master WHERE type = 'table' AND name NOT LIKE 'sqlite_%'",+ ).flatten+ indexes = db.query_array("SELECT name FROM sqlite_master WHERE type = 'index'").flatten++ expect(tables).to include("widgets") # schema is there+ expect(indexes).to include("widgets_name") # indexes too+ expect(db.query_single_splat("SELECT COUNT(*) FROM widgets")).to eq(0) # but no rows+ ensure+ db&.close+ manager&.cleanup+ end++ it "hands out a distinct shard path each time" do+ manager = build_manager+ expect(manager.create_shard).not_to eq(manager.create_shard)+ ensure+ manager&.cleanup+ end+enddiff --git a/migrations/core/spec/lib/conversion/step_base_spec.rb b/migrations/core/spec/lib/conversion/step_base_spec.rbdeleted file mode 100644index 0fa13656d5c..00000000000--- a/migrations/core/spec/lib/conversion/step_base_spec.rb+++ /dev/null@@ -1,39 +0,0 @@-# frozen_string_literal: true--RSpec.describe Migrations::Conversion::StepBase do- before do- Object.const_set(- "TemporaryStepsModule",- Module.new do- const_set("Topics", Class.new(Migrations::Conversion::ProgressStep))- const_set("TopicUsers", Class.new(Migrations::Conversion::Step))- const_set("Users", Class.new(Migrations::Conversion::Step))- end,- )- end-- after { Object.send(:remove_const, "TemporaryStepsModule") }-- describe "dependency metadata" do- it "exposes the StepDependencies macros at class level" do- expect(described_class).to be_a(Migrations::StepDependencies)- expect(described_class).to respond_to(:depends_on, :dependencies, :priority)- end-- it "lets steps depend on both `Step` and `ProgressStep` subclasses" do- TemporaryStepsModule::TopicUsers.depends_on(:topics, :users)-- expect(TemporaryStepsModule::TopicUsers.dependencies).to eq(- [TemporaryStepsModule::Topics, TemporaryStepsModule::Users],- )- end-- it "doesn't bleed metadata between sibling step classes" do- TemporaryStepsModule::TopicUsers.depends_on(:users)- TemporaryStepsModule::TopicUsers.priority 1-- expect(TemporaryStepsModule::Users.dependencies).to eq([])- expect(TemporaryStepsModule::Users.priority).to be_nil- end- end-enddiff --git a/migrations/core/spec/lib/conversion/step_executor_spec.rb b/migrations/core/spec/lib/conversion/step_executor_spec.rbdeleted file mode 100644index d0654446675..00000000000--- a/migrations/core/spec/lib/conversion/step_executor_spec.rb+++ /dev/null@@ -1,28 +0,0 @@-# frozen_string_literal: true--RSpec.describe Migrations::Conversion::StepExecutor do- subject(:executor) { described_class.new(step, reporter:) }-- let(:handle) { instance_double(Migrations::Reporting::Reporter::StepHandle, finish: nil) }- let(:reporter) { instance_double(Migrations::Reporting::Plain, start_step: handle) }- let(:step_class) { Class.new(Migrations::Conversion::Step) { title "Fixture step" } }- let(:step) { step_class.new(settings: {}) }-- describe "#execute" do- it "announces the step through the reporter and executes it" do- allow(step).to receive(:execute)-- executor.execute-- expect(reporter).to have_received(:start_step).with("Fixture step")- expect(step).to have_received(:execute)- end-- it "finishes the step's handle even when the step fails" do- allow(step).to receive(:execute).and_raise("boom")-- expect { executor.execute }.to raise_error("boom")- expect(handle).to have_received(:finish)- end- end-enddiff --git a/migrations/core/spec/lib/conversion/step_runner_spec.rb b/migrations/core/spec/lib/conversion/step_runner_spec.rbnew file mode 100644index 00000000000..3faff1c2d6a--- /dev/null+++ b/migrations/core/spec/lib/conversion/step_runner_spec.rb@@ -0,0 +1,151 @@+# frozen_string_literal: true++require "extralite"++RSpec.describe Migrations::Conversion::StepRunner do+ around do |example|+ Dir.mktmpdir do |dir|+ @shard_path = File.join(dir, "shard.db")+ db = Extralite::Database.new(@shard_path)+ db.execute("CREATE TABLE notes (id INTEGER, body TEXT NOT NULL)")+ db.execute(+ "CREATE TABLE log_entries (created_at TEXT, type INTEGER, message TEXT, exception TEXT, details TEXT)",+ )+ db.close++ example.run+ ensure+ Migrations::Database::IntermediateDB.setup(nil)+ end+ end++ class RecordingReporter+ attr_reader :max_progress, :progress, :warnings, :errors++ def initialize+ @progress = @warnings = @errors = 0+ end++ def report_max_progress(value)+ @max_progress = value+ end++ def report_progress(progress:, warnings:, errors:)+ @progress += progress+ @warnings += warnings+ @errors += errors+ end+ end++ let(:step_class) do+ Class.new(Migrations::Conversion::Step) do+ source do+ def max_progress+ 3+ end++ def items+ [{ id: 1, body: "ok" }, { id: 2, body: nil }, { id: 3, body: "ok" }]+ end+ end++ processor do+ def process(item)+ Migrations::Database::IntermediateDB.insert(+ "INSERT INTO notes (id, body) VALUES (?, ?)",+ item[:id],+ item[:body],+ )+ end+ end+ end+ end++ let(:reporter) { RecordingReporter.new }++ def shard_rows(table)+ db = Extralite::Database.new(@shard_path)+ db.query_splat("SELECT id FROM #{table} ORDER BY id")+ ensure+ db&.close+ end++ def shard_count(table)+ db = Extralite::Database.new(@shard_path)+ db.query_single_splat("SELECT COUNT(*) FROM #{table}")+ ensure+ db&.close+ end++ it "reads the source, writes every good row to the shard, and reports progress" do+ described_class.new(step: step_class.new, shard_path: @shard_path, reporter:).run++ expect(shard_rows("notes")).to eq([1, 3])+ expect(shard_count("log_entries")).to eq(1)++ # the failed row still counts toward progress+ expect(reporter.max_progress).to eq(3)+ expect(reporter.progress).to eq(3)+ expect(reporter.errors).to eq(1)+ end++ it "closes the source when it's done" do+ step = step_class.new+ allow(step.source).to receive(:cleanup).and_call_original++ described_class.new(step:, shard_path: @shard_path, reporter:).run++ expect(step.source).to have_received(:cleanup)+ end++ context "when given a chunk" do+ let(:partitioned_step_class) do+ Class.new(Migrations::Conversion::Step) do+ source do+ def all+ (1..10).map { |id| { id:, body: "ok" } }+ end++ def items+ return all unless chunk++ lower, upper = chunk+ all.select { |it| it[:id] >= lower && (upper.nil? || it[:id] < upper) }+ end+ end++ processor do+ def process(item)+ Migrations::Database::IntermediateDB.insert(+ "INSERT INTO notes (id, body) VALUES (?, ?)",+ item[:id],+ item[:body],+ )+ end+ end+ end+ end++ it "reads only its own chunk of the source" do+ described_class.new(+ step: partitioned_step_class.new,+ shard_path: @shard_path,+ reporter:,+ chunk: [1, 6],+ ).run++ expect(shard_rows("notes")).to eq([1, 2, 3, 4, 5])+ end++ it "reads an open-ended last chunk" do+ described_class.new(+ step: partitioned_step_class.new,+ shard_path: @shard_path,+ reporter:,+ chunk: [6, nil],+ ).run++ expect(shard_rows("notes")).to eq([6, 7, 8, 9, 10])+ end+ end+enddiff --git a/migrations/core/spec/lib/conversion/step_scheduler_spec.rb b/migrations/core/spec/lib/conversion/step_scheduler_spec.rbnew file mode 100644index 00000000000..f0e6d77af6d--- /dev/null+++ b/migrations/core/spec/lib/conversion/step_scheduler_spec.rb@@ -0,0 +1,247 @@+# frozen_string_literal: true++RSpec.describe Migrations::Conversion::StepScheduler do+ let(:events) { Queue.new }++ # Fake StepCoordinator: announces its step, then blocks until the example+ # releases it, so we can drive ordering and concurrency without forking.+ class FakeCoordinator+ attr_reader :step_class++ def initialize(step_class, scheduler, events:, outcome: :done, raise_in_run: false)+ @step_class = step_class+ @scheduler = scheduler+ @events = events+ @outcome = outcome+ @raise_in_run = raise_in_run+ @finish = Queue.new+ end++ def run+ @events << @step_class.name.demodulize+ @finish.pop++ raise "unhandled error in #{@step_class.name.demodulize}" if @raise_in_run++ if @outcome == :failed+ @scheduler.record_failure(+ @step_class,+ RuntimeError.new("boom in #{@step_class.name.demodulize}"),+ )+ end++ @outcome+ end++ def finish!+ @finish << true+ end+ end++ # Builds named step classes in a throwaway namespace so `depends_on` (which+ # resolves names lexically) and the class-name tie-break have real classes to+ # work with. `stub_const` removes them after the example.+ def define_steps(*names, partitionable: [])+ namespace = Module.new+ names.each do |name|+ step_class =+ Class.new(Migrations::Conversion::Step) do+ if partitionable.include?(name)+ source do+ partition_by :id, from: "things"++ def items+ []+ end+ end+ end+ end+ namespace.const_set(name, step_class)+ end+ stub_const("TempSteps", namespace)+ names.map { |name| namespace.const_get(name) }+ end++ def build_scheduler(step_classes, budget: 4, max_parallel_steps: nil, config: {})+ reporter = instance_double(Migrations::Reporting::Reporter)+ allow(reporter).to receive(:finalizing).and_yield+ allow(reporter).to receive(:report_summary)++ scheduler =+ described_class.new(+ step_classes:,+ reporter:,+ step_factory: ->(_step_class) {},+ shard_manager: instance_double(Migrations::Conversion::ShardManager),+ writer: instance_double(Migrations::Database::Connection),+ budget:,+ max_parallel_steps:,+ )++ coordinators =+ step_classes.to_h do |step_class|+ options = config[step_class] || {}+ coordinator =+ FakeCoordinator.new(+ step_class,+ scheduler,+ events:,+ outcome: options.fetch(:outcome, :done),+ raise_in_run: options.fetch(:raise_in_run, false),+ )+ [step_class, coordinator]+ end++ allow(scheduler).to receive(:build_coordinator) { |step_class| coordinators.fetch(step_class) }++ by_name = coordinators.transform_keys { |step_class| step_class.name.demodulize }+ [scheduler, by_name]+ end++ def run_in_background(scheduler)+ Thread.new do+ Thread.current.report_on_exception = false+ scheduler.run+ end+ end++ it "starts a step only once its dependencies are done" do+ a, b = define_steps(:A, :B)+ b.depends_on(:a)+ scheduler, by_name = build_scheduler([a, b])++ runner = run_in_background(scheduler)++ expect(events.pop).to eq("A")+ expect(events).to be_empty+ by_name["A"].finish!++ expect(events.pop).to eq("B")+ by_name["B"].finish!++ runner.join+ end++ it "admits ready steps in a deterministic order on an empty graph" do+ steps = define_steps(:C, :A, :B)+ scheduler, by_name = build_scheduler(steps, budget: 1)++ runner = run_in_background(scheduler)++ %w[A B C].each do |name|+ expect(events.pop).to eq(name)+ by_name[name].finish!+ end++ runner.join+ end++ it "never runs more steps at once than the budget" do+ steps = define_steps(:A, :B, :C, :D)+ scheduler, by_name = build_scheduler(steps, budget: 2)++ runner = run_in_background(scheduler)++ expect([events.pop, events.pop].sort).to eq(%w[A B])+ expect(events).to be_empty++ by_name["A"].finish!+ expect(events.pop).to eq("C")+ expect(events).to be_empty++ by_name["B"].finish!+ expect(events.pop).to eq("D")++ by_name["C"].finish!+ by_name["D"].finish!+ runner.join+ end++ it "leaves a fork free for a single-fork step to overlap a partitioned step" do+ big, x, y = define_steps(:Big, :X, :Y, partitionable: %i[Big])+ scheduler, by_name = build_scheduler([big, x, y], budget: 4)++ runner = run_in_background(scheduler)++ # Big takes 3 of the 4 forks, so one single-fork step fills the last fork+ # and overlaps Big instead of queueing behind it.+ expect([events.pop, events.pop].sort).to eq(%w[Big X])+ expect(events).to be_empty+ by_name["X"].finish!++ expect(events.pop).to eq("Y")+ by_name["Big"].finish!+ by_name["Y"].finish!++ runner.join+ end++ it "caps the budget with --max-parallel-steps" do+ steps = define_steps(:A, :B, :C)+ scheduler, by_name = build_scheduler(steps, budget: 8, max_parallel_steps: 1)++ runner = run_in_background(scheduler)++ %w[A B C].each do |name|+ expect(events.pop).to eq(name)+ expect(events).to be_empty+ by_name[name].finish!+ end++ runner.join+ end++ it "skips a failed step's dependents, runs independent steps, and raises a summary" do+ a, b, c = define_steps(:A, :B, :C)+ b.depends_on(:a)++ scheduler, by_name = build_scheduler([a, b, c], config: { a => { outcome: :failed } })++ runner = run_in_background(scheduler)++ expect([events.pop, events.pop].sort).to eq(%w[A C])+ by_name["A"].finish!+ by_name["C"].finish!++ expect { runner.join }.to raise_error(Migrations::Conversion::ConvertError) do |error|+ expect(error.message).to match(/boom in A/)+ expect(error.message).to match(/skipped/)+ end+ expect(events).to be_empty+ end++ it "still finishes when a coordinator raises instead of reporting back" do+ a, b = define_steps(:A, :B)+ scheduler, by_name = build_scheduler([a, b], config: { a => { raise_in_run: true } })++ runner = run_in_background(scheduler)++ expect([events.pop, events.pop].sort).to eq(%w[A B])+ by_name["A"].finish!+ by_name["B"].finish!++ expect { runner.join }.to raise_error(+ Migrations::Conversion::ConvertError,+ /unhandled error in A/,+ )+ end++ it "fails the run when a shard merge errors, even though every step succeeded" do+ merge_error = StandardError.new("merge boom")+ allow(Migrations::Conversion::Consolidator).to receive(:new).and_return(+ instance_double(Migrations::Conversion::Consolidator, enqueue: nil, drain: [merge_error]),+ )++ a, b = define_steps(:A, :B)+ scheduler, by_name = build_scheduler([a, b])++ runner = run_in_background(scheduler)+ expect([events.pop, events.pop].sort).to eq(%w[A B])+ by_name["A"].finish!+ by_name["B"].finish!++ expect { runner.join }.to raise_error(Migrations::Conversion::ConvertError) do |error|+ expect(error.message).to include("merge boom")+ end+ end+enddiff --git a/migrations/core/spec/lib/conversion/step_spec.rb b/migrations/core/spec/lib/conversion/step_spec.rbindex f34f0d5397f..9d1340f4453 100644--- a/migrations/core/spec/lib/conversion/step_spec.rb+++ b/migrations/core/spec/lib/conversion/step_spec.rb@@ -1,76 +1,512 @@# frozen_string_literal: trueRSpec.describe Migrations::Conversion::Step do- before do- Object.const_set(- "TemporaryModule",- Module.new do- const_set("TopicUsers", Class.new(Migrations::Conversion::Step) {})- const_set("Users", Class.new(Migrations::Conversion::Step) {})- end,- )+ def define_step(&block)+ Class.new(described_class, &block)end- after do- TemporaryModule.send(:remove_const, "TopicUsers")- TemporaryModule.send(:remove_const, "Users")- Object.send(:remove_const, "TemporaryModule")+ describe ".source" do+ it "defines methods on the source role only" do+ step_class =+ define_step do+ source do+ def items+ [1, 2, 3]+ end+ end+ end++ source = step_class.source_class.new+ expect(source.items).to eq([1, 2, 3])++ expect(step_class.new).not_to respond_to(:items)+ expect(step_class.processor_class.new).not_to respond_to(:items)+ end++ it "defines private methods on the source role" do+ step_class =+ define_step do+ source do+ def items+ [build_item]+ end++ private++ def build_item+ :item+ end+ end+ end++ source = step_class.source_class.new+ expect(source.items).to eq([:item])+ expect(source.respond_to?(:build_item)).to be(false)+ expect(source.respond_to?(:build_item, true)).to be(true)+ end++ it "raises an error when called twice" do+ expect do+ define_step do+ source {}+ source {}+ end+ end.to raise_error(ArgumentError, /`source` is already defined/)+ endend- describe ".title" do- it "uses the classname within title" do- expect(TemporaryModule::TopicUsers.title).to eq("Converting topic users")- expect(TemporaryModule::Users.title).to eq("Converting users")+ describe ".processor" do+ it "defines methods on the processor role only" do+ step_class =+ define_step do+ processor do+ def process(item)+ item * 2+ end+ end+ end++ processor = step_class.processor_class.new+ expect(processor.process(21)).to eq(42)++ expect(step_class.new).not_to respond_to(:process)+ expect(step_class.source_class.new).not_to respond_to(:process)end- it "uses the `title` attribute if it has been set" do- TemporaryModule::Users.title "Foo bar"- expect(TemporaryModule::Users.title).to eq("Foo bar")+ it "defines private methods on the processor role" do+ step_class =+ define_step do+ processor do+ def process(item)+ transform(item)+ end++ private++ def transform(item)+ item.to_s+ end+ end+ end++ processor = step_class.processor_class.new+ expect(processor.process(1)).to eq("1")+ expect(processor.respond_to?(:transform)).to be(false)+ expect(processor.respond_to?(:transform, true)).to be(true)+ end++ it "raises an error when called twice" do+ expect do+ define_step do+ processor {}+ processor {}+ end+ end.to raise_error(ArgumentError, /`processor` is already defined/)+ end+ end++ describe ".helpers" do+ it "mixes the helpers into both roles" do+ step_class =+ define_step do+ helpers do+ def double_it(value)+ value * 2+ end+ end++ source do+ def items+ [double_it(1)]+ end+ end++ processor do+ def process(item)+ double_it(item)+ end+ end+ end++ expect(step_class.source_class.new.items).to eq([2])+ expect(step_class.processor_class.new.process(2)).to eq(4)+ end++ it "works independently of the order of the DSL calls" do+ step_class =+ define_step do+ source do+ def items+ [double_it(1)]+ end+ end++ helpers do+ def double_it(value)+ value * 2+ end+ end+ end++ expect(step_class.source_class.new.items).to eq([2])+ end++ it "raises an error when called twice" do+ expect do+ define_step do+ helpers {}+ helpers {}+ end+ end.to raise_error(ArgumentError, /`helpers` is already defined/)+ end+ end++ describe "role isolation" do+ it "raises a NameError when the processor calls a source method" do+ step_class =+ define_step do+ source do+ def items+ []+ end++ private++ def source_helper+ "source-side state"+ end+ end++ processor do+ def process(item)+ source_helper+ end+ end+ end++ processor = step_class.processor_class.new+ expect { processor.process(1) }.to raise_error(NameError, /source_helper/)+ end++ it "does not expose instance variables set in `items` to the processor" do+ step_class =+ define_step do+ source do+ def items+ @leaked_state = "set in items"+ [1]+ end+ end++ processor do+ def process(item)+ instance_variable_defined?(:@leaked_state)+ end+ end+ end++ step = step_class.new+ step.source.items++ expect(step.create_processor.process(1)).to be(false)+ end++ it "does not capture local variables in methods defined with `def`" do+ local_state = "step-level local"++ step_class =+ define_step do+ source do+ def items+ local_state+ end+ end+ end++ expect { step_class.source_class.new.items }.to raise_error(NameError, /local_state/)+ expect(local_state).to eq("step-level local") # silence the unused variable warning+ end+ end++ describe "role defaults" do+ let(:step_class) { define_step }++ it "uses `nil` as default `max_progress`" do+ expect(step_class.source_class.new.max_progress).to be_nil+ end++ it "raises `NotImplementedError` when `items` is not defined" do+ expect { step_class.source_class.new.items }.to raise_error(NotImplementedError)+ end++ it "raises `NotImplementedError` when `process` is not defined" do+ expect { step_class.processor_class.new.process(1) }.to raise_error(NotImplementedError)+ end++ it "uses a no-op as default `setup`" do+ expect { step_class.processor_class.new.setup }.not_to raise_error+ end++ it "uses a no-op `cleanup` when the source has no connection" do+ expect { step_class.source_class.new.cleanup }.not_to raise_error+ end+ end++ describe "reads_table defaults" do+ # Stands in for the source DB adapter, recording the calls it receives (the+ # adapter builds the actual SQL; that's covered in its own spec).+ let(:source_db) do+ Class+ .new do+ attr_reader :select_args, :count_args++ def chunk_filter(column, lower, upper, base:)+ "#{base} AND #{column} >= #{lower} AND #{column} < #{upper}"+ end++ def select_all(table, where:, order: nil)+ @select_args = [table, where, order]+ [:row]+ end++ def count_all(table, where:)+ @count_args = [table, where]+ 4+ end+ end+ .new+ end++ it "reads the whole table through the adapter" do+ step_class = define_step { source { reads_table "things", where: "active" } }+ source = step_class.source_class.new(source_db:)++ expect(source.items).to eq([:row])+ expect(source_db.select_args).to eq(["things", "active", nil]) # unordered when unpartitioned++ expect(source.max_progress).to eq(4)+ expect(source_db.count_args).to eq(%w[things active])+ end++ it "narrows to the chunk and orders by the key when also partitioned" do+ step_class =+ define_step do+ source do+ reads_table "things", where: "active"+ partition_by :id+ end+ end+ source = step_class.source_class.new(source_db:, chunk: [1, 5])++ source.items+ # ordered by the key so the worker writes its shard in key order+ expect(source_db.select_args).to eq(["things", "active AND id >= 1 AND id < 5", :id])+ end++ it "passes no filter when `reads_table` has no `where`" do+ step_class = define_step { source { reads_table "things" } }+ source = step_class.source_class.new(source_db:)++ source.items+ expect(source_db.select_args).to eq(["things", nil, nil])+ end++ it "orders an unpartitioned read by the declared `order`" do+ step_class = define_step { source { reads_table "things", where: "active", order: :id } }+ source = step_class.source_class.new(source_db:)++ source.items+ expect(source_db.select_args).to eq(["things", "active", :id])+ end+ end++ describe "#cleanup" do+ it "closes the per-step `source_db` connection" do+ step_class = define_step { source { attr_accessor :source_db } }+ source_db = instance_double(Migrations::Database::Connection, close: nil)+ source = step_class.source_class.new(source_db:)++ source.cleanup++ expect(source_db).to have_received(:close)+ end+ end++ describe "constant resolution" do+ # Real steps write their role blocks inside a step class body, so constants+ # in their methods resolve through the step's lexical scope and ancestry+ # (`Step` defines `IntermediateDB` and `Enums`). A `Class.new` block in this+ # spec file would have the wrong lexical scope, so we use a string eval.+ before { Object.class_eval <<~RUBY }+ class StepConstantsFixture < Migrations::Conversion::Step+ source do+ def items+ [IntermediateDB, Enums]+ end+ end++ processor do+ def process(item)+ [IntermediateDB, Enums]+ end+ end+ end+ RUBY++ after { Object.send(:remove_const, :StepConstantsFixture) }++ it "resolves `IntermediateDB` and `Enums` inside role blocks via the step's ancestry" do+ expected = [Migrations::Database::IntermediateDB, Migrations::Database::IntermediateDB::Enums]++ expect(StepConstantsFixture.source_class.new.items).to eq(expected)+ expect(StepConstantsFixture.processor_class.new.process(nil)).to eq(expected)+ end++ it "does not define the constants on the role classes themselves" do+ expect(StepConstantsFixture.source_class).not_to have_constant(:IntermediateDB)+ expect(StepConstantsFixture.processor_class).not_to have_constant(:IntermediateDB)endenddescribe "#initialize" do- it "works when no arguments are supplied" do- step = nil- expect { step = TemporaryModule::Users.new }.not_to raise_error- expect(step.settings).to be_nil+ it "routes args to the role that declares a matching setter" do+ step_class = define_step { source { attr_accessor :source_db } }++ settings = { a: 1 }+ step = step_class.new(settings:, source_db: "source db", unknown_arg: "dropped")++ expect(step.source.settings).to eq(settings)+ expect(step.source.source_db).to eq("source db")++ processor = step.create_processor+ expect(processor.settings).to eq(settings)+ expect(processor).not_to respond_to(:source_db)+ expect(processor).not_to respond_to(:unknown_arg)+ end++ it "keeps no per-step state on the coordinator" do+ step = define_step.new(settings: { a: 1 })++ expect(step).not_to respond_to(:tracker)+ expect(step).not_to respond_to(:step)+ expect(step).not_to respond_to(:settings)+ expect(step).not_to respond_to(:execute)+ end+ end++ describe "#source" do+ it "returns the same source instance every time" do+ step = define_step.new+ expect(step.source).to be(step.source)+ end+ end++ describe "#create_processor" do+ it "creates a new processor with its own tracker on every call" do+ step = define_step.new++ processor1 = step.create_processor+ processor2 = step.create_processor++ expect(processor1).not_to be(processor2)+ expect(processor1.tracker).to be_a(Migrations::Conversion::StepTracker)+ expect(processor1.tracker).not_to be(processor2.tracker)+ end+ end++ describe "step subclasses" do+ it "inherits the roles of the parent step" do+ parent_class =+ define_step do+ source do+ def items+ [1]+ end+ end++ processor do+ def process(item)+ item + 1+ end+ end+ end++ step_class =+ Class.new(parent_class) do+ source do+ def max_progress+ 1+ end+ end+ end++ source = step_class.source_class.new+ expect(source.items).to eq([1])+ expect(source.max_progress).to eq(1)+ expect(step_class.processor_class.new.process(1)).to eq(2)end+ end- it "creates its own tracker" do- step = TemporaryModule::Users.new- expect(step.tracker).to be_a(Migrations::Conversion::StepTracker)- expect(step.tracker).not_to be(TemporaryModule::Users.new.tracker)+ describe ".partitionable?" do+ it "defaults to false" do+ expect(define_step.partitionable?).to be(false)end- it "initializes the `settings` attribute if given" do- settings = { a: 1, b: 2 }- step = TemporaryModule::Users.new(settings:)- expect(step.settings).to eq(settings)+ it "is true when the source declares a partition key" do+ step =+ define_step do+ source do+ partition_by :id, from: "things", base: "active"++ def items+ []+ end+ end+ end++ expect(step.partitionable?).to be(true)end- it "initializes additional attributes if they exist" do- TemporaryModule::Users.class_eval { attr_accessor :foo, :bar }+ it "is true for a composite partition key" do+ step = define_step { source { partition_by %i[topic_id user_id], from: "topic_users" } }++ expect(step.partitionable?).to be(true)+ end+ end++ describe "dependency metadata" do+ before do+ Object.const_set(+ "TemporaryStepsModule",+ Module.new do+ const_set("Topics", Class.new(Migrations::Conversion::Step))+ const_set("TopicUsers", Class.new(Migrations::Conversion::Step))+ const_set("Users", Class.new(Migrations::Conversion::Step))+ end,+ )+ end++ after { Object.send(:remove_const, "TemporaryStepsModule") }++ it "exposes the StepDependencies macros at class level" do+ expect(described_class).to be_a(Migrations::StepDependencies)+ expect(described_class).to respond_to(:depends_on, :dependencies, :priority)+ end- settings = { a: 1, b: 2 }- foo = "a string"- bar = false+ it "lets steps depend on other steps" do+ TemporaryStepsModule::TopicUsers.depends_on(:topics, :users)- step = TemporaryModule::Users.new(settings:, foo:, bar:, non_existent: 123)- expect(step.settings).to eq(settings)- expect(step.foo).to eq(foo)- expect(step.bar).to eq(bar)- expect(step).to_not respond_to(:non_existent)+ expect(TemporaryStepsModule::TopicUsers.dependencies).to eq(+ [TemporaryStepsModule::Topics, TemporaryStepsModule::Users],+ )end- it "skips attributes with a private setter" do- TemporaryModule::Users.class_eval do- attr_writer :secret- private :secret=- end+ it "doesn't bleed metadata between sibling step classes" do+ TemporaryStepsModule::TopicUsers.depends_on(:users)+ TemporaryStepsModule::TopicUsers.priority 1- step = nil- expect { step = TemporaryModule::Users.new(secret: 123) }.not_to raise_error- expect(step.instance_variable_defined?(:@secret)).to be(false)+ expect(TemporaryStepsModule::Users.dependencies).to eq([])+ expect(TemporaryStepsModule::Users.priority).to be_nilendendenddiff --git a/migrations/core/spec/lib/conversion/worker_budget_spec.rb b/migrations/core/spec/lib/conversion/worker_budget_spec.rbnew file mode 100644index 00000000000..9f91be50ef0--- /dev/null+++ b/migrations/core/spec/lib/conversion/worker_budget_spec.rb@@ -0,0 +1,33 @@+# frozen_string_literal: true++RSpec.describe Migrations::Conversion::WorkerBudget do+ describe ".available" do+ it "keeps the reserved cores back from the usable CPUs" do+ allow(described_class).to receive(:usable_cpus).and_return(20)++ expect(described_class.available).to eq(19) # reserve 1 by default+ expect(described_class.available(reserved: 4)).to eq(16)+ end++ it "never drops below a single worker" do+ allow(described_class).to receive(:usable_cpus).and_return(1)+ expect(described_class.available).to eq(1)+ end+ end++ describe ".usable_cpus" do+ it "takes the tighter of the affinity- and quota-aware counts" do+ allow(Etc).to receive(:nprocessors).and_return(64) # host affinity+ allow(Concurrent).to receive(:available_processor_count).and_return(4.0) # CFS quota++ expect(described_class.usable_cpus).to eq(4)+ end++ it "floors a fractional quota and never reports fewer than one" do+ allow(Etc).to receive(:nprocessors).and_return(8)+ allow(Concurrent).to receive(:available_processor_count).and_return(0.5)++ expect(described_class.usable_cpus).to eq(1)+ end+ end+enddiff --git a/migrations/core/spec/lib/conversion/worker_pool_spec.rb b/migrations/core/spec/lib/conversion/worker_pool_spec.rbdeleted file mode 100644index fb9f4afc3c3..00000000000--- a/migrations/core/spec/lib/conversion/worker_pool_spec.rb+++ /dev/null@@ -1,124 +0,0 @@-# frozen_string_literal: true--RSpec.describe Migrations::Conversion::WorkerPool do- subject(:pool) { described_class.new(size: pool_size) }-- let(:pool_size) { 2 }- let(:work_queue) { Queue.new }- let(:output_queue) { Queue.new }-- after do- work_queue.close if !work_queue.closed?- output_queue.close if !output_queue.closed?- end-- def create_echo_job- job = instance_double(Migrations::Conversion::ParallelJob, setup: nil, cleanup: nil)- allow(job).to receive(:run) { |item| { echoed: item[:id] } }- job- end-- def drained_output- results = []- results << output_queue.pop until output_queue.empty?- results- end-- describe "#start" do- it "forks one worker per pool slot by default and processes all queued items" do- allow(Migrations::ForkManager).to receive(:fork).and_call_original-- batch = pool.start(work_queue:, output_queue:) { create_echo_job }- 5.times { |id| work_queue << { id: } }- work_queue.close- batch.wait- output_queue.close-- expect(Migrations::ForkManager).to have_received(:fork).exactly(pool_size).times- expect(drained_output).to match_array(Array.new(5) { |id| { echoed: id } })- end-- def start_empty_batch(**kwargs)- batch = pool.start(work_queue:, output_queue:, **kwargs) { create_echo_job }- work_queue.close- batch.wait- output_queue.close- end-- it "forks a single worker when started with `size: 1`" do- allow(Migrations::ForkManager).to receive(:fork).and_call_original-- start_empty_batch(size: 1)-- expect(Migrations::ForkManager).to have_received(:fork).once- end-- it "clamps the requested size to the pool size" do- allow(Migrations::ForkManager).to receive(:fork).and_call_original-- start_empty_batch(size: pool_size + 10)-- expect(Migrations::ForkManager).to have_received(:fork).exactly(pool_size).times- end-- it "clamps the requested size to a minimum of one worker" do- allow(Migrations::ForkManager).to receive(:fork).and_call_original-- start_empty_batch(size: 0)-- expect(Migrations::ForkManager).to have_received(:fork).once- end-- it "runs the fork hooks once per `start` call, not once per worker" do- before_fork_count = 0- after_fork_count = 0- before_fork_hook = Migrations::ForkManager.before_fork { before_fork_count += 1 }- after_fork_hook = Migrations::ForkManager.after_fork_parent { after_fork_count += 1 }-- start_empty_batch-- expect(before_fork_count).to eq(1)- expect(after_fork_count).to eq(1)- ensure- Migrations::ForkManager.remove_before_fork(before_fork_hook)- Migrations::ForkManager.remove_after_fork_parent(after_fork_hook)- end-- it "builds a fresh job for every worker" do- jobs = []-- batch =- pool.start(work_queue:, output_queue:) do- job = create_echo_job- jobs << job- job- end- work_queue.close- batch.wait- output_queue.close-- expect(jobs.size).to eq(pool_size)- expect(jobs.uniq.size).to eq(pool_size)- end- end-- describe "Batch#wait" do- it "has run every worker's `cleanup` by the time `wait` returns" do- Dir.mktmpdir do |dir|- batch =- pool.start(work_queue:, output_queue:) do- job = instance_double(Migrations::Conversion::ParallelJob, setup: nil)- allow(job).to receive(:cleanup) do- File.write(File.join(dir, "worker_#{Process.pid}"), "exited")- end- job- end- work_queue.close- batch.wait- output_queue.close-- expect(Dir.children(dir).size).to eq(pool_size)- end- end- end-enddiff --git a/migrations/core/spec/lib/conversion/worker_spec.rb b/migrations/core/spec/lib/conversion/worker_spec.rbdeleted file mode 100644index 74d35d142ab..00000000000--- a/migrations/core/spec/lib/conversion/worker_spec.rb+++ /dev/null@@ -1,302 +0,0 @@-# frozen_string_literal: true--RSpec.describe Migrations::Conversion::Worker do- subject(:worker) { described_class.new(index, input_queue, output_queue, job) }-- let(:index) { 1 }- let(:input_queue) { Queue.new }- let(:output_queue) { Queue.new }- let(:job) do- instance_double(Migrations::Conversion::ParallelJob, setup: nil, run: "result", cleanup: nil)- end-- after do- input_queue.close if !input_queue.closed?- output_queue.close if !output_queue.closed?- end-- describe "#start" do- it "works when `input_queue` is empty" do- expect do- worker.start- input_queue.close- worker.wait- output_queue.close- end.not_to raise_error- end-- it "uses `ForkManager.fork`" do- allow(Migrations::ForkManager).to receive(:fork).and_call_original-- worker.start- input_queue.close- worker.wait- output_queue.close-- expect(Migrations::ForkManager).to have_received(:fork)- end-- it "writes the output of `job.run` into `output_queue`" do- allow(job).to receive(:run) { |data| "run: #{data[:text]}" }-- worker.start- input_queue << { text: "Item 1" } << { text: "Item 2" } << { text: "Item 3" }- input_queue.close- worker.wait- output_queue.close-- expect(output_queue).to have_queue_contents("run: Item 1", "run: Item 2", "run: Item 3")- end-- def create_progress_stats(progress: 1, warning_count: 0, error_count: 0)- Migrations::Conversion::StepStats.new(progress:, warning_count:, error_count:)- end-- it "writes objects to the `output_queue`" do- all_stats = [- create_progress_stats,- create_progress_stats(warning_count: 1),- create_progress_stats(warning_count: 1, error_count: 1),- create_progress_stats(warning_count: 2, error_count: 1),- ]-- allow(job).to receive(:run) do |data|- index = data[:index]- [index, all_stats[index]]- end-- worker.start- input_queue << { index: 0 } << { index: 1 } << { index: 2 } << { index: 3 }- input_queue.close- worker.wait- output_queue.close-- expect(output_queue).to have_queue_contents(- [0, all_stats[0]],- [1, all_stats[1]],- [2, all_stats[2]],- [3, all_stats[3]],- )- end-- # Characterization specs: they pin the wire contract of the IPC pipeline- # (currently Oj in `:object` mode) with payload shapes that real steps- # produce. Items are echoed back by the job, so each spec exercises both- # pipe directions with the same payload.- describe "wire format" do- def round_trip(*items)- allow(job).to receive(:run) { |data| data }-- worker.start- items.each { |item| input_queue << item }- input_queue.close- worker.wait- output_queue.close-- results = []- results << output_queue.pop until output_queue.empty?- results- end-- it "round-trips a site-settings-shaped item with nested hashes and a Time value" do- item = {- name: "logo_url",- value: "/uploads/default/original/1X/logo.png",- data_type: 18,- updated_at: Time.utc(2023, 5, 17, 12, 34, 56, 789_123),- uploads: [- { id: 1, url: "/uploads/default/original/1X/logo.png", filename: "logo.png" },- { id: 2, url: "/uploads/default/original/1X/logo2.png", filename: "logo2.png" },- ],- }-- result = round_trip(item).first-- expect(result).to eq(item)- expect(result.keys).to all(be_a(Symbol))- expect(result[:uploads].flat_map(&:keys)).to all(be_a(Symbol))- expect(result[:updated_at]).to be_a(Time)- expect(result[:updated_at]).to eq(item[:updated_at])- end-- it "round-trips a wide users-shaped item with full fidelity" do- item = {- id: 42,- username: "alice",- name: "Alice Άλφα 𝒜",- active: true,- admin: false,- moderator: false,- staged: false,- approved: true,- approved_at: Time.utc(2020, 1, 2, 3, 4, 5),- approved_by_id: 1,- created_at: Time.utc(2019, 12, 31, 23, 59, 59, 999_999),- first_seen_at: Time.utc(2020, 1, 2, 3, 4, 6),- last_seen_at: Time.utc(2024, 11, 5, 6, 7, 8),- silenced_till: nil,- suspended_at: nil,- suspended_till: nil,- date_of_birth: Date.new(1990, 4, 1),- ip_address: IPAddr.new("192.168.0.1"),- registration_ip_address: IPAddr.new("2001:db8::1"),- locale: nil,- title: nil,- trust_level: 2,- group_locked_trust_level: nil,- manual_locked_trust_level: nil,- primary_group_id: nil,- flair_group_id: nil,- uploaded_avatar_id: 99,- views: 1234,- avatar_url: "/uploads/default/original/1X/avatar.png",- avatar_filename: "avatar.png",- avatar_origin: nil,- avatar_user_id: 42,- }-- result = round_trip(item).first-- expect(result).to eq(item)- expect(result[:created_at]).to be_a(Time)- expect(result[:date_of_birth]).to be_a(Date)- expect(result[:ip_address]).to be_a(IPAddr)- end-- it "round-trips string edge cases" do- item = {- emoji: "héllo wörld 👋🏼 🇦🇹 文字",- newlines: "line1\nline2\r\nline3",- quotes_and_backslashes: %q(she said "hi\there" \\ and left),- control_chars: "tab\there\u0000null",- large: "Ω" * 1_000_000,- }-- result = round_trip(item).first-- expect(result).to eq(item)- expect(result[:large].bytesize).to be >= 1_000_000- end-- it "round-trips a realistic statements batch with stats on the return path" do- statements = [- [- "INSERT INTO users (original_id, username, created_at, active) VALUES (?, ?, ?, ?)",- [42, "alice", "2019-12-31T23:59:59Z", 1],- ],- [- "INSERT INTO user_emails (user_id, email, \"primary\", created_at) VALUES (?, ?, ?, ?)",- [42, "[email protected]", 1, "2019-12-31T23:59:59Z"],- ],- [- "INSERT INTO user_options (user_id, timezone, dark_scheme_id) VALUES (?, ?, ?)",- [42, nil, nil],- ],- ]- stats = create_progress_stats(progress: 1, warning_count: 2, error_count: 3)-- allow(job).to receive(:run).and_return([statements, stats])-- worker.start- input_queue << { id: 42 }- input_queue.close- worker.wait- output_queue.close-- result_statements, result_stats = output_queue.pop- expect(result_statements).to eq(statements)- expect(result_stats).to eq(stats)- end-- it "preserves FIFO ordering under volume" do- item_count = 2000-- results = round_trip(*(0...item_count).map { |i| { index: i, payload: "item #{i}" } })-- expect(results.map { |item| item[:index] }).to eq((0...item_count).to_a)- end- end-- it "runs `job.setup` once in the worker process before processing items" do- temp_file = Tempfile.new("setup_check")- temp_file_path = temp_file.path- parent_pid = Process.pid-- allow(job).to receive(:setup) do- process = Process.pid == parent_pid ? "parent" : "worker"- File.write(temp_file_path, "setup in #{process}\n", mode: "a+")- end- allow(job).to receive(:run) do |data|- File.write(temp_file_path, "run: #{data[:text]}\n", mode: "a+")- data[:text]- end-- worker.start- input_queue << { text: "Item 1" } << { text: "Item 2" }- input_queue.close- worker.wait- output_queue.close-- expect(File.read(temp_file_path)).to eq <<~LOG- setup in worker- run: Item 1- run: Item 2- LOG- ensure- temp_file.unlink- end-- it "raises an error in the parent process when `job.setup` fails in the worker process" do- allow(job).to receive(:setup) do- $stderr.reopen(File::NULL) # silence the worker's backtrace in the test output- raise Migrations::Conversion::SetupGuard::SetupError, "setup failed"- end-- worker.start- input_queue << { text: "Item 1" }- input_queue.close-- expect { worker.wait }.to raise_error(- described_class::CrashedError,- /Worker process #{index} exited unexpectedly/,- )- end-- it "raises an error when the worker process dies even though no pipe write failed" do- allow(job).to receive(:setup) { exit!(1) }-- worker.start- input_queue.close-- expect { worker.wait }.to raise_error(described_class::CrashedError)- end-- it "runs `job.cleanup` at the end" do- temp_file = Tempfile.new("method_call_check")- temp_file_path = temp_file.path-- allow(job).to receive(:run) do |data|- File.write(temp_file_path, "run: #{data[:text]}\n", mode: "a+")- data[:text]- end- allow(job).to receive(:cleanup) do- File.write(temp_file_path, "cleanup\n", mode: "a+")- end-- worker.start- input_queue << { text: "Item 1" } << { text: "Item 2" } << { text: "Item 3" }- input_queue.close- worker.wait- output_queue.close-- expect(File.read(temp_file_path)).to eq <<~LOG- run: Item 1- run: Item 2- run: Item 3- cleanup- LOG- ensure- temp_file.unlink- end- end-enddiff --git a/migrations/core/spec/lib/database/intermediate_db_spec.rb b/migrations/core/spec/lib/database/intermediate_db_spec.rbindex 87f2c01bbef..8ac93a4efc3 100644--- a/migrations/core/spec/lib/database/intermediate_db_spec.rb+++ b/migrations/core/spec/lib/database/intermediate_db_spec.rb@@ -29,23 +29,6 @@ RSpec.describe Migrations::Database::IntermediateDB doendend- it "works with `Migrations::Database::OfflineConnection`" do- connection = Migrations::Database::OfflineConnection.new-- described_class.setup(connection)- described_class.insert("INSERT INTO foo (id, name) VALUES (?, ?)", 1, "Alice")- described_class.insert("INSERT INTO foo (id, name) VALUES (?, ?)", 2, "Bob")-- expect(connection.parametrized_insert_statements).to eq(- [- ["INSERT INTO foo (id, name) VALUES (?, ?)", [1, "Alice"]],- ["INSERT INTO foo (id, name) VALUES (?, ?)", [2, "Bob"]],- ],- )-- connection.close- end-it "switches the connection" doold_connection = create_connection_doublenew_connection = create_connection_doublediff --git a/migrations/core/spec/lib/database/migrator_spec.rb b/migrations/core/spec/lib/database/migrator_spec.rbindex b0a1f675769..55cebc4fe91 100644--- a/migrations/core/spec/lib/database/migrator_spec.rb+++ b/migrations/core/spec/lib/database/migrator_spec.rb@@ -75,22 +75,4 @@ RSpec.describe Migrations::Database::Migrator doendendend-- describe "#reset!" do- it "deletes all DB related files" do- migrate(migrations_directory: "invalid", ignore_errors: true) do |db_path, storage_path|- File.write(File.join(storage_path, "hello_world.txt"), "Hello World!")-- expect(Dir.children(storage_path)).to contain_exactly(- "intermediate.db",- "intermediate.db-shm",- "intermediate.db-wal",- "hello_world.txt",- )-- described_class.new(db_path).reset!- expect(Dir.children(storage_path)).to contain_exactly("hello_world.txt")- end- end- endenddiff --git a/migrations/core/spec/lib/database/offline_connection_spec.rb b/migrations/core/spec/lib/database/offline_connection_spec.rbdeleted file mode 100644index d36b5ed89b0..00000000000--- a/migrations/core/spec/lib/database/offline_connection_spec.rb+++ /dev/null@@ -1,67 +0,0 @@-# frozen_string_literal: true--RSpec.describe Migrations::Database::OfflineConnection do- subject(:connection) { described_class.new }-- let!(:sql) { "INSERT INTO foo (id, name) VALUES (?, ?)" }-- it_behaves_like "a database connection"-- describe "#close" do- it "removes the cached statements" do- connection.insert(sql, [1, "Alice"])- connection.insert(sql, [2, "Bob"])-- expect(connection.parametrized_insert_statements).to_not be_empty-- connection.close- expect(connection.parametrized_insert_statements).to be_nil- end- end-- describe "#closed?" do- it "correctly reports if connection is closed" do- expect(connection.closed?).to be false- connection.close- expect(connection.closed?).to be true- end- end-- describe "#insert" do- it "can be called without errors" do- expect { connection.insert(sql, [1, "Alice"]) }.not_to raise_error- end- end-- describe "#parametrized_insert_statements" do- it "returns an empty array if nothing has been cached" do- expect(connection.parametrized_insert_statements).to eq([])- end-- it "returns the cached INSERT statements and parameters in original order" do- connection.insert(sql, [1, "Alice"])- connection.insert(sql, [2, "Bob"])- connection.insert(sql, [3, "Carol"])-- expected_data = [[sql, [1, "Alice"]], [sql, [2, "Bob"]], [sql, [3, "Carol"]]]- expect(connection.parametrized_insert_statements).to eq(expected_data)-- # multiple calls return the same data- expect(connection.parametrized_insert_statements).to eq(expected_data)- expect(connection.parametrized_insert_statements).to eq(expected_data)- end- end-- describe "#clear!" do- it "clears all cached data" do- connection.insert(sql, [1, "Alice"])- connection.insert(sql, [2, "Bob"])- connection.insert(sql, [3, "Carol"])-- expect(connection.parametrized_insert_statements).to_not be_empty-- connection.clear!- expect(connection.parametrized_insert_statements).to eq([])- end- end-enddiff --git a/migrations/core/spec/lib/database_spec.rb b/migrations/core/spec/lib/database_spec.rbindex 08f4b48173e..7e78cf3e13e 100644--- a/migrations/core/spec/lib/database_spec.rb+++ b/migrations/core/spec/lib/database_spec.rb@@ -26,15 +26,27 @@ RSpec.describe Migrations::Database doexpect(migrator_instance).to have_received(:migrate).with(migrations_path)endend+ end- describe ".reset!" do- it "resets the database" do- allow(migrator_instance).to receive(:reset!)+ describe ".database_files" do+ it "lists the database file and its WAL/SHM sidecars" do+ expect(described_class.database_files("path/to/db")).to eq(+ %w[path/to/db path/to/db-wal path/to/db-shm],+ )+ end+ end- described_class.reset!(db_path)+ describe ".delete_database" do+ it "removes the database and its sidecars, leaves other files, ignores missing ones" do+ Dir.mktmpdir do |storage_path|+ db_path = File.join(storage_path, "test.db")+ FileUtils.touch(db_path)+ FileUtils.touch("#{db_path}-wal")+ # no -shm file: deletion must not raise on the missing one+ FileUtils.touch(File.join(storage_path, "keep.txt"))- expect(Migrations::Database::Migrator).to have_received(:new).with(db_path)- expect(migrator_instance).to have_received(:reset!)+ expect { described_class.delete_database(db_path) }.not_to raise_error+ expect(Dir.children(storage_path)).to contain_exactly("keep.txt")endendend@@ -161,7 +173,7 @@ RSpec.describe Migrations::Database donumber: 123,date: DateTime.new(2023, 10, 5, 17, 30, 0),),- ).to eq(%q|{"text":"foo","number":123,"date":"2023-10-05T17:30:00.000+00:00"}|)+ ).to eq(%q|{"text":"foo","number":123,"date":"2023-10-05T17:30:00+00:00"}|)endit "returns nil for nil input" dodiff --git a/migrations/importer/lib/migrations/importer/executor.rb b/migrations/importer/lib/migrations/importer/executor.rbindex ee9c6333f6e..b8eafec71a1 100644--- a/migrations/importer/lib/migrations/importer/executor.rb+++ b/migrations/importer/lib/migrations/importer/executor.rb@@ -36,7 +36,7 @@ module Migrationsprivatedef attach_mappings_db(db_path, reset)- Database.reset!(db_path) if reset+ Database.delete_database(db_path) if resetmigrate_and_attach(db_path, Database::MAPPINGS_DB_SCHEMA_PATH, "mapped")enddiff --git a/migrations/importer/lib/migrations/importer/uploads/tasks/base.rb b/migrations/importer/lib/migrations/importer/uploads/tasks/base.rbindex bd152da436c..2d4676272f2 100644--- a/migrations/importer/lib/migrations/importer/uploads/tasks/base.rb+++ b/migrations/importer/lib/migrations/importer/uploads/tasks/base.rb@@ -8,9 +8,6 @@ module Migrationsmodule Uploadsmodule Tasksclass Base- class NotImplementedError < StandardError- end-TRANSACTION_SIZE = 1000QUEUE_SIZE = 1000DEFAULT_THREAD_FACTOR = 1.5diff --git a/migrations/tooling/config/schema/intermediate_db/ignored.rb b/migrations/tooling/config/schema/intermediate_db/ignored.rbindex f8ae0657803..f316a4d3dde 100644--- a/migrations/tooling/config/schema/intermediate_db/ignored.rb+++ b/migrations/tooling/config/schema/intermediate_db/ignored.rb@@ -168,6 +168,7 @@ Migrations::Tooling::Schema.ignored do:bookmarks,:browser_pageview_event_scores,:browser_pageview_events,+ :browser_pageview_session_engagement_daily_rollups,:browser_pageview_session_engagements,:custom_emojis,:developers,diff --git a/migrations/tooling/config/schema/plugin_manifest.yml b/migrations/tooling/config/schema/plugin_manifest.ymlindex 95faf61c1d8..546f77285c1 100644--- a/migrations/tooling/config/schema/plugin_manifest.yml+++ b/migrations/tooling/config/schema/plugin_manifest.yml@@ -167,6 +167,7 @@ plugins:- discourse_rewind_share_publiclydiscourse-rss-polling:tables:+ - discourse_rss_polling_poll_attempts- discourse_rss_polling_rss_feedscolumns: {}discourse-solved:@@ -218,10 +219,10 @@ plugins:columns: {}plugin_checksums:automation: e69972690991bc04d92b2d161aac8921- chat: 93de6fd994b8fbd1a3921c3deacd1d7a+ chat: 43ee047298d56af4fad013d534155812discourse-adplugin: 9e52419decf1099c5bc301a90e6cc91adiscourse-affiliate: '08c6e1a851441a082e9be9ddddc63910'- discourse-ai: 3063169bf8303c6c3f944391e3e42142+ discourse-ai: 166cf3770ac2b5524b22a0e33cbfa7eddiscourse-assign: 7a7a36e8ac0214e8a6aca1a1ed9899aadiscourse-cakeday: a5bb91366c8dc8d267088eb0c2679187discourse-calendar: 07aa9951f5fde7daf03be9edfbda4dec@@ -241,7 +242,7 @@ plugin_checksums:discourse-post-voting: ece13b87f2fbac3e64f8aa3abe07fe3ddiscourse-reactions: e9de82d2ebb48eb434be2b646c7fec26discourse-rewind: 1e0f2acf6e8409f2d2f011e14405fa06- discourse-rss-polling: b7cb3eeb38beff3ce10b333f24128d2f+ discourse-rss-polling: 526db84c3d2ade0fa03497031fea2333discourse-solved: 67743e14f4c5c2fae4487193c374e429discourse-subscriptions: 05eb12877a6a842a290a98fa0e70d462discourse-templates: a131c89a3e2b765f1eb6ec811c0feceadiff --git a/migrations/tooling/scripts/compare_intermediate_dbs.rb b/migrations/tooling/scripts/compare_intermediate_dbs.rbnew file mode 100644index 00000000000..3a3a7e7e67e--- /dev/null+++ b/migrations/tooling/scripts/compare_intermediate_dbs.rb@@ -0,0 +1,69 @@+# frozen_string_literal: true++# Compares two IntermediateDB files for order-insensitive equality. The+# concurrent converter writes rows in no fixed order, so we sort each table's+# rows and compare a digest instead of diffing the files.+#+# Usage (from the repo root, under the migrations bundle):+#+# cd migrations/core+# bundle exec ruby ../tooling/scripts/compare_intermediate_dbs.rb \+# /path/to/sequential.db /path/to/concurrent.db++require "extralite"+require "digest"++# `schema_migrations.applied_at` is a wall-clock time and always differs.+IGNORED_TABLES = %w[schema_migrations].freeze++def tables(db)+ (db.tables - IGNORED_TABLES).sort+end++def quote(identifier)+ %("#{identifier.gsub('"', '""')}")+end++def table_digest(db, table)+ rows = []+ count = 0+ db.query_array("SELECT * FROM #{quote(table)}") do |row|+ rows << row.map { |value| value.is_a?(String) ? value.b : value.inspect }.join("")+ count += 1+ end+ rows.sort!+ [count, Digest::SHA256.hexdigest(rows.join(""))]+end++seq_path, conc_path = ARGV+abort "usage: compare_intermediate_dbs.rb SEQUENTIAL_DB CONCURRENT_DB" unless seq_path && conc_path++seq = Extralite::Database.new(seq_path)+conc = Extralite::Database.new(conc_path)++seq_tables = tables(seq)+conc_tables = tables(conc)++ok = true++if seq_tables != conc_tables+ ok = false+ warn "Table sets differ:"+ warn " only in #{seq_path}: #{(seq_tables - conc_tables).join(", ")}"+ warn " only in #{conc_path}: #{(conc_tables - seq_tables).join(", ")}"+end++(seq_tables & conc_tables).each do |table|+ seq_count, seq_digest = table_digest(seq, table)+ conc_count, conc_digest = table_digest(conc, table)++ if seq_digest == conc_digest+ puts " ok #{table.ljust(32)} #{seq_count} rows"+ else+ ok = false+ puts " DIFF #{table.ljust(32)} sequential=#{seq_count} rows, concurrent=#{conc_count} rows"+ end+end++puts(ok ? "\nIdentical (order-insensitive)." : "\nDifferences found.")+exit(ok ? 0 : 1)