Open Source Platform for DTC, B2B Commerce, Marketplaces & Omnichannel. REST APIs, TypeScript SDKs, and production-ready Next.js storefront. Self-host it. Own your data. No vendor lock-in. Zero platform fees.
Ruby b2b-commercee-commerceecommerceecommerce-apiecommerce-frameworkecommerce-platformheadlessheadless-commerceheadless-ecommercemarketplacemulti-tenantmulti-vendormulti-vendor-ecommerceopen-sourcespree-commerce
Phase P3.1: Type the stock history and reconcile open fulfillments
Łukasz Adamczyk187aa1b69bd8
diff --git a/spree/core/lib/spree/upgrades/5_6_to_6_0/manifest.yml b/spree/core/lib/spree/upgrades/5_6_to_6_0/manifest.ymlindex 39774b7d45..f41351ca06 100644--- a/spree/core/lib/spree/upgrades/5_6_to_6_0/manifest.yml+++ b/spree/core/lib/spree/upgrades/5_6_to_6_0/manifest.yml@@ -270,6 +270,43 @@ steps:to retry those rows — the task only looks at what isn't migrated yet — orpass SKIP_FAILED_ROWS=true to accept the gap and continue.+ - id: migrate_stock_movements_to_typed_rows+ name: "Type stock movements and reconcile open fulfillments"+ task: "spree:migrate_stock_movements_to_typed_rows"+ notes: |+ Gives every pre-6.0 stock movement a kind and a concrete cause key+ derived from its polymorphic originator: a fulfillment originator+ becomes shipped or released by sign, a stock transfer becomes received+ or shipped, a legacy return authorization becomes received against the+ Return or Exchange it turned into, and a movement with no originator+ becomes an adjustment reading "Legacy manual adjustment". A cause that+ cannot be resolved is still typed by sign and simply carries no cause+ key — a movement with no cause is still a true statement about stock.++ Then reconciles the representation change. Stock leaves the shelf at+ dispatch in 6.0, not at placement, so the units of every open+ fulfillment on a placed order go back onto count_on_hand and are+ allocated at the same time, with an allocated movement recording the+ jump. Availability is unchanged by construction, since both sides of+ "on hand minus allocated" move together, and backordered units need no+ special case — the pair of increments cancels out the negative on-hand+ that used to represent them.++ Run AFTER migrate_shipping_to_delivery, which rewrites the+ 'Spree::Shipment' originator strings, and AFTER migrate_returns,+ because a legacy authorization id says nothing about the new record:+ the returns migrator preserves the number, not the id, and one+ authorization can become either a Return or an Exchange.++ Idempotent. Typing narrows to rows without a kind and reconciliation+ skips fulfillments that already hold an allocation, so an interrupted+ run resumes for free. That same predicate makes the deploy window safe:+ a fulfillment shipped before the task runs never gets an allocation,+ and dispatch only ships allocated units, so its stock is never+ decremented twice. originator_type and originator_id stay in place as+ the rollback path and are dropped in 6.1. Tune the load size with+ BATCH_SIZE (default 5000).+- id: backfill_delivery_and_stock_store_idsname: "Bind delivery methods and stock locations to the default store"task: "spree:backfill_delivery_and_stock_store_ids"diff --git a/spree/core/lib/tasks/typed_stock_movements_migration.rake b/spree/core/lib/tasks/typed_stock_movements_migration.rakenew file mode 100644index 0000000000..71f4974f07--- /dev/null+++ b/spree/core/lib/tasks/typed_stock_movements_migration.rake@@ -0,0 +1,194 @@+# frozen_string_literal: true++# Defined here rather than in lib/spree/core/ — this is a one-release upgrade+# step that dies with the polymorphic originator in 6.1, not engine+# infrastructure. Same placement as Spree::ReturnsMigrator in a sibling task.+module Spree+ # Types the legacy stock-movement history and reconciles open fulfillments+ # onto the allocation model (docs/plans/6.0-typed-stock-movements.md).+ #+ # Two jobs in one idempotent task. Typing narrows to rows without a kind;+ # reconciliation skips fulfillments that already hold an allocation. Both+ # predicates are queries over the data itself, so an interrupted run resumes+ # for free and a second run finds nothing to do. That same predicate is what+ # makes the deploy window safe: a fulfillment shipped before the task runs+ # never gets an allocation, and dispatch only ships allocated units.+ #+ # Runs after spree:migrate_shipping_to_delivery, which rewrites the+ # 'Spree::Shipment' originator strings, and after+ # spree:upgrade:migrate_returns, because a legacy authorization id says+ # nothing about the new record — the returns migrator preserves the number,+ # not the id, and one authorization can become either a Return or an+ # Exchange.+ class TypedStockMovementsMigration+ LEGACY_ADJUSTMENT_REASON = 'Legacy manual adjustment'+ LEGACY_RETURN_TABLE = 'spree_return_authorizations'+ FULFILLMENT_ORIGINATOR_TYPES = ['Spree::Fulfillment', 'Spree::Shipment'].freeze++ def initialize(batch_size: 5_000)+ @batch_size = batch_size+ @typed = 0+ @reconciled = 0+ end++ # @return [Hash] counts of rows typed and fulfillments reconciled+ def call+ type_history+ reconcile_open_fulfillments++ { typed: @typed, reconciled: @reconciled }+ end++ private++ attr_reader :batch_size++ def type_history+ Spree::StockMovement.unscoped.where(kind: nil).find_each(batch_size: batch_size) do |movement|+ attributes = attributes_for(movement)+ Spree::StockMovement.unscoped.where(id: movement.id).update_all(attributes.merge(updated_at: Time.current))+ @typed += 1+ end+ end++ def attributes_for(movement)+ return { kind: 'adjusted', reason: LEGACY_ADJUSTMENT_REASON } if movement.originator_type.blank?++ case movement.originator_type+ when *FULFILLMENT_ORIGINATOR_TYPES then fulfillment_attributes(movement)+ when 'Spree::StockTransfer' then stock_transfer_attributes(movement)+ when 'Spree::ReturnAuthorization' then return_attributes(movement)+ else by_sign(movement)+ end+ end++ # A departure, or the restock of a cancellation or relocation, carrying+ # the order the fulfillment belonged to so "which order was this for?"+ # costs no join.+ def fulfillment_attributes(movement)+ fulfillment = Spree::Fulfillment.find_by(id: movement.originator_id)+ return by_sign(movement) if fulfillment.nil?++ by_sign(movement, negative: 'shipped', positive: 'released').+ merge(fulfillment_id: fulfillment.id, order_id: fulfillment.order_id)+ end++ def stock_transfer_attributes(movement)+ return by_sign(movement) unless Spree::StockTransfer.exists?(id: movement.originator_id)++ by_sign(movement).merge(stock_transfer_id: movement.originator_id)+ end++ # The legacy id is worthless on its own, so the authorization is read by+ # name and its preserved number matched against whichever record the+ # returns migrator produced.+ def return_attributes(movement)+ attributes = { kind: 'received', quantity: movement.quantity.to_i.abs }+ record = migrated_return_for(movement.originator_id)++ case record+ when Spree::Return then attributes.merge(return_id: record.id, order_id: record.order_id)+ when Spree::Exchange then attributes.merge(exchange_id: record.id, order_id: record.order_id)+ else attributes+ end+ end++ def migrated_return_for(authorization_id)+ return nil unless legacy_returns++ number = legacy_returns.where(id: authorization_id).pick(:number)+ return nil if number.blank?++ Spree::Return.find_by(number: number) || Spree::Exchange.find_by(number: number)+ end++ def legacy_returns+ return @legacy_returns if defined?(@legacy_returns)++ @legacy_returns =+ if ActiveRecord::Base.connection.table_exists?(LEGACY_RETURN_TABLE)+ Class.new(Spree.base_class) do+ self.table_name = LEGACY_RETURN_TABLE+ def self.name = 'LegacyReturnAuthorization'+ end+ end+ end++ # A movement whose cause cannot be resolved is still a true statement+ # about stock, so it is typed by which way the stock went and left with no+ # cause key rather than skipped. The three order-driven kinds are written+ # positive — the kind carries the direction now.+ def by_sign(movement, negative: 'shipped', positive: 'received')+ quantity = movement.quantity.to_i++ { kind: quantity.negative? ? negative : positive, quantity: quantity.abs }+ end++ # Under the old model an unfulfilled fulfillment on a placed order had+ # already taken its units off the shelf. Under the new one those units are+ # on the shelf and allocated, so both counters move together and+ # availability is unchanged by construction — backordered units included,+ # since the pair of increments cancels out the negative on-hand that used+ # to represent them.+ def reconcile_open_fulfillments+ open_fulfillments.find_each(batch_size: batch_size) do |fulfillment|+ next if fulfillment.stock_location.nil?+ next if Spree::StockMovement.exists?(fulfillment_id: fulfillment.id, kind: 'allocated')++ reconcile(fulfillment)+ end+ end++ def open_fulfillments+ Spree::Fulfillment.+ where(status: 'unfulfilled').+ joins(:order).+ where.not(Spree::Order.table_name => { completed_at: nil })+ end++ def reconcile(fulfillment)+ quantities = fulfillment.fulfillment_items.group(:variant_id).sum(:quantity)+ return if quantities.empty?++ ActiveRecord::Base.transaction do+ quantities.each do |variant_id, quantity|+ quantity = quantity.to_i+ next unless quantity.positive?++ variant = Spree::Variant.with_deleted.find_by(id: variant_id)+ next if variant.nil? || !variant.should_track_inventory?++ stock_level = fulfillment.stock_location.stock_level_or_create(variant)+ # A direct column write, not a `received` movement: no goods+ # arrived. The allocation below is what gives the jump a cause.+ stock_level.update_columns(count_on_hand: stock_level.count_on_hand + quantity, updated_at: Time.current)+ fulfillment.stock_location.allocate(variant, quantity, fulfillment)+ end+ end++ @reconciled += 1+ end+ end+end++namespace :spree do+ desc <<~DESC+ Type legacy stock movements and reconcile open fulfillments onto allocations.++ Gives every pre-6.0 movement a kind and a concrete cause key derived from+ its polymorphic originator, then puts the units of every open fulfillment+ on a placed order back on the shelf and allocates them, which is where+ they belong now that stock leaves at dispatch rather than at placement.++ Run after spree:migrate_shipping_to_delivery and after+ spree:upgrade:migrate_returns. Idempotent. Tune the load size with+ BATCH_SIZE (default 5000).+ DESC+ task migrate_stock_movements_to_typed_rows: :environment do+ result = Spree::TypedStockMovementsMigration.new(batch_size: ENV.fetch('BATCH_SIZE', 5_000).to_i).call++ puts "spree_stock_movements: #{result[:typed]} legacy rows typed"+ puts "spree_fulfillments: #{result[:reconciled]} open fulfillments reconciled onto allocations"+ puts 'migrate_stock_movements_to_typed_rows done.'+ end+enddiff --git a/spree/core/spec/lib/tasks/six_zero_data_migrations_spec.rb b/spree/core/spec/lib/tasks/six_zero_data_migrations_spec.rbindex 33684d8f84..ee9e494af0 100644--- a/spree/core/spec/lib/tasks/six_zero_data_migrations_spec.rb+++ b/spree/core/spec/lib/tasks/six_zero_data_migrations_spec.rb@@ -14,6 +14,7 @@ describe '6.0 data migration tasks' doload Spree::Core::Engine.root.join('lib', 'tasks', 'fulfillment_statuses_migration.rake')load Spree::Core::Engine.root.join('lib', 'tasks', 'tax_zones_migration.rake')load Spree::Core::Engine.root.join('lib', 'tasks', 'capture_methods_migration.rake')+ load Spree::Core::Engine.root.join('lib', 'tasks', 'typed_stock_movements_migration.rake')endlet(:store) { @default_store }@@ -660,4 +661,176 @@ describe '6.0 data migration tasks' doexpect(payment_method.reload.capture_method).to eq('manual')endend++ describe 'spree:migrate_stock_movements_to_typed_rows' do+ let(:stock_location) { create(:stock_location) }+ let(:variant) { create(:variant) }+ let(:stock_level) { stock_location.stock_level_or_create(variant) }+ # The legacy authorizations table survives to 6.1 and is reached the same+ # way the task reaches it.+ let(:legacy_authorizations) { Class.new(ActiveRecord::Base) { self.table_name = 'spree_return_authorizations' } }++ # A pre-6.0 row: no kind, cause carried by the polymorphic originator.+ def legacy_movement(quantity, originator_type: nil, originator_id: nil, level: nil)+ movement = Spree::StockMovement.new(stock_level: level || stock_level, quantity: quantity)+ movement.originator_type = originator_type+ movement.originator_id = originator_id+ movement.save(validate: false)+ movement+ end++ def open_order+ create(:order_ready_to_ship, store: store, line_items_count: 1)+ end++ describe 'typing the history' do+ it 'types a fulfillment departure and carries its order' do+ order = open_order+ fulfillment = order.fulfillments.first+ movement = legacy_movement(-3, originator_type: 'Spree::Fulfillment', originator_id: fulfillment.id)++ run_task('spree:migrate_stock_movements_to_typed_rows')++ movement.reload+ expect(movement.kind).to eq('shipped')+ expect(movement.quantity).to eq(3)+ expect(movement.fulfillment_id).to eq(fulfillment.id)+ expect(movement.order_id).to eq(order.id)+ end++ # Installs whose rows were never rewritten by migrate_shipping_to_delivery+ # still say Spree::Shipment.+ it 'accepts the pre-rename originator string' do+ fulfillment = open_order.fulfillments.first+ movement = legacy_movement(-1, originator_type: 'Spree::Shipment', originator_id: fulfillment.id)++ run_task('spree:migrate_stock_movements_to_typed_rows')++ expect(movement.reload.kind).to eq('shipped')+ expect(movement.fulfillment_id).to eq(fulfillment.id)+ end++ it 'types a fulfillment restock as a release' do+ fulfillment = open_order.fulfillments.first+ movement = legacy_movement(2, originator_type: 'Spree::Fulfillment', originator_id: fulfillment.id)++ run_task('spree:migrate_stock_movements_to_typed_rows')++ movement.reload+ expect(movement.kind).to eq('released')+ expect(movement.quantity).to eq(2)+ end++ it 'types both sides of a stock transfer' do+ transfer = create(:stock_transfer)+ arrival = legacy_movement(4, originator_type: 'Spree::StockTransfer', originator_id: transfer.id)+ departure = legacy_movement(-4, originator_type: 'Spree::StockTransfer', originator_id: transfer.id)++ run_task('spree:migrate_stock_movements_to_typed_rows')++ expect(arrival.reload.kind).to eq('received')+ expect(arrival.stock_transfer_id).to eq(transfer.id)+ expect(departure.reload.kind).to eq('shipped')+ expect(departure.quantity).to eq(4)+ expect(departure.stock_transfer_id).to eq(transfer.id)+ end++ # The legacy id is worthless: the returns migrator preserves the number,+ # not the id, and an authorization can become either record.+ it 'resolves a legacy return authorization by its preserved number' do+ return_record = create(:return)+ authorization = legacy_authorizations.create!(number: return_record.number, order_id: return_record.order_id)+ movement = legacy_movement(2, originator_type: 'Spree::ReturnAuthorization', originator_id: authorization.id)++ run_task('spree:migrate_stock_movements_to_typed_rows')++ movement.reload+ expect(movement.kind).to eq('received')+ expect(movement.return_id).to eq(return_record.id)+ expect(movement.order_id).to eq(return_record.order_id)+ end++ it 'resolves an authorization that became an exchange' do+ exchange = create(:exchange)+ authorization = legacy_authorizations.create!(number: exchange.number, order_id: exchange.order_id)+ movement = legacy_movement(1, originator_type: 'Spree::ReturnAuthorization', originator_id: authorization.id)++ run_task('spree:migrate_stock_movements_to_typed_rows')++ movement.reload+ expect(movement.kind).to eq('received')+ expect(movement.exchange_id).to eq(exchange.id)+ expect(movement.return_id).to be_nil+ end++ it 'types a row with no originator as a legacy adjustment, keeping its sign' do+ movement = legacy_movement(-6)++ run_task('spree:migrate_stock_movements_to_typed_rows')++ movement.reload+ expect(movement.kind).to eq('adjusted')+ expect(movement.quantity).to eq(-6)+ expect(movement.reason).to eq('Legacy manual adjustment')+ end++ # A movement with no resolvable cause is still a true statement about+ # stock, so it is typed rather than skipped.+ it 'types an unresolvable cause by sign and leaves the cause keys empty' do+ movement = legacy_movement(-5, originator_type: 'Spree::Fulfillment', originator_id: 0)++ run_task('spree:migrate_stock_movements_to_typed_rows')++ movement.reload+ expect(movement.kind).to eq('shipped')+ expect(movement.quantity).to eq(5)+ expect(movement.fulfillment_id).to be_nil+ expect(movement.order_id).to be_nil+ end+ end++ describe 'reconciling open fulfillments' do+ let!(:order) { open_order }+ let(:fulfillment) { order.fulfillments.first }+ let(:reconciled_variant) { fulfillment.fulfillment_items.first.variant }+ let(:level) { fulfillment.stock_location.stock_level(reconciled_variant) }+ let(:quantity) { fulfillment.fulfillment_items.where(variant_id: reconciled_variant.id).sum(:quantity) }++ it 'puts the units back on the shelf and allocates them, leaving availability alone' do+ count_before = level.reload.count_on_hand+ available_before = level.available_count++ run_task('spree:migrate_stock_movements_to_typed_rows')++ level.reload+ expect(level.count_on_hand).to eq(count_before + quantity)+ expect(level.allocated_count).to eq(quantity)+ expect(level.available_count).to eq(available_before)+ expect(Spree::StockMovement.where(fulfillment_id: fulfillment.id, kind: 'allocated').sum(:quantity)).to eq(quantity)+ end++ # Backordered units need no special case: the pair of increments cancels+ # out the negative on-hand that used to represent them.+ it 'reconciles a backordered fulfillment the same way' do+ fulfillment.fulfillment_items.update_all(status: 'backordered')+ level.update_column(:count_on_hand, -quantity)+ available_before = level.reload.available_count++ run_task('spree:migrate_stock_movements_to_typed_rows')++ level.reload+ expect(level.count_on_hand).to eq(0)+ expect(level.allocated_count).to eq(quantity)+ expect(level.available_count).to eq(available_before)+ end++ it 'changes nothing on a second run' do+ run_task('spree:migrate_stock_movements_to_typed_rows')+ level.reload++ expect { run_task('spree:migrate_stock_movements_to_typed_rows') }.+ not_to change { [level.reload.count_on_hand, level.allocated_count, Spree::StockMovement.count] }+ end+ end+ endend