Skip to content

armgit / spree

Public Imported from GitHub
0

Open Source Platform for DTC, B2B Commerce, Marketplaces & Omnichannel. REST APIs, TypeScript SDKs, and production-ready Next.js storefront. Self-host it. Own your data. No vendor lock-in. Zero platform fees.

Phase P3.1: Type the stock history and reconcile open fulfillments

Łukasz Adamczyk187aa1b69bd8
diff --git a/spree/core/lib/spree/upgrades/5_6_to_6_0/manifest.yml b/spree/core/lib/spree/upgrades/5_6_to_6_0/manifest.yml
index 39774b7d45..f41351ca06 100644
--- a/spree/core/lib/spree/upgrades/5_6_to_6_0/manifest.yml
+++ b/spree/core/lib/spree/upgrades/5_6_to_6_0/manifest.yml
@@ -270,6 +270,43 @@ steps:
to retry those rows — the task only looks at what isn't migrated yet — or
pass SKIP_FAILED_ROWS=true to accept the gap and continue.
+ - id: migrate_stock_movements_to_typed_rows
+ name: "Type stock movements and reconcile open fulfillments"
+ task: "spree:migrate_stock_movements_to_typed_rows"
+ notes: |
+ Gives every pre-6.0 stock movement a kind and a concrete cause key
+ derived from its polymorphic originator: a fulfillment originator
+ becomes shipped or released by sign, a stock transfer becomes received
+ or shipped, a legacy return authorization becomes received against the
+ Return or Exchange it turned into, and a movement with no originator
+ becomes an adjustment reading "Legacy manual adjustment". A cause that
+ cannot be resolved is still typed by sign and simply carries no cause
+ key — a movement with no cause is still a true statement about stock.
+
+ Then reconciles the representation change. Stock leaves the shelf at
+ dispatch in 6.0, not at placement, so the units of every open
+ fulfillment on a placed order go back onto count_on_hand and are
+ allocated at the same time, with an allocated movement recording the
+ jump. Availability is unchanged by construction, since both sides of
+ "on hand minus allocated" move together, and backordered units need no
+ special case — the pair of increments cancels out the negative on-hand
+ that used to represent them.
+
+ Run AFTER migrate_shipping_to_delivery, which rewrites the
+ 'Spree::Shipment' originator strings, and AFTER migrate_returns,
+ because a legacy authorization id says nothing about the new record:
+ the returns migrator preserves the number, not the id, and one
+ authorization can become either a Return or an Exchange.
+
+ Idempotent. Typing narrows to rows without a kind and reconciliation
+ skips fulfillments that already hold an allocation, so an interrupted
+ run resumes for free. That same predicate makes the deploy window safe:
+ a fulfillment shipped before the task runs never gets an allocation,
+ and dispatch only ships allocated units, so its stock is never
+ decremented twice. originator_type and originator_id stay in place as
+ the rollback path and are dropped in 6.1. Tune the load size with
+ BATCH_SIZE (default 5000).
+
- id: backfill_delivery_and_stock_store_ids
name: "Bind delivery methods and stock locations to the default store"
task: "spree:backfill_delivery_and_stock_store_ids"
diff --git a/spree/core/lib/tasks/typed_stock_movements_migration.rake b/spree/core/lib/tasks/typed_stock_movements_migration.rake
new file mode 100644
index 0000000000..71f4974f07
--- /dev/null
+++ b/spree/core/lib/tasks/typed_stock_movements_migration.rake
@@ -0,0 +1,194 @@
+# frozen_string_literal: true
+
+# Defined here rather than in lib/spree/core/ — this is a one-release upgrade
+# step that dies with the polymorphic originator in 6.1, not engine
+# infrastructure. Same placement as Spree::ReturnsMigrator in a sibling task.
+module Spree
+ # Types the legacy stock-movement history and reconciles open fulfillments
+ # onto the allocation model (docs/plans/6.0-typed-stock-movements.md).
+ #
+ # Two jobs in one idempotent task. Typing narrows to rows without a kind;
+ # reconciliation skips fulfillments that already hold an allocation. Both
+ # predicates are queries over the data itself, so an interrupted run resumes
+ # for free and a second run finds nothing to do. That same predicate is what
+ # makes the deploy window safe: a fulfillment shipped before the task runs
+ # never gets an allocation, and dispatch only ships allocated units.
+ #
+ # Runs after spree:migrate_shipping_to_delivery, which rewrites the
+ # 'Spree::Shipment' originator strings, and after
+ # spree:upgrade:migrate_returns, because a legacy authorization id says
+ # nothing about the new record — the returns migrator preserves the number,
+ # not the id, and one authorization can become either a Return or an
+ # Exchange.
+ class TypedStockMovementsMigration
+ LEGACY_ADJUSTMENT_REASON = 'Legacy manual adjustment'
+ LEGACY_RETURN_TABLE = 'spree_return_authorizations'
+ FULFILLMENT_ORIGINATOR_TYPES = ['Spree::Fulfillment', 'Spree::Shipment'].freeze
+
+ def initialize(batch_size: 5_000)
+ @batch_size = batch_size
+ @typed = 0
+ @reconciled = 0
+ end
+
+ # @return [Hash] counts of rows typed and fulfillments reconciled
+ def call
+ type_history
+ reconcile_open_fulfillments
+
+ { typed: @typed, reconciled: @reconciled }
+ end
+
+ private
+
+ attr_reader :batch_size
+
+ def type_history
+ Spree::StockMovement.unscoped.where(kind: nil).find_each(batch_size: batch_size) do |movement|
+ attributes = attributes_for(movement)
+ Spree::StockMovement.unscoped.where(id: movement.id).update_all(attributes.merge(updated_at: Time.current))
+ @typed += 1
+ end
+ end
+
+ def attributes_for(movement)
+ return { kind: 'adjusted', reason: LEGACY_ADJUSTMENT_REASON } if movement.originator_type.blank?
+
+ case movement.originator_type
+ when *FULFILLMENT_ORIGINATOR_TYPES then fulfillment_attributes(movement)
+ when 'Spree::StockTransfer' then stock_transfer_attributes(movement)
+ when 'Spree::ReturnAuthorization' then return_attributes(movement)
+ else by_sign(movement)
+ end
+ end
+
+ # A departure, or the restock of a cancellation or relocation, carrying
+ # the order the fulfillment belonged to so "which order was this for?"
+ # costs no join.
+ def fulfillment_attributes(movement)
+ fulfillment = Spree::Fulfillment.find_by(id: movement.originator_id)
+ return by_sign(movement) if fulfillment.nil?
+
+ by_sign(movement, negative: 'shipped', positive: 'released').
+ merge(fulfillment_id: fulfillment.id, order_id: fulfillment.order_id)
+ end
+
+ def stock_transfer_attributes(movement)
+ return by_sign(movement) unless Spree::StockTransfer.exists?(id: movement.originator_id)
+
+ by_sign(movement).merge(stock_transfer_id: movement.originator_id)
+ end
+
+ # The legacy id is worthless on its own, so the authorization is read by
+ # name and its preserved number matched against whichever record the
+ # returns migrator produced.
+ def return_attributes(movement)
+ attributes = { kind: 'received', quantity: movement.quantity.to_i.abs }
+ record = migrated_return_for(movement.originator_id)
+
+ case record
+ when Spree::Return then attributes.merge(return_id: record.id, order_id: record.order_id)
+ when Spree::Exchange then attributes.merge(exchange_id: record.id, order_id: record.order_id)
+ else attributes
+ end
+ end
+
+ def migrated_return_for(authorization_id)
+ return nil unless legacy_returns
+
+ number = legacy_returns.where(id: authorization_id).pick(:number)
+ return nil if number.blank?
+
+ Spree::Return.find_by(number: number) || Spree::Exchange.find_by(number: number)
+ end
+
+ def legacy_returns
+ return @legacy_returns if defined?(@legacy_returns)
+
+ @legacy_returns =
+ if ActiveRecord::Base.connection.table_exists?(LEGACY_RETURN_TABLE)
+ Class.new(Spree.base_class) do
+ self.table_name = LEGACY_RETURN_TABLE
+ def self.name = 'LegacyReturnAuthorization'
+ end
+ end
+ end
+
+ # A movement whose cause cannot be resolved is still a true statement
+ # about stock, so it is typed by which way the stock went and left with no
+ # cause key rather than skipped. The three order-driven kinds are written
+ # positive — the kind carries the direction now.
+ def by_sign(movement, negative: 'shipped', positive: 'received')
+ quantity = movement.quantity.to_i
+
+ { kind: quantity.negative? ? negative : positive, quantity: quantity.abs }
+ end
+
+ # Under the old model an unfulfilled fulfillment on a placed order had
+ # already taken its units off the shelf. Under the new one those units are
+ # on the shelf and allocated, so both counters move together and
+ # availability is unchanged by construction — backordered units included,
+ # since the pair of increments cancels out the negative on-hand that used
+ # to represent them.
+ def reconcile_open_fulfillments
+ open_fulfillments.find_each(batch_size: batch_size) do |fulfillment|
+ next if fulfillment.stock_location.nil?
+ next if Spree::StockMovement.exists?(fulfillment_id: fulfillment.id, kind: 'allocated')
+
+ reconcile(fulfillment)
+ end
+ end
+
+ def open_fulfillments
+ Spree::Fulfillment.
+ where(status: 'unfulfilled').
+ joins(:order).
+ where.not(Spree::Order.table_name => { completed_at: nil })
+ end
+
+ def reconcile(fulfillment)
+ quantities = fulfillment.fulfillment_items.group(:variant_id).sum(:quantity)
+ return if quantities.empty?
+
+ ActiveRecord::Base.transaction do
+ quantities.each do |variant_id, quantity|
+ quantity = quantity.to_i
+ next unless quantity.positive?
+
+ variant = Spree::Variant.with_deleted.find_by(id: variant_id)
+ next if variant.nil? || !variant.should_track_inventory?
+
+ stock_level = fulfillment.stock_location.stock_level_or_create(variant)
+ # A direct column write, not a `received` movement: no goods
+ # arrived. The allocation below is what gives the jump a cause.
+ stock_level.update_columns(count_on_hand: stock_level.count_on_hand + quantity, updated_at: Time.current)
+ fulfillment.stock_location.allocate(variant, quantity, fulfillment)
+ end
+ end
+
+ @reconciled += 1
+ end
+ end
+end
+
+namespace :spree do
+ desc <<~DESC
+ Type legacy stock movements and reconcile open fulfillments onto allocations.
+
+ Gives every pre-6.0 movement a kind and a concrete cause key derived from
+ its polymorphic originator, then puts the units of every open fulfillment
+ on a placed order back on the shelf and allocates them, which is where
+ they belong now that stock leaves at dispatch rather than at placement.
+
+ Run after spree:migrate_shipping_to_delivery and after
+ spree:upgrade:migrate_returns. Idempotent. Tune the load size with
+ BATCH_SIZE (default 5000).
+ DESC
+ task migrate_stock_movements_to_typed_rows: :environment do
+ result = Spree::TypedStockMovementsMigration.new(batch_size: ENV.fetch('BATCH_SIZE', 5_000).to_i).call
+
+ puts "spree_stock_movements: #{result[:typed]} legacy rows typed"
+ puts "spree_fulfillments: #{result[:reconciled]} open fulfillments reconciled onto allocations"
+ puts 'migrate_stock_movements_to_typed_rows done.'
+ end
+end
diff --git a/spree/core/spec/lib/tasks/six_zero_data_migrations_spec.rb b/spree/core/spec/lib/tasks/six_zero_data_migrations_spec.rb
index 33684d8f84..ee9e494af0 100644
--- a/spree/core/spec/lib/tasks/six_zero_data_migrations_spec.rb
+++ b/spree/core/spec/lib/tasks/six_zero_data_migrations_spec.rb
@@ -14,6 +14,7 @@ describe '6.0 data migration tasks' do
load Spree::Core::Engine.root.join('lib', 'tasks', 'fulfillment_statuses_migration.rake')
load Spree::Core::Engine.root.join('lib', 'tasks', 'tax_zones_migration.rake')
load Spree::Core::Engine.root.join('lib', 'tasks', 'capture_methods_migration.rake')
+ load Spree::Core::Engine.root.join('lib', 'tasks', 'typed_stock_movements_migration.rake')
end
let(:store) { @default_store }
@@ -660,4 +661,176 @@ describe '6.0 data migration tasks' do
expect(payment_method.reload.capture_method).to eq('manual')
end
end
+
+ describe 'spree:migrate_stock_movements_to_typed_rows' do
+ let(:stock_location) { create(:stock_location) }
+ let(:variant) { create(:variant) }
+ let(:stock_level) { stock_location.stock_level_or_create(variant) }
+ # The legacy authorizations table survives to 6.1 and is reached the same
+ # way the task reaches it.
+ let(:legacy_authorizations) { Class.new(ActiveRecord::Base) { self.table_name = 'spree_return_authorizations' } }
+
+ # A pre-6.0 row: no kind, cause carried by the polymorphic originator.
+ def legacy_movement(quantity, originator_type: nil, originator_id: nil, level: nil)
+ movement = Spree::StockMovement.new(stock_level: level || stock_level, quantity: quantity)
+ movement.originator_type = originator_type
+ movement.originator_id = originator_id
+ movement.save(validate: false)
+ movement
+ end
+
+ def open_order
+ create(:order_ready_to_ship, store: store, line_items_count: 1)
+ end
+
+ describe 'typing the history' do
+ it 'types a fulfillment departure and carries its order' do
+ order = open_order
+ fulfillment = order.fulfillments.first
+ movement = legacy_movement(-3, originator_type: 'Spree::Fulfillment', originator_id: fulfillment.id)
+
+ run_task('spree:migrate_stock_movements_to_typed_rows')
+
+ movement.reload
+ expect(movement.kind).to eq('shipped')
+ expect(movement.quantity).to eq(3)
+ expect(movement.fulfillment_id).to eq(fulfillment.id)
+ expect(movement.order_id).to eq(order.id)
+ end
+
+ # Installs whose rows were never rewritten by migrate_shipping_to_delivery
+ # still say Spree::Shipment.
+ it 'accepts the pre-rename originator string' do
+ fulfillment = open_order.fulfillments.first
+ movement = legacy_movement(-1, originator_type: 'Spree::Shipment', originator_id: fulfillment.id)
+
+ run_task('spree:migrate_stock_movements_to_typed_rows')
+
+ expect(movement.reload.kind).to eq('shipped')
+ expect(movement.fulfillment_id).to eq(fulfillment.id)
+ end
+
+ it 'types a fulfillment restock as a release' do
+ fulfillment = open_order.fulfillments.first
+ movement = legacy_movement(2, originator_type: 'Spree::Fulfillment', originator_id: fulfillment.id)
+
+ run_task('spree:migrate_stock_movements_to_typed_rows')
+
+ movement.reload
+ expect(movement.kind).to eq('released')
+ expect(movement.quantity).to eq(2)
+ end
+
+ it 'types both sides of a stock transfer' do
+ transfer = create(:stock_transfer)
+ arrival = legacy_movement(4, originator_type: 'Spree::StockTransfer', originator_id: transfer.id)
+ departure = legacy_movement(-4, originator_type: 'Spree::StockTransfer', originator_id: transfer.id)
+
+ run_task('spree:migrate_stock_movements_to_typed_rows')
+
+ expect(arrival.reload.kind).to eq('received')
+ expect(arrival.stock_transfer_id).to eq(transfer.id)
+ expect(departure.reload.kind).to eq('shipped')
+ expect(departure.quantity).to eq(4)
+ expect(departure.stock_transfer_id).to eq(transfer.id)
+ end
+
+ # The legacy id is worthless: the returns migrator preserves the number,
+ # not the id, and an authorization can become either record.
+ it 'resolves a legacy return authorization by its preserved number' do
+ return_record = create(:return)
+ authorization = legacy_authorizations.create!(number: return_record.number, order_id: return_record.order_id)
+ movement = legacy_movement(2, originator_type: 'Spree::ReturnAuthorization', originator_id: authorization.id)
+
+ run_task('spree:migrate_stock_movements_to_typed_rows')
+
+ movement.reload
+ expect(movement.kind).to eq('received')
+ expect(movement.return_id).to eq(return_record.id)
+ expect(movement.order_id).to eq(return_record.order_id)
+ end
+
+ it 'resolves an authorization that became an exchange' do
+ exchange = create(:exchange)
+ authorization = legacy_authorizations.create!(number: exchange.number, order_id: exchange.order_id)
+ movement = legacy_movement(1, originator_type: 'Spree::ReturnAuthorization', originator_id: authorization.id)
+
+ run_task('spree:migrate_stock_movements_to_typed_rows')
+
+ movement.reload
+ expect(movement.kind).to eq('received')
+ expect(movement.exchange_id).to eq(exchange.id)
+ expect(movement.return_id).to be_nil
+ end
+
+ it 'types a row with no originator as a legacy adjustment, keeping its sign' do
+ movement = legacy_movement(-6)
+
+ run_task('spree:migrate_stock_movements_to_typed_rows')
+
+ movement.reload
+ expect(movement.kind).to eq('adjusted')
+ expect(movement.quantity).to eq(-6)
+ expect(movement.reason).to eq('Legacy manual adjustment')
+ end
+
+ # A movement with no resolvable cause is still a true statement about
+ # stock, so it is typed rather than skipped.
+ it 'types an unresolvable cause by sign and leaves the cause keys empty' do
+ movement = legacy_movement(-5, originator_type: 'Spree::Fulfillment', originator_id: 0)
+
+ run_task('spree:migrate_stock_movements_to_typed_rows')
+
+ movement.reload
+ expect(movement.kind).to eq('shipped')
+ expect(movement.quantity).to eq(5)
+ expect(movement.fulfillment_id).to be_nil
+ expect(movement.order_id).to be_nil
+ end
+ end
+
+ describe 'reconciling open fulfillments' do
+ let!(:order) { open_order }
+ let(:fulfillment) { order.fulfillments.first }
+ let(:reconciled_variant) { fulfillment.fulfillment_items.first.variant }
+ let(:level) { fulfillment.stock_location.stock_level(reconciled_variant) }
+ let(:quantity) { fulfillment.fulfillment_items.where(variant_id: reconciled_variant.id).sum(:quantity) }
+
+ it 'puts the units back on the shelf and allocates them, leaving availability alone' do
+ count_before = level.reload.count_on_hand
+ available_before = level.available_count
+
+ run_task('spree:migrate_stock_movements_to_typed_rows')
+
+ level.reload
+ expect(level.count_on_hand).to eq(count_before + quantity)
+ expect(level.allocated_count).to eq(quantity)
+ expect(level.available_count).to eq(available_before)
+ expect(Spree::StockMovement.where(fulfillment_id: fulfillment.id, kind: 'allocated').sum(:quantity)).to eq(quantity)
+ end
+
+ # Backordered units need no special case: the pair of increments cancels
+ # out the negative on-hand that used to represent them.
+ it 'reconciles a backordered fulfillment the same way' do
+ fulfillment.fulfillment_items.update_all(status: 'backordered')
+ level.update_column(:count_on_hand, -quantity)
+ available_before = level.reload.available_count
+
+ run_task('spree:migrate_stock_movements_to_typed_rows')
+
+ level.reload
+ expect(level.count_on_hand).to eq(0)
+ expect(level.allocated_count).to eq(quantity)
+ expect(level.available_count).to eq(available_before)
+ end
+
+ it 'changes nothing on a second run' do
+ run_task('spree:migrate_stock_movements_to_typed_rows')
+ level.reload
+
+ expect { run_task('spree:migrate_stock_movements_to_typed_rows') }.
+ not_to change { [level.reload.count_on_hand, level.allocated_count, Spree::StockMovement.count] }
+ end
+ end
+ end
end