Skip to content

armgit / spree

Public Imported from GitHub
0

Open Source Platform for DTC, B2B Commerce, Marketplaces & Omnichannel. REST APIs, TypeScript SDKs, and production-ready Next.js storefront. Self-host it. Own your data. No vendor lock-in. Zero platform fees.

Make the product type backfill resumable

Damian Legawiec0666c8bd3949
diff --git a/docs/plans/6.0-product-types.md b/docs/plans/6.0-product-types.md
index 17773eb832..810f45dc91 100644
--- a/docs/plans/6.0-product-types.md
+++ b/docs/plans/6.0-product-types.md
@@ -24,7 +24,7 @@ This is the Shopify-shaped behavior (category changes affect nothing retroactive
1. **Seeding trigger fixed** — `sync_associations_from_product_type` now also runs `after_update, if: :saved_change_to_product_type_id?`, so assigning a type to an existing product and reassigning both seed (detach is a no-op). Presence guards query the join tables rather than the cached associations, which are stale when pending variants insert option-type joins in the same save.
2. **`ProductTypeCustomFieldDefinition`** — table + model with `required`/`sort_order`, `required`/`ordered` scopes, per-type uniqueness, and a validation that the definition's `resource_type` is `Spree::Product`. **No `Spree::CustomFieldDefinition` constant exists in Ruby** (the 5.5 alias this plan assumed is API-layer only), so the association carries custom-field vocabulary while `class_name` points at `Spree::MetafieldDefinition`.
3. **`required` is advisory — no server-side enforcement** (revised 2026-08-06, see Key Decisions). The join keeps the flag and the dashboard marks the field; nothing rejects a blank value.
-4. **`ProductTypes::ApplyToProducts`** + `ApplyToProductsJob` + `POST /api/v3/admin/product_types/:id/apply_to_products` (202 with `products_count`). Batched, additive, idempotent; the category half delegates to `Categories::AddProducts`, which owns the counter-cache/position/search-index bookkeeping a raw `insert_all` would skip.
+4. **`ProductTypes::ApplyToProducts`** + `ApplyToProductsJob` + `POST /api/v3/admin/product_types/:id/apply_to_products` (202 with `products_count`). Batched, additive, idempotent. Both halves bulk-insert, and the counter caches, branch counts and reindexing that `insert_all` skips settle once per run rather than per category per batch. The job runs on **ActiveJob Continuations** (`Spree::Imports::ProcessJob` is the reference): the cursor is the last product id applied, so a deploy or timeout mid-backfill resumes there instead of restarting a catalog-sized run.
5. **Admin write surface** — `option_type_ids`, `category_ids` and the nested `custom_field_definitions` replace-set are permitted; the serializer returns association ids as prefixed ids (read/write symmetry). Category ids are re-scoped to `current_store` before assignment (OptionType is global — nothing to scope there).
6. **Dashboard** — the type sheet edits option types, categories and the dnd-kit-sortable custom-field list with per-row `required` toggles, carries the "changes apply to new products" note, and offers the confirmed apply action. The product form renders custom fields from its type via a new `definitionIds` prop on `FormBackedCustomFieldsProvider` (omitted ⇒ all definitions, so typeless products are unchanged). Keys added to all six locales.
7. **No immutability guard** — the superseded 2026-07-27 model-level block was never shipped and was not added; specs cover detach/reassign being additive and non-destructive.
@@ -204,7 +204,7 @@ There is **no** `type_option_types_present` validation, no `type_option_type?`/`
`Spree::ProductTypes::ApplyToProducts` (Tier 1 service per `6.0-service-workflows.md`) is the ONLY path by which a type edit reaches existing products. Additive and idempotent: safe to re-run, never removes anything, never touches variants or option values. It batches with `find_in_batches`, bulk-inserts the missing option-type joins (continuing each product's own `acts_as_list` position sequence), and delegates the category half to `Categories::AddProducts` — that service owns the counter-cache, position and search-index bookkeeping that a raw `insert_all` would silently skip. Returns the count of products changed.
-Invoked from the Admin API via a background job (`ProductTypes::ApplyToProductsJob`, `Spree.queues.products`); the dashboard confirms with the type's `products_count` before enqueueing. Adding an option type to a product is safe by construction: existing variants simply have no values for it yet — no validation starts failing.
+Invoked from the Admin API via a background job (`ProductTypes::ApplyToProductsJob`, `Spree.queues.products`), which includes `ActiveJob::Continuable` and walks products in id order with the last applied id as its cursor — a large catalog survives a deploy mid-run. The dashboard confirms with the type's `products_count` before enqueueing. Adding an option type to a product is safe by construction: existing variants simply have no values for it yet — no validation starts failing.
### Admin API
diff --git a/docs/plans/6.0-service-workflows.md b/docs/plans/6.0-service-workflows.md
index b0a3f3b8b5..c0c9ad071d 100644
--- a/docs/plans/6.0-service-workflows.md
+++ b/docs/plans/6.0-service-workflows.md
@@ -389,7 +389,9 @@ plan.
4. **Rolling adoption:** services migrate when meaningfully touched — never a
big-bang sweep. `ServiceModule::Base` is not deprecated in 6.0.
5. **Phase 2 — resolved:** plain `ActiveJob::Continuable` jobs are the
- durability tool (`Spree::Imports::ProcessJob` is the reference); the
+ durability tool (`Spree::Imports::ProcessJob` is the reference;
+ `Spree::ProductTypes::ApplyToProductsJob` is the second adopter, cursoring
+ on the last product id of a catalog-sized backfill); the
upgrade data migrations adopt the same pattern. Workflow-level durable
execution is deliberately not shipped — revisit with the payout run
per the note above.
diff --git a/spree/core/app/jobs/spree/product_types/apply_to_products_job.rb b/spree/core/app/jobs/spree/product_types/apply_to_products_job.rb
index 974959d99b..38f8bd984b 100644
--- a/spree/core/app/jobs/spree/product_types/apply_to_products_job.rb
+++ b/spree/core/app/jobs/spree/product_types/apply_to_products_job.rb
@@ -1,13 +1,71 @@
module Spree
module ProductTypes
+ # Backfills a product type's option types and categories onto its existing
+ # products, on ActiveJob Continuations: the cursor is the last product id
+ # applied, so a deploy or timeout mid-backfill resumes from there instead of
+ # restarting a catalog-sized run.
+ #
+ # Products are walked in id order — a stable ordering is what makes the
+ # cursor meaningful, unlike `find_in_batches`' batching alone. Each batch is
+ # additive and idempotent, so re-running the batch that was in flight when
+ # the job died changes nothing.
class ApplyToProductsJob < ::Spree::BaseJob
+ include ActiveJob::Continuable
+
queue_as Spree.queues.products
def perform(product_type_id)
- product_type = Spree::ProductType.find_by(id: product_type_id)
- return if product_type.nil?
+ # Outside the steps on purpose — runs on every execution, resumes
+ # included.
+ @product_type = Spree::ProductType.find_by(id: product_type_id)
+ return if @product_type.nil?
+
+ service = Spree::ProductTypes::ApplyToProducts.new
+ return if service.nothing_to_apply?(@product_type)
+
+ step :apply_to_products, start: 0
+ step :settle_bookkeeping
+ end
+
+ private
+
+ # Cursor = the highest product id already applied. Checkpointed after each
+ # batch, so a resume re-reads nothing it has finished.
+ def apply_to_products(step)
+ service = Spree::ProductTypes::ApplyToProducts.new
+
+ loop do
+ product_ids = @product_type.products.where(Spree::Product.arel_table[:id].gt(step.cursor)).
+ order(:id).limit(Spree::ProductTypes::ApplyToProducts::BATCH_SIZE).pluck(:id)
+ break if product_ids.empty?
+
+ service.apply_batch(@product_type, product_ids)
+ step.set!(product_ids.last)
+ end
+ end
+
+ # Counter caches, branch counts and reindexing settle once for the whole
+ # run — each is a subtree-wide recount or a job per product, so doing it
+ # per batch would cost more than the backfill itself.
+ #
+ # The set of touched products is re-derived from the join table rather
+ # than carried across batches: a resumed job has no memory of what earlier
+ # executions changed, and every product of the type is in the right state
+ # by now regardless of which execution put it there.
+ def settle_bookkeeping
+ category_ids = @product_type.category_ids
+ return if category_ids.empty?
+
+ service = Spree::ProductTypes::ApplyToProducts.new
+ # Per-product counters batch — the whole point of the cursor is not
+ # holding a catalog-sized id list in memory.
+ @product_type.products.select(:id).
+ find_in_batches(batch_size: Spree::ProductTypes::ApplyToProducts::BATCH_SIZE) do |products|
+ service.settle_product_counters(products.map(&:id))
+ end
- Spree::ProductTypes::ApplyToProducts.call(product_type: product_type)
+ # The subtree-wide category recount runs once.
+ service.settle_category_counters(category_ids)
end
end
end
diff --git a/spree/core/app/services/spree/product_types/apply_to_products.rb b/spree/core/app/services/spree/product_types/apply_to_products.rb
index 36861735d6..b059183de2 100644
--- a/spree/core/app/services/spree/product_types/apply_to_products.rb
+++ b/spree/core/app/services/spree/product_types/apply_to_products.rb
@@ -6,6 +6,12 @@ module Spree
#
# Additive and idempotent: products keep everything they already have, so a
# re-run after a crash is safe.
+ #
+ # `ApplyToProductsJob` drives this in id order with a resumable cursor, so
+ # the entry points are deliberately separate: {#call} runs the whole
+ # backfill in one pass (console, specs, small catalogs), while
+ # {#apply_batch} plus {#settle_product_counters} / {#settle_category_counters}
+ # let the job checkpoint between batches.
class ApplyToProducts
prepend ::Spree::ServiceModule::Base
@@ -14,10 +20,7 @@ module Spree
# @param product_type [Spree::ProductType]
# @return [Spree::ServiceModule::Base::Result] count of products changed
def call(product_type:)
- option_type_ids = product_type.option_type_ids
- category_ids = product_type.category_ids
-
- return success(0) if option_type_ids.empty? && category_ids.empty?
+ return success(0) if nothing_to_apply?(product_type)
changed_product_ids = Set.new
categorized_product_ids = Set.new
@@ -25,21 +28,59 @@ module Spree
# Only ids are ever needed — hydrating products would run their
# after_initialize callbacks for nothing.
product_type.products.select(:id).find_in_batches(batch_size: BATCH_SIZE) do |products|
- product_ids = products.map(&:id)
+ result = apply_batch(product_type, products.map(&:id))
+ changed_product_ids.merge(result[:changed_ids])
+ categorized_product_ids.merge(result[:categorized_ids])
+ end
+
+ settle_category_bookkeeping(categorized_product_ids.to_a, product_type.category_ids)
+
+ success(changed_product_ids.size)
+ end
+
+ # @return [Boolean] true when the type defines nothing to seed
+ def nothing_to_apply?(product_type)
+ product_type.option_type_ids.empty? && product_type.category_ids.empty?
+ end
+
+ # One batch of the backfill. Split out so the job can checkpoint between
+ # batches; safe to re-run on the same ids after an interruption.
+ #
+ # @return [Hash] :changed_ids and :categorized_ids for this batch
+ def apply_batch(product_type, product_ids)
+ changed = add_option_types(product_ids, product_type.option_type_ids)
+ categorized = add_categories(product_ids, product_type.category_ids)
- changed_product_ids.merge(add_option_types(product_ids, option_type_ids))
+ { changed_ids: changed | categorized, categorized_ids: categorized }
+ end
+
+ # insert_all skips the ProductCategory callbacks, so the per-product
+ # counts they maintain are recomputed here. Safe to call per batch.
+ def settle_product_counters(product_ids)
+ return if product_ids.empty?
- newly_categorized = add_categories(product_ids, category_ids)
- categorized_product_ids.merge(newly_categorized)
- changed_product_ids.merge(newly_categorized)
+ counts = Spree::ProductCategory.where(product_id: product_ids).group(:product_id).count
+ counts.each do |product_id, count|
+ Spree::Product.where(id: product_id).update_all(categories_count: count)
end
- # Counter caches, branch counts and reindexing are settled once for the
- # whole run rather than per category per batch — each is a subtree-wide
- # recount or a job per product.
- settle_category_bookkeeping(categorized_product_ids.to_a, category_ids)
+ Spree::Product.where(id: product_ids).find_each(&:enqueue_search_index)
+ end
- success(changed_product_ids.size)
+ # The category side settles once for the whole run: `products_count` is a
+ # subtree-wide recount over each category and its ancestors, so repeating
+ # it per batch would cost more than the backfill.
+ def settle_category_counters(category_ids)
+ return if category_ids.empty?
+
+ Spree::Category.recalculate_products_count(category_ids)
+ Spree::Category.where(id: category_ids).touch_all
+ end
+
+ # Both halves, for callers that run the backfill in one pass.
+ def settle_category_bookkeeping(product_ids, category_ids)
+ settle_product_counters(product_ids)
+ settle_category_counters(category_ids) if product_ids.any?
end
private
@@ -105,22 +146,6 @@ module Spree
Spree::ProductCategory.insert_all(rows)
rows.pluck(:product_id).uniq
end
-
- # insert_all skips the ProductCategory callbacks, so the counts they
- # maintain are recomputed here — once, for every product touched.
- def settle_category_bookkeeping(product_ids, category_ids)
- return if product_ids.empty?
-
- counts = Spree::ProductCategory.where(product_id: product_ids).group(:product_id).count
- counts.each do |product_id, count|
- Spree::Product.where(id: product_id).update_all(categories_count: count)
- end
-
- Spree::Category.recalculate_products_count(category_ids)
- Spree::Category.where(id: category_ids).touch_all
-
- Spree::Product.where(id: product_ids).find_each(&:enqueue_search_index)
- end
end
end
end
diff --git a/spree/core/spec/jobs/spree/product_types/apply_to_products_job_spec.rb b/spree/core/spec/jobs/spree/product_types/apply_to_products_job_spec.rb
new file mode 100644
index 0000000000..3d10c9f751
--- /dev/null
+++ b/spree/core/spec/jobs/spree/product_types/apply_to_products_job_spec.rb
@@ -0,0 +1,81 @@
+require 'spec_helper'
+require 'active_job/continuation/test_helper'
+
+RSpec.describe Spree::ProductTypes::ApplyToProductsJob, type: :job do
+ let(:store) { @default_store }
+ let(:option_type) { create(:option_type, name: 'size') }
+ let(:category) { create(:category) }
+ let(:product_type) { create(:product_type, option_types: [option_type], categories: [category]) }
+
+ # update_column so the seeding callback doesn't do the job's work for it.
+ def product_carrying_the_type
+ create(:product).tap { |product| product.update_column(:product_type_id, product_type.id) }
+ end
+
+ it 'applies the type to its products' do
+ product = product_carrying_the_type
+
+ described_class.perform_now(product_type.id)
+
+ expect(product.reload.option_types).to eq([option_type])
+ expect(product.categories).to eq([category])
+ end
+
+ it 'settles the category counter cache' do
+ product_carrying_the_type
+
+ described_class.perform_now(product_type.id)
+
+ expect(category.reload.products_count).to eq(1)
+ end
+
+ it 'does nothing for a type that defines neither option types nor categories' do
+ bare_type = create(:product_type)
+ product = create(:product)
+ product.update_column(:product_type_id, bare_type.id)
+
+ described_class.perform_now(bare_type.id)
+
+ expect(product.reload.option_types).to be_empty
+ end
+
+ it 'ignores a deleted product type' do
+ expect { described_class.perform_now('nonexistent') }.not_to raise_error
+ end
+
+ describe 'interruption and resume' do
+ include ActiveJob::Continuation::TestHelper
+
+ around do |example|
+ original = ActiveJob::Base.queue_adapter
+ ActiveJob::Base.queue_adapter = :test
+ example.run
+ ensure
+ ActiveJob::Base.queue_adapter = original
+ end
+
+ # One product per batch, so the cursor advances mid-run.
+ before { stub_const('Spree::ProductTypes::ApplyToProducts::BATCH_SIZE', 1) }
+
+ it 'resumes from the cursor instead of restarting the backfill' do
+ first = product_carrying_the_type
+ second = product_carrying_the_type
+
+ described_class.perform_later(product_type.id)
+
+ interrupt_job_during_step(described_class, :apply_to_products, cursor: first.id) do
+ perform_enqueued_jobs
+ end
+
+ expect(first.reload.option_types).to eq([option_type])
+ expect(second.reload.option_types).to be_empty
+
+ perform_enqueued_jobs
+
+ expect(second.reload.option_types).to eq([option_type])
+ # Re-running the finished batch adds nothing — the backfill is additive.
+ expect(first.reload.option_types).to eq([option_type])
+ expect(category.reload.products_count).to eq(2)
+ end
+ end
+end