Open Source Platform for DTC, B2B Commerce, Marketplaces & Omnichannel. REST APIs, TypeScript SDKs, and production-ready Next.js storefront. Self-host it. Own your data. No vendor lock-in. Zero platform fees.
Ruby b2b-commercee-commerceecommerceecommerce-apiecommerce-frameworkecommerce-platformheadlessheadless-commerceheadless-ecommercemarketplacemulti-tenantmulti-vendormulti-vendor-ecommerceopen-sourcespree-commerce
Make the product type backfill resumable
Damian Legawiec0666c8bd3949
diff --git a/docs/plans/6.0-product-types.md b/docs/plans/6.0-product-types.mdindex 17773eb832..810f45dc91 100644--- a/docs/plans/6.0-product-types.md+++ b/docs/plans/6.0-product-types.md@@ -24,7 +24,7 @@ This is the Shopify-shaped behavior (category changes affect nothing retroactive1. **Seeding trigger fixed** — `sync_associations_from_product_type` now also runs `after_update, if: :saved_change_to_product_type_id?`, so assigning a type to an existing product and reassigning both seed (detach is a no-op). Presence guards query the join tables rather than the cached associations, which are stale when pending variants insert option-type joins in the same save.2. **`ProductTypeCustomFieldDefinition`** — table + model with `required`/`sort_order`, `required`/`ordered` scopes, per-type uniqueness, and a validation that the definition's `resource_type` is `Spree::Product`. **No `Spree::CustomFieldDefinition` constant exists in Ruby** (the 5.5 alias this plan assumed is API-layer only), so the association carries custom-field vocabulary while `class_name` points at `Spree::MetafieldDefinition`.3. **`required` is advisory — no server-side enforcement** (revised 2026-08-06, see Key Decisions). The join keeps the flag and the dashboard marks the field; nothing rejects a blank value.-4. **`ProductTypes::ApplyToProducts`** + `ApplyToProductsJob` + `POST /api/v3/admin/product_types/:id/apply_to_products` (202 with `products_count`). Batched, additive, idempotent; the category half delegates to `Categories::AddProducts`, which owns the counter-cache/position/search-index bookkeeping a raw `insert_all` would skip.+4. **`ProductTypes::ApplyToProducts`** + `ApplyToProductsJob` + `POST /api/v3/admin/product_types/:id/apply_to_products` (202 with `products_count`). Batched, additive, idempotent. Both halves bulk-insert, and the counter caches, branch counts and reindexing that `insert_all` skips settle once per run rather than per category per batch. The job runs on **ActiveJob Continuations** (`Spree::Imports::ProcessJob` is the reference): the cursor is the last product id applied, so a deploy or timeout mid-backfill resumes there instead of restarting a catalog-sized run.5. **Admin write surface** — `option_type_ids`, `category_ids` and the nested `custom_field_definitions` replace-set are permitted; the serializer returns association ids as prefixed ids (read/write symmetry). Category ids are re-scoped to `current_store` before assignment (OptionType is global — nothing to scope there).6. **Dashboard** — the type sheet edits option types, categories and the dnd-kit-sortable custom-field list with per-row `required` toggles, carries the "changes apply to new products" note, and offers the confirmed apply action. The product form renders custom fields from its type via a new `definitionIds` prop on `FormBackedCustomFieldsProvider` (omitted ⇒ all definitions, so typeless products are unchanged). Keys added to all six locales.7. **No immutability guard** — the superseded 2026-07-27 model-level block was never shipped and was not added; specs cover detach/reassign being additive and non-destructive.@@ -204,7 +204,7 @@ There is **no** `type_option_types_present` validation, no `type_option_type?`/``Spree::ProductTypes::ApplyToProducts` (Tier 1 service per `6.0-service-workflows.md`) is the ONLY path by which a type edit reaches existing products. Additive and idempotent: safe to re-run, never removes anything, never touches variants or option values. It batches with `find_in_batches`, bulk-inserts the missing option-type joins (continuing each product's own `acts_as_list` position sequence), and delegates the category half to `Categories::AddProducts` — that service owns the counter-cache, position and search-index bookkeeping that a raw `insert_all` would silently skip. Returns the count of products changed.-Invoked from the Admin API via a background job (`ProductTypes::ApplyToProductsJob`, `Spree.queues.products`); the dashboard confirms with the type's `products_count` before enqueueing. Adding an option type to a product is safe by construction: existing variants simply have no values for it yet — no validation starts failing.+Invoked from the Admin API via a background job (`ProductTypes::ApplyToProductsJob`, `Spree.queues.products`), which includes `ActiveJob::Continuable` and walks products in id order with the last applied id as its cursor — a large catalog survives a deploy mid-run. The dashboard confirms with the type's `products_count` before enqueueing. Adding an option type to a product is safe by construction: existing variants simply have no values for it yet — no validation starts failing.### Admin APIdiff --git a/docs/plans/6.0-service-workflows.md b/docs/plans/6.0-service-workflows.mdindex b0a3f3b8b5..c0c9ad071d 100644--- a/docs/plans/6.0-service-workflows.md+++ b/docs/plans/6.0-service-workflows.md@@ -389,7 +389,9 @@ plan.4. **Rolling adoption:** services migrate when meaningfully touched — never abig-bang sweep. `ServiceModule::Base` is not deprecated in 6.0.5. **Phase 2 — resolved:** plain `ActiveJob::Continuable` jobs are the- durability tool (`Spree::Imports::ProcessJob` is the reference); the+ durability tool (`Spree::Imports::ProcessJob` is the reference;+ `Spree::ProductTypes::ApplyToProductsJob` is the second adopter, cursoring+ on the last product id of a catalog-sized backfill); theupgrade data migrations adopt the same pattern. Workflow-level durableexecution is deliberately not shipped — revisit with the payout runper the note above.diff --git a/spree/core/app/jobs/spree/product_types/apply_to_products_job.rb b/spree/core/app/jobs/spree/product_types/apply_to_products_job.rbindex 974959d99b..38f8bd984b 100644--- a/spree/core/app/jobs/spree/product_types/apply_to_products_job.rb+++ b/spree/core/app/jobs/spree/product_types/apply_to_products_job.rb@@ -1,13 +1,71 @@module Spreemodule ProductTypes+ # Backfills a product type's option types and categories onto its existing+ # products, on ActiveJob Continuations: the cursor is the last product id+ # applied, so a deploy or timeout mid-backfill resumes from there instead of+ # restarting a catalog-sized run.+ #+ # Products are walked in id order — a stable ordering is what makes the+ # cursor meaningful, unlike `find_in_batches`' batching alone. Each batch is+ # additive and idempotent, so re-running the batch that was in flight when+ # the job died changes nothing.class ApplyToProductsJob < ::Spree::BaseJob+ include ActiveJob::Continuable+queue_as Spree.queues.productsdef perform(product_type_id)- product_type = Spree::ProductType.find_by(id: product_type_id)- return if product_type.nil?+ # Outside the steps on purpose — runs on every execution, resumes+ # included.+ @product_type = Spree::ProductType.find_by(id: product_type_id)+ return if @product_type.nil?++ service = Spree::ProductTypes::ApplyToProducts.new+ return if service.nothing_to_apply?(@product_type)++ step :apply_to_products, start: 0+ step :settle_bookkeeping+ end++ private++ # Cursor = the highest product id already applied. Checkpointed after each+ # batch, so a resume re-reads nothing it has finished.+ def apply_to_products(step)+ service = Spree::ProductTypes::ApplyToProducts.new++ loop do+ product_ids = @product_type.products.where(Spree::Product.arel_table[:id].gt(step.cursor)).+ order(:id).limit(Spree::ProductTypes::ApplyToProducts::BATCH_SIZE).pluck(:id)+ break if product_ids.empty?++ service.apply_batch(@product_type, product_ids)+ step.set!(product_ids.last)+ end+ end++ # Counter caches, branch counts and reindexing settle once for the whole+ # run — each is a subtree-wide recount or a job per product, so doing it+ # per batch would cost more than the backfill itself.+ #+ # The set of touched products is re-derived from the join table rather+ # than carried across batches: a resumed job has no memory of what earlier+ # executions changed, and every product of the type is in the right state+ # by now regardless of which execution put it there.+ def settle_bookkeeping+ category_ids = @product_type.category_ids+ return if category_ids.empty?++ service = Spree::ProductTypes::ApplyToProducts.new+ # Per-product counters batch — the whole point of the cursor is not+ # holding a catalog-sized id list in memory.+ @product_type.products.select(:id).+ find_in_batches(batch_size: Spree::ProductTypes::ApplyToProducts::BATCH_SIZE) do |products|+ service.settle_product_counters(products.map(&:id))+ end- Spree::ProductTypes::ApplyToProducts.call(product_type: product_type)+ # The subtree-wide category recount runs once.+ service.settle_category_counters(category_ids)endendenddiff --git a/spree/core/app/services/spree/product_types/apply_to_products.rb b/spree/core/app/services/spree/product_types/apply_to_products.rbindex 36861735d6..b059183de2 100644--- a/spree/core/app/services/spree/product_types/apply_to_products.rb+++ b/spree/core/app/services/spree/product_types/apply_to_products.rb@@ -6,6 +6,12 @@ module Spree## Additive and idempotent: products keep everything they already have, so a# re-run after a crash is safe.+ #+ # `ApplyToProductsJob` drives this in id order with a resumable cursor, so+ # the entry points are deliberately separate: {#call} runs the whole+ # backfill in one pass (console, specs, small catalogs), while+ # {#apply_batch} plus {#settle_product_counters} / {#settle_category_counters}+ # let the job checkpoint between batches.class ApplyToProductsprepend ::Spree::ServiceModule::Base@@ -14,10 +20,7 @@ module Spree# @param product_type [Spree::ProductType]# @return [Spree::ServiceModule::Base::Result] count of products changeddef call(product_type:)- option_type_ids = product_type.option_type_ids- category_ids = product_type.category_ids-- return success(0) if option_type_ids.empty? && category_ids.empty?+ return success(0) if nothing_to_apply?(product_type)changed_product_ids = Set.newcategorized_product_ids = Set.new@@ -25,21 +28,59 @@ module Spree# Only ids are ever needed — hydrating products would run their# after_initialize callbacks for nothing.product_type.products.select(:id).find_in_batches(batch_size: BATCH_SIZE) do |products|- product_ids = products.map(&:id)+ result = apply_batch(product_type, products.map(&:id))+ changed_product_ids.merge(result[:changed_ids])+ categorized_product_ids.merge(result[:categorized_ids])+ end++ settle_category_bookkeeping(categorized_product_ids.to_a, product_type.category_ids)++ success(changed_product_ids.size)+ end++ # @return [Boolean] true when the type defines nothing to seed+ def nothing_to_apply?(product_type)+ product_type.option_type_ids.empty? && product_type.category_ids.empty?+ end++ # One batch of the backfill. Split out so the job can checkpoint between+ # batches; safe to re-run on the same ids after an interruption.+ #+ # @return [Hash] :changed_ids and :categorized_ids for this batch+ def apply_batch(product_type, product_ids)+ changed = add_option_types(product_ids, product_type.option_type_ids)+ categorized = add_categories(product_ids, product_type.category_ids)- changed_product_ids.merge(add_option_types(product_ids, option_type_ids))+ { changed_ids: changed | categorized, categorized_ids: categorized }+ end++ # insert_all skips the ProductCategory callbacks, so the per-product+ # counts they maintain are recomputed here. Safe to call per batch.+ def settle_product_counters(product_ids)+ return if product_ids.empty?- newly_categorized = add_categories(product_ids, category_ids)- categorized_product_ids.merge(newly_categorized)- changed_product_ids.merge(newly_categorized)+ counts = Spree::ProductCategory.where(product_id: product_ids).group(:product_id).count+ counts.each do |product_id, count|+ Spree::Product.where(id: product_id).update_all(categories_count: count)end- # Counter caches, branch counts and reindexing are settled once for the- # whole run rather than per category per batch — each is a subtree-wide- # recount or a job per product.- settle_category_bookkeeping(categorized_product_ids.to_a, category_ids)+ Spree::Product.where(id: product_ids).find_each(&:enqueue_search_index)+ end- success(changed_product_ids.size)+ # The category side settles once for the whole run: `products_count` is a+ # subtree-wide recount over each category and its ancestors, so repeating+ # it per batch would cost more than the backfill.+ def settle_category_counters(category_ids)+ return if category_ids.empty?++ Spree::Category.recalculate_products_count(category_ids)+ Spree::Category.where(id: category_ids).touch_all+ end++ # Both halves, for callers that run the backfill in one pass.+ def settle_category_bookkeeping(product_ids, category_ids)+ settle_product_counters(product_ids)+ settle_category_counters(category_ids) if product_ids.any?endprivate@@ -105,22 +146,6 @@ module SpreeSpree::ProductCategory.insert_all(rows)rows.pluck(:product_id).uniqend-- # insert_all skips the ProductCategory callbacks, so the counts they- # maintain are recomputed here — once, for every product touched.- def settle_category_bookkeeping(product_ids, category_ids)- return if product_ids.empty?-- counts = Spree::ProductCategory.where(product_id: product_ids).group(:product_id).count- counts.each do |product_id, count|- Spree::Product.where(id: product_id).update_all(categories_count: count)- end-- Spree::Category.recalculate_products_count(category_ids)- Spree::Category.where(id: category_ids).touch_all-- Spree::Product.where(id: product_ids).find_each(&:enqueue_search_index)- endendendenddiff --git a/spree/core/spec/jobs/spree/product_types/apply_to_products_job_spec.rb b/spree/core/spec/jobs/spree/product_types/apply_to_products_job_spec.rbnew file mode 100644index 0000000000..3d10c9f751--- /dev/null+++ b/spree/core/spec/jobs/spree/product_types/apply_to_products_job_spec.rb@@ -0,0 +1,81 @@+require 'spec_helper'+require 'active_job/continuation/test_helper'++RSpec.describe Spree::ProductTypes::ApplyToProductsJob, type: :job do+ let(:store) { @default_store }+ let(:option_type) { create(:option_type, name: 'size') }+ let(:category) { create(:category) }+ let(:product_type) { create(:product_type, option_types: [option_type], categories: [category]) }++ # update_column so the seeding callback doesn't do the job's work for it.+ def product_carrying_the_type+ create(:product).tap { |product| product.update_column(:product_type_id, product_type.id) }+ end++ it 'applies the type to its products' do+ product = product_carrying_the_type++ described_class.perform_now(product_type.id)++ expect(product.reload.option_types).to eq([option_type])+ expect(product.categories).to eq([category])+ end++ it 'settles the category counter cache' do+ product_carrying_the_type++ described_class.perform_now(product_type.id)++ expect(category.reload.products_count).to eq(1)+ end++ it 'does nothing for a type that defines neither option types nor categories' do+ bare_type = create(:product_type)+ product = create(:product)+ product.update_column(:product_type_id, bare_type.id)++ described_class.perform_now(bare_type.id)++ expect(product.reload.option_types).to be_empty+ end++ it 'ignores a deleted product type' do+ expect { described_class.perform_now('nonexistent') }.not_to raise_error+ end++ describe 'interruption and resume' do+ include ActiveJob::Continuation::TestHelper++ around do |example|+ original = ActiveJob::Base.queue_adapter+ ActiveJob::Base.queue_adapter = :test+ example.run+ ensure+ ActiveJob::Base.queue_adapter = original+ end++ # One product per batch, so the cursor advances mid-run.+ before { stub_const('Spree::ProductTypes::ApplyToProducts::BATCH_SIZE', 1) }++ it 'resumes from the cursor instead of restarting the backfill' do+ first = product_carrying_the_type+ second = product_carrying_the_type++ described_class.perform_later(product_type.id)++ interrupt_job_during_step(described_class, :apply_to_products, cursor: first.id) do+ perform_enqueued_jobs+ end++ expect(first.reload.option_types).to eq([option_type])+ expect(second.reload.option_types).to be_empty++ perform_enqueued_jobs++ expect(second.reload.option_types).to eq([option_type])+ # Re-running the finished batch adds nothing — the backfill is additive.+ expect(first.reload.option_types).to eq([option_type])+ expect(category.reload.products_count).to eq(2)+ end+ end+end