Open Source Platform for DTC, B2B Commerce, Marketplaces & Omnichannel. REST APIs, TypeScript SDKs, and production-ready Next.js storefront. Self-host it. Own your data. No vendor lock-in. Zero platform fees.
Ruby b2b-commercee-commerceecommerceecommerce-apiecommerce-frameworkecommerce-platformheadlessheadless-commerceheadless-ecommercemarketplacemulti-tenantmulti-vendormulti-vendor-ecommerceopen-sourcespree-commerce
Plan: Solid Queue as the default job backend — single-container, Postgres-only (#14325)
Damian Legawiece28ee384ce38
diff --git a/docs/plans/5.6-solid-queue-single-container.md b/docs/plans/5.6-solid-queue-single-container.mdnew file mode 100644index 0000000000..ea5e355525--- /dev/null+++ b/docs/plans/5.6-solid-queue-single-container.md@@ -0,0 +1,99 @@+# Solid Queue as the Default Job Backend — Single-Container, Postgres-Only Deployment++**Status:** Done (starter + CLI + docs implemented — spree-starter#1368, spree#14323)+**Target:** Spree 5.6 (spree-starter + tooling change only — no core gem dependency; can ship in a 5.6.x starter update if RC feedback warrants)+**Depends on:** Single-node Docker deployment (spree-starter#1368 + spree#14323, shipped); `6.0-stock-reservations.md` (consumes the recurring-jobs mechanism this plan standardizes)+**Author:** Damian Legawiec (design discussion with Claude, 2026-07-18)+**Last updated:** 2026-07-18 (implementation landed same day; verified on the built image)++## Summary++Replace Sidekiq + Redis with the Rails 8 Solid stack in spree-starter, making the default Spree deployment **one process, one container, one datastore**: Puma runs the Solid Queue supervisor in-process (`plugin :solid_queue`), jobs live in Postgres (which Spree already requires), Solid Cache replaces the Redis cache store, and Solid Cable replaces the Redis Action Cable adapter. Redis disappears from the default stack entirely — dev compose, production compose, and the Render Blueprint each lose a service. This is the "ONCE"/Rails 8 deployment model, and it beats the simplest stories in the category: Medusa's `workerMode: "shared"` still wants Redis in production; Vendure's DB-backed JobQueue is the same bet without the in-process default; Saleor has no single-process story at all. Scale-out is a config flag, not a code change.++## Key Decisions (do not deviate without discussion)++- **Solid Queue is the default adapter in spree-starter; Sidekiq remains fully supported.** Core is adapter-agnostic (audited 2026-07-18 — see Design Details) and stays that way. Existing projects keep whatever their repo configures; nothing in the gems forces a migration.+- **In-Puma supervisor is the default topology.** `puma.rb` gets `plugin :solid_queue if ENV.fetch("SOLID_QUEUE_IN_PUMA", "true") == "true"`. The escape hatch for scale is `SOLID_QUEUE_IN_PUMA=false` on web plus a worker service running `bin/jobs` **from the same image** — one env var and one service definition, no rebuild.+- **Single database by default; the queue-DB split is a documented scaling step, not the default.** Single-DB gives commerce the property that matters: `perform_later` inside a transaction commits atomically with the business data ("order created → confirmation email" cannot race). The Rails 8 two-DB convention (`spree_production_queue`, same Postgres server) is documented for high-churn installs alongside the worker split.+- **Full no-Redis:** Solid Cache for `Rails.cache`, Solid Cable for Action Cable (the legacy admin's import live-feed broadcasts still need a cable adapter until the engine is removed in 6.0). Meilisearch stays — it is a search engine, not incidental infra.+- **Solid Cache lives in the primary database — no separate cache DB.** Headless reframing: the Russian-doll fragment caching that justified Redis-sized caches died with the ERB storefront, and the legacy admin's partial caching dies in 6.0. What remains is ~20 low-level `Rails.cache.fetch` sites (taxon descendant ids, states helper, the preferences store) plus rate-limit counters — a workload where splitting databases is over-engineering. Redis/Valkey stays a documented one-line swap (`config.cache_store = :redis_cache_store`) for high-traffic installs; wherever Redis remains in docs, say "Redis or Valkey" (post-relicensing, platforms steer to Valkey). It's a package deal — no half-migration that keeps Sidekiq while adopting Solid Cache; the payoff is deleting the Redis requirement entirely.+- **Mission Control – Jobs replaces the `/sidekiq` UI** (`mission_control-jobs`, mounted at `/jobs`). Guarded by Mission Control's own HTTP Basic auth (`MISSION_CONTROL_USER` / `MISSION_CONTROL_PASSWORD`, seed-admin defaults in dev/test, Render Blueprint generates the password) — deliberately **not** the admin Devise session, which `6.0-platform-auth.md` removes from the Rails app.+- **Recurring jobs standardize on Solid Queue's `config/recurring.yml`** — this is the mechanism `6.0-stock-reservations.md` should name for `ExpireJob` (it currently hedges "sidekiq-cron / solid_queue").++## Design Details++### Core audit (2026-07-18) — why the gems need zero changes++- **No Sidekiq-specific APIs anywhere in `spree/core`, `spree/api`, `spree/admin`, `spree/emails`**: no `Sidekiq::` constants, no batches, no unique-jobs middleware, no server middleware. The only mentions are comments.+- **Scheduling is plain ActiveJob**: five `set(wait: 2.seconds..30.seconds)` sites (product taxon auto-match, import row fan-out, theme duplication, payment webhook retry). Solid Queue's dispatcher handles scheduled jobs natively at this granularity.+- **`Spree::BaseJob`'s `retry_on ActiveRecord::RecordNotFound` exists to absorb the enqueue-before-commit race** — which single-DB Solid Queue *eliminates* (the enqueue commits with the transaction; Solid Queue ≥1.2 exposes `enqueue_after_transaction_commit` to control this). The retry stays for adapter-agnosticism, but the default deployment no longer exercises it.+- **No Redis usage in core**: the events system is explicitly adapter-abstracted; no advisory locks, no Redlock, no direct `Redis.new`.+- **Action Cable surface**: `Spree::Import` + the legacy admin's import subscribers broadcast Turbo Streams (live import feed). The React Dashboard polls instead (by design — see `5.6-admin-spa-csv-import.md`), so this coupling dies with the legacy admin in 6.0; until then Solid Cable carries it.+- **No `sidekiq-cron`/schedule file exists today** — there is no recurring-job migration, only a green field for `recurring.yml`.+- **Extensions are clean too** (confirmed by Damian, 2026-07-18): all extensions — including `spree_stripe`, `spree_adyen`, `spree_paypal_checkout` — use plain ActiveJob with no Sidekiq-specific APIs.+- **Cache audit**: 20 `Rails.cache` call sites across core+api, all small-value `fetch` memoization — no fragment caching in the API surface. The highest-frequency cache operation is **rate limiting**: the v3 API uses Rails 8's built-in `rate_limit` with `store: Rails.cache` on the base controller (per-key, every request) plus tighter limits on login/refresh/register/password-reset/newsletter/webhooks. Verified (2026-07-18): Solid Cache implements atomic `increment`, and Rails' `rate_limit` was deliberately built over the generic `ActiveSupport::Cache` contract (rails/rails#50781) — the combination is supported and correct across processes. Cost model: one Postgres `UPDATE` per API request; fine for the median install, and the swap knob exists for hot paths.++### spree-starter changes++| File | Change |+|---|---|+| `Gemfile` | − `sidekiq`, `sentry-sidekiq`, `redis`; + `solid_queue`, `solid_cache`, `solid_cable`, `mission_control-jobs` (sentry-rails already instruments ActiveJob) |+| `config/application.rb` | `config.active_job.queue_adapter = :solid_queue` |+| `config/puma.rb` | `plugin :solid_queue if ENV.fetch("SOLID_QUEUE_IN_PUMA", "true") == "true"` |+| `config/environments/production.rb` | `config.cache_store = :solid_cache_store` (drop the `REDIS_CACHE_URL` branch); `config.solid_queue.connects_to` **not** set (single-DB default) |+| `config/cable.yml` | `adapter: solid_cable` (dev + production) |+| `config/database.yml` | Pool sizing comment + formula: Puma threads + Solid Queue worker threads + ~3 (dispatcher/scheduler/supervisor). This is the one place in-Puma mode bites people — pool exhaustion that presents as a Puma problem. |+| `config/routes.rb` | `mount MissionControl::Jobs::Engine => "/jobs"` behind admin auth, replacing `Sidekiq::Web` |+| `db/` | Solid Queue/Cache/Cable schemas in the main migrations path (single-DB) |+| `config/recurring.yml` | Empty scaffold with a commented example (ready for stock-reservation expiry etc.) |+| `docker-compose.yml` / `docker-compose.dev.yml` | Drop the `redis` service and the `worker` service; drop `REDIS_URL` env; web is the only app container |+| `render.yaml` | Web service unchanged (Docker, repo-root context); redis service deleted; the commented worker block becomes the scale-out example: same image, `dockerCommand: bundle exec bin/jobs`, `SOLID_QUEUE_IN_PUMA=false` on web |+| `.env.example` | − `REDIS_URL`, `SIDEKIQ_*`; + `SOLID_QUEUE_IN_PUMA` (documented, default true) |++### Deployment shapes++```+Default (create-spree-app, docker run, Render free tier):+┌─────────────────────────────┐ ┌──────────────┐+│ app container │ │ Postgres │+│ Puma │─────▶│ (one DB) │+│ └─ plugin :solid_queue │ └──────────────┘+│ ├─ dispatcher │+│ ├─ worker(s) │+│ └─ scheduler │+└─────────────────────────────┘++Scale-out (flag flip, same image):+web: SOLID_QUEUE_IN_PUMA=false → Puma only+worker: bundle exec bin/jobs → Solid Queue supervisor only+db: optionally spree_production_queue (second DB, same server)+ via config.solid_queue.connects_to + db/queue_migrate+```++### CLI / create-spree-app changes++- `spree dev` runs `docker compose up web worker` today — with no `worker` service it must detect the compose shape (worker service present → include it; absent → web only). Same for `primeBundleVolume`'s assumptions and `spree logs worker` (error message pointing at Mission Control / `SOLID_QUEUE_IN_PUMA` instead).+- Setup summary/README/docs drop the Sidekiq UI link; add `/jobs` (Mission Control).+- `SIDEKIQ_DB_POOL` env handling removed; pool guidance moves to the database.yml comment.+- Generated project docs (README/CLAUDE.md templates) describe the single-process model and the two scale-out steps.++### Competitor context (for docs/marketing framing)++- **Medusa**: `workerMode: "server" | "worker" | "shared"` — shared in one process for dev/simple deploys, but production guidance splits and the event bus/queue want Redis. Spree's story: same simplicity, Postgres-only.+- **Vendure**: explicit `bootstrap()` vs `bootstrapWorker()` split; default JobQueue is DB-backed (same bet), BullMQ/Redis as the production-grade plugin.+- **Saleor**: Django + Celery + Redis broker, mandatory separate workers — no single-process story.++## Migration Path++1. **spree-starter PR**: the table above, in one change. CI: boot + a job-exercising smoke (e.g. seed triggers `AutoMatchTaxonsJob`; assert it executes in-Puma).+2. **CLI + create-spree-app PR**: compose-shape detection in `spree dev`/`logs`, template/doc updates. Backward-compatible — existing Sidekiq-shaped projects keep working (worker service detected and included).+3. **Docs**: deployment guide gains the two scale-out steps (worker split, queue-DB split); environment-variables page updated.+4. **Existing projects**: no forced migration. A docs recipe covers switching (add gems, flip adapter, drain Sidekiq queues before cutover — Sidekiq jobs don't transfer).+5. **6.0**: `6.0-stock-reservations.md` names `recurring.yml` for `ExpireJob`; legacy-admin removal deletes the last Action Cable broadcasts (Solid Cable stays for storefront/custom use).++## Constraints on Current Work++- **Keep core adapter-agnostic**: no Sidekiq-specific APIs (batches, unique-jobs, middleware) and no direct Redis usage in any Spree gem. New jobs inherit `Spree::BaseJob` and use plain ActiveJob scheduling (`set(wait:)`).+- **New recurring-work designs should name `config/recurring.yml`** as the mechanism, not sidekiq-cron/whenever.+- **Don't add Redis-backed features to the starter** (e.g. Rack::Attack with a Redis store) without routing them through `Rails.cache`.+- CLI work touching `spree dev`/compose should not hard-code the `worker` service name — detect it.diff --git a/docs/plans/6.0-stock-reservations.md b/docs/plans/6.0-stock-reservations.mdindex 80e94bc60a..efa4d23d2e 100644--- a/docs/plans/6.0-stock-reservations.md+++ b/docs/plans/6.0-stock-reservations.md@@ -236,7 +236,7 @@ module Spreeend```-Scheduled via `sidekiq-cron` or `solid_queue` recurring task — runs every 1 minute.+Scheduled via Solid Queue's `config/recurring.yml` — runs every 1 minute. (Mechanism standardized by `6.0-solid-queue-single-container.md`; sidekiq-cron is not part of the default stack.)### Integration points (5.5 — Store API only)