Skip to content

armgit / spree

Public Imported from GitHub
0

Open Source Platform for DTC, B2B Commerce, Marketplaces & Omnichannel. REST APIs, TypeScript SDKs, and production-ready Next.js storefront. Self-host it. Own your data. No vendor lock-in. Zero platform fees.

23.8 KB
Raw Blame

Preferences Stored as JSON, Secrets Encrypted

Status: In Progress — everything this plan targets for 6.0 is built (JSON coder, encrypted secret column, conversion migration, upgrade tasks, tiered calculators, docs); waiting on review and CI Target: Spree 6.0 Depends on: nothing. Follows the precedent of the metadata consolidation (20260809120001_consolidate_metadata_columns.rb) Author: Damian + Claude Last updated: 2026-10-01 (implementation) Tracking: Linear V-3730 · GitHub spree/spree#14786

Summary

Every model's preferences column is a YAML document, written by one line in Spree::Preferences::Preferable (serialize :preferences, type: Hash, coder: YAML). This plan moves them to a JSON column (jsonb on PostgreSQL, json on MySQL and SQLite) in 6.0, converting the stored data during the 6.0 upgrade. At the same time, preferences declared :password (gateway and integration secrets) move out of preferences into an encrypted secret_preferences column, so the main column never holds a secret. Declarations do not change: preference :secret_key, :password and preferred_secret_key work as before.

Reasons, in order: gateway secrets — including live Stripe secret keys — sit in the database in plain text today; loading YAML is an attack surface Rails has to fence with yaml_column_permitted_classes, a list Spree injects into every application and every Rails upgrade touches; and two tables built in 6.0 already declare preferences as JSON but, through the shared coder, hold a YAML document wrapped in a JSON string. Being able to query or index preferences is a side benefit only — nothing queries them today. 6.0 is a clean-break release and already carries a data-converting migration of the same shape, which makes it the right release for this.

Inventory

Measured on main at 2026-10-01. Smaller than expected: there is no log entry table any more, and global configuration (Spree::Config) is held in memory and environment variables, not in the database.

Column Today Notes
preferences on spree_calculators, spree_payment_methods, spree_promotion_rules, spree_stores, spree_integrations, spree_imports, spree_exports, spree_price_rules, spree_channels, spree_order_routing_rules, spree_seller_requirements text, YAML The main body of the work
preferences on spree_delivery_method_rules, spree_commission_rules jsonb/json column holding a YAML string Already the right column type; only the data is wrong
spree_preferences.value text, YAML One live row: the installation id. The table is dropped without copying it; the default store generates a new id in an internal install_id preference when it is next saved (settled during implementation)
spree_gateways.preferences text Leftover 4.3 table, no model reads it. Not touched: the migration converts only tables a model storing Spree preferences reads, so a table another library owns is never changed
Spree::Address serialize :preferences declaration only spree_addresses has no such column. Kept, with the JSON coder: removing it changes the address attribute set that duplicate detection compares

metadata is already JSON (Spree::Metadata::HashSerializer) and is out of scope. No provider gem (spree_stripe, spree_easypost, spree_meilisearch) has a preferences table of its own.

Values found in the YAML today, by declared preference type (counted across core and providers): string 69, boolean 58, integer 54, decimal 36 (stored as BigDecimal), password 7 (Stripe publishable key, secret key and two webhook signing secrets; EasyPost API key and webhook secret; the test gateway's key), array 16, hash 4, any 3, date 2 (already stored as a yyyy-MM-dd string), text 1. No datetime preference is declared anywhere. Keys are symbols.

Key Decisions (do not deviate without discussion)

  1. One JSON coder replaces YAML in Preferable. It writes a plain JSON object with string keys and loads it as a HashWithIndifferentAccess, so preferences[:amount], fetch(:amount) and the generated preferred_* readers keep working unchanged.
  2. Decimals and times are restored on read from their declared type (narrowed 2026-10-01 during implementation). A :decimal preference read from JSON comes back as a BigDecimal and a :datetime one as a time, exactly as they did from YAML; both are stored as their exact JSON strings. Only these two: every other declared type survives JSON unchanged, and running the full writer cast on read would turn a stored nil into "" or 0 for non-nullable strings and numbers. The preference declarations are the schema; no second schema is introduced.
  3. Symbols are not stored. Keys become strings; symbol values become strings. Code comparing preference values to symbols must compare strings.
  4. The conversion runs inside the 6.0 migration, not in an upgrade task. spree:upgrade runs its manifest after db:migrate (see upgrade.rake and the CLI's upgrade command), so a backfill scheduled there would find the YAML column already replaced. This is the reasoning ConsolidateMetadataColumns recorded for the same problem. The proposal's "add a column, backfill with a task, swap later" order does not work with Spree's upgrade runner.
  5. The migration never uses models. It reads rows with select_all, parses with YAML.safe_load and an explicit permitted-class list, and writes with exec_update. A model would load through whichever coder the current code declares, which is the thing being changed. The one thing it asks a class is which of its preferences are :password (Decision 12), from the declaration alone, without loading a record.
  6. No batching or background job. Every table in the inventory is configuration-sized (one row per calculator, rule, gateway, store, import). The append-only log table the proposal anticipated no longer exists.
  7. A row still holding YAML fails when it is read, not at boot (decided 2026-10-01). A YAML document cannot be loaded as a hash, so reading that record raises, and the spree:upgrade:preferences_json task finds and converts any leftovers. A dedicated error naming the task was built first and then dropped with the custom coder (see Coder below): the migration converts every row it can read and stops on any it cannot, so the error only covered rows written by pre-6.0 code after the migration ran. A boot check was rejected: boot also happens where there is no database (asset builds) and before the converting migration has run, so it would block the very migration that repairs the data, and it would cost a query per table on every start. Accepted trade-off: a missed row breaks the requests that touch that record rather than the whole application, so the failure is loud but local.
  8. Decimals are stored as exact strings (decided 2026-10-01): {"amount": "9.99"}, which is what BigDecimal#as_json produces and what the API already returns. The writer keeps that same string in memory (as do tier ladders), so a record just written and one read back hold identical values and setting a decimal to the value it already has is not a change; the preferred_* reader returns the BigDecimal. A JSON number was rejected because Ruby and JavaScript parse it as a float before the read cast can restore it. Accepted trade-off: a numeric comparison in SQL needs an explicit cast ((preferences->>'amount')::numeric).
  9. Tiered calculators store tiers as a list, not a hash with number keys (decided 2026-10-01). Spree::Calculator::TieredPercent and TieredFlatRate change preference :tiers, :hash to preference :tiers, :array of { "threshold", "value" } objects, sorted by threshold. One value key for both calculators (settled during implementation, after review found the dashboard's tiers editor cannot tell which calculator it is editing): the calculator's type already says whether the value is a percentage or an amount, and per-calculator keys would have needed a new field in the preference schema. JSON object keys are always strings, so the hash shape could only survive with a hidden cast; a list reads naturally in JSON and TypeScript. This changes the tiers value these calculators show through the Admin API, which is a breaking change listed in the upgrade guide. The migration converts the shape. Dropping both calculators was considered and rejected (2026-10-01): they discount by spend, while price-list and catalog volume pricing works per variant by quantity, so they are not duplicates; banded promotions with the item-total rule could replace them, but only as several promotions for what is one today.
  10. Secrets live in their own encrypted column, secret_preferences (decided 2026-10-01). A :password preference is read from and written to secret_preferences, a text column holding a JSON object and declared with Rails' standard encrypts. Everything else stays in preferences. The declaration DSL, the preferred_* methods, masking and PreferenceSchema are unchanged — only where the value is stored moves. Rejected: encrypting each secret inside the preferences JSON (it would call Rails' encryptor by hand and rebuild key rotation, plain-text fallback and log filtering that encrypts already provides, and secrets would still travel wherever preferences is exported or logged); and a polymorphic Spree::Credential model (an extra query per gateway load, preference dirty tracking that must look across a child record, secrets silently lost when a record is duplicated — a credential lifecycle nothing needs yet; it can be built later on top of this without touching declarations again).
  11. Encryption follows Spree's existing rule: on whenever keys are configured. New projects get Active Record encryption keys at setup, so their secrets are encrypted from the first save. An installation without keys keeps working with secrets in plain text inside secret_preferences, and gets the production boot warning Spree already logs for its other encrypted fields. This is the same condition Spree::WebhookEndpoint, Spree::GatewayCustomer and Spree::UserIdentity use.
  12. Existing secrets are moved by the migration and encrypted by an upgrade task; nothing breaks in between. The migration copies each :password value out of preferences into secret_preferences as plain JSON — it needs no encryption keys and so can never fail for lack of them. encrypts :secret_preferences, support_unencrypted_data: true keeps those plain values readable (the precedent is Spree::UserIdentity), so every gateway keeps working the moment the migration finishes. The spree:upgrade:encrypt_secret_preferences task then encrypts every row that is still plain; without it a row is encrypted on its next save anyway. A secret's reader looks in preferences first and then in secret_preferences (settled during implementation): a secret assigned as part of a whole hash waits there until before_save moves it, and a secret of a class the migration could not load stays readable until spree:upgrade:preferences_json moves it. The writer always writes the encrypted column and removes any copy from preferences.

Design Details

Coder

No Spree-specific coder (settled during implementation). The preferences column is a native JSON column, which Rails already converts to and from a hash, so the only coder is Spree::Metadata::HashSerializer, the one metadata uses, whose sole job is to let the loaded hash answer to symbol and string keys alike. Without it, a symbol lookup on the raw hash (calculator.preferences[:amount], common in Spree and in applications) would quietly return nil instead of the value. Decimals are kept as exact strings by the preference writer, so the coder does not need to normalize them.

serialize :preferences, coder: Spree::Metadata::HashSerializer

as_json turns BigDecimal into its exact string ("9.99"), Date into ISO 8601 and symbols into strings, so dump never hands the adapter a Ruby-only type. Rails' JSON column type has already decoded the value before load sees it, on every adapter, so a string reaching load can only be a leftover YAML document.

Reads cast through the declared type

define_method preference_getter_method(name) do
  stored = (preferences || {}).fetch(name) { return default.call }
  convert_preference_value(stored, type, nullable: nullable)
end

convert_preference_value is idempotent for every type it handles, so casting on read changes nothing for values written in this process.

Places that assume symbol keys

Three raw reads (provision_defaults.rb, promotion/rules/country.rb twice) keep working through indifferent access. Two places compare key sets and need a stringify: Spree::Base's after_initialize backfill (default_preferences.except(*preferences.keys) would otherwise see every default as missing and dirty every record on load) and Preferable#preference_change.

Migration

One migration, ConvertPreferencesToJson, for every table in the inventory:

  1. For each table that has a preferences column whose values are YAML: add preferences_json (jsonb if the adapter has it, else json), convert each row, drop preferences, rename preferences_json to preferences.
  2. For the two tables already typed JSON: rewrite each row whose value is a string in place.
  3. Drop spree_preferences. The installation id is not copied; rolling back recreates the table empty.

Conversion per row: YAML.safe_load(text, permitted_classes: [Symbol, BigDecimal, Date, Time, ActiveSupport::TimeWithZone, ActiveSupport::TimeZone, ActiveSupport::HashWithIndifferentAccess], aliases: true) — the list Spree's engine already permits, plus Date — then as_json. Tables are discovered from the schema (any table with a preferences column), as the metadata migration does, so an application's own Preferable table converts too.

A row that does not parse — typically an old !ruby/hash-with-ivars:ActionController::Parameters dump, which the current code already cannot load — stops the migration with the table name and row id, rather than being converted to {}.

Secret preferences

# PreferableClassMethods — routing by declared type; the DSL is untouched
store_accessor(type == :password ? :secret_preferences : :preferences, name, prefix: :preferred)

# A model whose table has the column
encrypts :secret_preferences, support_unencrypted_data: true if ActiveRecord::Encryption.config.has_primary_key?
serialize :secret_preferences, coder: Spree::SecretPreferences::TextHashSerializer # JSON in a text column, then HashSerializer
  • Which tables get the column: spree_payment_methods and spree_integrations, the only tables whose classes declare :password preferences today. A model opts in by including Spree::SecretPreferences; an application's own table adds the column itself.
  • Declaring a secret on a model that has not opted in raises ArgumentError at declaration time, naming the concern and the column to add. A secret is never silently written to the plain column.
  • A secret's default is never copied into storage. The backfill that fills in missing defaults on load skips secrets; their defaults are read through the preferred_* reader. Masking reads only the stored value, so an unset secret never reveals its default.
  • Gateway connection options (Spree::Gateway#options) merge both columns and every secret's reader, so gateways still receive their keys.
  • An empty secret_preferences column reads as nil, not {}. Encrypted attributes compare the stored value with the loaded one, so an empty hash would mark every record without secrets as changed the moment a secret was read.
  • Declarations stay current down the class tree. Each class keeps only its own declarations and merges its ancestors' on demand, cached until the next declaration anywhere — so a secret a decorator adds to Spree::PaymentMethod after its subclasses loaded is still routed to the encrypted column.
  • Code that writes preferences without callbacks writes both columns. update_columns(preferences: ...) skips the save hook that moves secrets, so a writer like Stripe's migrate_stripe_webhook_keys sets values through the preferred_* writers and then writes preferences and secret_preferences together.
  • The safety-net task converts a column that is still text — a table whose model the migration did not load — with the same steps as the migration, instead of pointing back at a migration that has already run.
  • Each preference is a Rails store accessor (decided 2026-10-01, at the author's request, in the same pull request). preference registers a store accessor on preferences (or secret_preferences for a secret) with the preferred prefix, so Rails supplies the change methods; the hand-written ones are gone, and their after-save forms take Rails' names (saved_change_to_preferred_<name>?, preferred_<name>_before_last_save). One table of definitions per class (preference_definitions) replaces the five helper methods every preference generated and the methods.grep that found them. The raw preferences hash still holds every default (the author's requirement), so the backfill on load stays. activerecord-typedstore was considered and rejected: it wants all fields declared in one block per column, fixes defaults at class load, and would still need Spree's layer for parse_on_set, nullable values, choices and deprecations.
  • Rows of an uninstalled class are skipped by both upgrade tasks, with a line naming the class, so one leftover gateway from a removed gem cannot stop the run before the other rows. Their secrets stay readable where they are.
  • Change detection across both columns: previously_changed_preference_names lists what the last save changed in either column. Stripe uses it to recognise a save that only stored webhook registration results.
  • preferences is the only column the Admin API, exports and event payloads read. Secrets reach the API only through the existing masking, which now reads through preferred_* rather than the raw hash.
  • Copying a record (dup) carries secret_preferences along with preferences, so duplicating a payment method keeps its keys.
  • Which keys are secrets comes from the class declarations (declared_preference_types, recorded per class when a preference is declared, so no record is built). The migration resolves each row's class from its type column; a row whose class cannot be loaded (an uninstalled gem) keeps its secrets in preferences and is reported, and the safety-net task moves them once the class is back.

Tiered calculators

# Before: { 100.0 => 10.0, 250.0 => 15.0 }
preference :tiers, :array, default: []
# After:  [{ "threshold" => "100.0", "value" => "10.0" }, { "threshold" => "250.0", "value" => "15.0" }]

compute picks the last tier whose threshold the amount reaches. Validation checks that each entry has a positive threshold and, for the percent calculator, a rate from 0 to 100, and that thresholds are unique. The migration converts the hash for rows whose type is either calculator or an application's subclass of one, keeping the decimal-string rule above. The dashboard's tiers editor (preferences-form.tsx in @spree/dashboard-core) reads and writes the list.

Afterwards

yaml_column_permitted_classes injection is removed from Spree::Core::Engine and from the dummy app generator. The comments in preferable.rb and base.rb that explain YAML behaviour go with it.

Migration Path

  1. Change the tiered calculators to the list shape (code and specs).
  2. Switch Preferable to JSON through Spree::Metadata::HashSerializer; restore decimals on read; stringify the two key-set comparisons; delete the dead serialize on Spree::Address.
  3. Add the secret_preferences column and routing for :password preferences; raise when a secret is declared on a table without it; read masking through the preferred_* methods.
  4. Add the ConvertPreferencesToJson migration, including the tiers reshaping and moving secrets into secret_preferences, with a reversible down (JSON back to YAML restores every value through the declared types; tiers convert back to the hash).
  5. Add two steps to the 5.6→6.0 manifest: spree:upgrade:preferences_json (safety net: reports and converts any row still holding YAML or a secret left in preferences) and spree:upgrade:encrypt_secret_preferences (encrypts every plain secret_preferences row; reports and skips when no keys are configured). Both idempotent.
  6. Remove the YAML permitted-classes configuration, Spree::Preference, Spree::Preferences::Store and Spree::Preferences::ScopedStore.
  7. Document the change in docs/developer/upgrades/5.6-to-6.0.mdx (including that gateway secrets are now encrypted when keys are set, and linking the existing encryption-keys section), list gateway secrets among encrypted values in docs/developer/deployment/environment_variables.mdx, and update docs/developer/customization/model-preferences.mdx (keys are strings in the raw hash; values come back typed through the preferred_* readers). (docs/developer/core-concepts/promotions.mdx names the tiered calculators but shows no tiers, so it needed no change.)
  8. Specs: a coder round trip for every declared type, secrets landing only in secret_preferences (encrypted with keys, plain without), a moved plain secret staying readable, the encrypt task, the migration on each adapter, the existing preference specs passing unchanged, and the tiered calculator specs moved to the list shape.

Constraints on Current Work

  • New preferences columns are JSON. Use the t.respond_to?(:jsonb) guard from the root CLAUDE.md. Do not add another text preferences column.
  • Read preferences through preferred_* or get_preference, not the raw hash. The raw hash is where YAML and JSON differ; the readers hide it.
  • Do not store symbols, BigDecimal or Time objects in a preference you expect back unchanged without a declared type that casts it.
  • Every secret is a :password preference. That type is what sends a value to the encrypted column. A secret declared :string, or nested inside a :hash or :array preference, stays in plain text.
  • New gateway or integration tables need a secret_preferences text column if their classes declare :password preferences.
  • Never write preferences with update_columns/update_all when it may hold a secret. Those skip the hook that moves secrets to the encrypted column. Set values through the preferred_* writers and write both preferences and secret_preferences.
  • No number keys in :hash preferences. JSON turns them into strings. Use a list of objects, as the tiered calculators now do.
  • Do not query preferences in SQL until this lands. On the two JSON-typed tables the value is a YAML string today and any JSON operator silently matches nothing.

Open Questions

None blocking. Deferred:

  • A credential lifecycle (rotation history, last used, expiry, its own permission). Nothing needs it yet; it can be built on secret_preferences without changing declarations.
  • Dropping spree_gateways, the unused 4.3 table. Unblocked by confirming no extension still reads it; a candidate for 6.1 cleanup.

References

  • docs/plans/decisions.md — 2026-08-09 metadata consolidation (the migration precedent) and the Stripe port entry naming encryption of preferences as a live question
  • docs/plans/6.0-store-scoped-configuration.md — store preferences and the backfill-marker wrinkle
  • docs/developer/customization/model-preferences.mdx
  • docs/developer/deployment/environment_variables.mdx — encryption keys, enabling encryption on an existing installation
  • spree/core/app/models/spree/user_identity.rb — the plain-text-then-encrypt precedent
  • spree/core/lib/spree/core/preferences/preferable.rb, preferable_class_methods.rb
  • spree/core/db/migrate/20260809120001_consolidate_metadata_columns.rb