Skip to content

armgit / spree

Public Imported from GitHub
0

Open Source Platform for DTC, B2B Commerce, Marketplaces & Omnichannel. REST APIs, TypeScript SDKs, and production-ready Next.js storefront. Self-host it. Own your data. No vendor lock-in. Zero platform fees.

Added carts, promotions, metric filters and lifetime value to reporting (#14627)

Damian Legawiecbede249d96dd
diff --git a/docs/plans/6.0-reporting-additional-metrics.md b/docs/plans/6.0-reporting-additional-metrics.md
new file mode 100644
index 0000000000..32f5633b23
--- /dev/null
+++ b/docs/plans/6.0-reporting-additional-metrics.md
@@ -0,0 +1,481 @@
+# Reporting: Additional Metrics, Filters and Grains
+
+**Status:** Implemented — carts family, promotion vocabulary, metric filters, `include_empty`, hour grain, year-over-year, lifetime metrics and the two refusal messages all ship; seeded classic reports for the new questions are the one deferred item (see Open Questions)
+**Target:** Spree 6.0 (carts family, promotion vocabulary, metric filters, hour grain, year-over-year, lifetime value, refusal messages)
+**Depends on:** `6.0-analytics-semantic-layer.md` (the registry, contract and compiler this extends), `6.0-cart-order-split.md` (the `Spree::Cart` table the carts family reads)
+**Author:** Damian + Claude
+**Last updated:** 2026-09-12 (Decisions 11–13: abandonment ignores the reaper's money-safety carve-out, lifetime value is net of refunds, the hour cap refuses and the picker greys out)
+
+## Summary
+
+The semantic layer shipped a vocabulary that answers what merchants ask about
+completed orders. A pass over the questions merchants actually type found nine
+it cannot answer. Seven are genuine gaps in the vocabulary or the contract and
+this plan closes them: abandoned carts and cart conversion (a new **carts
+family**), sales by promotion and coupon code (a new **promotion dimension
+family** on `spree_discounts`), "which products never sold" (**metric filters**,
+a `HAVING` clause the contract has no concept of), sales by hour (**an `hour`
+grain**), "versus the same month last year" (**a `previous_year` compare
+mode**), and customer lifetime value (**a lifetime metric** whose window is
+deliberately not the report's).
+
+The remaining two are not gaps. "Total sales per product" and "revenue and
+units shipped together" are refused by design — the first would double-count
+order-level money onto line items, the second would put two different clocks in
+one table. Both refusals stay; what changes is that the 422 now explains itself
+and names the metric or the split that answers the question. A merchant hitting
+a wall should learn the vocabulary from the error, not file a bug.
+
+One question in the original list stays unanswerable and is recorded as such:
+**traffic-based conversion rate** needs sessions and referrers that a headless
+storefront never sends us. Cart-level conversion — of the carts that were
+started, how many became orders — *is* computable and ships here; it is a
+different and narrower number, and the docs must not let the two be confused.
+
+## Problem
+
+The vocabulary is anchored on one root: a completed, non-canceled order. That
+single choice is what makes the shipped reports trustworthy, and it is also
+what puts these seven questions out of reach.
+
+| Merchant asks | Today | Why it is out of reach |
+|---|---|---|
+| "Abandoned carts" | No metric | Every sales base filters `completed_at` in range; a cart has no `completed_at` and lives in a different table |
+| "Cart conversion rate" | No metric | Same — needs the cart as a root, with the order as the optional join |
+| "Sales by promotion or coupon code" | No dimension | `discounts` is money on a line item with no attribution axis |
+| "Which products have never sold?" | Impossible | `FILTER_OPS = %w[eq in]` filters dimensions only; nothing compiles to `HAVING` |
+| "Sales by hour of day" | Grains stop at day | Every time dimension declares `grains: %i[day week month]`; `time_bucket_sql` handles exactly those three |
+| "Versus the same month last year" | Only previous period | `COMPARE_MODES = %w[previous_period]`, shifted by an equal day count |
+| "Customer lifetime value" | No metric | Every base relation hardcodes `completed_at: range`, so no per-customer all-time aggregate is reachable |
+| "Total sales per product" | Refused | Correct — `total_sales` is order-level money; per line item it double-counts |
+| "Revenue and units shipped" | Refused | Correct — sales and inventory are different families on different clocks |
+| "Conversion rate" (traffic) | Unavailable | Correct and permanent — no session data reaches Spree |
+
+The last three rows are the design working. The first seven are the vocabulary
+being smaller than the questions.
+
+## Key Decisions (do not deviate without discussion)
+
+### 1. Carts are a fourth family, not a filter on orders
+
+In 6.0 a cart is its own table (`spree_carts`), joined to its order by a unique
+`spree_orders.cart_id`. It has no `status`, no `seller_id`, and its customer FK
+is `customer_id`, not `user_id`. It is therefore **a new base in a new
+`:carts` family**, registered like any other:
+
+```ruby
+base :carts, family: :carts, table: '%{carts}',
+ time_column: '%{carts}.created_at',
+ relation: lambda { |store, range, currency|
+ scope = store.carts.where(Spree::Cart.table_name => { created_at: range })
+ currency ? scope.where(Spree::Cart.table_name => { currency: currency }) : scope
+ }
+```
+
+A cart is anchored on **`created_at`**, not `completed_at`: the question is
+"of the carts started in this period, how many converted", and anchoring on
+completion would silently drop every cart that never converted — the exact
+population being measured. This is a different clock from the sales family,
+which is precisely why it is a separate family and why the one-family rule
+(Decision 14 of the semantic layer plan) refuses to mix the two. A merchant
+who wants carts beside revenue runs two queries; the alternative is a table
+where one row means "started in March" and the next means "completed in March".
+
+`store.carts` already exists, so the store-scoped relation needs no new
+association.
+
+### 2. Abandonment is the definition the reaper already uses, and it is a store preference
+
+`Spree::Carts::ReapExpiredJob` is the codebase's existing answer to "when is a
+cart abandoned": an incomplete cart whose **`updated_at`** has gone quiet.
+Reporting adopts that column — it is indexed on `spree_carts` — rather than
+inventing a second definition of the same word.
+
+The quiet window is a **store preference**, `abandoned_cart_after_hours`
+(integer, default 24), read through `preferred_abandoned_cart_after_hours` and
+edited under Settings → Store, exactly as `low_stock_threshold` was settled in
+Decision 15 of the semantic layer plan. It is not a request parameter and not a
+global. A cart is abandoned when it is incomplete and untouched for longer than
+that window; a cart that is merely incomplete and still being worked on is not
+abandoned, and a report that counted it would move every time someone was
+mid-checkout.
+
+**The reaper caps this history and the docs must say so.** Abandoned carts are
+destroyed after 30 days (guest) or 90 days (customer) by default, so an
+abandoned-cart report over a longer range under-reports — not because the query
+is wrong, but because the rows are gone. This is a documented property of the
+metric, not something the metric should paper over.
+
+### 3. Cart conversion is a cart-level rate, and it is not the conversion rate merchants read about
+
+`carts_converted / carts_started`, computed over the carts *started* in the
+period. The join is safe to aggregate over because `spree_orders.cart_id`
+carries a unique index — one cart yields at most one order, so no fan-out.
+
+This is emphatically **not** the storefront conversion rate (sessions →
+purchase), which stays unavailable forever because sessions never reach Spree.
+The metric is named `cart_conversion_rate` rather than `conversion_rate` for
+that reason, and its description says what it measures and what it does not. A
+merchant comparing it against an analytics tool's number will otherwise
+conclude one of the two is broken.
+
+A cart that completes into an **order group** (a multi-seller basket splitting
+into several orders) is one conversion, not three — the same
+`COALESCE(order_group_id, id)` checkout identity that `Promotion#credits_count`
+already uses.
+
+### 4. Promotion attribution reads `spree_discounts`, not the join table
+
+`spree_discounts` carries `promotion_id`, `promotion_action_id` and a
+**snapshot of the redeemed `code`** on every money row, all indexed. So "sales
+by promotion" and "sales by coupon code" are a plain `GROUP BY` on the money
+itself — no walk through `spree_order_promotions`, which carries no amount and
+would need a correlated subquery to produce one.
+
+Three consequences the implementation must respect:
+
+- **`amount` is non-positive** (enforced by a DB check constraint). Every
+ discount metric sums `-amount`, so a merchant reads "£4,210 discounted", not
+ "−£4,210".
+- **`promotion_id` is nullified, never cascaded, on promotion deletion.** The
+ dimension must therefore tolerate NULL and group it as "Unassigned" — which
+ the LEFT-join rule (Decision 13 of the semantic layer plan) already
+ guarantees. The row keeps its `code` and `label`, so a deleted promotion's
+ discounts stay attributable by code.
+- **The owner is `order_id` XOR `cart_id`.** Sales-family promotion reporting
+ filters to the order side explicitly; cart-owned rows belong to the carts
+ family. A join that forgets this silently mixes a discount that was never
+ purchased into revenue.
+
+Promotion redemptions are counted in **checkouts, not orders**, for the same
+reason `Promotion#credits_count` does: a multi-seller basket becomes several
+orders, and counting those separately would report one customer's single
+redemption as three.
+
+### 5. Metric filters are a separate contract key, never mixed into `filters`
+
+Filtering on a metric is a `HAVING` clause and behaves differently from a
+`WHERE` on a dimension — it runs after aggregation, it cannot use the
+id-subquery path, and it interacts with `limit`. Conflating the two in one
+`filters` array would hide that difference behind identical-looking JSON.
+
+```json
+{ "metrics": ["units_sold", "net_sales"],
+ "dimensions": ["product"],
+ "metric_filters": [{ "metric": "units_sold", "op": "eq", "value": 0 }],
+ "time_range": { "preset": "last_quarter" } }
+```
+
+`METRIC_FILTER_OPS = %w[eq gt gte lt lte]`. A metric filter may only name a
+metric the query already requests — the same rule `sort` follows — so the
+number a merchant filters on is a number they can see.
+
+**Derived metrics are refused**, naming the ingredient to filter instead:
+`average_order_value` is computed in Ruby after aggregation (its ratio never
+reaches SQL), so a `HAVING` on it cannot compile. This matches the existing
+rule that a derived metric cannot be SQL-sorted.
+
+### 6. "Never sold" needs the zero rows, so it is a left join from the dimension
+
+A `HAVING units_sold = 0` over the line-items base finds nothing: a product
+that never sold has no line item, so it is not in the result set to be
+filtered. The natural question needs the *dimension's* population as the root.
+
+This ships as an explicit `include_empty: true` on the dimension entry, which
+compiles the dimension's lookup relation as the left side and the aggregate as
+the right:
+
+```json
+{ "metrics": ["units_sold"],
+ "dimensions": [{ "name": "product", "include_empty": true }],
+ "metric_filters": [{ "metric": "units_sold", "op": "eq", "value": 0 }] }
+```
+
+It is opt-in and only valid for dimensions with a `lookup` (a relation exists
+to enumerate), because the alternative — making every breakdown include its
+empty members — would put every unsold product in every product report.
+`COALESCE(aggregate, 0)` supplies the zero so the filter compares a number
+rather than NULL.
+
+### 7. Hour grain resolves in the store timezone, like every other grain
+
+`hour` joins `%i[day week month]` on the time dimensions and gets its three
+adapter branches in `time_bucket_sql`. The store-timezone rule (Decision 7 of
+the semantic layer plan) matters more here than anywhere else: "sales by hour
+of day" is a question about the merchant's trading day, and reading it in UTC
+shifts every peak by the store's offset — the difference between "we sell at
+lunchtime" and "we sell mid-morning".
+
+Hour is the one grain where the range genuinely needs a guard: a year at hourly
+grain is 8,760 buckets. `MAX_BUCKETS` (2,000) is checked when the grain is
+resolved and refused with a message naming the range and a coarser grain,
+rather than being silently truncated.
+
+### 8. Year-over-year shifts by calendar, not by day count
+
+`COMPARE_MODES` gains `previous_year`. The existing `previous_period` shifts by
+an equal number of days, which is right for an arbitrary range and wrong for
+this one: "the same month last year" means February to February, and a
+day-count shift off a 28-day February lands in January.
+
+`previous_year` therefore shifts the range **by one calendar year in the store's
+timezone**, and February 29 maps to February 28 in a non-leap year. Bucket
+alignment stays positional, as `previous_period` already does — the first
+bucket of this year's range pairs with the first of last year's — so a
+year-over-year day series does not misalign on the leap day.
+
+### 9. Lifetime value is lifetime; the range selects the customers
+
+An LTV that only counted orders inside the report's range would be `net_sales`
+per customer wearing a misleading name. So `customer_lifetime_value` breaks the
+rule every other metric follows, deliberately and in exactly one way: **the
+time range selects which customers appear in the report; their value covers
+their whole history.**
+
+A merchant running "top customers by lifetime value, last 30 days" gets the
+customers who ordered in the last 30 days, ranked by what they have spent
+ever — which is the question. The metric's description says this outright,
+because a number that ignores the date picker must explain itself where it is
+read, not only in the docs.
+
+It compiles as a correlated subquery over the customer's completed,
+non-canceled orders — the pattern Decision 14 of the semantic layer plan
+already set for order-level money living on another table. It is only valid
+grouped by `customer` (or filtered to one); ungrouped it would sum every
+customer's lifetime into a figure that means nothing, so an ungrouped request
+is refused with that reason. `orders_lifetime` ships beside it, since "spent a
+lot across many small orders" and "spent a lot once" are different customers.
+
+### 10. The two refusals stay; the errors start teaching
+
+Neither refusal is relaxed. Both messages are rewritten to name the reason and
+the way through, because the current wording states a rule without a remedy:
+
+| Query | Today | New |
+|---|---|---|
+| `total_sales` by `product` | "metrics total_sales cannot be grouped by product" | "total_sales is order-level money (it includes shipping, duties and tax that belong to the whole order), so it cannot be split per product without counting the same money under several products. Use net_sales for per-product revenue." |
+| `net_sales` + `units_shipped` | "metrics from different families cannot be combined in one query: inventory, sales" | "net_sales measures sales (anchored on when the order completed) and units_shipped measures inventory (anchored on when stock moved). One query cannot report both. Run them as two queries." |
+
+This is the Decision 4 principle — reject, never silently narrow — carried
+through to the wording: the rejection should leave the caller knowing what to
+send instead. It matters twice over for the agent surface, where the error is
+the only teacher the model has.
+
+### 11. Abandonment does not inherit the reaper's money-safety carve-out (2026-09-12)
+
+`ReapExpiredJob` never destroys a cart carrying a live payment session or a
+valid payment. That exclusion exists so the reaper cannot delete rows with
+money attached — it is a **deletion-safety rule, not a definition of
+abandonment**. Reporting does not inherit it.
+
+A cart that stalled mid-payment is abandoned by any merchant's reading, and it
+is the most recoverable kind there is: there is money on the table and a known
+customer to chase. Excluding it would drop precisely the rows an
+abandoned-cart report exists to surface.
+
+So the predicate stays the simple one — incomplete, and untouched for longer
+than the store's window — and the report and the reaper legitimately disagree
+about which carts they mean. That divergence is recorded here because the two
+otherwise look like they should match.
+
+### 12. Lifetime value is net of refunds (2026-09-12)
+
+`customer_lifetime_value` subtracts refunds, consistent with `net_sales` and
+with the question being asked: a customer who bought heavily and returned most
+of it is not a high-value customer, and a gross figure would rank them as one.
+
+The cost is the Decision 14 reversal rule — a refund restates the figure after
+the fact, so a customer's lifetime value can fall. That is correct and the
+metric's description says so.
+
+### 13. The hour cap refuses, and the picker prevents it (2026-09-12)
+
+A range too wide for hourly buckets is a 422 naming the range and a coarser
+grain, never a silently coarsened result — Decision 4 of the semantic layer
+plan applies unchanged.
+
+An error the UI lets a merchant walk into is still a UI failure, so the two
+ship together: the backend refuses, and the builder disables the hour grain for
+ranges it cannot serve. The backend guard is the correctness boundary; the
+picker is what stops anyone meeting it.
+
+## Design Details
+
+### New registry members
+
+**Carts family** — one base, five metrics, three dimensions:
+
+| Member | Kind | Definition |
+|---|---|---|
+| `carts_started` | metric | `COUNT(*)` over the base |
+| `carts_converted` | metric | `COUNT(DISTINCT COALESCE(o.order_group_id, o.id))` through the cart's order |
+| `carts_abandoned` | metric | `COUNT(*)` where incomplete and `updated_at` older than the store's window |
+| `cart_conversion_rate` | metric | derived, `ratio: %i[carts_converted carts_started]`, `format: :percent` |
+| `abandoned_value` | metric | `SUM(total)` over the abandoned population, `format: :money` |
+| `started_at` | dimension | time, `grains: %i[hour day week month]` |
+| `cart_status` | dimension | expression → `converted` / `abandoned` / `active`, enumerated |
+| `cart_channel` | dimension | `channel_id`, reusing the existing channel lookup |
+
+`carts_abandoned` and `abandoned_value` read the same preference-driven
+predicate, defined once and referenced by both — two definitions of "abandoned"
+is the failure this decision exists to prevent.
+
+**Promotion dimensions** (sales family, `:line_items` base, joined through
+`spree_discounts`):
+
+| Member | Kind | Definition |
+|---|---|---|
+| `promotion` | dimension | `%{discounts}.promotion_id`, lookup `Spree::Promotion`, `subject: Spree::Promotion`, `key_scope: 'read_promotions'` |
+| `coupon_code` | dimension | `%{discounts}.code`, no lookup (the code is its own label) |
+| `promotion_discounts` | metric | `SUM(-%{discounts}.amount)`, `format: :money` |
+| `promotion_redemptions` | metric | `COUNT(DISTINCT COALESCE(order_group_id, id))` |
+
+A `discount_kind` axis was considered and dropped: all three discount
+dimensions join the same promotion-filtered association (two joins onto
+`spree_discounts` would alias the second, and both would then read whichever
+was joined first), so the kind could only ever report `promotion` — a field
+with one possible value.
+
+Both dimensions declare a `subject`, so a role without promotion access cannot
+break down revenue by promotion — and, per the semantic layer's fail-closed
+rule, declaring a subject obliges them to declare a `key_scope` too.
+
+**Lifetime metrics** (sales family, `:orders` base): `customer_lifetime_value`
+(money) and `orders_lifetime` (integer), both correlated subqueries over the
+customer's whole history, both valid only grouped by `customer`.
+
+### Contract changes
+
+```
+metric_filters: [{ metric:, op:, value: }] # HAVING; ops eq|gt|gte|lt|lte
+dimensions: [{ name:, grain:, include_empty: }] # include_empty is new
+compare: "previous_period" | "previous_year" # previous_year is new
+grains: hour | day | week | month # hour is new
+```
+
+Every addition is additive: an existing saved report parses and returns the same
+numbers. `GET /reporting/schema` publishes the new ops, grains, compare modes
+and the `include_empty` capability per dimension, so the picker builder and the
+agent tool schema pick them up without a second source of truth.
+
+### Compiler work
+
+- **`HAVING`** — `Adapters::Live` gains an `apply_metric_filters` that appends
+ to the grouped query only. The ungrouped totals query ignores metric filters
+ by construction: the Total is the period's real figure, not the sum of the
+ rows that survived a filter. (Rows already need not sum to the total under
+ Decision 13; this is the same rule, and the docs say so in the same place.)
+- **`include_empty`** — compiles the dimension's lookup relation as the left
+ side of a LEFT JOIN onto the aggregate, with `COALESCE(metric, 0)`. Valid
+ only with exactly one dimension, which must have a `lookup`.
+- **`hour`** — three branches in `time_bucket_sql` and three in
+ `expected_buckets` (zero-fill), plus the `MAX_BUCKETS` guard. Note the
+ existing shape: an unhandled grain currently falls through every `case` and
+ returns `nil`, which group-bys a NULL — so the guard is that both methods
+ learn `hour` together.
+- **`previous_year`** — a calendar-year shift in `previous_time_range`, with
+ the Feb 29 → Feb 28 rule.
+- **Cross-base join** — `join_for` currently hardcodes the single
+ `:line_items → order → :orders` hop. The carts base reaches its order and the
+ promotion dimensions reach `spree_discounts`, so this becomes a small
+ declared map (`base → dimension base → association path`) rather than a third
+ and fourth `if`. `resolve_sql`'s fixed table map gains `%{carts}`,
+ `%{discounts}` and `%{promotions}`.
+
+### Docs
+
+The metrics whose window or population is not what a reader assumes carry their
+caveat in the schema `description`, not only in the guide: lifetime value
+(ignores the range), abandoned carts (capped by the reaper), cart conversion
+(not storefront conversion). The description is what the picker shows and what
+the agent reads, so a caveat that lives only in Mintlify reaches neither.
+
+## Migration Path
+
+No data migration and no schema change beyond the store preference. Every
+change is additive vocabulary or an additive contract key, so saved reports
+written against the current vocabulary keep running unchanged.
+
+1. **Contract + compiler** — `metric_filters`, `include_empty`, `hour`,
+ `previous_year`, the declared cross-base join map, `MAX_BUCKETS`. Specs at
+ the compiler level, where the semantic layer's other numbers are proven.
+2. **Refusal messages** — the two rewrites, with specs asserting the remedy is
+ named (the message is the deliverable, so it is what the spec asserts).
+3. **Promotion vocabulary** — dimensions and metrics on `spree_discounts`,
+ including the order-side owner filter and the NULL-promotion group.
+4. **Carts family** — the `abandoned_cart_after_hours` preference and its
+ Settings field, the base, metrics and dimensions.
+5. **Lifetime metrics** — with the ungrouped refusal.
+6. **Schema, SDK, dashboard** — `GET /reporting/schema` publishes the
+ additions; `@spree/admin-sdk` contract types gain the new keys; the picker
+ builder grows a metric-filter row and the hour grain. New keys land in every
+ locale file, not only `en.json`.
+7. **Seeded reports** — the questions that prompted this plan become seeded
+ classics: abandoned carts, sales by promotion, sales by hour of day, never-sold
+ products, top customers by lifetime value. *(Deferred — the vocabulary and
+ the builder answer all of them today; seeding is a separate change to
+ `Seeds::SavedReports` with its own upgrade question, see Open Questions.)*
+
+## Constraints on Current Work
+
+- **A cart metric is never an order metric with a filter.** Carts are their own
+ base in their own family, anchored on `created_at`; mixing the two clocks in
+ one query is refused for the same reason sales and inventory are.
+- **One definition of "abandoned"**, from the store preference, shared by every
+ metric that uses it — and never a second window hardcoded in a query or a
+ dashboard filter.
+- **Discount money sums `-amount`.** The column is non-positive by DB
+ constraint; a metric that forgets this reports negative discounts.
+- **Promotion joins filter the owner side explicitly.** `spree_discounts` rows
+ are `order_id` XOR `cart_id`; a sales-family join that omits the filter mixes
+ unpurchased cart discounts into revenue.
+- **Metric filters never touch the ungrouped totals query.** The Total stays the
+ period's real figure.
+- **`include_empty` is opt-in and single-dimension**, and only on a dimension
+ with a `lookup`.
+- **A new grain is added to `time_bucket_sql` and `expected_buckets`
+ together** — an unhandled grain falls through to `nil` and silently groups by
+ NULL rather than raising.
+- **Lifetime metrics are grouped by `customer` or refused**, and say in their
+ own description that they ignore the report's range.
+- **Abandonment is incomplete + quiet, and nothing else** (Decision 11) — the
+ reaper's payment exclusion is a deletion-safety rule and must not be copied
+ into a reporting predicate.
+- **The two refusals stay refusals.** `total_sales` is not made groupable by
+ line-item dimensions, and families are not merged; only the messages improve.
+- **Traffic conversion stays unavailable.** `cart_conversion_rate` is not
+ renamed to `conversion_rate`, and no metric implies session data we do not
+ have.
+
+## Open Questions
+
+- **Seeding the new classics**: the five reports this plan set out to make
+ askable are all reachable through the builder, but none ships as a seeded
+ row. Adding them means touching `Seeds::SavedReports`, which the semantic
+ layer plan already flags as unable to tell "never created" from "deliberately
+ deleted" — so it waits on that question rather than adding five more rows a
+ merchant cannot permanently remove.
+
+- **Should `promotion` reach the orders base?** The dimensions are registered on
+ `:line_items` because that is where discount rows attach. "Orders that used
+ promotion X" is a reasonable question that would need an id-subquery from the
+ orders base; deferred until a merchant asks, since the filter path already
+ supports it.
+- **Lifetime value and canceled history:** the subquery excludes canceled
+ orders, consistent with every sales metric. (Refunds settled — Decision 12.)
+
+## References
+
+- `docs/plans/6.0-analytics-semantic-layer.md` — the registry, contract,
+ compiler and every decision this plan extends (especially 4 rejection, 7
+ timezone, 13 totals/LEFT joins, 14 families and correlated subqueries, 15
+ counters and store preferences).
+- `docs/plans/6.0-cart-order-split.md` — the `Spree::Cart` table, the
+ `cart_id` uniqueness that makes conversion safe to aggregate, and the tiered
+ reaper that caps abandoned-cart history.
+- `spree/core/app/jobs/spree/carts/reap_expired_job.rb` — the existing
+ abandonment definition this plan adopts.
+- `spree/core/app/models/spree/discount.rb` — per-row promotion attribution,
+ the non-positive amount constraint, and the dual owner.
+- `spree/core/app/models/spree/promotion.rb` — `credits_count` and
+ `checkouts_credited`, the checkout-identity rule for redemptions.
diff --git a/docs/plans/decisions.md b/docs/plans/decisions.md
index 52133a546a..ef62cf580f 100644
--- a/docs/plans/decisions.md
+++ b/docs/plans/decisions.md
@@ -5416,6 +5416,65 @@ audience; and a new `q[<column>_eq]` filter on either ledger model needs that
column in `whitelisted_ransackable_attributes` — a whitelisted association
name does not make its foreign key filterable.
+## 2026-09-12 — Reporting: the carts family, and abandonment is not the reaper's population
+
+The reporting vocabulary gains a fourth family. A cart is its own table in 6.0,
+so **abandoned carts and cart conversion are a `:carts` base anchored on
+`created_at`**, never a filter on orders: the population being measured is the
+carts that were *started*, and anchoring on completion drops every cart that
+never converted — the exact rows the report is about. The one-family rule
+refuses to mix carts with sales for the same reason it refuses sales with
+inventory; the two clocks mean different things.
+
+**Abandonment is incomplete + untouched for the store's
+`abandoned_cart_after_hours` window, and nothing else.**
+`Carts::ReapExpiredJob` additionally excludes carts carrying a live payment
+session or a valid payment, and reporting deliberately **does not** inherit
+that: the exclusion is there so the reaper cannot destroy rows with money
+attached, not because such a cart is un-abandoned. A cart stalled mid-payment
+is the most recoverable kind there is, and dropping it would hide the rows the
+report exists to surface. The report and the reaper therefore disagree about
+which carts they mean, which is recorded because they otherwise look like they
+should match. Consequence worth knowing: the reaper destroys abandoned carts at
+30 days (guest) / 90 days (customer), so an abandoned-cart report over a longer
+range under-reports because the rows are gone.
+
+**Promotion attribution reads `spree_discounts`**, which carries
+`promotion_id`, `promotion_action_id` and a snapshot of the redeemed `code` on
+every money row. Three rules bind anyone adding to it: `amount` is non-positive
+by DB constraint so metrics sum `-amount`; `promotion_id` is nullified rather
+than cascaded on deletion so the dimension must group NULL as "Unassigned"; and
+rows are `order_id` XOR `cart_id`, so a sales-family join that omits the owner
+filter mixes unpurchased cart discounts into revenue. Redemptions count
+checkouts (`COALESCE(order_group_id, id)`), not orders — a multi-seller basket
+is one redemption.
+
+**Constraints now:** a metric filter (`HAVING`) never touches the ungrouped
+totals query — the Total stays the period's real figure; a new grain is taught
+to `time_bucket_sql` and `expected_buckets` together, because an unhandled
+grain falls through to `nil` and silently groups by NULL rather than raising;
+and `cart_conversion_rate` is never renamed to `conversion_rate` — storefront
+conversion needs session data that never reaches Spree, and the two numbers
+must not be confusable.
+
+**Reporting refusal data is registry data, not compiler data.** The compiler
+briefly held three hardcoded lists — which metrics need grouping, what to
+suggest when an order-level metric is refused for a per-line breakdown, and
+what each family's clock means. All three are now fields on the definitions
+(`requires_grouping:` and `suggests:` on a metric, `clock:` on a base), for the
+same reason a base is registered rather than hardcoded: an extension's own
+metric must get the same refusals and the same advice without editing core.
+`per_group` as a boolean was the wrong shape — it could say a metric needed
+grouping but not *by what*, so the dimension was hardcoded in the validator
+beside it.
+
+**Customer lifetime value needs `spree_orders (store_id, email)`.** The metric
+correlates a customer's whole history on email, which had no index at all, so
+each customer group drove a full scan; and it walked that history twice, once
+for totals and again for refunds. One pass over the new index. Anything else
+correlating on order email needs the same index to be viable.
+
+
## 2026-09-13 — Stored value is not a promotion: gift cards and store credits live under Loyalty
Plan: `6.0-6.1-loyalty-gift-cards-store-credits.md`.
diff --git a/packages/admin-sdk/src/admin-client.ts b/packages/admin-sdk/src/admin-client.ts
index a103b0245e..c79c44e3d6 100644
--- a/packages/admin-sdk/src/admin-client.ts
+++ b/packages/admin-sdk/src/admin-client.ts
@@ -14,13 +14,35 @@ import { getParams, transformListParams } from '@spree/sdk-core'
// Semantic reporting contract (docs/plans/6.0-analytics-semantic-layer.md)
// ============================================
-export type ReportingGrain = 'day' | 'week' | 'month'
+export type ReportingGrain = 'hour' | 'day' | 'week' | 'month'
+
+export type ReportingCompare = 'previous_period' | 'previous_year'
+
+export type ReportingMetricFilterOp = 'eq' | 'gt' | 'gte' | 'lt' | 'lte'
export interface ReportingQuery {
/** Registered metric names — discover via `reporting.schema()`. */
metrics: string[]
- dimensions?: Array<string | { name: string; grain?: ReportingGrain }>
+ dimensions?: Array<
+ | string
+ | {
+ name: string
+ grain?: ReportingGrain
+ /**
+ * Include members with no matching rows, so a breakdown lists what had
+ * no activity ("which products never sold"). One dimension only, and
+ * only where `supports_include_empty` is set on the schema entry.
+ */
+ include_empty?: boolean
+ }
+ >
filters?: Array<{ dimension: string; op: 'eq' | 'in'; value: string | string[] }>
+ /**
+ * Filter on an aggregate (a HAVING clause), e.g. `units_sold = 0`. Narrows
+ * the rows only — the totals stay the period's real figure. The metric must
+ * also appear in `metrics`, and cannot be a derived one.
+ */
+ metric_filters?: Array<{ metric: string; op: ReportingMetricFilterOp; value: number }>
/**
* A named preset (`last_month`, `week_to_date`, `last_4_weeks`, any
* `last_<n>_<days|weeks|months>`), or ISO 8601 dates/datetimes in
@@ -28,7 +50,7 @@ export interface ReportingQuery {
* last 30 days. Everything resolves in the store timezone.
*/
time_range?: { preset?: string; since?: string; until?: string }
- compare?: 'previous_period'
+ compare?: ReportingCompare
/** Metric name, `-` prefix for descending. */
sort?: string
limit?: number
@@ -80,6 +102,8 @@ export interface ReportingSchemaMetric {
format: 'money' | 'integer' | 'decimal' | 'percent' | string
currency?: string
derived: boolean
+ /** Derived metrics are computed after aggregation, so they cannot be sorted or filtered in SQL. */
+ filterable?: boolean
}
export interface ReportingSchemaDimension {
@@ -91,13 +115,15 @@ export interface ReportingSchemaDimension {
/** Set when rows carry hydrated `{ id, label, meta }` payloads for this dimension. */
lookup?: string
filter_ops: Array<'eq' | 'in'>
+ /** Whether this dimension can lead a query with `include_empty`. */
+ supports_include_empty?: boolean
/** Enumerated raw values for status-like dimensions. */
values?: Array<{ name: string; label: string }>
/** Metrics this dimension can break down without double counting. */
compatible_metrics: string[]
}
-/** Members that can be queried together: sales, payments, inventory. */
+/** Members that can be queried together: sales, payments, inventory, carts. */
export interface ReportingSchemaFamily {
name: string
label: string
@@ -117,7 +143,10 @@ export interface ReportingSchema {
absolute: string
}
/** Ranking defaults the server applies (`limit` when omitted, and its ceiling). */
- limits: { default: number; max: number }
+ limits: { default: number; max: number; max_buckets?: number }
+ /** Ops accepted by `metric_filters`. */
+ metric_filter_ops?: ReportingMetricFilterOp[]
+ compare_modes?: ReportingCompare[]
}
export interface DashboardCounters {
diff --git a/packages/admin-sdk/src/index.ts b/packages/admin-sdk/src/index.ts
index 3fc5ba4d53..2169b8dce2 100644
--- a/packages/admin-sdk/src/index.ts
+++ b/packages/admin-sdk/src/index.ts
@@ -11,8 +11,10 @@ export type {
LoginCredentials,
MeResponse,
PermissionRule,
+ ReportingCompare,
ReportingDimensionValue,
ReportingGrain,
+ ReportingMetricFilterOp,
ReportingMetricValue,
ReportingQuery,
ReportingResult,
diff --git a/packages/dashboard/src/components/spree/reporting/report-builder.tsx b/packages/dashboard/src/components/spree/reporting/report-builder.tsx
index afe160ae73..79a56dcf3e 100644
--- a/packages/dashboard/src/components/spree/reporting/report-builder.tsx
+++ b/packages/dashboard/src/components/spree/reporting/report-builder.tsx
@@ -39,7 +39,15 @@ import { channelAutocompleteProps, useChannels } from '../../../hooks/use-channe
import { customerAutocompleteProps } from '../../../hooks/use-customers'
import { useAllMarkets } from '../../../hooks/use-markets'
import { productAutocompleteProps } from '../../../hooks/use-products'
-import { findDimension, isTimeDimension, type ReportDraft, type ReportFilter } from './report-draft'
+import {
+ maxBuckets as bucketCeiling,
+ findDimension,
+ hourGrainFitsRange,
+ isTimeDimension,
+ type ReportDraft,
+ type ReportFilter,
+ servableGrain,
+} from './report-draft'
const NONE = '__none__'
const CUSTOM_RANGE = '__custom__'
@@ -92,10 +100,15 @@ export function ReportBuilder({ draft, onChange, schema }: ReportBuilderProps) {
...groupableDimensions.map((d) => ({ value: d.name, label: d.label })),
]
- const grainOptions = (dimension?.grains ?? []).map((grain) => ({
- value: grain,
- label: t(`admin.reports.grains.${grain}`),
- }))
+ const maxBuckets = bucketCeiling(schema)
+ const hourGrainFits = hourGrainFitsRange(draft.timeRange, maxBuckets)
+
+ const grainOptions = (dimension?.grains ?? [])
+ .filter((grain) => grain !== 'hour' || hourGrainFits)
+ .map((grain) => ({
+ value: grain,
+ label: t(`admin.reports.grains.${grain}`),
+ }))
// Presets come from the schema (named + common relative ranges); whatever a
// saved report carries stays selectable even if it is not among them.
@@ -156,18 +169,27 @@ export function ReportBuilder({ draft, onChange, schema }: ReportBuilderProps) {
const definition = findDimension(schema, next)
update({
dimension: next,
- grain: definition?.grains?.includes(draft.grain)
- ? draft.grain
- : (definition?.grains?.[0] ?? 'day'),
+ // Falls back to day, not grains[0]: hour leads the list, and silently
+ // switching a merchant to an hourly series is never what they meant.
+ // The range matters too — a dimension that offers hour cannot keep it
+ // over a range the server would refuse.
+ grain: servableGrain(draft, definition, maxBuckets),
})
}
- function setPreset(value: string) {
- if (value === CUSTOM_RANGE) {
- update({ timeRange: dateRangeValue(customRange) })
- } else {
- update({ timeRange: { preset: value } })
+ // Widening the range can put the hour grain out of reach. The draft keeps
+ // whatever it had, so the grain is stepped down here rather than left to
+ // send a query the picker no longer offers an option for.
+ function setTimeRange(timeRange: ReportDraft['timeRange']) {
+ const patch: Partial<ReportDraft> = { timeRange }
+ if (draft.grain === 'hour' && !hourGrainFitsRange(timeRange, maxBuckets)) {
+ patch.grain = 'day'
}
+ update(patch)
+ }
+
+ function setPreset(value: string) {
+ setTimeRange(value === CUSTOM_RANGE ? dateRangeValue(customRange) : { preset: value })
}
function addFilter(name: string) {
@@ -266,7 +288,7 @@ export function ReportBuilder({ draft, onChange, schema }: ReportBuilderProps) {
{presetValue === CUSTOM_RANGE && (
<DateRangePicker
value={customRange}
- onChange={(range) => update({ timeRange: dateRangeValue(range) })}
+ onChange={(range) => setTimeRange(dateRangeValue(range))}
timezone={timezone}
/>
)}
diff --git a/packages/dashboard/src/components/spree/reporting/report-draft.ts b/packages/dashboard/src/components/spree/reporting/report-draft.ts
index 16c5ebeea0..ba3bb6096c 100644
--- a/packages/dashboard/src/components/spree/reporting/report-draft.ts
+++ b/packages/dashboard/src/components/spree/reporting/report-draft.ts
@@ -90,6 +90,55 @@ export function draftFromQuery(query: ReportingQuery, schema?: ReportingSchema):
}
}
+/**
+ * Whether an hourly series over this range fits inside the server's bucket
+ * ceiling. A custom range is measured; a preset is matched against the day
+ * count in its own name, since the server's preset vocabulary is wider than
+ * the client's date helper knows. Anything unrecognised counts as too wide, so
+ * the grain is withheld rather than offered and then refused.
+ */
+export function hourGrainFitsRange(timeRange: ReportTimeRange, maxBuckets: number): boolean {
+ if ('since' in timeRange) {
+ const hours = (Date.parse(timeRange.until) - Date.parse(timeRange.since)) / 3_600_000 + 24
+ return hours <= maxBuckets
+ }
+ const preset = timeRange.preset
+ if (preset === 'today' || preset === 'yesterday') return true
+ // The server's relative grammar is `last_<n>_<days|weeks|months>`; each unit
+ // is converted at its longest so the estimate never claims a range fits when
+ // the server would refuse it.
+ const relative = preset.match(/^last_(\d+)_(days|weeks|months)$/)
+ if (relative) {
+ const perUnit = { days: 1, weeks: 7, months: 31 }[relative[2] as 'days' | 'weeks' | 'months']
+ return (Number(relative[1]) * perUnit + 1) * 24 <= maxBuckets
+ }
+ // Named presets bounded by a month or less; a quarter is 2,208 hours, over
+ // the ceiling, so it and everything longer fall through to false.
+ return ['week_to_date', 'last_week', 'month_to_date', 'last_month'].includes(preset)
+}
+
+/**
+ * The draft's grain, stepped down when it is no longer servable — because the
+ * dimension stopped offering it, or because the range grew too wide to chart
+ * hourly. A saved report carrying a stale hourly grain would otherwise fire a
+ * query the server refuses, with nothing in the picker the merchant could
+ * change to clear it.
+ */
+export function servableGrain(
+ draft: ReportDraft,
+ dimension: ReportingSchemaDimension | undefined,
+ maxBuckets: number,
+): ReportingGrain {
+ if (!dimension?.grains?.includes(draft.grain)) return 'day'
+ if (draft.grain === 'hour' && !hourGrainFitsRange(draft.timeRange, maxBuckets)) return 'day'
+ return draft.grain
+}
+
+/** The server's bucket ceiling, or its published default until the schema loads. */
+export function maxBuckets(schema?: ReportingSchema): number {
+ return schema?.limits?.max_buckets ?? 2000
+}
+
export function queryFromDraft(draft: ReportDraft, schema?: ReportingSchema): ReportingQuery {
const dimension = findDimension(schema, draft.dimension)
const timeDimension = isTimeDimension(dimension)
@@ -97,7 +146,7 @@ export function queryFromDraft(draft: ReportDraft, schema?: ReportingSchema): Re
if (draft.dimension) {
query.dimensions = timeDimension
- ? [{ name: draft.dimension, grain: draft.grain }]
+ ? [{ name: draft.dimension, grain: servableGrain(draft, dimension, maxBuckets(schema)) }]
: [draft.dimension]
}
diff --git a/packages/dashboard/src/components/spree/reporting/report-view.tsx b/packages/dashboard/src/components/spree/reporting/report-view.tsx
index 6dee8b868b..de052c42b8 100644
--- a/packages/dashboard/src/components/spree/reporting/report-view.tsx
+++ b/packages/dashboard/src/components/spree/reporting/report-view.tsx
@@ -170,7 +170,9 @@ export function formatMetric(
metric: ReportingSchemaMetric,
locale: string,
): string {
- if (!value) return '—'
+ // A metric that only means something per group (a lifetime figure) reports
+ // a null total rather than a misleading store-wide one.
+ if (!value || value.value == null) return '—'
if (value.display) return value.display
const formatted = value.value.toLocaleString(locale, {
maximumFractionDigits: metric.format === 'integer' ? 0 : 2,
@@ -250,22 +252,30 @@ function StatTiles({
}
const BUCKET_FORMATS: Record<ReportingGrain, Intl.DateTimeFormatOptions> = {
+ hour: { hour: 'numeric' },
day: { month: 'short', day: 'numeric' },
week: { month: 'short', day: 'numeric' },
month: { month: 'short', year: 'numeric' },
}
export function formatBucket(value: string, grain: ReportingGrain, locale: string, long = false) {
- // Buckets arrive as store-local dates (`yyyy-MM-dd`); parsing them as UTC
- // midnight keeps the day stable whatever the browser's zone.
- const date = new Date(value.length === 10 ? `${value}T00:00:00Z` : value)
+ // Buckets arrive as store-local wall-clock times — a date (`yyyy-MM-dd`) or
+ // a datetime (`yyyy-MM-dd HH:00:00`). Both are read as UTC so the label
+ // shows what the merchant's clock said, whatever zone the browser is in.
+ const normalized = value.length === 10 ? `${value}T00:00:00Z` : `${value.replace(' ', 'T')}Z`
+ const date = new Date(normalized)
if (Number.isNaN(date.getTime())) return value
const options: Intl.DateTimeFormatOptions = {
...BUCKET_FORMATS[grain],
...(long && grain !== 'month' ? { year: 'numeric' } : {}),
+ // An hour bucket needs its date alongside the time to be readable on its
+ // own (a tooltip, a table row), but not on a dense axis.
+ ...(grain === 'hour' && long ? { month: 'short', day: 'numeric' } : {}),
timeZone: 'UTC',
}
- return date.toLocaleDateString(locale, options)
+ return grain === 'hour'
+ ? date.toLocaleString(locale, options)
+ : date.toLocaleDateString(locale, options)
}
function BucketLabel({ value, grain }: { value: string; grain: ReportingGrain }) {
diff --git a/packages/dashboard/src/locales/ar.json b/packages/dashboard/src/locales/ar.json
index b137ec8207..66768d5a10 100644
--- a/packages/dashboard/src/locales/ar.json
+++ b/packages/dashboard/src/locales/ar.json
@@ -5904,6 +5904,7 @@
"title": "إعدادات التقرير"
},
"grains": {
+ "hour": "كل ساعة",
"day": "يومي",
"week": "أسبوعي",
"month": "شهري"
diff --git a/packages/dashboard/src/locales/de.json b/packages/dashboard/src/locales/de.json
index 5d54f8145d..8ca6d77ae5 100644
--- a/packages/dashboard/src/locales/de.json
+++ b/packages/dashboard/src/locales/de.json
@@ -5867,6 +5867,7 @@
"title": "Berichtseinstellungen"
},
"grains": {
+ "hour": "Stündlich",
"day": "Täglich",
"week": "Wöchentlich",
"month": "Monatlich"
diff --git a/packages/dashboard/src/locales/en.json b/packages/dashboard/src/locales/en.json
index 5dab51cf98..e5c203a140 100644
--- a/packages/dashboard/src/locales/en.json
+++ b/packages/dashboard/src/locales/en.json
@@ -5867,6 +5867,7 @@
"title": "Report settings"
},
"grains": {
+ "hour": "Hourly",
"day": "Daily",
"week": "Weekly",
"month": "Monthly"
diff --git a/packages/dashboard/src/locales/fr.json b/packages/dashboard/src/locales/fr.json
index 419187165c..6e47363bb7 100644
--- a/packages/dashboard/src/locales/fr.json
+++ b/packages/dashboard/src/locales/fr.json
@@ -5867,6 +5867,7 @@
"title": "Paramètres du rapport"
},
"grains": {
+ "hour": "Par heure",
"day": "Quotidien",
"week": "Hebdomadaire",
"month": "Mensuel"
diff --git a/packages/dashboard/src/locales/pl.json b/packages/dashboard/src/locales/pl.json
index ef1b266dd2..fd95bf9ff5 100644
--- a/packages/dashboard/src/locales/pl.json
+++ b/packages/dashboard/src/locales/pl.json
@@ -5933,6 +5933,7 @@
"title": "Ustawienia raportu"
},
"grains": {
+ "hour": "Godzinowo",
"day": "Dziennie",
"week": "Tygodniowo",
"month": "Miesięcznie"
diff --git a/packages/dashboard/src/locales/zh-CN.json b/packages/dashboard/src/locales/zh-CN.json
index 1eafe026d9..c2200b806e 100644
--- a/packages/dashboard/src/locales/zh-CN.json
+++ b/packages/dashboard/src/locales/zh-CN.json
@@ -5854,6 +5854,7 @@
"title": "报表设置"
},
"grains": {
+ "hour": "每小时",
"day": "按天",
"week": "按周",
"month": "按月"
diff --git a/spree/api/app/controllers/spree/api/v3/admin/reporting_controller.rb b/spree/api/app/controllers/spree/api/v3/admin/reporting_controller.rb
index 1fb8b1de2e..fd8ac6063c 100644
--- a/spree/api/app/controllers/spree/api/v3/admin/reporting_controller.rb
+++ b/spree/api/app/controllers/spree/api/v3/admin/reporting_controller.rb
@@ -12,7 +12,7 @@ module Spree
rescue_from Spree::Reporting::UnknownMember, Spree::Reporting::InvalidQuery, with: :render_invalid_query
- QUERY_KEYS = %w[metrics dimensions filters time_range compare sort limit currency].freeze
+ QUERY_KEYS = %w[metrics dimensions filters metric_filters time_range compare sort limit currency].freeze
# POST /api/v3/admin/reporting/query
def query
diff --git a/spree/api/spec/controllers/spree/api/v3/admin/reporting_controller_spec.rb b/spree/api/spec/controllers/spree/api/v3/admin/reporting_controller_spec.rb
index 5bcb80fb31..5e63ed2129 100644
--- a/spree/api/spec/controllers/spree/api/v3/admin/reporting_controller_spec.rb
+++ b/spree/api/spec/controllers/spree/api/v3/admin/reporting_controller_spec.rb
@@ -93,6 +93,23 @@ RSpec.describe Spree::Api::V3::Admin::ReportingController, type: :controller do
expect(json_response['totals']['orders']['value']).to eq(1)
end
+ # The endpoint slices the payload against an allowlist, so a contract key
+ # missing from it would be dropped and the caller would get unfiltered
+ # rows back with no error at all.
+ it 'applies metric filters sent by a client' do
+ create(:completed_order_with_totals, store: store, completed_at: 3.days.ago)
+
+ post :query, params: {
+ metrics: %w[units_sold],
+ dimensions: %w[product],
+ metric_filters: [{ metric: 'units_sold', op: 'gt', value: 10_000 }]
+ }, as: :json
+
+ expect(response).to have_http_status(:ok)
+ expect(json_response['rows']).to be_empty
+ expect(json_response['totals']['units_sold']['value']).to be > 0
+ end
+
it 'returns 404 for a channel filter from another store' do
foreign_channel = create(:channel, store: create(:store))
diff --git a/spree/core/app/models/spree/line_item.rb b/spree/core/app/models/spree/line_item.rb
index e1068d03ec..99c43fd277 100644
--- a/spree/core/app/models/spree/line_item.rb
+++ b/spree/core/app/models/spree/line_item.rb
@@ -26,6 +26,11 @@ module Spree
has_many :tax_lines, class_name: 'Spree::TaxLine', dependent: :destroy, inverse_of: :line_item
has_many :discounts, class_name: 'Spree::Discount', dependent: :destroy, inverse_of: :line_item
+ # Promotion attribution reads this rather than every discount: joining the
+ # manual ones too would group a hand-discounted line under "no promotion"
+ # alongside its real promotion row, counting the same line twice.
+ has_many :promotion_discounts, -> { promotion },
+ class_name: 'Spree::Discount', inverse_of: :line_item
has_many :fees, class_name: 'Spree::Fee', dependent: :destroy, inverse_of: :line_item
has_many :fulfillment_items, class_name: 'Spree::FulfillmentItem', inverse_of: :line_item, dependent: :destroy
has_many :fulfillments, through: :fulfillment_items, source: :fulfillment
diff --git a/spree/core/app/models/spree/store.rb b/spree/core/app/models/spree/store.rb
index 44ba24702a..baa24583ee 100644
--- a/spree/core/app/models/spree/store.rb
+++ b/spree/core/app/models/spree/store.rb
@@ -119,6 +119,11 @@ module Spree
# On-hand quantity at or below which a tracked variant counts as running
# low on the home screen. 0 turns the warning off.
preference :low_stock_threshold, :integer, default: 5
+ # How long a cart must sit untouched before reporting counts it abandoned.
+ # Deliberately shorter than the cart-expiry reaper's windows: that job
+ # decides when a cart is deleted, this decides when a merchant should
+ # chase it.
+ preference :abandoned_cart_after_hours, :integer, default: 24
preference :show_products_without_price, :boolean, default: false
preference :disable_sku_validation, :boolean, default: false
# Records price changes so the storefront can show the lowest price of the
@@ -319,6 +324,7 @@ module Spree
numericality: { only_integer: true, greater_than: 0, less_than_or_equal_to: 1.hour.to_i }
validates :preferred_stock_reservation_ttl_minutes, numericality: { only_integer: true, greater_than: 0 }
validates :preferred_low_stock_threshold, numericality: { only_integer: true, greater_than_or_equal_to: 0 }
+ validates :preferred_abandoned_cart_after_hours, numericality: { only_integer: true, greater_than: 0 }
# A fraction, not a percentage: 0.21 is 21%. Bounded because the value is
# multiplied straight into what a seller is charged, so a negative would
# credit them and a figure above 1 would bill more tax than fee.
diff --git a/spree/core/config/brakeman.ignore b/spree/core/config/brakeman.ignore
index b556546c78..869f46ca39 100644
--- a/spree/core/config/brakeman.ignore
+++ b/spree/core/config/brakeman.ignore
@@ -1,97 +1,120 @@
{
+ "brakeman_version": "8.0.5",
"ignored_warnings": [
{
- "warning_type": "SQL Injection",
- "warning_code": 0,
- "fingerprint": "5ea89a73bb4996140bbba012ae1b3112199826c65f3d699e4d5a49b8dc5b98cf",
"check_name": "SQL",
- "message": "Possible SQL injection",
+ "code": "scope.joins(:default_variant).select(\"#{(\"\" or \"#{Spree::Product.table_name}.*, \")}#{Spree::Variant.table_name}.sku\").order(\"#{Spree::Variant.table_name}.sku #{sort_by?(\"sku\")[1]}\")",
+ "confidence": "Weak",
+ "cwe_id": [
+ 89
+ ],
"file": "app/sorters/spree/products/sort.rb",
+ "fingerprint": "5ea89a73bb4996140bbba012ae1b3112199826c65f3d699e4d5a49b8dc5b98cf",
"line": 44,
"link": "https://brakemanscanner.org/docs/warning_types/sql_injection/",
- "code": "scope.joins(:default_variant).select(\"#{(\"\" or \"#{Spree::Product.table_name}.*, \")}#{Spree::Variant.table_name}.sku\").order(\"#{Spree::Variant.table_name}.sku #{sort_by?(\"sku\")[1]}\")",
- "render_path": null,
"location": {
- "type": "method",
"class": "Spree::Products::Sort",
- "method": "by_sku"
+ "method": "by_sku",
+ "type": "method"
},
+ "message": "Possible SQL injection",
+ "note": "False positive: interpolates only Model.table_name constants and a normalized :asc/:desc direction (Spree::BaseSorter#order_direction); no user input reaches the SQL. CI Brakeman flags it only because it cannot detect the engine Rails version (no Gemfile.lock under core/) and assumes Rails 4.x.",
+ "render_path": null,
"user_input": "sort_by?(\"sku\")[1]",
+ "warning_code": 0,
+ "warning_type": "SQL Injection"
+ },
+ {
+ "check_name": "SQL",
+ "code": "scope.joins(:variants => :prices).select(\"#{Spree::Product.table_name}.*, #{Spree::Price.table_name}.amount\").distinct.where(:\"#{Spree::Price.table_name}.currency\" => currency).order(\"#{Spree::Price.table_name}.amount #{sort_by?(\"price\")[1]}\")",
"confidence": "Weak",
"cwe_id": [
89
],
- "note": "False positive: interpolates only Model.table_name constants and a normalized :asc/:desc direction (Spree::BaseSorter#order_direction); no user input reaches the SQL. CI Brakeman flags it only because it cannot detect the engine Rails version (no Gemfile.lock under core/) and assumes Rails 4.x."
- },
- {
- "warning_type": "SQL Injection",
- "warning_code": 0,
- "fingerprint": "72eaf56a0fb1ac0ecd3e6a78f4d85c1b17cc199cd5c2c502d9f8abb3066bc6b4",
- "check_name": "SQL",
- "message": "Possible SQL injection",
"file": "app/sorters/spree/products/sort.rb",
+ "fingerprint": "72eaf56a0fb1ac0ecd3e6a78f4d85c1b17cc199cd5c2c502d9f8abb3066bc6b4",
"line": 30,
"link": "https://brakemanscanner.org/docs/warning_types/sql_injection/",
- "code": "scope.joins(:variants => :prices).select(\"#{Spree::Product.table_name}.*, #{Spree::Price.table_name}.amount\").distinct.where(:\"#{Spree::Price.table_name}.currency\" => currency).order(\"#{Spree::Price.table_name}.amount #{sort_by?(\"price\")[1]}\")",
- "render_path": null,
"location": {
- "type": "method",
"class": "Spree::Products::Sort",
- "method": "by_price"
+ "method": "by_price",
+ "type": "method"
},
+ "message": "Possible SQL injection",
+ "note": "False positive: interpolates only Model.table_name constants and a normalized :asc/:desc direction (Spree::BaseSorter#order_direction); no user input reaches the SQL. CI Brakeman flags it only because it cannot detect the engine Rails version (no Gemfile.lock under core/) and assumes Rails 4.x.",
+ "render_path": null,
"user_input": "sort_by?(\"price\")[1]",
+ "warning_code": 0,
+ "warning_type": "SQL Injection"
+ },
+ {
+ "check_name": "SQL",
+ "code": "scope.joins(:variants => :prices).select(\"#{Spree::Product.table_name}.* , min(#{Spree::Price.table_name}.amount)\").where(Spree::Price.table_name => ({ :currency => currency })).where.not(Spree::Price.table_name => ({ :amount => nil })).order(\"min(#{Spree::Price.table_name}.amount) #{sort_order}\")",
"confidence": "Weak",
"cwe_id": [
89
],
- "note": "False positive: interpolates only Model.table_name constants and a normalized :asc/:desc direction (Spree::BaseSorter#order_direction); no user input reaches the SQL. CI Brakeman flags it only because it cannot detect the engine Rails version (no Gemfile.lock under core/) and assumes Rails 4.x."
- },
- {
- "warning_type": "SQL Injection",
- "warning_code": 0,
- "fingerprint": "b29c921d4c4aab845ee22266e80e5a295edb67b6bf3e5d40361a9f91753daaff",
- "check_name": "SQL",
- "message": "Possible SQL injection",
"file": "app/finders/spree/products/find.rb",
+ "fingerprint": "b29c921d4c4aab845ee22266e80e5a295edb67b6bf3e5d40361a9f91753daaff",
"line": 309,
"link": "https://brakemanscanner.org/docs/warning_types/sql_injection/",
- "code": "scope.joins(:variants => :prices).select(\"#{Spree::Product.table_name}.* , min(#{Spree::Price.table_name}.amount)\").where(Spree::Price.table_name => ({ :currency => currency })).where.not(Spree::Price.table_name => ({ :amount => nil })).order(\"min(#{Spree::Price.table_name}.amount) #{sort_order}\")",
- "render_path": null,
"location": {
- "type": "method",
"class": "Spree::Products::Find",
- "method": "order_by_price"
+ "method": "order_by_price",
+ "type": "method"
},
+ "message": "Possible SQL injection",
+ "note": "False positive: interpolates only Model.table_name constants and a normalized :asc/:desc direction (Spree::BaseSorter#order_direction); no user input reaches the SQL. CI Brakeman flags it only because it cannot detect the engine Rails version (no Gemfile.lock under core/) and assumes Rails 4.x.",
+ "render_path": null,
"user_input": "sort_order",
+ "warning_code": 0,
+ "warning_type": "SQL Injection"
+ },
+ {
+ "check_name": "SQL",
+ "code": "scope.joins(:product => ({ :variants => :prices })).select(\"#{Spree::Classification.table_name}.*, min(#{Spree::Price.table_name}.amount)\").distinct.where(:spree_prices => ({ :currency => currency })).order(\"min(#{Spree::Price.table_name}.amount) #{sort_by?(\"price\")[1]}\")",
"confidence": "Weak",
"cwe_id": [
89
],
- "note": "False positive: interpolates only Model.table_name constants and a normalized :asc/:desc direction (Spree::BaseSorter#order_direction); no user input reaches the SQL. CI Brakeman flags it only because it cannot detect the engine Rails version (no Gemfile.lock under core/) and assumes Rails 4.x."
- },
- {
- "warning_type": "SQL Injection",
- "warning_code": 0,
- "fingerprint": "ee3a8b8b2f5f3c94db405aa73fec5cf247392037b3e61804771a41e2a5d07702",
- "check_name": "SQL",
- "message": "Possible SQL injection",
"file": "app/sorters/spree/classifications/sort.rb",
+ "fingerprint": "ee3a8b8b2f5f3c94db405aa73fec5cf247392037b3e61804771a41e2a5d07702",
"line": 59,
"link": "https://brakemanscanner.org/docs/warning_types/sql_injection/",
- "code": "scope.joins(:product => ({ :variants => :prices })).select(\"#{Spree::Classification.table_name}.*, min(#{Spree::Price.table_name}.amount)\").distinct.where(:spree_prices => ({ :currency => currency })).order(\"min(#{Spree::Price.table_name}.amount) #{sort_by?(\"price\")[1]}\")",
- "render_path": null,
"location": {
- "type": "method",
"class": "Spree::Classifications::Sort",
- "method": "by_price"
+ "method": "by_price",
+ "type": "method"
},
+ "message": "Possible SQL injection",
+ "note": "False positive: interpolates only Model.table_name constants and a normalized :asc/:desc direction (Spree::BaseSorter#order_direction); no user input reaches the SQL. CI Brakeman flags it only because it cannot detect the engine Rails version (no Gemfile.lock under core/) and assumes Rails 4.x.",
+ "render_path": null,
"user_input": "sort_by?(\"price\")[1]",
- "confidence": "Weak",
+ "warning_code": 0,
+ "warning_type": "SQL Injection"
+ },
+ {
+ "check_name": "SQL",
+ "code": "scope.having(Arel.sql(\"#{resolve_sql(filter[:metric].sql_for(:grouped => true))} #{COMPARISONS.fetch(filter[:op]).first} ?\"), filter[:value])",
+ "confidence": "Medium",
"cwe_id": [
89
],
- "note": "False positive: interpolates only Model.table_name constants and a normalized :asc/:desc direction (Spree::BaseSorter#order_direction); no user input reaches the SQL. CI Brakeman flags it only because it cannot detect the engine Rails version (no Gemfile.lock under core/) and assumes Rails 4.x."
+ "file": "lib/spree/reporting/adapters/live.rb",
+ "fingerprint": "9370b80276052cef88afe7154cc83064b1c6e503be8fde70c911959d05a95ac1",
+ "line": 148,
+ "link": "https://brakemanscanner.org/docs/warning_types/sql_injection/",
+ "location": {
+ "class": "Spree::Reporting::Adapters::Live",
+ "method": "apply_metric_filters",
+ "type": "method"
+ },
+ "message": "Possible SQL injection",
+ "note": "False positive: the interpolated aggregate is developer-registered SQL from Spree.reporting, reached only after Registry#metric! matched the requested name against that allowlist, and the operator comes from COMPARISON_OPERATORS.fetch. The caller's own value is a bind parameter, not interpolated.",
+ "render_path": null,
+ "user_input": "resolve_sql(filter[:metric].sql_for(:grouped => true))",
+ "warning_code": 0,
+ "warning_type": "SQL Injection"
}
- ],
- "brakeman_version": "8.0.5"
+ ]
}
diff --git a/spree/core/config/locales/en.yml b/spree/core/config/locales/en.yml
index f420258cea..e6a2a45149 100644
--- a/spree/core/config/locales/en.yml
+++ b/spree/core/config/locales/en.yml
@@ -1681,6 +1681,12 @@ en:
sales_total: Sales total
reporting:
dimensions:
+ cart_channel:
+ description: The channel the cart was started in.
+ label: Sales channel
+ cart_status:
+ description: Whether the cart converted into an order, was abandoned, or is still active.
+ label: Cart status
category:
description: The category the sold product belongs to.
label: Category
@@ -1696,6 +1702,9 @@ en:
country:
description: Country of the shipping address (ISO code).
label: Shipping country
+ coupon_code:
+ description: The discount code redeemed, as recorded on the order.
+ label: Coupon code
customer:
description: The customer who placed the order, by email.
label: Customer
@@ -1732,9 +1741,15 @@ en:
product:
description: The product sold.
label: Product
+ promotion:
+ description: The promotion the discount came from.
+ label: Promotion
seller:
description: The seller who sold the item.
label: Seller
+ started_at:
+ description: When the cart was started, bucketed by hour, day, week or month in the store timezone.
+ label: Cart started
stock_location:
description: The location the stock moved in or out of.
label: Stock location
@@ -1744,6 +1759,9 @@ en:
export:
total: Total
families:
+ carts:
+ description: Baskets started, converted and abandoned.
+ label: Carts
inventory:
label: Inventory
payments:
@@ -1751,15 +1769,33 @@ en:
sales:
label: Sales
metrics:
+ abandoned_value:
+ description: What the abandoned carts were worth. Carts are deleted once they expire, so a range longer than the cart expiry window under-reports.
+ label: Abandoned cart value
average_order_value:
description: Total sales divided by the number of orders.
label: Average order value
+ cart_conversion_rate:
+ description: Share of carts started in the period that became orders. This counts baskets, not visits — it is not a storefront conversion rate.
+ label: Cart conversion rate
+ carts_abandoned:
+ description: Carts that never completed and have been untouched for longer than the store abandoned-cart window.
+ label: Abandoned carts
+ carts_converted:
+ description: Carts that became an order. A basket split across sellers counts once.
+ label: Converted carts
+ carts_started:
+ description: Carts created in the period.
+ label: Carts started
commission:
description: What the marketplace earned on these sales, including its tax.
label: Commission
cost_of_goods:
description: What the goods sold cost you. Variants with no cost price count as zero.
label: Cost of goods
+ customer_lifetime_value:
+ description: 'Everything a customer has ever spent, net of refunds. Ignores the report date range: the range selects which customers appear, not which of their orders count.'
+ label: Customer lifetime value
customers:
description: Distinct customer emails that placed orders.
label: Customers
@@ -1790,6 +1826,9 @@ en:
orders:
description: Number of placed orders.
label: Orders
+ orders_lifetime:
+ description: How many orders a customer has ever placed. Ignores the report date range, like customer lifetime value.
+ label: Lifetime orders
payments_count:
description: Number of successful charges.
label: Payments
@@ -1802,6 +1841,12 @@ en:
payments_refunded:
description: Money refunded through these payments.
label: Payments refunded
+ promotion_discounts:
+ description: Money discounted by promotions, shown as a positive amount.
+ label: Promotion discounts
+ promotion_redemptions:
+ description: How many checkouts used a promotion. A basket split across sellers counts once.
+ label: Promotion redemptions
returns:
description: Money refunded, counted in the period the refund was issued.
label: Returns
diff --git a/spree/core/db/migrate/20260912120000_index_reporting_lifetime_and_promotion_columns.rb b/spree/core/db/migrate/20260912120000_index_reporting_lifetime_and_promotion_columns.rb
new file mode 100644
index 0000000000..327588fab9
--- /dev/null
+++ b/spree/core/db/migrate/20260912120000_index_reporting_lifetime_and_promotion_columns.rb
@@ -0,0 +1,19 @@
+class IndexReportingLifetimeAndPromotionColumns < ActiveRecord::Migration[8.1]
+ # Customer lifetime value correlates a customer's whole order history on
+ # email, which `spree_orders` had no index for at all — so every customer
+ # group in a report drove a full scan of the orders table, and the metric
+ # became unusable well before a store got large. Store first, because every
+ # reporting query is store-scoped before it is anything else.
+ #
+ # The promotion breakdown probes discounts per line item, filtered to
+ # promotion rows; carrying `kind` in the index keeps those probes from
+ # visiting the row itself.
+ # Cart reporting anchors on when a basket was started, and `spree_carts` had
+ # no index pairing the store with that column — so every cart report scanned
+ # the store's whole cart history to find one period's rows.
+ def change
+ add_index :spree_orders, %i[store_id email]
+ add_index :spree_discounts, %i[line_item_id kind]
+ add_index :spree_carts, %i[store_id created_at]
+ end
+end
diff --git a/spree/core/lib/spree/reporting/adapters/live.rb b/spree/core/lib/spree/reporting/adapters/live.rb
index 449a66885b..9fcca66a87 100644
--- a/spree/core/lib/spree/reporting/adapters/live.rb
+++ b/spree/core/lib/spree/reporting/adapters/live.rb
@@ -14,6 +14,15 @@ module Spree
# Roughly a century — long enough for any report a merchant would read,
# short enough that the widest one still fits in memory at day grain.
MAX_RANGE_DAYS = 36_600
+ # SQL operator and Ruby message for each metric-filter op, paired so the
+ # HAVING path and the Ruby path for include_empty cannot drift apart.
+ COMPARISONS = { eq: ['=', :==], gt: ['>', :>], gte: ['>=', :>=],
+ lt: ['<', :<], lte: ['<=', :<=] }.freeze
+ # Every hour bucket label, from the three adapters' SQL and the Ruby
+ # zero-fill alike, has to render identically or each bucket duplicates.
+ HOUR_BUCKET_FORMAT = '%Y-%m-%d %H:00:00'.freeze
+ # Upper bound on one population batch when filling empty rows.
+ MEMBER_BATCH_SIZE = 1_000
def execute(query)
@query = query
@@ -50,10 +59,13 @@ module Spree
# Ungrouped aggregates use the single [] key.
def aggregate(range, grouped:, push_sort: false, key_filter: nil)
rows = Hash.new { |h, k| h[k] = {} }
+ # Read by #abandoned_cutoff: a cart is judged by the clock at the end
+ # of the period being aggregated, not by today's.
+ self.aggregating_range = range
query.aggregated_metrics.group_by(&:base).each do |base, metrics|
scope = base_scope(base, range, grouped: grouped)
- selects = metrics.map { |m| "#{resolve_sql(m.sql)} AS #{metric_alias(m)}" }
+ selects = metrics.map { |m| "#{resolve_sql(m.sql_for(grouped: grouped))} AS #{metric_alias(m)}" }
if grouped
dimension_selects = query.dimensions.map do |d|
@@ -61,6 +73,7 @@ module Spree
end
scope = scope.group(query.dimensions.map { |d| Arel.sql(group_by_term(d)) })
scope = apply_key_filter(scope, key_filter) if key_filter
+ scope = apply_metric_filters(scope, metrics) unless query.include_empty?
scope = apply_sql_sort(scope, metrics) if push_sort
selects = dimension_selects + selects
end
@@ -108,6 +121,39 @@ module Spree
scope
end
+ # Metric filters compile to HAVING and so apply to the grouped query
+ # only — the Total row stays the period's real figure rather than the
+ # sum of the rows that happened to survive the filter (Decision 13:
+ # rows need not sum to the total).
+ #
+ # Skipped entirely under include_empty: HAVING would drop the rows that
+ # do not match, and the population merge would then reintroduce them as
+ # zeros — so a bestseller would come back looking like it never sold.
+ # Those queries filter once, in Ruby, over the merged values.
+ #
+ # Only filters whose metric this base aggregates are applied; a query
+ # spanning two bases runs one SQL statement per base, and a HAVING
+ # naming another base's column would not compile.
+ def apply_metric_filters(scope, metrics)
+ names = metrics.map(&:name)
+
+ query.metric_filters.each do |filter|
+ metric = filter[:metric]
+ next unless names.include?(metric.name)
+
+ # The value rides as a bind rather than an interpolation: it is
+ # the only piece of request data that reaches the statement, and
+ # the aggregate beside it is registered SQL.
+ scope = scope.having(
+ Arel.sql("#{resolve_sql(metric.sql_for(grouped: true))} " \
+ "#{COMPARISONS.fetch(filter[:op]).first} ?"),
+ filter[:value]
+ )
+ end
+
+ scope
+ end
+
# A filter on a joined dimension narrows through an id subquery rather
# than joining the aggregating scope itself: `category IN (a, b)` must
# keep a line item once even when its product sits in both categories.
@@ -152,12 +198,18 @@ module Spree
# while the ungrouped Total still counts them, so the rows quietly fail
# to add up. Left-joining produces a NULL key instead, which is a real
# group the hydration already renders as "Unassigned".
+ # How a dimension's joins are reached from the executing base is
+ # registry data (`reaches`), so nothing here branches on a base name:
+ # a dimension on this base joins directly, one on a reachable base
+ # joins through the association that base declared.
def join_for(scope, dimension, base)
return scope if dimension.joins.blank?
- return scope.left_joins(dimension.joins) if dimension.base == base
- return scope.left_joins(order: dimension.joins) if base == :line_items && dimension.base == :orders
- scope
+ definition = query.registry.base!(base)
+ return scope unless definition.reaches?(dimension.base)
+
+ path = definition.path_to(dimension.base)
+ scope.left_joins(path ? { path => dimension.joins } : dimension.joins)
end
# ---- sorted/limited rankings ----
@@ -165,11 +217,16 @@ module Spree
# ORDER BY + LIMIT can move into SQL when one base group carries every
# aggregated metric (so the sort metric and the limit apply to the
# same query) and the grouping is by value, not time buckets.
+ # include_empty is excluded: pushing LIMIT into the grouped SQL would
+ # cut the result before the dimension's full population is merged in,
+ # so the members with no rows — the ones being asked about — would be
+ # the first to fall off.
def sql_sortable?
return @sql_sortable unless @sql_sortable.nil?
@sql_sortable = query.sort && query.limit && query.time_dimension.nil? && query.dimensions.any? &&
- !sort_metric.derived? && query.aggregated_metrics.map(&:base).uniq.one? || false
+ !query.include_empty? && !sort_metric.derived? &&
+ query.aggregated_metrics.map(&:base).uniq.one? || false
end
def sort_metric
@@ -179,7 +236,8 @@ module Spree
def apply_sql_sort(scope, metrics)
return scope unless metrics.any? { |m| m.name == sort_metric.name }
- scope.order(Arel.sql("#{resolve_sql(sort_metric.sql)} #{query.sort[:direction] == :desc ? 'DESC' : 'ASC'}"))
+ scope.order(Arel.sql("#{resolve_sql(sort_metric.sql_for(grouped: true))} " \
+ "#{query.sort[:direction] == :desc ? 'DESC' : 'ASC'}"))
.limit(query.limit)
end
@@ -213,20 +271,62 @@ module Spree
# Registered fragments defer table names as %{placeholders} because
# model classes cannot load while initializers register the vocabulary.
+ # The cutoff rides along because it is the one value a fragment needs
+ # that is not a table name — see #abandoned_cutoff.
def resolve_sql(fragment)
- format(fragment,
- orders: Spree::Order.table_name,
- line_items: Spree::LineItem.table_name,
- variants: Spree::Variant.table_name,
- products: Spree::Product.table_name,
- addresses: Spree::Address.table_name,
- product_categories: Spree::ProductCategory.table_name,
- refunds: Spree::Refund.table_name,
- fees: Spree::Fee.table_name,
- commission_lines: Spree::CommissionLine.table_name,
- payments: Spree::Payment.table_name,
- stock_movements: Spree::StockMovement.table_name,
- stock_levels: Spree::StockLevel.table_name)
+ format(fragment, placeholders)
+ end
+
+ # Table names never vary, so they are built once; the cutoff is the one
+ # placeholder that depends on which period is being aggregated.
+ def placeholders
+ table_placeholders.merge(abandoned_cutoff: abandoned_cutoff)
+ end
+
+ def table_placeholders
+ @table_placeholders ||= {
+ orders: Spree::Order.table_name,
+ line_items: Spree::LineItem.table_name,
+ variants: Spree::Variant.table_name,
+ products: Spree::Product.table_name,
+ addresses: Spree::Address.table_name,
+ product_categories: Spree::ProductCategory.table_name,
+ refunds: Spree::Refund.table_name,
+ fees: Spree::Fee.table_name,
+ commission_lines: Spree::CommissionLine.table_name,
+ payments: Spree::Payment.table_name,
+ stock_movements: Spree::StockMovement.table_name,
+ stock_levels: Spree::StockLevel.table_name,
+ carts: Spree::Cart.table_name,
+ discounts: Spree::Discount.table_name,
+ promotions: Spree::Promotion.table_name
+ }
+ end
+
+ # The instant a cart must have been quiet since to count as abandoned:
+ # the store's window measured back from the end of the period being
+ # aggregated, not from now.
+ #
+ # Measuring from now would make every unconverted cart in a comparison
+ # period abandoned — that whole window lies further in the past than
+ # any cutoff — so a year-on-year abandonment chart would show a
+ # fabricated collapse. A period's carts are judged by the clock at the
+ # end of that period.
+ def abandoned_cutoff
+ hours = query.store.preferred_abandoned_cart_after_hours.to_i.hours
+ connection.quote((aggregating_range.last - hours).utc)
+ end
+
+ # The period currently being aggregated. Defaults to the query's own
+ # range so anything reading placeholders before #aggregate runs (a sort
+ # term, a dimension expression) still resolves.
+ def aggregating_range
+ @aggregating_range || query.time_range
+ end
+
+ def aggregating_range=(range)
+ @aggregating_range = range
+ @placeholders = nil
end
def dimension_alias(dim)
@@ -276,36 +376,55 @@ module Spree
# Time buckets come back as Date on PostgreSQL and String elsewhere —
# normalize to ISO strings so keys merge and zero-fill consistently.
+ # Time keys are normalized to the string form #expected_buckets emits,
+ # because each database returns a different Ruby type for a bucket
+ # (Postgres a Time for date_trunc, SQLite a String) and the zero-fill
+ # has to match the aggregated rows exactly or every bucket duplicates.
def dimension_key(dim, raw)
- dim[:dimension].time? ? raw.to_s : raw
+ return raw unless dim[:dimension].time?
+ return raw.to_s unless dim[:grain] == :hour
+
+ # MySQL and SQLite format the bucket themselves; only PostgreSQL's
+ # date_trunc hands back a Time.
+ raw.respond_to?(:strftime) ? raw.strftime(HOUR_BUCKET_FORMAT) : raw.to_s
end
# Store-timezone day/month buckets, per database. PostgreSQL converts
# properly (DST-aware); SQLite/MySQL shift by the timezone's current
# UTC offset — a documented approximation near DST boundaries.
+ # A grain this method does not know would return nil and silently
+ # group every row under one NULL bucket, so an unhandled grain raises
+ # here rather than producing a plausible-looking wrong series. Adding a
+ # grain means teaching this method and #expected_buckets together.
def time_bucket_sql(column, grain)
- case connection.adapter_name
- when /postgres/i
- local = "#{column} AT TIME ZONE 'UTC' AT TIME ZONE '#{query.time_zone.tzinfo.identifier}'"
- case grain
- when :day then "(#{local})::date"
- when :week then "date_trunc('week', #{local})::date"
- when :month then "date_trunc('month', #{local})::date"
- end
- when /mysql/i
- local = "CONVERT_TZ(#{column}, '+00:00', '#{format_offset(utc_offset)}')"
- case grain
- when :day then "DATE(#{local})"
- when :week then "DATE(DATE_SUB(#{local}, INTERVAL WEEKDAY(#{local}) DAY))"
- when :month then "DATE_FORMAT(#{local}, '%Y-%m-01')"
- end
- else # SQLite — weeks start on Monday (ISO), matching the other adapters
- case grain
- when :day then "DATE(#{column}, '#{utc_offset} seconds')"
- when :week then "DATE(#{column}, '#{utc_offset} seconds', '+1 day', 'weekday 1', '-7 days')"
- when :month then "strftime('%Y-%m-01', #{column}, '#{utc_offset} seconds')"
+ sql =
+ case connection.adapter_name
+ when /postgres/i
+ local = "#{column} AT TIME ZONE 'UTC' AT TIME ZONE '#{query.time_zone.tzinfo.identifier}'"
+ case grain
+ when :hour then "date_trunc('hour', #{local})"
+ when :day then "(#{local})::date"
+ when :week then "date_trunc('week', #{local})::date"
+ when :month then "date_trunc('month', #{local})::date"
+ end
+ when /mysql/i
+ local = "CONVERT_TZ(#{column}, '+00:00', '#{format_offset(utc_offset)}')"
+ case grain
+ when :hour then "DATE_FORMAT(#{local}, '#{HOUR_BUCKET_FORMAT}')"
+ when :day then "DATE(#{local})"
+ when :week then "DATE(DATE_SUB(#{local}, INTERVAL WEEKDAY(#{local}) DAY))"
+ when :month then "DATE_FORMAT(#{local}, '%Y-%m-01')"
+ end
+ else # SQLite — weeks start on Monday (ISO), matching the other adapters
+ case grain
+ when :hour then "strftime('#{HOUR_BUCKET_FORMAT}', #{column}, '#{utc_offset} seconds')"
+ when :day then "DATE(#{column}, '#{utc_offset} seconds')"
+ when :week then "DATE(#{column}, '#{utc_offset} seconds', '+1 day', 'weekday 1', '-7 days')"
+ when :month then "strftime('%Y-%m-01', #{column}, '#{utc_offset} seconds')"
+ end
end
- end
+
+ sql || raise(ArgumentError, "reporting adapter has no SQL for the #{grain} grain")
end
@@ -320,8 +439,13 @@ module Spree
# ---- result assembly ----
+ # A per-group metric has no dimensionless figure: MIN() of every
+ # customer's lifetime value is one arbitrary customer's, not the
+ # store's, so it is reported as nil rather than rendered as a tile.
def build_totals(current, previous)
query.metrics.to_h do |metric|
+ next [metric.name, { value: nil }] if metric.per_group?
+
value = current[:totals].fetch([], {})[metric.name] || zero_for(metric)
prev = previous && (previous[:totals].fetch([], {})[metric.name] || zero_for(metric))
[metric.name, metric_payload(value, prev)]
@@ -333,9 +457,10 @@ module Spree
keys = row_keys(current)
prev_key_map = previous ? previous_key_map(keys) : {}
+ metrics_by_name = query.metrics.index_by(&:name)
rows = keys.map do |key|
- metrics = query.metrics.index_by(&:name).transform_values do |metric|
+ metrics = metrics_by_name.transform_values do |metric|
value = current[:groups].fetch(key, {})[metric.name] || zero_for(metric)
prev = previous && (previous[:groups].fetch(prev_key_map.fetch(key, key), {})[metric.name] || zero_for(metric))
metric_payload(value, prev)
@@ -348,7 +473,7 @@ module Spree
{ dimensions: dimensions, metrics: metrics }
end
- sort_rows(rows)
+ sort_rows(filter_merged_rows(rows))
end
# Grouped keys observed in the data, plus zero-filled time buckets
@@ -356,9 +481,92 @@ module Spree
def row_keys(current)
keys = current[:groups].keys
time_dim = query.time_dimension
- return keys unless time_dim && query.dimensions.size == 1
- expected_buckets(query.time_range, time_dim[:grain]).map { |bucket| [bucket] }
+ if time_dim && query.dimensions.size == 1
+ return expected_buckets(query.time_range, time_dim[:grain]).map { |bucket| [bucket] }
+ end
+
+ empty_dim = query.dimensions.find { |d| d[:include_empty] }
+ return keys unless empty_dim
+
+ # The dimension's whole population leads, so members with no rows in
+ # the period still appear (and read zero through build_rows' own
+ # `|| zero_for`). This is what answers "which products never sold" —
+ # a HAVING alone cannot, because a product that never sold has no
+ # row to filter.
+ (empty_member_keys(empty_dim, keys) | keys)
+ end
+
+ # Members merged in by `include_empty` never went through the grouped
+ # SQL, so HAVING never saw them and they arrive zero-filled. Without
+ # re-applying the filters here, "which products never sold" would list
+ # every product: the bestsellers HAVING removed come back as zeros.
+ #
+ # Only needed for the include_empty path — everything else was already
+ # filtered in SQL, and re-testing it would be a second implementation
+ # of the same predicate.
+ def filter_merged_rows(rows)
+ return rows if query.metric_filters.empty? || !query.include_empty?
+
+ rows.select do |row|
+ query.metric_filters.all? do |filter|
+ value = row[:metrics].dig(filter[:metric].name, :value)
+ next true if value.nil?
+
+ value.to_d.public_send(COMPARISONS.fetch(filter[:op]).last, filter[:value])
+ end
+ end
+ end
+
+ # Applies any filter on the dimension itself to its own population. A
+ # filter on a *different* dimension cannot narrow this list — it
+ # describes the facts, not the members — so a query filtered by
+ # something else refuses rather than silently listing members the
+ # filter never applied to.
+ def narrow_population(relation, dimension)
+ query.filters.each do |filter|
+ next unless filter[:dimension].name == dimension.name
+
+ relation = relation.where(id: filter[:values])
+ end
+ relation
+ end
+
+ # Ids of every record the dimension can group by, store-scoped through
+ # its own lookup relation.
+ # Bounded by the query's own limit: these rows all read zero, so beyond
+ # a page of them there is nothing more to say, and an unbounded pluck
+ # would load a whole catalogue to render fifty rows.
+ def empty_member_keys(dim, observed)
+ relation = dim[:dimension].population&.call(query.store)
+ return [] if relation.nil?
+
+ # The aggregating scope is filtered, so the population must be too:
+ # asking "which of these shoes never sold" must not answer with every
+ # unsold product in the catalogue.
+ relation = narrow_population(relation, dim[:dimension])
+ # Ordered before limiting: without it two identical "which products
+ # never sold" requests can come back with different products.
+ relation = relation.reorder(:id)
+ # Only as many rows as the page can show, plus room to drop the ones
+ # that already have facts. Excluding `observed` through a NOT IN list
+ # instead would send one bind per observed key — tens of thousands on
+ # a large catalogue, past what PostgreSQL accepts at all — so the
+ # rows are fetched and filtered here.
+ wanted = query.limit
+ return fetch_member_keys(relation, observed) if wanted.nil?
+
+ keys = []
+ relation.in_batches(of: [wanted * 4, MEMBER_BATCH_SIZE].min) do |batch|
+ keys.concat(fetch_member_keys(batch, observed))
+ break if keys.size >= wanted
+ end
+ keys.first(wanted)
+ end
+
+ def fetch_member_keys(relation, observed)
+ seen = observed.to_set { |key| key.first }
+ relation.pluck(:id).reject { |id| seen.include?(id) }.map { |id| [id] }
end
# Every bucket in the range becomes a row, so an absurd range is an
@@ -376,12 +584,31 @@ module Spree
end
case grain
+ when :hour then hour_buckets(range)
when :day then (from..to).map(&:to_s)
when :week then step_buckets(from.beginning_of_week(:monday), to) { |d| d + 7 }
when :month then step_buckets(from.beginning_of_month, to, &:next_month)
end
end
+ # Hour buckets are datetimes rather than dates, so they zero-fill from
+ # the range's own edges. Query::MAX_BUCKETS has already refused a range
+ # too wide to chart at this grain.
+ # Stepping by an hour crosses the repeated local hour on a DST
+ # fall-back, where two distinct instants render as the same wall-clock
+ # string — so the labels are de-duplicated. Keeping both would render
+ # that hour twice and collide the comparison-period keys.
+ def hour_buckets(range)
+ cursor = range.first.in_time_zone(query.time_zone).change(min: 0, sec: 0)
+ last = range.last.in_time_zone(query.time_zone)
+ buckets = []
+ while cursor <= last
+ buckets << cursor.strftime(HOUR_BUCKET_FORMAT)
+ cursor += 1.hour
+ end
+ buckets.uniq
+ end
+
def step_buckets(cursor, to)
buckets = []
while cursor <= to
diff --git a/spree/core/lib/spree/reporting/default_vocabulary.rb b/spree/core/lib/spree/reporting/default_vocabulary.rb
index 63da98e408..b4186b193d 100644
--- a/spree/core/lib/spree/reporting/default_vocabulary.rb
+++ b/spree/core/lib/spree/reporting/default_vocabulary.rb
@@ -8,6 +8,103 @@ module Spree
# Model classes appear only inside lambdas and %{table} placeholders —
# nothing here may autoload during initialization.
module DefaultVocabulary
+ # A cart converts when an order was completed from it. Counted as
+ # checkouts rather than orders: a basket spanning several sellers becomes
+ # several orders from one cart, and that is one conversion.
+ # What counts as a real sale, in SQL. Spelled once because copies of it
+ # would silently diverge the first time an order gains a status.
+ COMPLETED_ORDER = "%{alias}.completed_at IS NOT NULL AND %{alias}.status <> 'canceled'".freeze
+
+ CART_CONVERTED_CONDITION = <<~SQL.squish.freeze
+ EXISTS (SELECT 1 FROM %{orders} o WHERE o.cart_id = %{carts}.id
+ AND #{format(COMPLETED_ORDER, alias: 'o')})
+ SQL
+
+ CARTS_CONVERTED_SUM = "SUM(CASE WHEN #{CART_CONVERTED_CONDITION} THEN 1 ELSE 0 END)".freeze
+
+ # A cart is abandoned when it never completed and has gone quiet for
+ # longer than the store's window. `updated_at` is the codebase's own
+ # measure of a cart going quiet — Carts::ReapExpiredJob reads the same
+ # column, and it is indexed.
+ #
+ # Deliberately NOT the reaper's population: that job additionally skips
+ # carts carrying a live payment session, because it must not destroy
+ # rows with money attached. A cart stalled mid-payment is abandoned by
+ # any merchant's reading, and it is the most recoverable kind there is.
+ #
+ # `%{abandoned_cutoff}` is filled in per query from the store's
+ # preference, so every cart metric in one report reads one cutoff.
+ ABANDONED_CONDITION = <<~SQL.squish.freeze
+ %{carts}.completed_at IS NULL AND %{carts}.updated_at < %{abandoned_cutoff}
+ SQL
+
+ ABANDONED_CARTS_SUM = "SUM(CASE WHEN #{ABANDONED_CONDITION} THEN 1 ELSE 0 END)".freeze
+ ABANDONED_VALUE_SUM = "SUM(CASE WHEN #{ABANDONED_CONDITION} THEN %{carts}.total ELSE 0 END)".freeze
+
+ # 'converted' wins over 'abandoned' so the three states stay exclusive:
+ # a cart that completed is never abandoned, however long it sat first.
+ CART_STATUS_SQL = <<~SQL.squish.freeze
+ CASE WHEN #{CART_CONVERTED_CONDITION} THEN 'converted'
+ WHEN #{ABANDONED_CONDITION} THEN 'abandoned'
+ ELSE 'active' END
+ SQL
+
+ # One customer's whole order history, correlated on the email the
+ # `customer` dimension groups by and scoped to the same store.
+ LIFETIME_HISTORY_CONDITION = <<~SQL.squish.freeze
+ h.email = %{orders}.email AND h.store_id = %{orders}.store_id
+ AND #{format(COMPLETED_ORDER, alias: 'h')}
+ SQL
+
+ # A customer's whole history, correlated on the email the `customer`
+ # dimension groups by and scoped to the same store. MIN() rather than
+ # SUM(): the subquery already covers every one of that customer's
+ # orders, so summing it once per order in the period would multiply the
+ # figure by how often they bought — and every row in a customer group
+ # carries the identical value, so the minimum is that value.
+ #
+ # Net of refunds, consistent with net_sales: a customer who returned most
+ # of what they bought is not a high-value customer. The consequence is
+ # that a refund restates the figure after the fact.
+ LIFETIME_VALUE_SUBQUERY = <<~SQL.squish.freeze
+ MIN((SELECT COALESCE(SUM(h.total), 0) - COALESCE(SUM(
+ (SELECT SUM(hr.amount) FROM %{refunds} hr WHERE hr.order_id = h.id)), 0)
+ FROM %{orders} h
+ WHERE #{LIFETIME_HISTORY_CONDITION}))
+ SQL
+
+ LIFETIME_ORDERS_SUBQUERY = <<~SQL.squish.freeze
+ MIN((SELECT COUNT(*) FROM %{orders} h WHERE #{LIFETIME_HISTORY_CONDITION}))
+ SQL
+
+ # Discount money summed from the joined discount rows themselves.
+ #
+ # Read through the join rather than a per-line subquery because a line
+ # carrying two promotions must credit each with what it actually took
+ # off: a subquery over the line gives both rows the combined figure, and
+ # the breakdown then adds up to double what the store discounted.
+ #
+ # The grouped query is the only one that carries the join (Decision 13),
+ # so the dimensionless total uses the subquery form below instead.
+ PROMOTION_DISCOUNTS_JOINED = 'SUM(-COALESCE(%{discounts}.amount, 0))'.freeze
+
+ PROMOTION_DISCOUNTS_SUBQUERY = <<~SQL.squish.freeze
+ SUM(COALESCE((SELECT -SUM(d.amount) FROM %{discounts} d
+ WHERE d.line_item_id = %{line_items}.id AND d.kind = 'promotion'), 0))
+ SQL
+
+ # Checkouts that used a promotion. Counted in checkouts rather than
+ # orders (a basket split across sellers is one redemption), and narrowed
+ # to lines actually carrying a promotion discount — without the EXISTS
+ # the ungrouped total would count every order in the period, promoted or
+ # not, since the totals query carries no dimension join to narrow it.
+ PROMOTION_REDEMPTIONS_SQL = <<~SQL.squish.freeze
+ COUNT(DISTINCT CASE WHEN EXISTS (
+ SELECT 1 FROM %{discounts} d
+ WHERE d.line_item_id = %{line_items}.id AND d.kind = 'promotion'
+ ) THEN COALESCE(%{orders}.order_group_id, %{orders}.id) END)
+ SQL
+
# Order-level money that lives on another table. Correlated on the
# order's own id so the aggregate stays one value per order — joining
# would multiply the base row and inflate every other metric in the
@@ -115,23 +212,27 @@ module Spree
registry.instance_eval do
# ---- bases ----
#
- # Two families, because they answer different questions. Sales money
+ # Four families, because they answer different questions. Sales money
# is anchored on when an order completed; a payment is anchored on
- # when it was taken, and inventory on when stock moved. Mixing them
- # in one row would put three different clocks in one table.
+ # when it was taken, inventory on when stock moved, and a cart on
+ # when it was started. Mixing them in one row would put four
+ # different clocks in one table.
# `currency` is nil when the question does not involve money — see
# Query#scope_currency. Counting orders in one currency and calling
# it "orders" would understate a multi-currency store's trade.
base :orders, family: :sales, table: '%{orders}',
time_column: '%{orders}.completed_at',
+ clock: 'anchored on when the order completed',
relation: lambda { |store, range, currency|
scope = store.orders.not_canceled.where(completed_at: range)
currency ? scope.where(currency: currency) : scope
}
- base :line_items, family: :sales, table: '%{line_items}', reaches: %i[line_items orders],
+ base :line_items, family: :sales, table: '%{line_items}',
+ reaches: { line_items: nil, orders: :order },
time_column: '%{orders}.completed_at',
+ clock: 'anchored on when the order completed',
relation: lambda { |store, range, currency|
scope = store.line_items.merge(Spree::Order.not_canceled).
where(Spree::Order.table_name => { completed_at: range })
@@ -150,6 +251,7 @@ module Spree
# join this reads is the association's own.
base :payments, family: :payments, table: '%{payments}',
time_column: '%{payments}.created_at',
+ clock: 'anchored on when the payment was taken',
relation: lambda { |store, range, currency|
scope = store.payments.where(Spree::Payment.table_name => { created_at: range })
currency ? scope.where(Spree::Order.table_name => { currency: currency }) : scope
@@ -159,11 +261,24 @@ module Spree
# through stock level → variant → product, which `for_store` owns.
base :stock_movements, family: :inventory, table: '%{stock_movements}', reaches: %i[stock_movements],
time_column: '%{stock_movements}.created_at',
+ clock: 'anchored on when stock moved',
relation: lambda { |store, range, _currency|
Spree::StockMovement.for_store(store).
where(Spree::StockMovement.table_name => { created_at: range })
}
+ # A cart is anchored on when it was started, not when it completed:
+ # the population being measured is the carts that were *started* in
+ # the period, and anchoring on completion would drop every cart that
+ # never converted — exactly the rows an abandonment report is about.
+ base :carts, family: :carts, table: '%{carts}', reaches: %i[carts],
+ time_column: '%{carts}.created_at',
+ clock: 'anchored on when the cart was started',
+ relation: lambda { |store, range, currency|
+ scope = store.carts.where(Spree::Cart.table_name => { created_at: range })
+ currency ? scope.where(Spree::Cart.table_name => { currency: currency }) : scope
+ }
+
# ---- the sales chain ----
#
# gross_sales - discounts - returns = net_sales
@@ -183,16 +298,16 @@ module Spree
# merchant recognises as "what we actually sold".
metric :net_sales, sql: "SUM(%{line_items}.pre_tax_amount) - #{LINE_ITEM_REFUNDS_SUBQUERY}",
base: :line_items, format: :money
- metric :shipping, sql: 'SUM(%{orders}.delivery_total)', base: :orders, format: :money
- metric :duties, sql: DUTIES_SUBQUERY, base: :orders, format: :money
+ metric :shipping, sql: 'SUM(%{orders}.delivery_total)', base: :orders, format: :money, suggests: 'net_sales'
+ metric :duties, sql: DUTIES_SUBQUERY, base: :orders, format: :money, suggests: 'net_sales'
# Every shopper-visible fee except duties, which the chain counts separately.
metric :fees, sql: "SUM(COALESCE(%{orders}.fee_total, 0)) - #{DUTIES_SUBQUERY}",
- base: :orders, format: :money
+ base: :orders, format: :money, suggests: 'net_sales'
metric :taxes, sql: 'SUM(%{orders}.additional_tax_total + %{orders}.included_tax_total)',
- base: :orders, format: :money
+ base: :orders, format: :money, suggests: 'net_sales'
# The order grand total, net of what came back.
metric :total_sales, sql: "SUM(%{orders}.total) - #{REFUNDS_SUBQUERY}",
- base: :orders, format: :money
+ base: :orders, format: :money, suggests: 'net_sales'
# ---- volume ----
@@ -257,7 +372,54 @@ module Spree
# the same unit, so this reads as a percentage.
metric :sell_through, ratio: %i[units_shipped units_received], format: :percent
- dimension :completed_at, base: :orders, column: :completed_at, type: :time, grains: %i[day week month]
+ # ---- carts ----
+ #
+ # What happened to the baskets that were started. Anchored on the
+ # cart's own clock, which is why these never share a query with sales.
+
+ metric :carts_started, sql: 'COUNT(*)', base: :carts, format: :integer
+ metric :carts_converted, sql: CARTS_CONVERTED_SUM, base: :carts, format: :integer
+ metric :carts_abandoned, sql: ABANDONED_CARTS_SUM, base: :carts, format: :integer
+ # What the abandoned baskets were worth — the recoverable figure.
+ metric :abandoned_value, sql: ABANDONED_VALUE_SUM, base: :carts, format: :money
+ # Of the carts started, the share that became orders. NOT the
+ # storefront conversion rate: sessions never reach Spree, so this
+ # counts baskets, not visits, and the two numbers differ.
+ metric :cart_conversion_rate, ratio: %i[carts_converted carts_started], format: :percent
+
+ # ---- promotions ----
+ #
+ # Discount money is stored non-positive (a DB check constraint), so
+ # these negate it: a merchant reads "4,210 discounted", not "-4,210".
+
+ metric :promotion_discounts, sql: PROMOTION_DISCOUNTS_SUBQUERY,
+ grouped_sql: PROMOTION_DISCOUNTS_JOINED,
+ base: :line_items, format: :money,
+ subject: -> { Spree::Promotion }, key_scope: 'read_promotions'
+ # Counted in checkouts, not orders: a basket spanning several sellers
+ # becomes several orders, and charging one customer's single
+ # redemption three times would spend a limited promotion too fast.
+ metric :promotion_redemptions, sql: PROMOTION_REDEMPTIONS_SQL,
+ base: :line_items, format: :integer,
+ subject: -> { Spree::Promotion }, key_scope: 'read_promotions'
+
+ # ---- lifetime ----
+ #
+ # A customer's whole history, deliberately ignoring the query's time
+ # range: the range picks *which* customers appear, their figure
+ # covers everything they have ever bought. Query#validate_lifetime_
+ # metrics! refuses these ungrouped, where they would add every
+ # customer's history into one meaningless total.
+
+ metric :customer_lifetime_value, sql: LIFETIME_VALUE_SUBQUERY, base: :orders, format: :money,
+ requires_grouping: :customer,
+ subject: -> { Spree.customer_class }, key_scope: 'read_customers'
+ metric :orders_lifetime, sql: LIFETIME_ORDERS_SUBQUERY, base: :orders, format: :integer,
+ requires_grouping: :customer,
+ subject: -> { Spree.customer_class }, key_scope: 'read_customers'
+
+ dimension :completed_at, base: :orders, column: :completed_at, type: :time,
+ grains: %i[hour day week month]
dimension :payment_status, base: :orders, column: :payment_status,
values: -> { Spree::Order::PAYMENT_STATUSES }
# The rollup values Orders::UpdateStatuses writes today; the model
@@ -357,6 +519,10 @@ module Spree
dimension :product, base: :line_items, column: '%{variants}.product_id', joins: [:variant],
lookup: :product,
subject: -> { Spree::Product }, key_scope: 'read_products',
+ # Live products only: a deleted product that never sold is
+ # not a merchandising gap worth reporting, though a deleted
+ # one that did sell still hydrates through the resolve path.
+ population: ->(store) { store.products },
resolve: ->(store, value) { store.products.with_deleted.find_by_prefix_id!(value).id },
# Four display fields, read directly: the product serializer
# would resolve buy-box variants, price lists, seller, type
@@ -378,6 +544,57 @@ module Spree
end
}
+ # ---- cart axes ----
+
+ dimension :started_at, base: :carts, column: :created_at, type: :time,
+ grains: %i[hour day week month]
+
+ dimension :cart_status, base: :carts, expression: CART_STATUS_SQL,
+ values: -> { %w[active abandoned converted] }
+
+ dimension :cart_channel, base: :carts, column: :channel_id, lookup: :channel,
+ resolve: ->(store, value) { store.channels.find_by_prefix_id!(value).id },
+ hydrate: lambda { |store, ids, _params|
+ store.channels.where(id: ids).to_h do |channel|
+ [channel.id, { id: channel.prefixed_id, label: channel.name, meta: {} }]
+ end
+ }
+
+ # ---- promotion axes ----
+ #
+ # Attribution reads the money rows themselves: every discount row
+ # carries promotion_id, promotion_action_id and a snapshot of the
+ # redeemed code, all indexed, so a breakdown is a plain GROUP BY.
+ # The join table carries no amount and would need a subquery to
+ # produce one.
+ #
+ # The join runs from the line item, and the :line_items base already
+ # restricts to completed orders, so cart-owned discount rows cannot
+ # reach a sales report. A discount dimension registered on a base
+ # that does include carts would need an explicit owner predicate:
+ # discount rows are order_id XOR cart_id.
+ #
+ # It joins only promotion discounts, not every discount. Including
+ # the manual ones would give a hand-discounted line a second group
+ # under "no promotion" beside its real promotion row, and the two
+ # rows would then count the same line twice.
+
+ dimension :promotion, base: :line_items, column: '%{discounts}.promotion_id',
+ joins: [:promotion_discounts], lookup: :promotion,
+ subject: -> { Spree::Promotion }, key_scope: 'read_promotions',
+ population: ->(store) { store.promotions },
+ resolve: ->(store, value) { store.promotions.find_by_prefix_id!(value).id },
+ hydrate: lambda { |store, ids, _params|
+ store.promotions.where(id: ids).to_h do |promotion|
+ [promotion.id, { id: promotion.prefixed_id, label: promotion.name, meta: {} }]
+ end
+ }
+
+ # The code is its own label, and it is snapshotted on the money row,
+ # so it survives the promotion being deleted.
+ dimension :coupon_code, base: :line_items, column: '%{discounts}.code', joins: [:promotion_discounts],
+ subject: -> { Spree::Promotion }, key_scope: 'read_promotions'
+
# ---- payment axes ----
dimension :payment_method, base: :payments, column: :payment_method_id, lookup: :payment_method,
@@ -395,12 +612,12 @@ module Spree
values: -> { Spree::Payment.statuses }
dimension :paid_at, base: :payments, column: :created_at, type: :time,
- grains: %i[day week month]
+ grains: %i[hour day week month]
# ---- inventory axes ----
dimension :moved_at, base: :stock_movements, column: :created_at, type: :time,
- grains: %i[day week month]
+ grains: %i[hour day week month]
dimension :movement_kind, base: :stock_movements, column: :kind,
values: -> { Spree::StockMovement::KINDS }
diff --git a/spree/core/lib/spree/reporting/query.rb b/spree/core/lib/spree/reporting/query.rb
index fb31fe752d..8869ab46a5 100644
--- a/spree/core/lib/spree/reporting/query.rb
+++ b/spree/core/lib/spree/reporting/query.rb
@@ -11,7 +11,15 @@ module Spree
DEFAULT_VALUE_LIMIT = 50
MAX_DIMENSIONS = 2
FILTER_OPS = %w[eq in].freeze
- COMPARE_MODES = %w[previous_period].freeze
+ # Comparison ops for metric filters. A metric filter compiles to HAVING,
+ # so it takes scalar comparisons rather than the set membership a
+ # dimension filter uses.
+ METRIC_FILTER_OPS = %w[eq gt gte lt lte].freeze
+ COMPARE_MODES = %w[previous_period previous_year].freeze
+ # Hourly buckets over a wide range produce a series nothing can read
+ # (a year is 8,760 points), so the grain is refused rather than
+ # silently coarsened (Decision 4).
+ MAX_BUCKETS = 2_000
LAST_N_RANGE = /\Alast_(\d+)_(days|weeks|months)\z/
# Named presets, resolved in the store's timezone (see #time_zone).
PRESETS = %w[today yesterday week_to_date month_to_date quarter_to_date year_to_date
@@ -21,7 +29,7 @@ module Spree
RELATIVE_PRESETS = %w[last_7_days last_30_days last_4_weeks last_90_days last_12_months].freeze
attr_reader :store, :registry, :currency, :metrics, :dimensions, :filters,
- :time_range, :compare, :sort, :limit
+ :metric_filters, :time_range, :compare, :sort, :limit
def initialize(store:, params:, registry: Spree.reporting)
@store = store
@@ -34,11 +42,15 @@ module Spree
@metrics = normalize_metrics(params[:metrics])
@dimensions = normalize_dimensions(params[:dimensions])
@filters = normalize_filters(params[:filters])
+ @metric_filters = normalize_metric_filters(params[:metric_filters])
@time_range = normalize_time_range(params[:time_range])
@compare = normalize_compare(params[:compare])
@sort = normalize_sort(params[:sort])
@limit = normalize_limit(params[:limit])
validate_bases!
+ validate_include_empty_filters!
+ validate_bucket_count!
+ validate_lifetime_metrics!
end
def execute(adapter: Spree::Dependencies.reporting_adapter.constantize.new)
@@ -106,20 +118,36 @@ module Spree
(dimensions.map { |d| d[:dimension] } + filters.map { |f| f[:dimension] }).uniq
end
- # The immediately preceding period of equal length, shifted by calendar
- # days in the store zone so a range that spans a DST change keeps its
- # midnight edges instead of drifting by an hour.
+ # The period this query compares against.
+ #
+ # `previous_period` shifts by an equal number of calendar days in the
+ # store zone, so a range spanning a DST change keeps its midnight edges.
+ # `previous_year` shifts by a calendar year instead: "the same month last
+ # year" means February to February, and an equal-day-count shift off a
+ # 28-day February would land in January. ActiveSupport maps February 29
+ # to February 28 in a non-leap year.
def previous_time_range
first = time_range.first.in_time_zone(time_zone)
last = time_range.last.in_time_zone(time_zone)
- days = (last.to_date - first.to_date).to_i + 1
- (first - days.days)..(last - days.days)
+
+ if compare == 'previous_year'
+ (first - 1.year)..(last - 1.year)
+ else
+ days = (last.to_date - first.to_date).to_i + 1
+ (first - days.days)..(last - days.days)
+ end
end
def compare?
compare.present?
end
+ # Whether a dimension's whole population leads the query, so members with
+ # no matching rows still produce a row.
+ def include_empty?
+ dimensions.any? { |d| d[:include_empty] }
+ end
+
# All metrics the adapter must aggregate: requested non-derived metrics
# plus the hidden components of requested ratios.
def aggregated_metrics
@@ -160,7 +188,8 @@ module Spree
raise InvalidQuery, 'dimensions must be an array' unless list.nil? || list.is_a?(Array)
dims = Array(list).map do |entry|
- name, grain = entry.is_a?(Hash) ? [entry[:name], entry[:grain]] : [entry, nil]
+ hash = entry.is_a?(Hash)
+ name, grain = hash ? [entry[:name], entry[:grain]] : [entry, nil]
dimension = registry.dimension!(member_name(name, 'dimension'))
if dimension.time?
@@ -172,15 +201,81 @@ module Spree
raise InvalidQuery, "dimension #{dimension.name} does not support grains"
end
- { dimension: dimension, grain: grain }
+ { dimension: dimension, grain: grain, include_empty: hash && ActiveModel::Type::Boolean.new.cast(entry[:include_empty]).present? }
end
raise InvalidQuery, "at most #{MAX_DIMENSIONS} dimensions per query" if dims.size > MAX_DIMENSIONS
raise InvalidQuery, 'at most one time dimension per query' if dims.count { |d| d[:dimension].time? } > 1
+ validate_include_empty!(dims)
dims
end
+ # `include_empty` roots the query in the dimension's own population so
+ # members with no matching rows still appear (the "which products never
+ # sold" question). It needs a relation to enumerate, and it only makes
+ # sense as the sole grouping — a second dimension would multiply the
+ # empty side into rows that never existed.
+ def validate_include_empty!(dims)
+ empty = dims.select { |d| d[:include_empty] }
+ return if empty.empty?
+
+ if dims.size > 1
+ raise InvalidQuery, 'include_empty supports exactly one dimension'
+ end
+
+ dimension = empty.first[:dimension]
+ unless dimension.population?
+ raise InvalidQuery,
+ "dimension #{dimension.name} does not support include_empty — it has no record list to draw empty rows from"
+ end
+ end
+
+ # Metric filters are a separate contract key rather than an op inside
+ # `filters` because they compile to HAVING: they run after aggregation,
+ # cannot use the id-subquery path a joined dimension filter needs, and
+ # never narrow the ungrouped totals.
+ def normalize_metric_filters(list)
+ raise InvalidQuery, 'metric_filters must be an array' unless list.nil? || list.is_a?(Array)
+
+ Array(list).map do |filter|
+ raise InvalidQuery, 'each metric filter must be an object with metric, op and value' unless filter.is_a?(Hash)
+
+ metric = registry.metric!(member_name(filter[:metric], 'metric'))
+ op = filter[:op].to_s
+ unless METRIC_FILTER_OPS.include?(op)
+ raise InvalidQuery, "invalid metric filter op #{op}. Valid ops: #{METRIC_FILTER_OPS.join(', ')}"
+ end
+
+ unless metrics.any? { |requested| requested.name == metric.name }
+ raise InvalidQuery,
+ "metric_filters may only reference a requested metric; add #{metric.name} to metrics or filter on one of #{metrics.map(&:name).join(', ')}"
+ end
+
+ # A ratio is divided in Ruby after aggregation, so its value never
+ # exists in SQL for HAVING to compare against.
+ if metric.derived?
+ raise InvalidQuery,
+ "#{metric.name} is a derived metric and cannot be filtered. Filter on one of its components instead: #{registry.components(metric).map(&:name).join(', ')}"
+ end
+
+ value = filter[:value]
+ raise InvalidQuery, "metric filter on #{metric.name} requires a numeric value" unless numeric?(value)
+
+ { metric: metric, op: op.to_sym, value: BigDecimal(value.to_s) }
+ end
+ end
+
+ # The filter value is the one piece of request data that reaches the
+ # statement as a number rather than a bound parameter, so anything that
+ # is not finite is refused here: BigDecimal('Infinity') parses happily
+ # and would render as the bare token `Infinity` in the HAVING clause.
+ def numeric?(value)
+ BigDecimal(value.to_s).finite?
+ rescue ArgumentError, TypeError
+ false
+ end
+
def normalize_filters(list)
raise InvalidQuery, 'filters must be an array' unless list.nil? || list.is_a?(Array)
@@ -283,6 +378,11 @@ module Spree
# :orders-based metrics cannot be grouped or filtered by :line_items
# dimensions (order totals per product/category would double count).
+ #
+ # The refusal names the reason and the way through: a caller who asked
+ # for total sales per product wants net_sales, and learning that from
+ # the error is the difference between a usable API and a wall. The agent
+ # surface has nothing else to learn the vocabulary from.
def validate_bases!
validate_one_family!
@@ -290,21 +390,98 @@ module Spree
return if incompatible.empty?
offenders = metrics.reject { |m| incompatible.all? { |d| registry.compatible?(m, d) } }
- raise InvalidQuery,
- "metrics #{offenders.map(&:name).join(', ')} cannot be grouped by #{incompatible.map(&:name).join(', ')}"
+ raise InvalidQuery, incompatible_base_message(offenders, incompatible)
+ end
+
+ def incompatible_base_message(offenders, dimensions)
+ names = offenders.map(&:name)
+ message = "#{names.join(', ')} cannot be grouped by #{dimensions.map(&:name).join(', ')}: " \
+ 'they measure money that belongs to the whole order (shipping, duties and tax included), ' \
+ 'so splitting them per line would count the same money under several rows.'
+
+ alternatives = offenders.filter_map(&:suggests).uniq
+ return message if alternatives.empty?
+
+ "#{message} Use #{alternatives.join(' or ')} for a per-line breakdown."
end
# Sales, payments and inventory answer different questions on different
# clocks — a payment total beside a units-received count is two reports
- # wearing one table. The refusal names the families so a caller can split
- # the query rather than guess why it was rejected.
+ # wearing one table. The refusal names each metric's clock so a caller
+ # can see why the two cannot share a row, and split the query.
def validate_one_family!
- families = aggregated_metrics.map { |metric| registry.family_of(metric) }.uniq
- return if families.size <= 1
+ by_family = aggregated_metrics.group_by { |metric| registry.family_of(metric) }
+ return if by_family.size <= 1
+
+ described = by_family.sort_by { |family, _| family.to_s }.map do |family, family_metrics|
+ clock = registry.base!(family_metrics.first.base).clock
+ "#{family_metrics.map(&:name).join(', ')} measures #{family}" \
+ "#{" (#{clock})" if clock.present?}"
+ end
raise InvalidQuery,
- "metrics from different families cannot be combined in one query: #{families.sort.join(', ')}"
+ "#{described.join(', while ')}. One query cannot report both — run them as separate queries."
end
+
+ # A filter on another dimension describes the facts, not the members, so
+ # it cannot narrow the population "include_empty" draws from: "which
+ # shoes never sold" filtered by category would list every unsold product
+ # in the store. Refused rather than answered wrongly.
+ def validate_include_empty_filters!
+ entry = dimensions.find { |d| d[:include_empty] }
+ return if entry.nil?
+
+ name = entry[:dimension].name
+ foreign = filters.map { |f| f[:dimension] }.reject { |d| d.name == name }.uniq
+ return if foreign.empty?
+
+ raise InvalidQuery,
+ "include_empty on #{name} cannot be combined with a filter on " \
+ "#{foreign.map(&:name).join(', ')} — that filter narrows what sold, not which " \
+ "#{name}s exist. Filter on #{name} instead."
+ end
+
+ def validate_bucket_count!
+ entry = time_dimension
+ return if entry.nil?
+
+ buckets = estimated_buckets(entry[:grain])
+ return if buckets.nil? || buckets <= MAX_BUCKETS
+
+ raise InvalidQuery,
+ "#{entry[:grain]} grain over this range produces #{buckets} points (the limit is #{MAX_BUCKETS}). " \
+ "Shorten the range or use a coarser grain: #{coarser_grains(entry).join(', ')}."
+ end
+
+ # Only the hour grain is capped here. Day and coarser grains are bounded
+ # by the adapter's own MAX_RANGE_DAYS, and checking them here too would
+ # answer "this range is absurdly long" with a message about grains.
+ def estimated_buckets(grain)
+ return unless grain == :hour
+
+ ((time_range.last - time_range.first) / 1.hour).ceil
+ end
+
+ def coarser_grains(entry)
+ available = entry[:dimension].grains
+ available.drop(available.index(entry[:grain]) + 1)
+ end
+
+ # A metric declaring `requires_grouping` covers more than the query's
+ # time range (a customer's whole history), so ungrouped it would collapse
+ # every group's figure into one number that answers no question. Which
+ # dimension it needs is registry data, so an extension's own metric gets
+ # the same refusal without touching the compiler.
+ def validate_lifetime_metrics!
+ aggregated_metrics.select(&:per_group?).group_by(&:requires_grouping).each do |needed, group|
+ next if dimensions.any? { |d| d[:dimension].name == needed }
+
+ raise InvalidQuery,
+ "#{group.map(&:name).join(', ')} covers more than the report's date range, " \
+ "so it must be grouped by #{needed}."
+ end
+ end
+
end
end
end
diff --git a/spree/core/lib/spree/reporting/registry.rb b/spree/core/lib/spree/reporting/registry.rb
index 318c0f8b4c..171a909455 100644
--- a/spree/core/lib/spree/reporting/registry.rb
+++ b/spree/core/lib/spree/reporting/registry.rb
@@ -19,8 +19,31 @@ module Spree
# like the marketplace's commission. Declared with key_scope.
# @!attribute key_scope
# API-key scope the same number requires.
- Metric = Struct.new(:name, :sql, :base, :format, :ratio, :subject, :key_scope, keyword_init: true) do
+ # @!attribute requires_grouping
+ # Dimension name this metric is only meaningful grouped by (e.g.
+ # :customer for a lifetime figure). Ungrouped it is refused, and it
+ # publishes no dimensionless total.
+ # @!attribute grouped_sql
+ # Optional aggregate used only for the grouped query, where dimension
+ # joins are present. See #sql_for.
+ # @!attribute suggests
+ # Metric to reach for instead when this one is refused for a breakdown
+ # it cannot answer — named in the error so a caller learns the
+ # vocabulary from the refusal rather than by trial.
+ Metric = Struct.new(:name, :sql, :base, :format, :ratio, :subject, :key_scope,
+ :requires_grouping, :suggests, :grouped_sql, keyword_init: true) do
+ # The aggregate to use when the query carries its dimension joins. A
+ # metric whose money lives on a joined row reads it directly there, and
+ # falls back to `sql` for the dimensionless total, which has no joins.
+ def sql_for(grouped:) = (grouped && grouped_sql.presence) || sql
+
def derived? = ratio.present?
+
+ # The dimension this metric only means anything inside of. A customer's
+ # lifetime value is not a figure the whole store has, so a query that
+ # does not group by that dimension is refused and the dimensionless
+ # total is suppressed rather than rendered as a headline.
+ def per_group? = requires_grouping.present?
def money? = format == :money
end
@@ -59,10 +82,18 @@ module Spree
# Enumerable raw values for status-like dimensions (an Array, or a lambda
# returning one so model constants load lazily). Published in the schema
# as the filter value list.
+ # @!attribute population
+ # Lambda (store) -> relation enumerating every record the dimension can
+ # group by. Declared rather than inferred from `lookup`, because it is
+ # what `include_empty` reads to produce rows for members with no
+ # matching facts ("which products never sold"); a dimension without one
+ # refuses `include_empty` instead of quietly returning only what sold.
Dimension = Struct.new(:name, :base, :column, :expression, :joins, :type, :grains, :lookup,
- :resolve, :hydrate, :subject, :key_scope, :values, keyword_init: true) do
+ :resolve, :hydrate, :subject, :key_scope, :values, :population, keyword_init: true) do
def time? = type == :time
+ def population? = population.present?
+
# A dimension whose key is computed rather than read from a column. The
# compiler validates `column` down to `table.column` so registration
# cannot smuggle SQL by accident; `expression` is how a developer says
@@ -101,8 +132,16 @@ module Spree
# Bases whose dimensions this one can group by, itself included. An
# :orders dimension is reachable from :line_items through the order join;
# not the reverse.
- Base = Struct.new(:name, :family, :table, :relation, :time_column, :reaches, keyword_init: true) do
- def reaches?(dimension_base) = Array(reaches).include?(dimension_base)
+ # @!attribute clock
+ # What this base's time column means, in a merchant's words ("anchored
+ # on when the order completed"). Named in the cross-family refusal so a
+ # caller can see why two metrics cannot share a row.
+ Base = Struct.new(:name, :family, :table, :relation, :time_column, :reaches, :clock, keyword_init: true) do
+ def reaches?(dimension_base) = reaches.key?(dimension_base)
+
+ # The association a dimension on `dimension_base` is joined through from
+ # here — nil when it is this base's own, so the join needs no hop.
+ def path_to(dimension_base) = reaches[dimension_base]
end
# A point-in-time count of things that need attention right now: orders
@@ -148,12 +187,23 @@ module Spree
# @param relation [Proc] ->(store, range, currency) → store-scoped relation
# @param time_column [String] table-qualified column the range filters on
# @param reaches [Array<Symbol>] bases whose dimensions this one can group by
- def base(name, replace: false, family:, table:, relation:, time_column:, reaches: nil)
+ # @param reaches [Hash, Array, nil] bases whose dimensions this one can
+ # group by, mapped to the association they are reached through (nil for
+ # its own). An Array is read as "reachable, on this base's own table".
+ def base(name, replace: false, family:, table:, relation:, time_column:, reaches: nil, clock: nil)
name = name.to_sym
raise ArgumentError, "base #{name} already registered (pass replace: true to override)" if @bases.key?(name) && !replace
@bases[name] = Base.new(name: name, family: family, table: table, relation: relation,
- time_column: time_column, reaches: reaches || [name])
+ time_column: time_column, reaches: normalize_reaches(name, reaches),
+ clock: clock)
+ end
+
+ def normalize_reaches(name, reaches)
+ return { name => nil } if reaches.nil?
+ return reaches.to_h { |base_name| [base_name, nil] } if reaches.is_a?(Array)
+
+ reaches
end
def base!(name)
@@ -182,6 +232,10 @@ module Spree
raise ArgumentError, "dimension #{name} needs a column or an expression"
end
+ if opts[:population] && !opts[:population].respond_to?(:call)
+ raise ArgumentError, "dimension #{name} needs a callable population"
+ end
+
opts[:type] ||= :value
@dimensions[name] = Dimension.new(name: name, **opts)
end
diff --git a/spree/core/lib/spree/reporting/schema.rb b/spree/core/lib/spree/reporting/schema.rb
index 6d9f838030..bc26b66a93 100644
--- a/spree/core/lib/spree/reporting/schema.rb
+++ b/spree/core/lib/spree/reporting/schema.rb
@@ -38,7 +38,12 @@ module Spree
},
# Ranking defaults the compiler applies, published so a client never
# has to mirror them.
- limits: { default: Query::DEFAULT_VALUE_LIMIT, max: Query::MAX_LIMIT }
+ limits: { default: Query::DEFAULT_VALUE_LIMIT, max: Query::MAX_LIMIT,
+ max_buckets: Query::MAX_BUCKETS },
+ # Filtering on an aggregate (HAVING) is a separate contract key from
+ # `filters`, which narrow on dimensions before aggregation.
+ metric_filter_ops: Query::METRIC_FILTER_OPS,
+ compare_modes: Query::COMPARE_MODES
}
end
@@ -72,7 +77,11 @@ module Spree
description: translate('metrics', metric.name, :description),
format: metric.format,
currency: (metric.money? ? @store.default_currency : nil),
- derived: metric.derived?
+ derived: metric.derived?,
+ # A ratio is divided after aggregation, so it can be neither sorted
+ # nor filtered in SQL. Published so a builder disables those controls
+ # rather than letting a merchant discover it through a 422.
+ filterable: !metric.derived?
}.compact
end
@@ -85,6 +94,9 @@ module Spree
grains: dimension.grains,
lookup: dimension.lookup,
filter_ops: Query::FILTER_OPS,
+ # Whether this dimension can lead a query with include_empty, which
+ # is what answers "which of these never had any activity".
+ supports_include_empty: dimension.population?,
values: dimension.enumerated_values&.map { |value| { name: value, label: self.class.value_label(dimension, value) } },
# Order-level metrics cannot be broken down by line-item dimensions
# (they would double count) — the compiler enforces the same rule.
diff --git a/spree/core/spec/lib/spree/reporting/query_spec.rb b/spree/core/spec/lib/spree/reporting/query_spec.rb
index e63dbc100a..7ce4d98135 100644
--- a/spree/core/spec/lib/spree/reporting/query_spec.rb
+++ b/spree/core/spec/lib/spree/reporting/query_spec.rb
@@ -357,9 +357,10 @@ RSpec.describe Spree::Reporting::Query do
expect(result.totals[:payments_failed][:value]).to eq(1)
end
- it 'refuses to answer a payments question and a sales question at once' do
+ it 'refuses to answer a payments question and a sales question at once, naming each clock' do
expect { run(metrics: %w[net_payments total_sales]) }
- .to raise_error(Spree::Reporting::InvalidQuery, /different families/)
+ .to raise_error(Spree::Reporting::InvalidQuery,
+ /net_payments measures payments.*total_sales measures sales.*separate queries/m)
end
it 'refuses a payments metric grouped by a sales axis' do
@@ -762,4 +763,307 @@ RSpec.describe Spree::Reporting::Query do
end
end
end
+
+ describe 'metric filters' do
+ let!(:sold) { create(:completed_order_with_totals, store: store, completed_at: 3.days.ago) }
+
+ it 'filters grouped rows on an aggregate value' do
+ result = run(metrics: %w[units_sold], dimensions: %w[product],
+ metric_filters: [{ metric: 'units_sold', op: 'gte', value: 1 }])
+
+ expect(result.rows).to be_present
+ expect(result.rows.map { |row| row[:metrics][:units_sold][:value] }).to all(be >= 1)
+ end
+
+ it 'excludes rows below the threshold' do
+ result = run(metrics: %w[units_sold], dimensions: %w[product],
+ metric_filters: [{ metric: 'units_sold', op: 'gt', value: 10_000 }])
+
+ expect(result.rows).to be_empty
+ end
+
+ it 'leaves the ungrouped total as the period figure' do
+ filtered = run(metrics: %w[units_sold], dimensions: %w[product],
+ metric_filters: [{ metric: 'units_sold', op: 'gt', value: 10_000 }])
+ unfiltered = run(metrics: %w[units_sold], dimensions: %w[product])
+
+ expect(filtered.rows).to be_empty
+ expect(filtered.totals[:units_sold][:value]).to eq(unfiltered.totals[:units_sold][:value])
+ expect(filtered.totals[:units_sold][:value]).to be > 0
+ end
+
+ it 'rejects a metric the query does not request' do
+ expect { run(metrics: %w[orders], metric_filters: [{ metric: 'units_sold', op: 'eq', value: 0 }]) }
+ .to raise_error(Spree::Reporting::InvalidQuery, /only reference a requested metric/)
+ end
+
+ it 'rejects a derived metric, naming its components' do
+ expect do
+ run(metrics: %w[average_order_value],
+ metric_filters: [{ metric: 'average_order_value', op: 'gt', value: 1 }])
+ end.to raise_error(Spree::Reporting::InvalidQuery, /derived metric.*total_sales/m)
+ end
+
+ it 'rejects a non-numeric value' do
+ expect { run(metrics: %w[orders], metric_filters: [{ metric: 'orders', op: 'gt', value: 'lots' }]) }
+ .to raise_error(Spree::Reporting::InvalidQuery, /numeric/)
+ end
+
+ it 'rejects an unknown op' do
+ expect { run(metrics: %w[orders], metric_filters: [{ metric: 'orders', op: 'like', value: 1 }]) }
+ .to raise_error(Spree::Reporting::InvalidQuery, /invalid metric filter op/)
+ end
+
+ # The value is the one piece of request data interpolated into the
+ # statement as a number, so a non-finite one must be refused rather than
+ # rendered as the bare token `Infinity`.
+ it 'rejects a non-finite value' do
+ expect { run(metrics: %w[orders], metric_filters: [{ metric: 'orders', op: 'gt', value: 'Infinity' }]) }
+ .to raise_error(Spree::Reporting::InvalidQuery, /numeric/)
+ end
+ end
+
+ describe 'include_empty' do
+ let!(:order) { create(:completed_order_with_totals, store: store, completed_at: 3.days.ago) }
+ let!(:never_sold) { create(:product, store: store, name: 'Never Sold') }
+
+ # The bestseller must be absent, not merely present-and-zero: HAVING drops
+ # it from the grouped SQL, and merging the population back in would
+ # otherwise reintroduce it as a zero and call it unsold.
+ it 'answers which products never sold' do
+ sold = order.line_items.first.variant.product
+ result = run(metrics: %w[units_sold],
+ dimensions: [{ name: 'product', include_empty: true }],
+ metric_filters: [{ metric: 'units_sold', op: 'eq', value: 0 }])
+
+ ids = result.rows.map { |row| row[:dimensions][:product] }
+ expect(ids).to include(never_sold.id)
+ expect(ids).not_to include(sold.id)
+ expect(result.rows.map { |row| row[:metrics][:units_sold][:value] }).to all(eq(0))
+ end
+
+ it 'omits the unsold product without it' do
+ result = run(metrics: %w[units_sold], dimensions: %w[product])
+
+ expect(result.rows.map { |row| row[:dimensions][:product] }).not_to include(never_sold.id)
+ end
+
+ it 'lists every unsold product, not just the first page' do
+ create_list(:product, 3, store: store)
+
+ result = run(metrics: %w[units_sold],
+ dimensions: [{ name: 'product', include_empty: true }],
+ sort: '-units_sold',
+ metric_filters: [{ metric: 'units_sold', op: 'eq', value: 0 }])
+
+ # Four unsold products exist; a pushed-down LIMIT would have cut the
+ # population merge before they were all in.
+ expect(result.rows.size).to eq(4)
+ end
+
+ # The filter narrows what sold, not which products exist, so answering
+ # would list every unsold product in the store rather than every unsold
+ # product in that category.
+ it 'refuses a filter on another dimension' do
+ category = create(:category, store: store)
+
+ expect do
+ run(metrics: %w[units_sold],
+ dimensions: [{ name: 'product', include_empty: true }],
+ filters: [{ dimension: 'category', op: 'eq', value: category.prefixed_id }])
+ end.to raise_error(Spree::Reporting::InvalidQuery, /cannot be combined with a filter on category/)
+ end
+
+ it 'refuses a dimension with no population to draw from' do
+ expect { run(metrics: %w[orders], dimensions: [{ name: 'payment_status', include_empty: true }]) }
+ .to raise_error(Spree::Reporting::InvalidQuery, /does not support include_empty/)
+ end
+
+ it 'refuses more than one dimension' do
+ expect do
+ run(metrics: %w[units_sold], dimensions: [{ name: 'product', include_empty: true }, 'category'])
+ end.to raise_error(Spree::Reporting::InvalidQuery, /exactly one dimension/)
+ end
+ end
+
+ describe 'the hour grain' do
+ # Pinned to midday: `3.hours.ago` run between midnight and 03:00 falls on
+ # the previous day, which the `today` preset would then exclude.
+ let(:midday) { Time.current.in_time_zone(store.preferred_timezone).change(hour: 12) }
+ let!(:order) { create(:completed_order_with_totals, store: store, completed_at: midday) }
+
+ it 'buckets sales by hour in the store timezone' do
+ result = Timecop.freeze(midday + 1.hour) do
+ run(metrics: %w[orders], dimensions: [{ name: 'completed_at', grain: 'hour' }],
+ time_range: { preset: 'today' })
+ end
+
+ expect(result.rows).to be_present
+ expect(result.rows.first[:dimensions][:completed_at]).to match(/\A\d{4}-\d{2}-\d{2} \d{2}:00:00\z/)
+ expect(result.rows.sum { |row| row[:metrics][:orders][:value] }).to eq(1)
+ end
+
+ it 'refuses a range too wide to chart hourly, naming a coarser grain' do
+ expect do
+ run(metrics: %w[orders], dimensions: [{ name: 'completed_at', grain: 'hour' }],
+ time_range: { preset: 'last_12_months' })
+ end.to raise_error(Spree::Reporting::InvalidQuery, /hour grain.*coarser grain.*day/m)
+ end
+ end
+
+ describe 'year-over-year comparison' do
+ let!(:this_year) { create(:completed_order_with_totals, store: store, completed_at: 2.days.ago) }
+ let!(:last_year) { create(:completed_order_with_totals, store: store, completed_at: 2.days.ago - 1.year) }
+
+ it 'compares against the same range a calendar year earlier' do
+ result = run(metrics: %w[orders], time_range: { preset: 'last_7_days' }, compare: 'previous_year')
+
+ expect(result.totals[:orders][:value]).to eq(1)
+ expect(result.totals[:orders][:previous]).to eq(1)
+ expect(result.meta[:previous_time_range].first.year).to eq(result.meta[:time_range].first.year - 1)
+ end
+
+ it 'rejects an unknown compare mode' do
+ expect { run(metrics: %w[orders], compare: 'previous_decade') }
+ .to raise_error(Spree::Reporting::InvalidQuery, /invalid compare mode/)
+ end
+ end
+
+ describe 'lifetime metrics' do
+ let(:customer) { create(:user, email: '[email protected]') }
+ let!(:old_order) do
+ create(:completed_order_with_totals, store: store, customer: customer, completed_at: 300.days.ago)
+ end
+ let!(:recent_order) do
+ create(:completed_order_with_totals, store: store, customer: customer, completed_at: 2.days.ago)
+ end
+
+ it 'counts history outside the report range' do
+ result = run(metrics: %w[orders_lifetime], dimensions: %w[customer],
+ time_range: { preset: 'last_7_days' })
+
+ expect(result.rows).to be_present
+ expect(result.rows.map { |r| r[:metrics][:orders_lifetime][:value] }).to include(2)
+ end
+
+ # MIN() over the period would report one arbitrary customer's history as
+ # though it were the store's.
+ it 'reports no dimensionless total for a lifetime metric' do
+ result = run(metrics: %w[orders_lifetime], dimensions: %w[customer],
+ time_range: { preset: 'last_7_days' })
+
+ expect(result.totals[:orders_lifetime][:value]).to be_nil
+ end
+
+ it 'refuses to report a lifetime figure ungrouped' do
+ expect { run(metrics: %w[customer_lifetime_value]) }
+ .to raise_error(Spree::Reporting::InvalidQuery, /grouped by customer/)
+ end
+ end
+
+ describe 'the carts family' do
+ let!(:abandoned) do
+ create(:cart, store: store).tap do |cart|
+ cart.update_columns(created_at: 3.days.ago, updated_at: 3.days.ago, total: 50)
+ end
+ end
+ let!(:active) do
+ create(:cart, store: store).tap { |cart| cart.update_columns(created_at: 2.days.ago, total: 20) }
+ end
+
+ it 'counts started and abandoned carts, and what the abandoned ones were worth' do
+ result = run(metrics: %w[carts_started carts_abandoned abandoned_value])
+
+ expect(result.totals[:carts_started][:value]).to eq(2)
+ expect(result.totals[:carts_abandoned][:value]).to eq(1)
+ expect(result.totals[:abandoned_value][:value]).to eq(50.0)
+ end
+
+ it 'reads the store preference for the quiet window' do
+ allow(store).to receive(:preferred_abandoned_cart_after_hours).and_return(24 * 10)
+
+ expect(run(metrics: %w[carts_abandoned]).totals[:carts_abandoned][:value]).to eq(0)
+ end
+
+ it 'breaks carts down by status' do
+ rows = run(metrics: %w[carts_started], dimensions: %w[cart_status]).rows
+ statuses = rows.to_h { |row| [row[:dimensions][:cart_status], row[:metrics][:carts_started][:value]] }
+
+ expect(statuses['abandoned']).to eq(1)
+ expect(statuses['active']).to eq(1)
+ end
+
+ # Measuring the cutoff from now rather than from each period's own end
+ # would make every unconverted cart in the comparison window abandoned.
+ it 'judges the comparison period by its own clock' do
+ result = run(metrics: %w[carts_abandoned], time_range: { preset: 'last_7_days' },
+ compare: 'previous_period')
+
+ expect(result.totals[:carts_abandoned][:value]).to eq(1)
+ expect(result.totals[:carts_abandoned][:previous]).to eq(0)
+ end
+
+ it 'refuses to report carts beside sales' do
+ expect { run(metrics: %w[carts_started total_sales]) }
+ .to raise_error(Spree::Reporting::InvalidQuery, /carts.*sales.*separate queries/m)
+ end
+ end
+
+ describe 'promotion attribution' do
+ let!(:order) { create(:completed_order_with_totals, store: store, completed_at: 3.days.ago) }
+ let!(:promotion) { create(:promotion, store: store, name: 'Spring Sale') }
+ let!(:discount) do
+ Spree::Discount.create!(order: order, line_item: order.line_items.first, promotion: promotion,
+ kind: 'promotion', label: 'Spring Sale', amount: -5, code: 'SPRING')
+ end
+
+ it 'reports discount money per promotion as a positive amount' do
+ result = run(metrics: %w[promotion_discounts], dimensions: %w[promotion])
+
+ expect(result.rows.first[:dimensions][:promotion]).to eq(promotion.id)
+ expect(result.rows.first[:metrics][:promotion_discounts][:value]).to eq(5.0)
+ end
+
+ # The ungrouped totals query carries no dimension joins (Decision 13), so a
+ # metric reading a joined column would fail to compile there.
+ it 'totals discount money without the grouping join' do
+ result = run(metrics: %w[promotion_discounts])
+
+ expect(result.totals[:promotion_discounts][:value]).to eq(5.0)
+ end
+
+ it 'breaks discounts down by the redeemed code' do
+ result = run(metrics: %w[promotion_discounts], dimensions: %w[coupon_code])
+
+ expect(result.rows.first[:dimensions][:coupon_code]).to eq('SPRING')
+ expect(result.rows.first[:metrics][:promotion_discounts][:value]).to eq(5.0)
+ end
+
+ it 'attributes revenue to the promotion that discounted it' do
+ result = run(metrics: %w[net_sales], dimensions: %w[promotion])
+
+ expect(result.rows.first[:metrics][:net_sales][:value]).to be > 0
+ end
+
+ # A per-line subquery would credit each promotion with the line's combined
+ # discount, so the two rows would sum to double what was taken off.
+ it 'splits a line carrying two promotions between them' do
+ second = create(:promotion, store: store, name: 'Summer Sale')
+ Spree::Discount.create!(order: order, line_item: order.line_items.first, promotion: second,
+ kind: 'promotion', label: 'Summer Sale', amount: -2)
+
+ result = run(metrics: %w[promotion_discounts], dimensions: %w[promotion])
+
+ by_promotion = result.rows.to_h { |row| [row[:dimensions][:promotion], row[:metrics][:promotion_discounts][:value]] }
+ expect(by_promotion[promotion.id]).to eq(5.0)
+ expect(by_promotion[second.id]).to eq(2.0)
+ expect(result.totals[:promotion_discounts][:value]).to eq(7.0)
+ end
+
+ it 'counts only checkouts that actually used a promotion' do
+ create(:completed_order_with_totals, store: store, completed_at: 2.days.ago)
+
+ expect(run(metrics: %w[promotion_redemptions]).totals[:promotion_redemptions][:value]).to eq(1)
+ end
+ end
end
diff --git a/spree/core/spec/lib/spree/reporting/schema_spec.rb b/spree/core/spec/lib/spree/reporting/schema_spec.rb
index 266f5fe14b..ef8add9228 100644
--- a/spree/core/spec/lib/spree/reporting/schema_spec.rb
+++ b/spree/core/spec/lib/spree/reporting/schema_spec.rb
@@ -29,7 +29,7 @@ RSpec.describe Spree::Reporting::Schema do
it 'describes the time range grammar and store meta' do
expect(schema[:time_range][:presets].map { |p| p[:name] }).to include('last_month', 'yesterday', 'last_30_days')
- expect(schema[:limits]).to eq(default: 50, max: 1000)
+ expect(schema[:limits]).to eq(default: 50, max: 1000, max_buckets: 2000)
expect(schema[:time_range][:presets].first[:label]).to be_present
expect(schema[:meta]).to include(:currency, :timezone, :supported_currencies)
end
@@ -44,7 +44,7 @@ RSpec.describe Spree::Reporting::Schema do
it 'groups members into the families that can be queried together' do
families = schema[:families].index_by { |f| f[:name] }
- expect(families.keys).to contain_exactly(:sales, :payments, :inventory)
+ expect(families.keys).to contain_exactly(:sales, :payments, :inventory, :carts)
expect(families[:sales][:metrics]).to include(:net_sales, :total_sales, :average_order_value)
expect(families[:payments][:metrics]).to include(:net_payments)
expect(families[:payments][:dimensions]).to include(:payment_method, :paid_at)