Skip to content

Open-source manufacturing ERP, MES and QMS. Quoting, MRP, inventory, shop floor, quality and lot/serial traceability on one Postgres schema, with a REST API and MCP server. Self-host or use Carbon Cloud.

fix(tables): a failed later page shows Retry instead of the error screen, export compose survives a retry, unordered RPC lists keep their order

aashuf2856f16cbf2
diff --git a/.claude/rules/table-csv-export.md b/.claude/rules/table-csv-export.md
index 88baa535ad..4a4e2af560 100644
--- a/.claude/rules/table-csv-export.md
+++ b/.claude/rules/table-csv-export.md
@@ -158,7 +158,8 @@ list and re-read every loaded page. The steps:
If `trigger` fails, the route deletes the lock.
5. Step `check-permission`: the job reads the user's claims again
(`canReadList`, `utils/table-jobs.server.ts`). If the user lost the list's
- permission, it notifies "Your export did not run" in-app and stops.
+ permission, it notifies "Your export did not run" in-app and stops. A
+ failed claims read throws, so the step retries instead.
6. Steps `write-part-0`, `write-part-1`, …: each step extends the lock, then
reads at most 20 pages of 500 rows (`TABLE_JOB_ROWS_PER_STEP`, 10,000,
`table-jobs/paging.ts`) as the user (`getUserScopedClient`, `fetchListPage`).
@@ -168,11 +169,12 @@ list and re-read every loaded page. The steps:
7. The parts stop at the end of the list, at 100,000 rows (`MAX_EXPORT_ROWS`),
or after the page that passes 200 MB (`MAX_EXPORT_BYTES`, `table-jobs/csv.ts`).
8. Step `compose` uploads one `ReadableStream` that downloads the parts one at
- a time (`composeExport`, `concatParts`). Only one part is in memory. Then
- it deletes the parts.
-9. Step `notify` sends `NotificationEvent.TableExport`. The title says "first
- N rows" when the export stopped at a limit.
-10. Step `release-lock` deletes the lock.
+ a time (`composeExport`, `concatParts`). Only one part is in memory.
+9. Step `remove-parts` deletes the parts (`removeExportParts`). It is its own
+ step so a retried `compose` still finds them.
+10. Step `notify` sends `NotificationEvent.TableExport`. The title says "first
+ N rows" when the export stopped at a limit.
+11. Step `release-lock` deletes the lock.
| File | Key (`table-jobs/storage.ts`, bucket `temp-staging`) |
|---|---|
diff --git a/.claude/rules/table-infinite-scroll.md b/.claude/rules/table-infinite-scroll.md
index 481fcb5fde..9e87149398 100644
--- a/.claude/rules/table-infinite-scroll.md
+++ b/.claude/rules/table-infinite-scroll.md
@@ -72,6 +72,9 @@ search the query reads. On an error:
- A later page logs it and returns `{ [dataKey]: [], nextCursor: null, listError }`
(`InfiniteListError`). The Table shows "Couldn't load more rows" with Retry.
+ A later page that throws (not a `Response`) is caught and answered the same
+ way: a fetcher whose loader throws replaces the list with the route's error
+ screen.
- Page 1 throws `redirect(onError.redirectTo)` with an error flash, or shows no
rows without `onError`.
@@ -108,9 +111,10 @@ infinite={{ dataKey: "customers", nextCursor, listKey: "customers" }}
- 🛑 Pass the loader's `data` array to `<Table infinite>`, or a memoized one.
`useInfiniteRows` reads a new array as a revalidation and re-reads every
loaded page. A copy built in the render body (`data.map(…)`) re-reads them
- on every render. `test/infinite-table-data-identity.test.ts` checks this, but
- its regex reads only the `<Table …>` elements with no `>` inside: most
- tables slip past it (open, see the 2026-10-14 review).
+ on every render. `test/infinite-table-data-identity.test.ts` checks this: it
+ parses every `.tsx` with the TypeScript compiler and fails on an infinite
+ `<Table>` whose `data` is a `.map`/`.filter`/`.slice`/`.concat`/`.toSorted`
+ call or a `[...x]` spread, inline or in a variable.
- Column widths: only the rows in view are rendered (`useVirtualRows`, 10
rows of overscan), so table layout alone would narrow and widen a column as
rows scroll by. `useGrowOnlyColumnWidths` holds each column at its widest so
diff --git a/apps/erp/app/components/Table/infiniteRowsStore.ts b/apps/erp/app/components/Table/infiniteRowsStore.ts
index ac75479ae4..60a32ce313 100644
--- a/apps/erp/app/components/Table/infiniteRowsStore.ts
+++ b/apps/erp/app/components/Table/infiniteRowsStore.ts
@@ -204,7 +204,9 @@ export type PageResult<T> =
/**
* Reads one fetched page from the fetcher's data. It is an error when:
* - the data is still the object seen before the request (a loader that
- * threw leaves no new data: the router drops the fetcher),
+ * threw leaves no new data: the router drops the fetcher and shows the
+ * route's error screen, so `loadListPage` turns a later page's throw into
+ * `listError`),
* - the loader returned `listError` (`InfiniteListError`),
* - the data has no rows array.
*
diff --git a/apps/erp/app/utils/list-registry/load.server.ts b/apps/erp/app/utils/list-registry/load.server.ts
index 84415fe440..caf2b18d76 100644
--- a/apps/erp/app/utils/list-registry/load.server.ts
+++ b/apps/erp/app/utils/list-registry/load.server.ts
@@ -132,43 +132,63 @@ export async function loadListPage<
await options.guard?.(context);
if (options.params) context.params = options.params(context);
- const [page, extra] = await Promise.all([
- fetchListPage(
- list,
- auth.client,
- auth.companyId,
- context.params,
- undefined,
- { companyGroupId: auth.companyGroupId }
- ),
- after ? undefined : options.firstPage?.(context)
- ]);
+ const loadPage = async () => {
+ const [page, extra] = await Promise.all([
+ fetchListPage(
+ list,
+ auth.client,
+ auth.companyId,
+ context.params,
+ undefined,
+ { companyGroupId: auth.companyGroupId }
+ ),
+ after ? undefined : options.firstPage?.(context)
+ ]);
- if (page.error) {
+ if (page.error) {
+ logger.error("Failed to load a page of the list {listKey}", {
+ listKey: list.key,
+ companyId: auth.companyId,
+ laterPage: Boolean(after),
+ error: page.error
+ });
+ if (after) return listErrorPage(options.dataKey);
+ if (options.onError) {
+ throw redirect(
+ options.onError.redirectTo,
+ await flash(request, error(page.error, options.onError.message))
+ );
+ }
+ }
+
+ const rows =
+ options.mapRows && !page.error
+ ? await options.mapRows(page.rows, context)
+ : page.rows;
+
+ return {
+ ...extra,
+ [options.dataKey]: rows,
+ count: page.count,
+ nextCursor: page.nextCursor
+ } as ListLoaderData<TKey, TOut> & TExtra;
+ };
+
+ if (!after) return loadPage();
+ // A later page is read by the Table's fetcher, and a fetcher whose loader
+ // throws replaces the whole list with the route's error screen. So any
+ // error becomes `listError` ("Couldn't load more rows" with Retry). A thrown
+ // Response (a redirect, a refusal) is the router's to handle.
+ try {
+ return await loadPage();
+ } catch (thrown) {
+ if (thrown instanceof Response) throw thrown;
logger.error("Failed to load a page of the list {listKey}", {
listKey: list.key,
companyId: auth.companyId,
- laterPage: Boolean(after),
- error: page.error
+ laterPage: true,
+ error: thrown
});
- if (after) return listErrorPage(options.dataKey);
- if (options.onError) {
- throw redirect(
- options.onError.redirectTo,
- await flash(request, error(page.error, options.onError.message))
- );
- }
+ return listErrorPage(options.dataKey);
}
-
- const rows =
- options.mapRows && !page.error
- ? await options.mapRows(page.rows, context)
- : page.rows;
-
- return {
- ...extra,
- [options.dataKey]: rows,
- count: page.count,
- nextCursor: page.nextCursor
- } as ListLoaderData<TKey, TOut> & TExtra;
}
diff --git a/apps/erp/app/utils/query.test.ts b/apps/erp/app/utils/query.test.ts
index fcfc072730..d4a51b5b5d 100644
--- a/apps/erp/app/utils/query.test.ts
+++ b/apps/erp/app/utils/query.test.ts
@@ -258,6 +258,51 @@ describe("setGenericQueryFilters paging", () => {
expect(query.limit).toHaveBeenCalledWith(101);
});
+ it("breaks ties by the id after the list's order", () => {
+ const query = createQueryStub();
+ setGenericQueryFilters(query as never, { limit: 100, cursor: null }, sorts);
+ expect(query.order.mock.calls).toEqual([
+ ["createdAt", { ascending: false, foreignTable: undefined }],
+ ["id", { ascending: true }]
+ ]);
+ });
+
+ it("breaks ties by the list's own id column", () => {
+ const query = createQueryStub();
+ setGenericQueryFilters(
+ query as never,
+ { sorts: [{ sortBy: "name", sortAsc: true }], limit: 50, offset: 0 },
+ { defaultSorts: sorts, idColumn: "itemId" }
+ );
+ expect(query.order.mock.calls).toEqual([
+ ["name", { ascending: true, foreignTable: undefined }],
+ ["itemId", { ascending: true }]
+ ]);
+ });
+
+ it("adds no second id order when the order already ends with the id", () => {
+ const query = createQueryStub();
+ setGenericQueryFilters(query as never, { limit: 100, cursor: null }, [
+ { column: "id", ascending: false }
+ ]);
+ expect(query.order.mock.calls).toEqual([
+ ["id", { ascending: false, foreignTable: undefined }]
+ ]);
+ });
+
+ it("orders by the id alone when a cursor list has no sorts", () => {
+ const query = createQueryStub();
+ setGenericQueryFilters(query as never, { limit: 100, cursor: null });
+ expect(query.order.mock.calls).toEqual([["id", { ascending: true }]]);
+ });
+
+ it("leaves a query nothing orders in its own order (an RPC's ORDER BY)", () => {
+ const query = createQueryStub();
+ setGenericQueryFilters(query as never, { limit: 1000, offset: 0 });
+ expect(query.order).not.toHaveBeenCalled();
+ expect(query.range).toHaveBeenCalledWith(0, 999);
+ });
+
it("does not page at all without a limit", () => {
const query = createQueryStub();
setGenericQueryFilters(query as never, {}, sorts);
diff --git a/apps/erp/app/utils/query.ts b/apps/erp/app/utils/query.ts
index 284cde9d10..e7f5419e3a 100644
--- a/apps/erp/app/utils/query.ts
+++ b/apps/erp/app/utils/query.ts
@@ -237,7 +237,10 @@ export function setGenericQueryFilters<
});
// The id breaks ties, so the order is total and every row has one place.
- if (keyset[keyset.length - 1]?.column !== idColumn) {
+ // A query nothing orders keeps its own order (an RPC's ORDER BY), unless
+ // it pages by cursor, which needs the id order.
+ const ordered = orders.length > 0 || args.cursor !== undefined;
+ if (ordered && keyset[keyset.length - 1]?.column !== idColumn) {
query = query.order(idColumn, { ascending: true });
}
diff --git a/apps/erp/test/infinite-table-data-identity.test.ts b/apps/erp/test/infinite-table-data-identity.test.ts
index 68ec4a1dc9..36176a6e87 100644
--- a/apps/erp/test/infinite-table-data-identity.test.ts
+++ b/apps/erp/test/infinite-table-data-identity.test.ts
@@ -4,6 +4,7 @@
import { readdirSync, readFileSync, statSync } from "node:fs";
import { join, relative } from "node:path";
+import ts from "typescript";
import { describe, expect, it } from "vitest";
/**
@@ -16,6 +17,8 @@ import { describe, expect, it } from "vitest";
const APP = join(__dirname, "..", "app");
+const COPY_METHODS = new Set(["map", "filter", "slice", "concat", "toSorted"]);
+
function tsxFiles(dir: string): string[] {
return readdirSync(dir).flatMap((name) => {
const path = join(dir, name);
@@ -24,24 +27,93 @@ function tsxFiles(dir: string): string[] {
});
}
-/** The identifier passed as `data={…}` on a JSX element that has `infinite=`. */
-function infiniteDataNames(source: string): string[] {
- const names: string[] = [];
- for (const match of source.matchAll(/<Table\b[^>]*?>/gs)) {
- const element = match[0];
- if (!/\binfinite=\{/.test(element)) continue;
- const data = /\bdata=\{\s*([A-Za-z_$][\w$]*)\s*\}/.exec(element);
- if (data) names.push(data[1]);
+function parse(source: string) {
+ return ts.createSourceFile(
+ "file.tsx",
+ source,
+ ts.ScriptTarget.Latest,
+ true,
+ ts.ScriptKind.TSX
+ );
+}
+
+function unwrap(node: ts.Expression): ts.Expression {
+ let current = node;
+ while (
+ ts.isParenthesizedExpression(current) ||
+ ts.isAsExpression(current) ||
+ ts.isNonNullExpression(current) ||
+ ts.isSatisfiesExpression(current)
+ ) {
+ current = current.expression;
}
- return names;
+ return current;
}
-/** True when `name` is built in the render body by copying an array. */
-function isRenderCopy(source: string, name: string): boolean {
- const copy = new RegExp(
- String.raw`const\s+${name}\s*=\s*(?:[\w$.?]+\s*\.\s*(?:map|filter|slice|concat|toSorted)\s*\(|\[\s*\.\.\.)`
+/** A new array on every evaluation: `x.map(…)` and the like, or `[...x]`. */
+function isArrayCopy(node: ts.Expression): boolean {
+ const expression = unwrap(node);
+ if (ts.isArrayLiteralExpression(expression)) {
+ return expression.elements.some(ts.isSpreadElement);
+ }
+ return (
+ ts.isCallExpression(expression) &&
+ ts.isPropertyAccessExpression(expression.expression) &&
+ COPY_METHODS.has(expression.expression.name.text)
);
- return copy.test(source);
+}
+
+function visit(node: ts.Node, onNode: (node: ts.Node) => void) {
+ onNode(node);
+ ts.forEachChild(node, (child) => visit(child, onNode));
+}
+
+/** The `data={…}` expression of every `<Table>` element that has `infinite`. */
+function infiniteDataExpressions(file: ts.SourceFile): ts.Expression[] {
+ const expressions: ts.Expression[] = [];
+ visit(file, (node) => {
+ if (!ts.isJsxOpeningElement(node) && !ts.isJsxSelfClosingElement(node)) {
+ return;
+ }
+ if (node.tagName.getText(file) !== "Table") return;
+ const attribute = (name: string) =>
+ node.attributes.properties.find(
+ (property): property is ts.JsxAttribute =>
+ ts.isJsxAttribute(property) && property.name.getText(file) === name
+ );
+ const data = attribute("data")?.initializer;
+ if (!attribute("infinite") || !data || !ts.isJsxExpression(data)) return;
+ if (data.expression) expressions.push(data.expression);
+ });
+ return expressions;
+}
+
+/** The names of the variables in the file that hold a render-body copy. */
+function renderCopyNames(file: ts.SourceFile): Set<string> {
+ const names = new Set<string>();
+ visit(file, (node) => {
+ if (
+ ts.isVariableDeclaration(node) &&
+ ts.isIdentifier(node.name) &&
+ node.initializer &&
+ isArrayCopy(node.initializer)
+ ) {
+ names.add(node.name.text);
+ }
+ });
+ return names;
+}
+
+/** How each infinite table's `data` is written that copies during render. */
+function renderCopies(source: string): string[] {
+ const file = parse(source);
+ const copies = renderCopyNames(file);
+ return infiniteDataExpressions(file).flatMap((expression) => {
+ const data = unwrap(expression);
+ if (isArrayCopy(data)) return [data.getText(file)];
+ if (ts.isIdentifier(data) && copies.has(data.text)) return [data.text];
+ return [];
+ });
}
describe("infinite tables pass a stable data array", () => {
@@ -49,34 +121,63 @@ describe("infinite tables pass a stable data array", () => {
readFileSync(file, "utf8").includes("infinite={")
);
- it("finds the infinite tables", () => {
- expect(files.length).toBeGreaterThan(50);
+ it("reads the data of every infinite table", () => {
+ const tables = files.reduce(
+ (total, file) =>
+ total + infiniteDataExpressions(parse(readFileSync(file, "utf8"))).length,
+ 0
+ );
+ // Most files with `infinite={` render one infinite `<Table>`.
+ expect(tables).toBeGreaterThan(100);
});
it("never passes an array copied during render", () => {
- const offenders = files.flatMap((file) => {
- const source = readFileSync(file, "utf8");
- return infiniteDataNames(source)
- .filter((name) => isRenderCopy(source, name))
- .map((name) => `${relative(APP, file)}: data={${name}}`);
- });
+ const offenders = files.flatMap((file) =>
+ renderCopies(readFileSync(file, "utf8")).map(
+ (data) => `${relative(APP, file)}: data={${data}}`
+ )
+ );
expect(offenders).toEqual([]);
});
- it("detects a copy built with .map", () => {
+ it("detects a copy built with .map, behind props that contain =>", () => {
const source = `
const rows = data.map((row) => ({ ...row }));
- return <Table data={rows} infinite={{ dataKey: "x", nextCursor }} />;
+ return (
+ <Table<Row>
+ onRowClick={(row) => navigate(row.id)}
+ data={rows}
+ infinite={{ dataKey: "x", nextCursor }}
+ />
+ );
`;
- expect(infiniteDataNames(source)).toEqual(["rows"]);
- expect(isRenderCopy(source, "rows")).toBe(true);
+ expect(renderCopies(source)).toEqual(["rows"]);
+ });
+
+ it("detects a copy written inline and a spread copy", () => {
+ expect(
+ renderCopies(
+ `<Table data={data.filter((r) => r.ok)} infinite={{ dataKey: "x" }} />`
+ )
+ ).toEqual(["data.filter((r) => r.ok)"]);
+ expect(
+ renderCopies(`
+ const rows = [...data];
+ <Table data={rows} infinite={{ dataKey: "x" }}></Table>
+ `)
+ ).toEqual(["rows"]);
});
- it("accepts a memoized array", () => {
+ it("accepts a memoized array and the loader's own array", () => {
const source = `
const rows = useMemo(() => data.map((row) => ({ ...row })), [data]);
- return <Table data={rows} infinite={{ dataKey: "x", nextCursor }} />;
+ <Table data={rows} infinite={{ dataKey: "x", nextCursor }} />;
+ <Table data={data} infinite={{ dataKey: "y", nextCursor }} />;
`;
- expect(isRenderCopy(source, "rows")).toBe(false);
+ expect(renderCopies(source)).toEqual([]);
+ });
+
+ it("ignores a table without infinite", () => {
+ expect(renderCopies(`<Table data={data.map((r) => r)} />`)).toEqual([]);
});
});
diff --git a/packages/jobs/src/inngest/functions/tasks/table-export.ts b/packages/jobs/src/inngest/functions/tasks/table-export.ts
index 6e7c3e1937..954cf6a58d 100644
--- a/packages/jobs/src/inngest/functions/tasks/table-export.ts
+++ b/packages/jobs/src/inngest/functions/tasks/table-export.ts
@@ -8,6 +8,7 @@ import { MAX_EXPORT_ROWS } from "../../../table-jobs/csv";
import {
composeExport,
type ExportDeps,
+ removeExportParts,
writeExportPart
} from "../../../table-jobs/export-run";
import {
@@ -162,11 +163,13 @@ export const tableExportFunction = inngest.createFunction(
if (!cursor) break;
}
+ const exportKey = tableExportStorageKey(companyId, userId, exportId);
await step.run("compose", () =>
- composeExport(deps(), {
- partKeys,
- key: tableExportStorageKey(companyId, userId, exportId)
- })
+ composeExport(deps(), { partKeys, key: exportKey })
+ );
+ // Its own step: a compose retried after the parts were gone would fail.
+ await step.run("remove-parts", () =>
+ removeExportParts(deps(), { partKeys, key: exportKey })
);
await step.run("notify", () =>
diff --git a/packages/jobs/src/table-jobs/export-run.ts b/packages/jobs/src/table-jobs/export-run.ts
index 350de9ec04..9e398775ea 100644
--- a/packages/jobs/src/table-jobs/export-run.ts
+++ b/packages/jobs/src/table-jobs/export-run.ts
@@ -116,17 +116,26 @@ export function concatParts(
});
}
-/** Streams the parts into the export file, then deletes the parts. A part
- * that cannot be deleted is logged; the nightly cleanup removes it. */
+/** Streams the parts into the export file. The parts stay, so a retried
+ * compose writes the same file again; `removeExportParts` deletes them in
+ * the step after it. */
export async function composeExport(
deps: Pick<ExportDeps, "storage">,
args: { partKeys: string[]; key: string }
-): Promise<{ partsRemoved: boolean }> {
+): Promise<void> {
await deps.storage.upload(
args.key,
concatParts(deps.storage, args.partKeys),
CSV_CONTENT_TYPE
);
+}
+
+/** Deletes the parts of a composed export. A part that cannot be deleted is
+ * logged; the nightly cleanup removes it. */
+export async function removeExportParts(
+ deps: Pick<ExportDeps, "storage">,
+ args: { partKeys: string[]; key: string }
+): Promise<{ partsRemoved: boolean }> {
try {
await deps.storage.remove(args.partKeys);
return { partsRemoved: true };
diff --git a/packages/jobs/src/table-jobs/run.test.ts b/packages/jobs/src/table-jobs/run.test.ts
index eb50e16b09..6b1b843b64 100644
--- a/packages/jobs/src/table-jobs/run.test.ts
+++ b/packages/jobs/src/table-jobs/run.test.ts
@@ -6,7 +6,11 @@ import { describe, expect, it } from "vitest";
import { planIdSlices } from "./bulk";
import { applyIdSlice, collectIds, storeSelectedIds } from "./bulk-run";
import { MAX_EXPORT_ROWS } from "./csv";
-import { composeExport, writeExportPart } from "./export-run";
+import {
+ composeExport,
+ removeExportParts,
+ writeExportPart
+} from "./export-run";
import type {
TableBulkActionCounts,
TableJobHandlers,
@@ -177,6 +181,7 @@ describe("the export steps", () => {
if (!cursor) break;
}
await composeExport({ storage }, { partKeys, key: "export.csv" });
+ await removeExportParts({ storage }, { partKeys, key: "export.csv" });
return { files, text, partKeys, rows };
}
@@ -195,6 +200,18 @@ describe("the export steps", () => {
expect([...files.keys()]).toEqual(["export.csv"]);
});
+ it("writes the same file when compose runs again before the parts are removed", async () => {
+ const { storage, text } = memoryStorage();
+ const partKeys = ["parts/0.csv", "parts/1.csv"];
+ await storage.upload(partKeys[0]!, "Part,Name\r\nrow_0,Row 0", "text/csv");
+ await storage.upload(partKeys[1]!, "\r\nrow_1,Row 1", "text/csv");
+
+ await composeExport({ storage }, { partKeys, key: "export.csv" });
+ await composeExport({ storage }, { partKeys, key: "export.csv" });
+
+ expect(text("export.csv")).toBe("Part,Name\r\nrow_0,Row 0\r\nrow_1,Row 1");
+ });
+
it("writes the header alone for an empty list", async () => {
const { text } = await runExport(0);
expect(text("export.csv")).toBe("Part,Name");