Skip to content

Open-source manufacturing ERP, MES and QMS. Quoting, MRP, inventory, shop floor, quality and lot/serial traceability on one Postgres schema, with a REST API and MCP server. Self-host or use Carbon Cloud.

13.6 KB
Raw Blame

@carbon/files

File handling, one subpath per file class. The goal is that "what is this file, can we store it, what does it become, how do we serve it" has exactly ONE answer in the codebase, shared by the browser and Node.

Subpath Provides
. (root) Cross-type helpers: MEDIA_CONTENT_TYPES + getContentType (the one extension→MIME map every file-serving route uses), getFileExtension / effectiveExtension (.zst unwrap), getDocumentType / documentTypes / isPreviewableDocumentType (file classification for the Documents UI), convertKbToString, downloadUrl(url, filename) (fetches first and throws on a failed response, so an expired session never saves an error page under the file's name), and downloadBlob / downloadText — THE browser "save this as a file" sequence (object URL → anchor click → revoke); never hand-roll it. storage(client) — the Carbon storage client, supabase-shaped: .from(bucket) is plain supabase for public / temp-staging; .company(companyId) is that company's private bucket (bucket id = companyId, THROWS on an empty id). Every read and write goes to the company bucket only; there is no shared private bucket. list(folder) takes no options: it reads the cursor-paged listV2 endpoint to the end and returns every entry directly inside the folder, sorted by name, in the old list shape (name is the entry's own name; a sub-folder has a null id). Every key must sit under ${companyId}/ and pass isUnsafeStoragePath; anything else gets { data: null, error } without touching storage. isUnsafeStoragePath(path) is THE storage-key traversal guard (dot segments, \, ?, #, control characters, checked as given and after every decode so %2e%2e / %252e%252e are caught; malformed escapes refused) — every route that takes a key from a URL or form calls it rather than a local copy; safeStorageFileName(name)** turns a caller-supplied file name into a safe last key segment (basename, no %) or null. This is the contract for document keys (${companyId}/parts/…, /job/…, /tmp/…, everything getPrivateUrl serves); the backup and audit-archive layouts (exports/…, audit/…) share the bucket under their own keys and keep using plain client.storage.from(companyId). getCompanyPrivateBucket / hasCompanyPrivateObjectPathPrefix are exported for the few places that need the bucket name (MediaUploader staging, route authorization, the assembler's bucket resolver). PRIVATE_PREVIEW_ALIAS ("private") is a stored URL segment, not a bucket: /file/preview/private/<path> links in rich text and inspection documents are served from the caller's own company bucket. storedDocumentPdfName(displayId, timestamp) is THE name of a document PDF stored when it is finalized, sent or posted (<display id> - <UTC second>.pdf: purchase order, purchase return, quote, sales order, sales return, sales invoice, shipment); **isStoredDocumentPdf(fileName, documentIds) recognises one at any revision, for documents that share a folder
./csv encodeCsv (object rows; header: false encodes a later page of a streamed file — pages join with \r\n, see the Sync Activity export) / encodeCsvTable (header + cells) — the ONE CSV encoder, injection-safe by construction (stripCsvFormulaPrefix on every string cell; formulaSafe: false turns that off for CSV a program reads and no spreadsheet opens, which today is only the MCP server's tool results); parseCsv (text) / parseCsvFile (browser File, streamed) / parseCsvLines (text → records with the line each starts on, blank and delimiter-only records left out; for messages that name a line); downloadCsv; CSV_CONTENT_TYPE. papaparse underneath, in every runtime. The Table export button, report exports, import templates, import-error re-exports and the sales CSV import all go through here — before this there were two libraries (json-2-csv + papaparse) and two hand-rolled encoders, and only ONE of the five export sites protected against formula injection
./markdown TiptapText(value, { format? }) — the ONE place a rich-text document is turned into text or read from it, with no DOM (server and browser). value is a stored document, a document encoded as text, or text. .toDocument() (text read as CommonMark through @tiptap/pm/markdown, raw HTML kept as text, the editor's node and mark names; format: "text" stores it as typed: a paragraph per line, **bold**), .toString() (plain text), .toMarkdown() (editor-only nodes such as mentions become text, unknown wrappers keep their children). Used by every API / MCP rich-text input (toTiptapDoc), the Linear sync, Paperless notes, and the shared generateHTML (@carbon/react), which reads a JSON string scalar an older API stored instead of crashing on it
./media Images. MediaUploader — the configured client (constructor takes the storage client + HEIC staging location once, files-sdk style): prepareForUpload(files) (HEIC → JPEG sequentially + duplicate-name refusal, throws DuplicateFileNameError), convertHeic(file), prepareImage(file, shape). Underneath: processImage (the pipeline), prepareImageUpload (pipeline → native-decode fallback → imgproxy fallback), convertHeicToJpeg / convertHeicFiles / findDuplicateFileName, transformImageViaStorage (imgproxy round-trip), isHeic, IMAGE_UPLOAD_MIME_TYPES, plus the storage-path helpers getPrivateUrl / getRawModelUrl / parseJobFilePath
./media/node initNodeImageCodecs() — Node-only wasm pre-instantiation; call once before the pipeline in Node (jobs, paperless, vitest). The wasm comes from the BUNDLER (?inline data URIs), never off disk — so this module is consumable only from a Vite pipeline, and that pipeline needs assetsInclude: ["**/*.wasm"] (already set in apps/erp/vite.config.ts and @carbon/config/vitest). renderLabelLogo(bytes, ext, { widthDots, threshold?, crop? }) — a logo as a mono PNG data URL + ZPL ^GFA field (inits the codecs itself; @carbon/documents/labels calls it). Never import from client code
./cad CAD/model formats: supportedModelTypes, optimizableModelFormat, modelPathOptimizeFormat, isModelRawDownloadable
./pdf PDF reading on unpdf: extractPdfText (--- Page N --- joined, what the AI extraction prompts consume), extractPdfPages, getPdfPageCount, getPdfMeta, and openPdf/closePdf for page-level work (the inspection overlay export and anchor crop drive pages themselves; closePdf destroys the loading task — cleanup() alone leaks the document transport). Generation stays in @carbon/documents — this is file handling, not templating
./pdf/worker registerReactPdfWorker(pdfjs) — browser-only, synchronous; both app entry.client.tsx files call it once. Nothing else may set GlobalWorkerOptions.workerSrc

The image pipeline

The implementation lives in src/media/image-pipeline.ts (its wasm module declarations in wasm-codecs.d.ts, triple-slash referenced); ./media/image.ts re-exports it. Rounding comes from @carbon/utils round. Do not duplicate the code.

Codecs are wasm, lazy-loaded per format and identical in every runtime: libheif-js (HEIC/HEIF decode), jSquash mozjpeg / png / webp (decode + encode), jSquash resize. No ImageMagick anywhere. Node cannot load wasm over fetch(file:), hence ./media/node. Codec packages are dependencies in this package's package.json.

Never resolve a codec's .wasm from disk. ./media/node used createRequire(import.meta.url).resolve(...), which works only while the file sits at its real path next to packages/files/node_modules; every server build inlines @carbon/files, which moves the anchor into the output directory and loses both the package and the asset — that is why every deployed paperless thumbnail failed with MODULE_NOT_FOUND on Vercel AND on the ECS image, while dev and vitest passed. A new codec's wasm is a ?inline import (a ?url one is a CLIENT public path and is not emitted into the server build at all).

Shape modes (ImageShapeOptions): default = center-crop 300×300 (avatars), contained = fit + 10% pad to 300×300 (item thumbnails), height: n = proportional (logos, 128/512), convert = format normalization only (attachments). JPEG/HEIC/AVIF sources re-encode as JPEG (alpha flattened onto white), everything else as PNG.

HEIC is never stored. Every upload chokepoint converts first, through MediaUploader: both apps' FileDropzone, the ERP useFileUpload hook (the one document-panel upload mutation — TanStack-shaped: options at the hook or per-call, onSuccess/onError, isUploading; all ten document panels go through it, including RecordDocuments for supplier/customer records), useImageUpload (ERP + MES editor hooks), Suggestion, slide uploads, DocumentCreateForm, AttachmentsList, the three curated forms (ItemThumbnailUpload, ProfilePhotoForm, CompanyLogoForm). Non-browser callers (REST/MCP/integrations) go through the MCP signed-URL flow (createDocumentUploadUrl in documents.service, which every module's create*DocumentUploadUrl delegates to) mints a .heic name into {companyId}/tmp/uploads/… staging, and insertUploadedDocument converts it to JPEG (imgproxy round-trip — the bytes are already in storage) at the real path before the row exists; the nightly cleanup job sweeps everything under {companyId}/tmp/ older than a day. Temp staging for the imgproxy fallback is {companyId}/tmp/ in the company bucket.

Fallback order in prepareImageUpload: wasm pipeline → (browser only) native createImageBitmap decode for formats the pipeline lacks (gif, avif) → imgproxy storage round-trip.

Adding a file class: create src/<type>/ with an index.ts, add the ./<type> export to package.json, and add a row above. Don't pre-create empty slots — json, text etc. get a subpath the day something needs unifying, not before.

Always

  • Export/download through encodeCsv + downloadCsv (or downloadBlob); never string-join cells or build a <a download> inline.
  • Add a new ERP document-panel upload via the useFileUpload hook; anywhere else, run picked files through MediaUploader.prepareForUpload first — never upload a picked File raw.
  • Serve files with getContentType(effectiveExtension(path)) — never a local MIME map — and build the response headers with fileResponseHeaders(type, cacheControl). Stored bytes are served from the app origin under a name the uploader chose: without it an uploaded SVG ran script as whoever opened the link (it adds nosniff, and for SVG/+xml/HTML attachment + a sandbox CSP).
  • Read PDFs through ./pdf only. Never import pdfjs-dist or pdfjs from react-pdf at a call site — react-pdf's <Document>/<Page> are the only things app code takes from that package.

One PDF.js engine per bundle

react-pdf hard-imports pdfjs-dist (5.4) and unpdf ships its own serverless build (6.1, worker inlined). Two engines in one browser bundle means two workers and ~2 MB of duplicate code, so ./pdf resolves the engine per runtime:

Runtime Engine Worker
Browser react-pdf's pdfjs-dist, via definePDFJSModule(() => import("pdfjs-dist")) — the bundler dedupes to one copy registerReactPdfWorker in entry.client
Node / edge / jobs unpdf's inlined serverless build built in — no worker file, no @ts-ignore legacy imports (the old extract-document hack)

unpdf still carries import("unpdf/pdfjs") as a never-reached fallback, which Vite emitted as a 1.5 MB lazy chunk in every deploy. Both apps resolve unpdf/pdfjs to app/ssr-shims/unpdf-pdfjs-stub.mjs in the client build only, with clientOnlyAlias from @carbon/dev/vite; the ERP client build was verified to contain exactly ONE engine chunk (388 KB). Any new app that consumes ./pdf in the browser needs the same plugin.

Never put that stub in resolve.alias: a top-level alias applies to every environment, and on the server unpdf/pdfjs IS the engine (the table above). It was there once, and every deployed document extraction failed with "Serverless PDF.js bundle could not be resolved" while dev worked, because dev leaves unpdf external and Node resolves the real module.

  • Classify with getDocumentType; narrow to preview-capable with isPreviewableDocumentType (a @ts-expect-error on a DocumentPreview type prop means a missing narrow, not a type bug).

Never

  • Import ./media/node from anything that can reach the browser bundle.
  • Store a .heic/.heif object. The preview routes' read-time transform is a safety net for legacy/API-stored files, not a licence to skip conversion.
  • Expose imgproxy outside the Docker network — it reads the storage volume with no RLS.
  • Report a failed storage image transform with a message of your own. Pass the error through imageTransformErrorMessage(error, fallback) (@carbon/files): the local dev stack leaves imgproxy off unless booted with crbn up --full, and that helper is what tells the developer to run crbn reload imgproxy.

Validation

pnpm --filter @carbon/files test        # vitest (includes a real HEIC fixture)
pnpm --filter @carbon/files typecheck

Cross-References

  • src/media/label-logo.ts — consumer of the pipeline primitives (decodeImage / resizeImage / encodeImage). Model thumbnails do not use it: the Rust assembler renders them (crates/thumbnail).
  • packages/dev/docker/docker-compose.dev.yml, contrib/deploying/simple-docker-caddy/docker-compose.prod.yml — imgproxy service
  • apps/{erp,mes}/app/routes/file+/preview+/$bucket.$.tsx — read-side serving