Open-source agentic AI Search Execution Platform. Track AI visibility, find opportunities, and let agents turn signals into prioritized actions and drafted fixes across ChatGPT, Claude, Gemini, Google AI Overviews, Google AI Mode, Perplexity, Grok, and Microsoft Copilot.
TypeScript aeoai-agentai-searchanswer-engine-optimizationbrand-monitoringchatgptclaudecopilotdockergeminigeogoogle-ai-overviewsgrokmcpnextjsopen-sourceperplexityseosupabasetypescript
feat(catalog): connect a store with its own sign-in, kept encrypted by the API, with a connector harness and CONNECTORS.md (00282, #1263 part 3) (#1501)
Gökhan Geyikc92f61d6314c
diff --git a/CONNECTORS.md b/CONNECTORS.mdnew file mode 100644index 000000000..35e21376b--- /dev/null+++ b/CONNECTORS.md@@ -0,0 +1,159 @@+# Building a catalog connector++Ansvisor keeps each brand's product catalog in one shape: products, variants, images and categories. A **connector** reads one kind of source into it. There are two kinds of connector:++- **A feed:** a public address read without a sign-in. The product feed URL connector (`server/src/connectors/feed-url/`) reads Google product feeds, Meta catalog CSVs and store CSV exports.+- **A store or marketplace:** read with the brand's own sign-in, for example WooCommerce, Walmart, Trendyol or Hepsiburada.++This guide is for writing a store connector. You can build and test one with no production keys and no real account.++## How a connector fits++A connector is one folder under `server/src/connectors/<kind>/`, exporting one object. You register it in `server/src/connectors/index.js` (`CONNECTORS`). Everything else is already there:++- **Settings › Integrations › Product catalog** shows a "Connect {your store}" button and a form built from your `fields`. You write no UI.+- **Connecting:**+ - Your `connect` checks what the person typed with the store.+ - The API makes the source and keeps the sign-in encrypted.+ - It then reads the catalog.+- **Syncing:**+ - The daily run reads every store again after 20 hours, and admins can press **Sync now**.+ - Each run's outcome is shown on the source: synced, couldn't sync, or sign-in refused.+- **The catalog:**+ - Whatever your `readCatalog` returns is written to the catalog.+ - A re-sync changes only what changed.+ - A product that's gone from the store is marked removed.+- **Removing:** removing the store deletes its sign-in and its products.++You write no migration, no route and no UI.++## The interface++| Member | What it is |+|---|---|+| `kind` | The source's kind, a short slug: `woocommerce`, `trendyol`. Lowercase letters, digits and `_`, 2 to 40 characters. |+| `label` | The platform's name as people know it: `WooCommerce`. |+| `fields` | What the person types to connect, in order: `{ name, label, secret?, placeholder?, help? }`. A `secret` field is typed like a password and never shown again. Use `help` to say where in the store to find it, and which access to give it. |+| `connect(input, ctx)` | Checks `input` (the fields' values, as strings) with the store. Returns `{ account, url?, label?, credentials }` (see below). |+| `readCatalog(source, ctx)` | Reads the store's products. Returns `{ items, errors, format }`, where `format` is your `kind`. |+| `disconnect(source, ctx)` | Optional. Revokes the sign-in at the store when the person removes it, where the platform allows. It's best effort: the source is removed anyway. |++What `connect` returns:+- `account` names the store to the person, such as the shop's domain or the seller id. A brand can connect each account once.+- `url` is the store's address, when it has one.+- `label` is the store's own name, when it gives one.+- `credentials` is an object of strings, kept encrypted for `readCatalog`.++What `ctx` holds:+- `ctx.request` is the HTTP client, and the only way to call the platform.+- In `readCatalog` and `disconnect`, `ctx.credentials` is what `connect` returned as `credentials`.+- `source` has `id`, `kind`, `url`, `account` and `format`.++### What `readCatalog` returns++Build each item with `toItem(attributes, n, 'Item')` from `server/src/lib/feeds/item.js`.++- **Attributes:** use Google product feed attribute names: `id`, `item_group_id`, `title`, `description`, `link`, `image_link`, `additional_image_link`, `brand`, `gtin`, `mpn`, `price`, `sale_price`, `availability`, `color`, `size`, `material`, `product_type`.+- **Required:** an item needs `id`, `title`, `link`, `price` (for example `"129.00 USD"`) and `availability`. An item without them comes back as an error, which you add to `errors`. The person sees the first five.+- **One item per variant.** Put the product's own id in `item_group_id`, and its own name in `title`. The variant's color and size go in `color` and `size`.++The example connector (`server/src/connectors/__example__/index.js`) shows all of this in about 100 lines.++### Calling the platform++Call the platform only through `ctx.request(url, { method, headers, query, body })` (`server/src/connectors/http.js`). It returns `{ status, headers, data }`, with the JSON body in `data`. It:++- reaches only public **https** addresses, on the standard web ports, and doesn't follow redirects. Only the test harness lets it call its fake API over http;+- tries again on 429. A call that changes nothing (GET, HEAD, PUT, DELETE) is also tried again on 502, 503 and 504 and on dropped connections; a POST isn't, since the store may have taken it. It waits as long as the store's `Retry-After` header says, or 1, 2 then 4 seconds. A wait longer than a minute ends the run, and the next daily run tries again;+- stops every call of one sync after 10 minutes, so one slow store can't hold the daily run;+- never puts the query, the headers or the body in an error message.++Send keys in a header (`Authorization`) where the platform allows it, not in the address.++For paging, `nextLink(headers)` reads a `Link: <…>; rel="next"` header. Cap the pages you read, as the example does (`MAX_PAGES`).++### Errors++| Throw | When | The person sees |+|---|---|---|+| `ConnectError` (`./errors.js`) | In `connect`, when what they typed is wrong: "That isn't a WooCommerce site's address." | Your words. Never repeat a secret in them. |+| Errors from `ctx.request` (let them through) | 401 or 403 | "Sign-in refused", and a prompt to update it. |+| Errors from `ctx.request` (let them through) | Anything else | The store couldn't be reached, asked to wait, or answered an error. |+| `FeedFormatError` (`server/src/lib/feeds/item.js`) | The store's data can't be read at all. | Your message. |++Anything else is logged, with every credential value taken out of its message, and shown as "couldn't be synced".++## Credentials++- **Where they're kept:**+ - The API encrypts the sign-in with AES-256-GCM, under a key only it holds (`CONNECTOR_CREDENTIALS_KEY`).+ - It's bound to its source: copied onto another source's row, it doesn't open.+ - It's kept in `catalog_source_credentials`, which only the service role can read. Members can't even see that a row exists.+- **What to ask for:** the least access the feature needs. Today that's read-only access to products (and, later, orders). Say which permission to tick in your field's `help`.+- **Self-hosting:** set `CONNECTOR_CREDENTIALS_KEY` in `server/.env` (`openssl rand -base64 32`). Without it, Settings says store connections aren't set up. If the key changes, saved sign-ins can't be opened: those stores show "Sign-in refused" until they're updated.++## Testing++Connector tests run in `npm test` (in `server/`), against a fake of the platform's API, with no network and no account. The harness is `server/src/connectors/testing.js`:++- **`startFakeApi(routes)`:**+ - It serves your routes, `{ 'GET /api/products': (req) => ({ status, headers, json }) }`, as `http://store.test:<port>`.+ - It gives you `request` to pass as `ctx.request`: the real HTTP client, with its waits recorded in `waits` instead of slept.+ - Every call is recorded in `calls`.+- **`runSync(connector, source, { credentials, request })`** syncs through the real plumbing into an in-memory catalog. It returns the outcome and the rows written (`products`, `variants`, `images`, `categories`).++Start from the example's test (`server/src/connectors/__example__/example.test.js`):++```js+import { afterEach, beforeEach, expect, it, vi } from 'vitest';++// The plumbing reaches the Supabase admin client at import time.+vi.mock('../../config/supabase.js', () => ({ default: {} }));++import { runSync, startFakeApi } from '../testing.js';+import woocommerce from './index.js';++let api;+beforeEach(async () => {+ api = await startFakeApi({+ 'GET /wp-json/wc/v3/products': () => ({ json: [/* products, from the platform's docs */] }),+ });+});+afterEach(() => api.close());++it('reads every product into the catalog', async () => {+ const { outcome, products } = await runSync(woocommerce, { url: api.origin }, {+ credentials: { consumerKey: 'ck_test', consumerSecret: 'cs_test' },+ request: api.request,+ });+ expect(outcome.status).toBe('ok');+});+```++Cover at least these:+- connecting, with a right and a wrong sign-in;+- paging;+- a 429;+- a product that can't be kept;+- one variant per item.++Fixtures are the platform's answers as its docs show them, or as your own sandbox returns them, with made-up products. Never a real store's data, a real customer or a real key.++## Review checklist++Copy this into your pull request. Every connector is reviewed against it.++- [ ] **Least access:** it asks only for what the feature needs (read-only products), and `help` says which permission to give.+- [ ] **Credentials** live only in the credential store. None is in code, logs, error messages, fixtures, screenshots or the PR. Keys go in headers, not addresses, where the platform allows.+- [ ] **No buyer or customer personal data** is read or kept. Orders, when they come, keep only the id, date, line items, revenue and channel.+- [ ] **Removal:** `disconnect` revokes the sign-in where the platform allows. The platform's data deletion requests are handled where it sends them.+- [ ] **Rate limits:** every call goes through `ctx.request`, pages are capped, and the platform's documented limits are respected.+- [ ] **The platform's data rules** are respected (for example, restricted data on marketplaces).+- [ ] **Tests** run with the harness, with no network and no account. Contributors build against their own sandbox or test accounts. Production keys live only with the core team.+- [ ] **Words:** errors a person sees say what to do, in plain words.++## Not here yet++- **Orders and webhooks** (`syncOrders(since)`, `handleWebhook(event)`) come with the first store that needs them (Shopify, [#1265](https://github.com/ansvisor/ansvisor/issues/1265)).+- **OAuth sign-in:** platforms that sign in through a redirect (Etsy, eBay, Amazon) need a callback the plumbing doesn't have yet. Open an issue before you start one.+- **API keys:** stores that sign in with keys (WooCommerce, Walmart, Trendyol, Hepsiburada) work today.diff --git a/CONTRIBUTING.md b/CONTRIBUTING.mdindex 74f98b6e2..af849f744 100644--- a/CONTRIBUTING.md+++ b/CONTRIBUTING.md@@ -7,6 +7,7 @@ Thank you for your interest in contributing to Ansvisor! This guide will help yo- **Bug reports** — found something broken? Open an issue.- **Feature requests** — have an idea? Start a discussion or open an issue.- **Code contributions** — fix a bug or implement a feature via pull request.+- **Store connectors** — bring a store's or marketplace's products into the catalog; see [CONNECTORS.md](CONNECTORS.md).- **Documentation** — improve guides, fix typos, add examples.## Prerequisitesdiff --git a/server/.env.example b/server/.env.exampleindex ee12d4864..d460f6fb4 100644--- a/server/.env.example+++ b/server/.env.example@@ -178,6 +178,14 @@ AI_VOLUME_MULTIPLIER=0.45# COMPOSIO_GA_MAX_PROPERTIES=500# COMPOSIO_GA_STREAM_LOOKUPS=100+# Store connections for the product catalog (#1263, CONNECTORS.md): the key+# a store's sign-in is encrypted with, 32 bytes in base64+# (openssl rand -base64 32). Only this server holds it; the web app never+# sees a store's sign-in. Optional: without it, Settings says store+# connections aren't set up, and feeds and uploads work as before. Changing+# it makes the saved sign-ins unreadable until they're entered again.+# CONNECTOR_CREDENTIALS_KEY=+# How long a run waits for Cloro's FIRST result before giving up (default 90# min). Nothing has come back yet at that point, so the stall and ghost exits# below — which compare successive pending counts — have nothing to measure.diff --git a/server/src/connectors/__example__/example.test.js b/server/src/connectors/__example__/example.test.jsnew file mode 100644index 000000000..9d7d09d96--- /dev/null+++ b/server/src/connectors/__example__/example.test.js@@ -0,0 +1,187 @@+import { afterEach, beforeEach, describe, expect, it, vi } from 'vitest';++// The plumbing reaches the Supabase admin client at import time, whose config+// exits without its env (as in CI). Every connector test starts like this.+vi.mock('../../config/supabase.js', () => ({ default: {} }));++import { logger } from '../../lib/logger.js';+import { ConnectError } from '../errors.js';+import { runSync, startFakeApi } from '../testing.js';+import example from './index.js';++/**+ * The example connector against a fake of its API: how a connector's tests+ * look (CONNECTORS.md), and the harness's own test.+ */++const KEY = 'ex_live_0123456789abcdef';++const product = (id, overrides = {}) => ({+ id,+ title: `Trail Boot ${id}`,+ description: 'Waterproof leather boot.',+ url: `https://shop.example/products/boot-${id}`,+ images: [`https://shop.example/img/${id}.jpg`],+ vendor: 'Acme',+ currency: 'USD',+ category: 'Footwear',+ variants: [+ {+ id: `${id}-42`,+ title: 'Brown / 42',+ price: '129.00',+ in_stock: true,+ options: { color: 'Brown', size: '42' },+ },+ {+ id: `${id}-43`,+ title: 'Brown / 43',+ price: '129.00',+ in_stock: false,+ options: { color: 'Brown', size: '43' },+ },+ ],+ ...overrides,+});++let api;+let catalog;+afterEach(async () => {+ await api?.close();+ vi.restoreAllMocks();+});++beforeEach(async () => {+ catalog = { 1: [product('p1'), product('p2')], 2: [product('p3')] };+ api = await startFakeApi({+ 'GET /api/shop': (req) =>+ req.headers.authorization === `Bearer ${KEY}`+ ? { json: { domain: 'shop.example', name: 'Acme Outdoor' } }+ : { status: 401, json: { error: 'bad key' } },+ 'GET /api/products': (req) => {+ if (req.headers.authorization !== `Bearer ${KEY}`) return { status: 401 };+ const page = Number(req.query.page ?? 1);+ const headers = catalog[page + 1]+ ? { link: `<${api.origin}/api/products?per_page=100&page=${page + 1}>; rel="next"` }+ : {};+ return { headers, json: { products: catalog[page] ?? [] } };+ },+ });+});++describe('connect', () => {+ it('checks the key with the store and keeps it, naming the store by its domain', async () => {+ const out = await example.connect(+ { url: `${api.origin}/admin`, apiKey: KEY },+ { request: api.request },+ );+ expect(out).toEqual({+ account: 'shop.example',+ url: api.origin,+ label: 'Acme Outdoor',+ credentials: { apiKey: KEY },+ });+ // The key goes in a header, never in the address.+ expect(api.calls[0]).toMatchObject({ path: '/api/shop', query: {} });+ });++ it('says what to fix, in words', async () => {+ await expect(+ example.connect({ url: 'shop', apiKey: KEY }, { request: api.request }),+ ).rejects.toThrow(ConnectError);+ await expect(+ example.connect({ url: api.origin, apiKey: '' }, { request: api.request }),+ ).rejects.toThrow('Paste the API key.');+ });++ it("lets the store's refusal through, for the plumbing to name", async () => {+ await expect(+ example.connect({ url: api.origin, apiKey: 'wrong' }, { request: api.request }),+ ).rejects.toMatchObject({ code: 'auth', status: 401 });+ });+});++describe('a sync through the harness', () => {+ const source = () => ({ url: api.origin, account: 'shop.example' });++ it('reads every page into the catalog, one item per variant under its product', async () => {+ const { outcome, products, variants } = await runSync(example, source(), {+ credentials: { apiKey: KEY },+ request: api.request,+ });+ expect(outcome).toMatchObject({+ status: 'ok',+ counts: {+ format: 'example_store',+ items: 6,+ skipped: 0,+ products: { added: 3, removed: 0 },+ variants: { added: 6, removed: 0 },+ },+ });+ expect(products.map((p) => p.title).sort()).toEqual([+ 'Trail Boot p1',+ 'Trail Boot p2',+ 'Trail Boot p3',+ ]);+ expect(variants.find((v) => v.external_id === 'p1-42')).toMatchObject({+ price: 129,+ currency: 'USD',+ availability: 'in_stock',+ attributes: { color: 'Brown', size: '42' },+ });+ expect(api.calls.filter((c) => c.path === '/api/products').map((c) => c.query.page)).toEqual([+ undefined,+ '2',+ ]);+ });++ it('waits as the store asks on a 429, and tries again', async () => {+ let first = true;+ const limited = await startFakeApi({+ 'GET /api/products': () => {+ if (first) {+ first = false;+ return { status: 429, headers: { 'retry-after': '2' } };+ }+ return { json: { products: [product('p1')] } };+ },+ });+ try {+ const { outcome } = await runSync(+ example,+ { url: limited.origin },+ {+ credentials: { apiKey: KEY },+ request: limited.request,+ },+ );+ expect(outcome.status).toBe('ok');+ expect(limited.waits).toHaveLength(1);+ expect(limited.waits[0]).toBeGreaterThanOrEqual(2000);+ } finally {+ await limited.close();+ }+ });++ it('records a refused key as a sign-in to update, without logging it', async () => {+ const error = vi.spyOn(logger, 'error');+ const { outcome } = await runSync(example, source(), {+ credentials: { apiKey: 'revoked-key-value' },+ request: api.request,+ });+ expect(outcome).toMatchObject({ status: 'auth_failed' });+ expect(JSON.stringify(error.mock.calls)).not.toContain('revoked-key-value');+ });++ it("names the store's address, not its key, when a product can't be kept", async () => {+ catalog[1] = [product('p1', { variants: [{ id: 'p1-x', price: null, in_stock: true }] })];+ catalog[2] = [];+ const { outcome } = await runSync(example, source(), {+ credentials: { apiKey: KEY },+ request: api.request,+ });+ expect(outcome).toMatchObject({ status: 'ok', counts: { items: 0, skipped: 1 } });+ expect(outcome.counts.errors[0]).toBe('Item 1: price is missing.');+ });+});diff --git a/server/src/connectors/__example__/index.js b/server/src/connectors/__example__/index.jsnew file mode 100644index 000000000..3c03d9f6d--- /dev/null+++ b/server/src/connectors/__example__/index.js@@ -0,0 +1,122 @@+import { toItem } from '../../lib/feeds/item.js';+import { ConnectError } from '../errors.js';+import { nextLink } from '../http.js';++/**+ * The example store connector (#1263): a template for a real one, and the+ * harness's own test. It isn't registered in `CONNECTORS`, so no one can+ * connect it.+ *+ * Its made-up platform has a JSON API at the store's own address:+ * - `GET /api/shop` → `{ domain, name }`, to check the key;+ * - `GET /api/products?per_page=100` → `{ products: [...] }`, paged by a+ * `Link: <…>; rel="next"` header, each product with its variants.+ * The key goes in an `Authorization: Bearer` header, never in the address.+ */++/** Pages read at most, so a store that pages forever can't hold the run. */+export const MAX_PAGES = 200;+const PER_PAGE = 100;++/** The store's address as its origin ("https://shop.example"), or a ConnectError. */+function storeOrigin(value) {+ let url;+ try {+ url = new URL(String(value ?? '').trim());+ } catch {+ throw new ConnectError(+ "That isn't a web address. Paste the store's address, like https://shop.example.",+ );+ }+ if (url.protocol !== 'https:' && url.protocol !== 'http:') {+ throw new ConnectError("The store's address must start with https://.");+ }+ return url.origin;+}++const auth = (credentials) => ({ authorization: `Bearer ${credentials.apiKey}` });++/** One variant as the catalog's feed attributes (lib/feeds/item.js). */+function attributesOf(product, variant) {+ return {+ id: String(variant.id),+ item_group_id: String(product.id),+ // The product's own name: the catalog names a product after its items' title.+ title: product.title,+ description: product.description,+ link: variant.url ?? product.url,+ image_link: variant.image ?? product.images?.[0],+ additional_image_link: product.images?.slice(1),+ brand: product.vendor,+ gtin: variant.barcode,+ mpn: variant.sku,+ price: variant.price == null ? undefined : `${variant.price} ${product.currency ?? ''}`.trim(),+ sale_price:+ variant.sale_price == null+ ? undefined+ : `${variant.sale_price} ${product.currency ?? ''}`.trim(),+ availability: variant.in_stock ? 'in stock' : 'out of stock',+ color: variant.options?.color,+ size: variant.options?.size,+ product_type: product.category,+ };+}++export default {+ kind: 'example_store',+ label: 'Example Store',+ fields: [+ { name: 'url', label: 'Store address', placeholder: 'https://shop.example' },+ {+ name: 'apiKey',+ label: 'API key',+ secret: true,+ help: 'Settings › API in the store, with read access to products only.',+ },+ ],++ async connect(input, { request }) {+ const origin = storeOrigin(input.url);+ const apiKey = String(input.apiKey ?? '').trim();+ if (!apiKey) throw new ConnectError('Paste the API key.');+ const { data } = await request(`${origin}/api/shop`, { headers: auth({ apiKey }) });+ if (!data?.domain) throw new ConnectError("That address didn't answer like an Example Store.");+ return {+ account: String(data.domain),+ url: origin,+ label: data.name ? String(data.name) : null,+ credentials: { apiKey },+ };+ },++ async readCatalog(source, { request, credentials }) {+ const items = [];+ const errors = [];+ let next = `${source.url}/api/products?per_page=${PER_PAGE}`;+ for (let page = 0; next && page < MAX_PAGES; page += 1) {+ const res = await request(next, { headers: auth(credentials) });+ for (const product of res.data?.products ?? []) {+ for (const variant of product.variants ?? []) {+ const out = toItem(+ attributesOf(product, variant),+ items.length + errors.length + 1,+ 'Item',+ );+ if (out.item) items.push(out.item);+ else errors.push(out.error);+ }+ }+ // Only a next page on the store's own address is followed.+ const link = nextLink(res.headers);+ next =+ link && new URL(link, source.url).origin === source.url+ ? new URL(link, source.url).href+ : null;+ }+ return { items, errors, format: 'example_store' };+ },++ async disconnect() {+ // The made-up platform has no way to revoke a key; a real one would call it here.+ },+};diff --git a/server/src/connectors/errors.js b/server/src/connectors/errors.jsnew file mode 100644index 000000000..5afb4cf35--- /dev/null+++ b/server/src/connectors/errors.js@@ -0,0 +1,11 @@+/**+ * A store couldn't be connected with what the person typed: the message says+ * why in words they can act on ("That isn't a WooCommerce site's address"),+ * and goes back to them as is. It never repeats a secret they typed.+ */+export class ConnectError extends Error {+ constructor(message) {+ super(message);+ this.name = 'ConnectError';+ }+}diff --git a/server/src/connectors/http.js b/server/src/connectors/http.jsnew file mode 100644index 000000000..225349095--- /dev/null+++ b/server/src/connectors/http.js@@ -0,0 +1,250 @@+import http from 'node:http';+import https from 'node:https';+import { pipeline } from 'node:stream';+import zlib from 'node:zlib';+import { PageFetchError, checkPageUrl, publicOnlyLookup } from '../lib/safe-fetch.js';++/**+ * The HTTP client a store connector calls its platform's API with (#1263).+ *+ * A store's address is often the brand's own (a WooCommerce site), so it is+ * reached like any address a person gives (lib/safe-fetch.js): on the+ * standard ports, every resolved address public, checked in the+ * connection's own lookup. Only https: the sign-in never crosses the network+ * in the clear. Redirects aren't followed: an API that redirects would carry+ * the sign-in to another host.+ *+ * It waits and tries again on 429, and, for a call that changes nothing+ * (GET, HEAD, PUT, DELETE), on 502, 503 and 504 and on a dropped connection:+ * as long as the API's Retry-After says, else 1, 2, 4 seconds (with a little+ * jitter), RETRIES times. A wait longer than MAX_WAIT_MS ends the call+ * ('rate_limited'), and the daily run tries the source again later.+ *+ * Errors name the method, host and path only: never the query (some APIs+ * take a key there), the headers or the body.+ */++export const REQUEST_TIMEOUT_MS = 30_000;+export const MAX_BODY_BYTES = 20 * 1024 * 1024;+export const RETRIES = 3;+export const MAX_WAIT_MS = 60_000;++const USER_AGENT = 'AnsvisorCatalog/1.0 (+https://ansvisor.com)';+const RETRY_STATUSES = new Set([429, 502, 503, 504]);+const RETRY_ERRNOS = new Set(['ECONNRESET', 'ECONNREFUSED', 'EPIPE', 'ETIMEDOUT', 'EAI_AGAIN']);++export class ConnectorHttpError extends Error {+ /**+ * @param {'invalid_url'|'blocked'|'unreachable'|'timeout'|'too_large'|'http_error'|'auth'|'rate_limited'|'bad_response'} code+ * 'auth': the API refused the sign-in (401, 403)+ * @param {string} message+ * @param {{ status?: number }} [extra]+ */+ constructor(code, message, { status } = {}) {+ super(message);+ this.name = 'ConnectorHttpError';+ this.code = code;+ this.status = status;+ }+}++/** How long a response asks to wait, from its Retry-After (seconds or a date), or null. */+export function retryAfterMs(headers, now = Date.now()) {+ const value = headers?.['retry-after'];+ if (value == null || value === '') return null;+ if (/^\s*\d+(\.\d+)?\s*$/.test(value)) return Math.round(Number(value) * 1000);+ const at = Date.parse(value);+ return Number.isNaN(at) ? null : Math.max(0, at - now);+}++/** The `rel="next"` address of a Link header (RFC 8288), as many APIs page with, or null. */+export function nextLink(headers) {+ const value = headers?.link;+ if (!value) return null;+ for (const part of String(value).split(/,(?=\s*<)/)) {+ const match = /^\s*<([^>]*)>(.*)$/.exec(part);+ if (!match) continue;+ for (const param of match[2].split(';')) {+ const [name, raw = ''] = param.split('=');+ if (name.trim().toLowerCase() !== 'rel') continue;+ const rels = raw.trim().replace(/^"|"$/g, '').toLowerCase().split(/\s+/);+ if (rels.includes('next')) return match[1];+ }+ }+ return null;+}++const where = (method, url) => `${method} ${url.host}${url.pathname}`;++function once(url, { method, headers, body, lookup, signal, maxBytes }) {+ return new Promise((resolve, reject) => {+ const client = url.protocol === 'https:' ? https : http;+ const req = client.request(+ url,+ {+ method,+ agent: false,+ lookup,+ signal,+ headers: {+ 'user-agent': USER_AGENT,+ accept: 'application/json',+ 'accept-encoding': 'gzip, deflate, br',+ ...(body != null ? { 'content-type': 'application/json' } : {}),+ ...headers,+ },+ },+ (res) => {+ const encoding = String(res.headers['content-encoding'] ?? '').toLowerCase();+ let decoder = null;+ if (encoding === 'gzip' || encoding === 'x-gzip') decoder = zlib.createGunzip();+ else if (encoding === 'deflate') decoder = zlib.createInflate();+ else if (encoding === 'br') decoder = zlib.createBrotliDecompress();+ // pipeline, not pipe: an answer cut off part way fails the call+ // instead of leaving its decoder waiting for an end that never comes.+ if (decoder) pipeline(res, decoder, (err) => err && reject(err));+ const stream = decoder ?? res;+ res.on('close', () => {+ if (!res.complete) {+ reject(new ConnectorHttpError('unreachable', `${where(method, url)} stopped part way`));+ }+ });+ const chunks = [];+ let size = 0;+ stream.on('data', (chunk) => {+ size += chunk.length;+ if (size > maxBytes) {+ req.destroy();+ reject(new ConnectorHttpError('too_large', `${where(method, url)} answered too much`));+ return;+ }+ chunks.push(chunk);+ });+ stream.on('end', () =>+ resolve({+ status: res.statusCode ?? 0,+ headers: res.headers,+ text: Buffer.concat(chunks).toString('utf8'),+ }),+ );+ stream.on('error', reject);+ },+ );+ req.on('error', reject);+ if (body != null) req.write(body);+ req.end();+ });+}++/**+ * Calls a JSON API (see above).+ *+ * @param {string} address+ * @param {{ method?: string, headers?: Record<string, string>, query?: Record<string, string|number>, body?: unknown, timeoutMs?: number, maxBytes?: number, retries?: number, maxWaitMs?: number, lookup?: Function, ports?: string[], allowHttp?: boolean, sleep?: (ms: number) => Promise<void>, signal?: AbortSignal }} [opts]+ * `body` is sent as JSON. Only https is called: a sign-in never crosses+ * the network in the clear. `lookup`, `ports` and `allowHttp` replace the+ * public-only lookup, the web ports and https, and `sleep` the waits;+ * tests use them.+ * @returns {Promise<{ status: number, headers: object, data: unknown }>} data: the parsed body, or null when empty+ */+export async function requestJson(address, opts = {}) {+ const {+ method = 'GET',+ headers = {},+ query,+ body,+ timeoutMs = REQUEST_TIMEOUT_MS,+ maxBytes = MAX_BODY_BYTES,+ retries = RETRIES,+ maxWaitMs = MAX_WAIT_MS,+ lookup = publicOnlyLookup,+ ports,+ allowHttp = false,+ sleep = (ms) => new Promise((resolve) => setTimeout(resolve, ms)),+ signal: outer = null,+ } = opts;+ let url;+ try {+ url = checkPageUrl(address, ports);+ } catch (err) {+ throw new ConnectorHttpError(err.code ?? 'invalid_url', err.message);+ }+ if (url.protocol !== 'https:' && !allowHttp) {+ throw new ConnectorHttpError('invalid_url', `${url.host} isn't an https address`);+ }+ // A call that changes something is tried again only when the API said it+ // didn't take it (429): after a 5xx or a dropped connection it may have.+ const idempotent = ['GET', 'HEAD', 'PUT', 'DELETE'].includes(method.toUpperCase());+ for (const [name, value] of Object.entries(query ?? {})) {+ if (value != null) url.searchParams.set(name, String(value));+ }+ const payload = body == null ? null : JSON.stringify(body);++ for (let attempt = 0; ; attempt += 1) {+ const signal = outer+ ? AbortSignal.any([outer, AbortSignal.timeout(timeoutMs)])+ : AbortSignal.timeout(timeoutMs);+ let res;+ let wait = null;+ try {+ res = await once(url, { method, headers, body: payload, lookup, signal, maxBytes });+ } catch (err) {+ if (err instanceof ConnectorHttpError) throw err;+ if (err instanceof PageFetchError || err?.cause instanceof PageFetchError) {+ throw new ConnectorHttpError('blocked', `${url.host} is not a public address`);+ }+ if (outer?.aborted)+ throw new ConnectorHttpError('timeout', `${where(method, url)} was stopped`);+ const retriable = idempotent && (signal.aborted || RETRY_ERRNOS.has(err?.code));+ if (!retriable || attempt >= retries) {+ throw signal.aborted+ ? new ConnectorHttpError('timeout', `${where(method, url)} took too long to answer`)+ : new ConnectorHttpError(+ 'unreachable',+ `${where(method, url)} couldn't be reached (${err?.code ?? 'network error'})`,+ );+ }+ wait = 1000 * 2 ** attempt;+ }++ if (res) {+ const { status } = res;+ if (status >= 200 && status < 300) {+ if (!res.text.trim()) return { status, headers: res.headers, data: null };+ try {+ return { status, headers: res.headers, data: JSON.parse(res.text) };+ } catch {+ throw new ConnectorHttpError('bad_response', `${where(method, url)} didn't answer JSON`, {+ status,+ });+ }+ }+ if (status === 401 || status === 403) {+ throw new ConnectorHttpError(+ 'auth',+ `${where(method, url)} refused the sign-in (${status})`,+ {+ status,+ },+ );+ }+ const retriable = RETRY_STATUSES.has(status) && (idempotent || status === 429);+ if (!retriable || attempt >= retries) {+ throw new ConnectorHttpError(+ status === 429 ? 'rate_limited' : 'http_error',+ `${where(method, url)} answered ${status}`,+ { status },+ );+ }+ wait = retryAfterMs(res.headers) ?? 1000 * 2 ** attempt;+ if (wait > maxWaitMs) {+ throw new ConnectorHttpError(+ 'rate_limited',+ `${where(method, url)} asked to wait ${Math.ceil(wait / 1000)} s`,+ { status },+ );+ }+ }+ await sleep(wait + Math.floor(Math.random() * 250));+ }+}diff --git a/server/src/connectors/http.test.js b/server/src/connectors/http.test.jsnew file mode 100644index 000000000..4f2c4dce2--- /dev/null+++ b/server/src/connectors/http.test.js@@ -0,0 +1,194 @@+import http from 'node:http';+import zlib from 'node:zlib';+import { afterEach, describe, expect, it, vi } from 'vitest';++// The harness reaches the Supabase admin client at import time.+vi.mock('../config/supabase.js', () => ({ default: {} }));++import { ConnectorHttpError, nextLink, requestJson, retryAfterMs } from './http.js';+import { startFakeApi } from './testing.js';++/** The HTTP client store connectors call their platform with (#1263). */++let api;+afterEach(async () => {+ await api?.close();+ api = null;+});++describe('retryAfterMs', () => {+ it('reads seconds or a date, else nothing', () => {+ const now = Date.parse('2026-10-25T12:00:00Z');+ expect(retryAfterMs({ 'retry-after': '3' }, now)).toBe(3000);+ expect(retryAfterMs({ 'retry-after': 'Sun, 25 Oct 2026 12:00:10 GMT' }, now)).toBe(10_000);+ expect(retryAfterMs({ 'retry-after': 'Sun, 25 Oct 2026 11:00:00 GMT' }, now)).toBe(0);+ expect(retryAfterMs({ 'retry-after': 'soon' }, now)).toBeNull();+ expect(retryAfterMs({}, now)).toBeNull();+ });+});++describe('nextLink', () => {+ it("finds the next page among a Link header's", () => {+ expect(+ nextLink({+ link: '<https://s.example/p?page=1>; rel="prev", <https://s.example/p?page=3>; rel="next"',+ }),+ ).toBe('https://s.example/p?page=3');+ expect(nextLink({ link: '<https://s.example/p?page=2>; rel="last next"' })).toBe(+ 'https://s.example/p?page=2',+ );+ expect(nextLink({ link: '<https://s.example/p?page=1>; rel="previous"' })).toBeNull();+ expect(nextLink({})).toBeNull();+ });+});++describe('requestJson against a fake API', () => {+ it('sends JSON, and reads JSON back', async () => {+ api = await startFakeApi({+ 'POST /echo': (req) => ({ json: { got: req.body, auth: req.headers.authorization } }),+ 'GET /empty': () => ({ status: 204 }),+ });+ await expect(+ api.request(`${api.origin}/echo`, {+ method: 'POST',+ headers: { authorization: 'Bearer k' },+ body: { a: 1 },+ }),+ ).resolves.toMatchObject({ status: 200, data: { got: { a: 1 }, auth: 'Bearer k' } });+ await expect(api.request(`${api.origin}/empty`)).resolves.toMatchObject({+ status: 204,+ data: null,+ });+ });++ it('adds the query, and never names it in an error', async () => {+ api = await startFakeApi({+ 'GET /items': (req) => ({ status: 500, json: { query: req.query } }),+ });+ const err = await api+ .request(`${api.origin}/items`, { query: { consumer_key: 'ck_secret', page: 2 }, retries: 0 })+ .catch((e) => e);+ expect(api.calls[0].query).toEqual({ consumer_key: 'ck_secret', page: '2' });+ expect(err).toBeInstanceOf(ConnectorHttpError);+ expect(err).toMatchObject({ code: 'http_error', status: 500 });+ expect(err.message).toBe(`GET store.test:${api.origin.split(':').pop()}/items answered 500`);+ expect(err.message).not.toContain('ck_secret');+ });++ it('tries again on 429 and 5xx, as long as Retry-After says, and stops past the limit', async () => {+ let n = 0;+ api = await startFakeApi({+ 'GET /flaky': () => {+ n += 1;+ if (n === 1) return { status: 503 };+ if (n === 2) return { status: 429, headers: { 'retry-after': '5' } };+ return { json: { ok: true } };+ },+ 'GET /busy': () => ({ status: 429, headers: { 'retry-after': '3600' } }),+ 'GET /down': () => ({ status: 502 }),+ });+ await expect(api.request(`${api.origin}/flaky`)).resolves.toMatchObject({ data: { ok: true } });+ expect(api.waits[0]).toBeGreaterThanOrEqual(1000);+ expect(api.waits[0]).toBeLessThan(1250);+ expect(api.waits[1]).toBeGreaterThanOrEqual(5000);+ await expect(api.request(`${api.origin}/busy`)).rejects.toMatchObject({+ code: 'rate_limited',+ status: 429,+ });+ await expect(api.request(`${api.origin}/down`, { retries: 2 })).rejects.toMatchObject({+ code: 'http_error',+ status: 502,+ });+ expect(api.calls.filter((c) => c.path === '/down')).toHaveLength(3);+ });++ it("names a refused sign-in, an answer that isn't JSON and a redirect, without retrying", async () => {+ api = await startFakeApi({+ 'GET /private': () => ({ status: 401 }),+ 'GET /html': () => ({ text: '<html>' }),+ 'GET /moved': () => ({ status: 301, headers: { location: 'https://elsewhere.example/' } }),+ });+ await expect(api.request(`${api.origin}/private`)).rejects.toMatchObject({+ code: 'auth',+ status: 401,+ });+ await expect(api.request(`${api.origin}/html`)).rejects.toMatchObject({ code: 'bad_response' });+ await expect(api.request(`${api.origin}/moved`)).rejects.toMatchObject({+ code: 'http_error',+ status: 301,+ });+ expect(api.calls).toHaveLength(3);+ });++ it('stops at its size limit', async () => {+ api = await startFakeApi({ 'GET /big': () => ({ json: { s: 'x'.repeat(5000) } }) });+ await expect(api.request(`${api.origin}/big`, { maxBytes: 1000 })).rejects.toMatchObject({+ code: 'too_large',+ });+ });++ it("tries a call that changes something again only when the API said it didn't take it", async () => {+ let n = 0;+ api = await startFakeApi({+ 'POST /orders': () => {+ n += 1;+ return n === 1 ? { status: 429 } : { status: 503 };+ },+ });+ await expect(+ api.request(`${api.origin}/orders`, { method: 'POST', body: {} }),+ ).rejects.toMatchObject({ code: 'http_error', status: 503 });+ // Once after the 429; the 503 may have been taken, so not again.+ expect(api.calls).toHaveLength(2);+ });+});++describe('requestJson on a connection cut part way', () => {+ it('fails a compressed answer cut off, instead of waiting for it forever', async () => {+ const server = http.createServer((req, res) => {+ res.writeHead(200, { 'content-type': 'application/json', 'content-encoding': 'gzip' });+ res.write(zlib.gzipSync(JSON.stringify({ s: 'x'.repeat(10_000) })).subarray(0, 40));+ setTimeout(() => res.socket.destroy(), 10);+ });+ await new Promise((resolve) => server.listen(0, '127.0.0.1', resolve));+ const port = String(server.address().port);+ const lookup = (hostname, options, callback) =>+ (typeof options === 'function' ? options : callback)(null, [+ { address: '127.0.0.1', family: 4 },+ ]);+ try {+ await expect(+ requestJson(`http://cut.test:${port}/`, {+ lookup,+ ports: [port],+ allowHttp: true,+ retries: 0,+ timeoutMs: 5000,+ }),+ ).rejects.toMatchObject({ code: 'unreachable' });+ } finally {+ await new Promise((resolve) => server.close(resolve));+ }+ });+});++describe('requestJson on the public-only lookup', () => {+ it('never reaches a private address', async () => {+ for (const address of ['https://127.0.0.1/api', 'https://localhost/api', 'https://[::1]/api']) {+ await expect(requestJson(address, { retries: 0 }), address).rejects.toMatchObject({+ code: 'blocked',+ });+ }+ await expect(requestJson('ftp://shop.example/')).rejects.toMatchObject({ code: 'invalid_url' });+ await expect(requestJson('https://shop.example:8443/')).rejects.toMatchObject({+ code: 'invalid_url',+ });+ });++ it('calls https only: a sign-in never goes in the clear', async () => {+ await expect(requestJson('http://shop.example/api')).rejects.toMatchObject({+ code: 'invalid_url',+ message: "shop.example isn't an https address",+ });+ });+});diff --git a/server/src/connectors/index.js b/server/src/connectors/index.jsindex db2859b95..b498babd9 100644--- a/server/src/connectors/index.js+++ b/server/src/connectors/index.js@@ -5,15 +5,31 @@ import feedUrl from './feed-url/index.js';* all writing the same catalog (00278) through `syncCatalog`* (lib/catalog/sync.js). A connector says what a source of its kind holds;* the plumbing (lib/catalog/sources.js) syncs it, records how it went, and- * runs it daily and on demand.+ * runs it daily and on demand. CONNECTORS.md, at the repo's root, is the+ * guide to writing one.*- * A connector is an object with:- * - `kind`: the `catalog_sources.kind` it reads (00279);+ * Every connector is an object with:+ * - `kind`: the `catalog_sources.kind` it reads, a short slug (00282);* - `readCatalog(source, ctx)`: the source's products, as the feed- * parsers' items (`{ items, errors, format }`); it throws- * FeedFormatError for a file that isn't a feed, or a fetch error;- * - later, for stores: `connect`, `syncOrders(since)`,- * `handleWebhook(event)` and `disconnect`.+ * parsers' items (`{ items, errors, format }`). It throws FeedFormatError+ * for data it can't read, and lets the HTTP client's errors through.+ * `ctx.request` is the HTTP client (./http.js `requestJson`), and for a+ * store `ctx.credentials` is what its `connect` kept.+ *+ * A store or marketplace, read with the brand's own sign-in, also has:+ * - `label`: the platform's name, as the Settings card shows it;+ * - `fields`: what the person types to connect it, in order:+ * `{ name, label, secret?, placeholder?, help? }`. A secret field is+ * kept encrypted and never shown again;+ * - `connect(input, ctx)`: checks what the person typed with the platform+ * and returns `{ account, url?, label?, credentials }`: `account` names+ * the store to the person (a shop domain, a seller id), `credentials` is+ * an object of strings kept encrypted for `readCatalog`. It throws+ * ConnectError (./errors.js) with words the person can act on;+ * - optionally `disconnect(source, ctx)`: revokes the sign-in at the+ * platform when the person removes the store, best effort.+ * Later, with orders and webhooks: `syncOrders(since)` and+ * `handleWebhook(event)`.** A connector never logs credentials or a buyer's personal data, and reads* only what its feature needs.@@ -23,6 +39,28 @@ export const CONNECTORS = {};/** The connector for a source's kind, or null. */-export function connectorFor(kind) {- return CONNECTORS[kind] ?? null;+export function connectorFor(kind, connectors = CONNECTORS) {+ return Object.hasOwn(connectors, kind) ? connectors[kind] : null;+}++/** Whether a connector reads a store with the brand's sign-in. */+export function isStore(connector) {+ return typeof connector?.connect === 'function';+}++/** The stores a person can connect, as the Settings card lists them. */+export function storeConnectors(connectors = CONNECTORS) {+ return Object.values(connectors)+ .filter(isStore)+ .map(({ kind, label, fields }) => ({+ kind,+ label,+ fields: (fields ?? []).map(({ name, label: fieldLabel, secret, placeholder, help }) => ({+ name,+ label: fieldLabel,+ secret: !!secret,+ placeholder: placeholder ?? null,+ help: help ?? null,+ })),+ }));}diff --git a/server/src/connectors/testing.js b/server/src/connectors/testing.jsnew file mode 100644index 000000000..86662ad95--- /dev/null+++ b/server/src/connectors/testing.js@@ -0,0 +1,135 @@+import http from 'node:http';+import { randomBytes, randomUUID } from 'node:crypto';+import { fakeDb } from '../lib/catalog/__fixtures__/fake-db.js';+import { saveCredentials } from '../lib/catalog/credentials.js';+import { syncSource } from '../lib/catalog/sources.js';+import { publicOnlyLookup } from '../lib/safe-fetch.js';+import { requestJson } from './http.js';++/**+ * The test harness for connectors (#1263, CONNECTORS.md): a connector's+ * tests run in `npm test` against a fake of its platform's API, with no+ * account and no network.+ *+ * - `startFakeApi(routes)` serves the routes on a local port, as+ * `http://store.test:<port>`, and gives the `request` to hand the+ * connector: the real HTTP client (./http.js), reaching only that fake+ * host, with its waits recorded instead of slept.+ * - `runSync(connector, source, { credentials, request })` syncs a source+ * through the real plumbing (lib/catalog/sources.js) into an in-memory+ * catalog, and returns the outcome and the rows written.+ *+ * A test file that uses it mocks the Supabase admin client first, which+ * exits without its env (see connectors/__example__/example.test.js).+ */++export const FAKE_HOST = 'store.test';++/**+ * @typedef {{ method: string, path: string, query: Record<string, string>, headers: Record<string, string|string[]|undefined>, body: unknown }} FakeRequest+ * @typedef {{ status?: number, headers?: Record<string, string>, json?: unknown, text?: string }} FakeResponse+ * @param {Record<string, (req: FakeRequest) => FakeResponse | Promise<FakeResponse>>} routes+ * by "METHOD /path" (no query); any other request answers 404+ */+export async function startFakeApi(routes) {+ const calls = [];+ const waits = [];+ const server = http.createServer((req, res) => {+ const chunks = [];+ req.on('data', (chunk) => chunks.push(chunk));+ req.on('end', async () => {+ const url = new URL(req.url, 'http://local');+ const text = Buffer.concat(chunks).toString('utf8');+ let body = null;+ try {+ body = text ? JSON.parse(text) : null;+ } catch {+ body = text;+ }+ const call = {+ method: req.method,+ path: url.pathname,+ query: Object.fromEntries(url.searchParams),+ headers: req.headers,+ body,+ };+ calls.push(call);+ const handler = routes[`${req.method} ${url.pathname}`];+ const out = handler ? await handler(call) : { status: 404, json: { error: 'not found' } };+ const payload = out.text ?? (out.json === undefined ? '' : JSON.stringify(out.json));+ res.writeHead(out.status ?? 200, {+ 'content-type': out.text !== undefined ? 'text/plain' : 'application/json',+ ...out.headers,+ });+ res.end(payload);+ });+ });+ await new Promise((resolve) => server.listen(0, '127.0.0.1', resolve));+ const port = String(server.address().port);+ const origin = `http://${FAKE_HOST}:${port}`;++ // The fake host is the local server; every other name goes through the real check.+ const lookup = (hostname, options, callback) => {+ const done = typeof options === 'function' ? options : callback;+ const all = typeof options === 'object' && options.all;+ if (hostname !== FAKE_HOST) return publicOnlyLookup(hostname, options, callback);+ return all ? done(null, [{ address: '127.0.0.1', family: 4 }]) : done(null, '127.0.0.1', 4);+ };+ // The fake serves plain http on its port; the real client calls https only.+ const request = (address, opts = {}) =>+ requestJson(address, {+ ...opts,+ lookup,+ ports: ['', '80', '443', port],+ allowHttp: true,+ sleep: async (ms) => {+ waits.push(ms);+ },+ });++ return {+ origin,+ request,+ calls,+ waits,+ close: () => new Promise((resolve) => server.close(resolve)),+ };+}++/**+ * Syncs `source` with `connector` into an in-memory catalog, as the daily+ * run does. A store's `credentials` are saved for it first, encrypted, as+ * connecting it would.+ *+ * @returns {Promise<{ outcome: object, products: object[], variants: object[], images: object[], categories: object[], fake: ReturnType<typeof fakeDb> }>}+ */+export async function runSync(connector, source, { credentials, request, now = new Date() } = {}) {+ // A key for this run, when the test has none: nothing outlives it.+ process.env.CONNECTOR_CREDENTIALS_KEY ??= randomBytes(32).toString('base64');+ const fake = fakeDb();+ const row = {+ id: randomUUID(),+ brand_id: 'brand-1',+ kind: connector.kind,+ format: 'auto',+ url: null,+ account: null,+ ...source,+ };+ fake.rows('catalog_sources').push({ ...row, last_synced_at: null });+ if (credentials) await saveCredentials(row.id, credentials, { db: fake.db });+ const outcome = await syncSource(row, {+ db: fake.db,+ now,+ request,+ connectors: { [connector.kind]: connector },+ });+ return {+ outcome,+ products: fake.rows('catalog_products'),+ variants: fake.rows('catalog_variants'),+ images: fake.rows('catalog_images'),+ categories: fake.rows('catalog_categories'),+ fake,+ };+}diff --git a/server/src/lib/catalog/__fixtures__/fake-db.js b/server/src/lib/catalog/__fixtures__/fake-db.jsindex a62170715..0c9979d49 100644--- a/server/src/lib/catalog/__fixtures__/fake-db.js+++ b/server/src/lib/catalog/__fixtures__/fake-db.js@@ -11,10 +11,11 @@ export function fakeDb() {return tables.get(name);};const writes = [];- const uniqueKey = (table, row) =>- table === 'catalog_categories'- ? `${row.brand_id}|${row.name}`- : `${row.brand_id}|${row.source}|${row.external_id}`;+ const uniqueKey = (table, row) => {+ if (table === 'catalog_categories') return `${row.brand_id}|${row.name}`;+ if (table === 'catalog_source_credentials') return row.source_id;+ return `${row.brand_id}|${row.source}|${row.external_id}`;+ };// The unique keys' rows, by table, as the database's unique indexes find them.const byKey = new Map();const keyed = (table) => {@@ -23,17 +24,28 @@ export function fakeDb() {};function from(table) {- const state = { filters: [], range: null, op: 'select', payload: null, returning: null };+ const state = {+ filters: [],+ range: null,+ op: 'select',+ payload: null,+ returning: null,+ one: null,+ };const run = () => {const all = rows(table);const match = (row) =>- state.filters.every(([kind, column, value]) =>- kind === 'eq' ? row[column] === value : value.includes(row[column]),- );+ state.filters.every(([kind, column, value]) => {+ if (kind === 'eq') return row[column] === value;+ if (kind === 'neq') return row[column] !== value;+ return value.includes(row[column]);+ });if (state.op === 'select') {const found = all.filter(match).sort((a, b) => (a.id < b.id ? -1 : 1));const [start, end] = state.range ?? [0, found.length];- return { data: found.slice(start, end + 1), error: null };+ const data = found.slice(start, end + 1);+ if (state.one) return { data: data[0] ?? null, error: null };+ return { data, error: null };}writes.push({table,@@ -42,7 +54,7 @@ export function fakeDb() {});if (state.op === 'upsert') {const out = [];- for (const values of state.payload) {+ for (const values of [state.payload].flat()) {const key = uniqueKey(table, values);let row = keyed(table).get(key);if (row) Object.assign(row, values);@@ -56,8 +68,9 @@ export function fakeDb() {return { data: out, error: null };}if (state.op === 'insert') {- for (const values of state.payload) all.push({ id: randomUUID(), ...values });- return { data: null, error: null };+ const added = [state.payload].flat().map((values) => ({ id: randomUUID(), ...values }));+ all.push(...added);+ return { data: state.one ? added[0] : null, error: null };}if (state.op === 'update') {for (const row of all.filter(match)) Object.assign(row, state.payload);@@ -78,6 +91,18 @@ export function fakeDb() {state.filters.push(['eq', column, value]);return builder;},+ neq: (column, value) => {+ state.filters.push(['neq', column, value]);+ return builder;+ },+ single: () => {+ state.one = 'single';+ return builder;+ },+ maybeSingle: () => {+ state.one = 'maybe';+ return builder;+ },in: (column, values) => {state.filters.push(['in', column, values]);return builder;diff --git a/server/src/lib/catalog/credentials.js b/server/src/lib/catalog/credentials.jsnew file mode 100644index 000000000..170d81e84--- /dev/null+++ b/server/src/lib/catalog/credentials.js@@ -0,0 +1,114 @@+import { createCipheriv, createDecipheriv, randomBytes } from 'node:crypto';+import supabaseAdmin from '../../config/supabase.js';++/**+ * A store source's sign-in (#1263, 00282), kept in+ * `catalog_source_credentials`, which only the service role reads.+ *+ * It is encrypted here, with a key only the API holds+ * (`CONNECTOR_CREDENTIALS_KEY`, 32 bytes in base64), so the database never+ * sees it in the clear. The envelope is the web's `key-encryption.ts` one:+ * AES-256-GCM with a fresh 12-byte IV, `{ v, iv, tag, ct }`. The source's id+ * is the GCM associated data, so an envelope copied onto another source's+ * row doesn't open.+ *+ * Credentials never go in a log, an error message or a response.+ */++const ALGO = 'aes-256-gcm';+const IV_BYTES = 12;+const KEY_BYTES = 32;+const VERSION = 1;++/** Credentials can't be kept or read: the API has no (valid) key. */+export class CredentialsKeyError extends Error {+ constructor() {+ super(+ `CONNECTOR_CREDENTIALS_KEY must be ${KEY_BYTES} bytes in base64 (openssl rand -base64 32)`,+ );+ this.name = 'CredentialsKeyError';+ }+}++function masterKey() {+ const raw = process.env.CONNECTOR_CREDENTIALS_KEY;+ const key = raw ? Buffer.from(raw, 'base64') : null;+ if (!key || key.length !== KEY_BYTES) throw new CredentialsKeyError();+ return key;+}++/** Whether the API can keep a store's sign-in. */+export function credentialsConfigured() {+ try {+ masterKey();+ return true;+ } catch {+ return false;+ }+}++/** A source's credentials (an object of strings), sealed for its row. */+export function sealCredentials(sourceId, credentials) {+ const key = masterKey();+ const iv = randomBytes(IV_BYTES);+ const cipher = createCipheriv(ALGO, key, iv);+ cipher.setAAD(Buffer.from(String(sourceId), 'utf8'));+ const ct = Buffer.concat([cipher.update(JSON.stringify(credentials), 'utf8'), cipher.final()]);+ return JSON.stringify({+ v: VERSION,+ iv: iv.toString('base64'),+ tag: cipher.getAuthTag().toString('base64'),+ ct: ct.toString('base64'),+ });+}++/**+ * The credentials sealed for this source, or null when the envelope is+ * malformed, another source's, or the key can't open it. Throws only+ * CredentialsKeyError, when the API has no key at all.+ */+export function openCredentials(sourceId, envelopeJson) {+ const key = masterKey();+ try {+ const envelope = JSON.parse(envelopeJson);+ if (envelope?.v !== VERSION || !envelope.iv || !envelope.tag || !envelope.ct) return null;+ const decipher = createDecipheriv(ALGO, key, Buffer.from(envelope.iv, 'base64'));+ decipher.setAAD(Buffer.from(String(sourceId), 'utf8'));+ decipher.setAuthTag(Buffer.from(envelope.tag, 'base64'));+ const text = Buffer.concat([+ decipher.update(Buffer.from(envelope.ct, 'base64')),+ decipher.final(),+ ]).toString('utf8');+ const credentials = JSON.parse(text);+ return credentials && typeof credentials === 'object' ? credentials : null;+ } catch {+ return null;+ }+}++/** Keeps (or replaces) a source's credentials. */+export async function saveCredentials(+ sourceId,+ credentials,+ { setBy = null, db = supabaseAdmin } = {},+) {+ const secret = sealCredentials(sourceId, credentials);+ const { error } = await db+ .from('catalog_source_credentials')+ .upsert(+ { source_id: sourceId, secret, set_at: new Date().toISOString(), set_by: setBy },+ { onConflict: 'source_id' },+ );+ if (error) throw new Error(`catalog_source_credentials write: ${error.code ?? 'error'}`);+}++/** A source's credentials, or null when it has none it can open. */+export async function readCredentials(sourceId, { db = supabaseAdmin } = {}) {+ const { data, error } = await db+ .from('catalog_source_credentials')+ .select('secret')+ .eq('source_id', sourceId)+ .maybeSingle();+ if (error) throw new Error(`catalog_source_credentials read: ${error.code ?? 'error'}`);+ return data?.secret ? openCredentials(sourceId, data.secret) : null;+}diff --git a/server/src/lib/catalog/credentials.test.js b/server/src/lib/catalog/credentials.test.jsnew file mode 100644index 000000000..58a2910be--- /dev/null+++ b/server/src/lib/catalog/credentials.test.js@@ -0,0 +1,78 @@+import { randomBytes } from 'node:crypto';+import { afterEach, beforeEach, describe, expect, it, vi } from 'vitest';++vi.mock('../../config/supabase.js', () => ({ default: {} }));++import { fakeDb } from './__fixtures__/fake-db.js';+import {+ CredentialsKeyError,+ credentialsConfigured,+ openCredentials,+ readCredentials,+ saveCredentials,+ sealCredentials,+} from './credentials.js';++/** A store's sign-in is kept sealed, for its own source only (#1263, 00282). */++const SOURCE = '0b8d6a1e-2a3c-4d5e-8f90-123456789abc';+const OTHER = '9f8e7d6c-5b4a-4321-8fed-cba987654321';+const SECRET = { apiKey: 'ck_live_secret_value', apiSecret: 'cs_live_other_value' };++let saved;+beforeEach(() => {+ saved = process.env.CONNECTOR_CREDENTIALS_KEY;+ process.env.CONNECTOR_CREDENTIALS_KEY = randomBytes(32).toString('base64');+});+afterEach(() => {+ if (saved === undefined) delete process.env.CONNECTOR_CREDENTIALS_KEY;+ else process.env.CONNECTOR_CREDENTIALS_KEY = saved;+});++describe('sealCredentials / openCredentials', () => {+ it('seals what opens again, with nothing of it in the clear', () => {+ const envelope = sealCredentials(SOURCE, SECRET);+ expect(envelope).not.toContain('secret_value');+ expect(JSON.parse(envelope)).toMatchObject({ v: 1 });+ expect(openCredentials(SOURCE, envelope)).toEqual(SECRET);+ });++ it("uses a fresh IV each time, so the same sign-in doesn't seal the same twice", () => {+ expect(sealCredentials(SOURCE, SECRET)).not.toBe(sealCredentials(SOURCE, SECRET));+ });++ it("doesn't open on another source's row, under another key, or changed", () => {+ const envelope = sealCredentials(SOURCE, SECRET);+ expect(openCredentials(OTHER, envelope)).toBeNull();+ const tampered = JSON.parse(envelope);+ tampered.ct = Buffer.from('x'.repeat(24)).toString('base64');+ expect(openCredentials(SOURCE, JSON.stringify(tampered))).toBeNull();+ expect(openCredentials(SOURCE, 'not json')).toBeNull();+ expect(openCredentials(SOURCE, JSON.stringify({ v: 2 }))).toBeNull();+ process.env.CONNECTOR_CREDENTIALS_KEY = randomBytes(32).toString('base64');+ expect(openCredentials(SOURCE, envelope)).toBeNull();+ });++ it('refuses to seal or open without a 32-byte key', () => {+ delete process.env.CONNECTOR_CREDENTIALS_KEY;+ expect(credentialsConfigured()).toBe(false);+ expect(() => sealCredentials(SOURCE, SECRET)).toThrow(CredentialsKeyError);+ process.env.CONNECTOR_CREDENTIALS_KEY = randomBytes(16).toString('base64');+ expect(credentialsConfigured()).toBe(false);+ expect(() => openCredentials(SOURCE, '{}')).toThrow(CredentialsKeyError);+ });+});++describe('saveCredentials / readCredentials', () => {+ it('keeps one sealed row a source, replaced on a new sign-in', async () => {+ const fake = fakeDb();+ await saveCredentials(SOURCE, SECRET, { setBy: 'user-1', db: fake.db });+ await saveCredentials(SOURCE, { apiKey: 'ck_new' }, { setBy: 'user-1', db: fake.db });+ const rows = fake.rows('catalog_source_credentials');+ expect(rows).toHaveLength(1);+ expect(rows[0]).toMatchObject({ source_id: SOURCE, set_by: 'user-1' });+ expect(rows[0].secret).not.toContain('ck_new');+ await expect(readCredentials(SOURCE, { db: fake.db })).resolves.toEqual({ apiKey: 'ck_new' });+ await expect(readCredentials(OTHER, { db: fake.db })).resolves.toBeNull();+ });+});diff --git a/server/src/lib/catalog/sources.js b/server/src/lib/catalog/sources.jsindex 7f1a3e229..55bbab8ff 100644--- a/server/src/lib/catalog/sources.js+++ b/server/src/lib/catalog/sources.js@@ -1,35 +1,71 @@import supabaseAdmin from '../../config/supabase.js';import { isSubscriptionActive } from '../../config/plans.js';-import { connectorFor } from '../../connectors/index.js';+import { connectorFor, isStore } from '../../connectors/index.js';+import { ConnectorHttpError, requestJson } from '../../connectors/http.js';import { FeedFormatError, parseFeed } from '../feeds/index.js';import { PageFetchError } from '../safe-fetch.js';import { logger } from '../logger.js';+import { CredentialsKeyError, readCredentials } from './credentials.js';import { syncCatalog } from './sync.js';/**- * A brand's catalog sources (#1263, 00279) synced into its catalog: a feed- * URL, read through its connector, daily and on demand; or an uploaded CSV.- * Each source keeps its own products under `sourceKey(id)`, and its last- * sync's outcome on its row (`last_*`), which the Settings card shows.+ * A brand's catalog sources (#1263, 00279, 00282) synced into its catalog: a+ * feed URL or a store, read through its connector, daily and on demand; or+ * an uploaded CSV. Each source keeps its own products under `sourceKey(id)`,+ * and its last sync's outcome on its row (`last_*`), which the Settings card+ * shows. A store is read with its saved sign-in (credentials.js).*/-/** Feeds read again after this long, by the daily run. */+/** Feeds and stores read again after this long, by the daily run. */export const SYNC_EVERY_MS = 20 * 60 * 60 * 1000;-/** Feeds a daily run reads at most, the longest waiting first, one at a time. */+/** Sources a daily run reads at most, the longest waiting first, one at a time. */export const SWEEP_LIMIT = 50;-/** A run stops starting feeds after this; the rest wait first in line for the next. */+/** A run stops starting sources after this; the rest wait first in line for the next. */export const SWEEP_DEADLINE_MS = 10 * 60 * 1000;/** Skipped products' errors kept on the source, for the person to see. */const ERRORS_KEPT = 5;+/** A store's read stops calling it after this: one slow store can't hold the daily run. */+export const STORE_READ_MS = 10 * 60 * 1000;++/** The sources being read in this process: a source isn't read twice at once. */+const reading = new Set();/** The catalog's `source` for a catalog source: `src_` and its id without dashes. */export function sourceKey(id) {return `src_${String(id).replace(/-/g, '')}`;}-/** Why a sync failed, in words a person can act on. No page or feed text. */+/** What a store's API said, in words a person can act on. */+const STORE_WORDS = {+ invalid_url: "The store's address isn't a valid http(s) address.",+ blocked: "The store's address isn't a public one.",+ unreachable: "The store couldn't be reached.",+ timeout: 'The store took too long to answer.',+ too_large: 'The store answered with more than can be read at once.',+ rate_limited: 'The store asked to wait. The next daily run tries again.',+ bad_response: "The store's answer couldn't be read.",+};++/** Why a sync failed, in words a person can act on. No page, feed or store text. */export function failureOf(err) {if (err instanceof FeedFormatError) return { status: 'format_error', error: err.message };+ if (err instanceof ConnectorHttpError) {+ if (err.code === 'auth') {+ return {+ status: 'auth_failed',+ error: 'The store refused the saved sign-in. Update it in Settings › Integrations.',+ };+ }+ return {+ status: 'failed',+ error:+ STORE_WORDS[err.code] ??+ `The store answered ${err.status ?? 'an error'}. The next daily run tries again.`,+ };+ }+ if (err instanceof CredentialsKeyError) {+ return { status: 'failed', error: "Store connections aren't set up on this server." };+ }if (err instanceof PageFetchError) {const words = {invalid_url: "The feed's address isn't a valid http(s) address.",@@ -43,7 +79,22 @@ export function failureOf(err) {};return { status: 'failed', error: words[err.code] ?? "The feed couldn't be read." };}- return { status: 'failed', error: "The feed couldn't be synced. Try again later." };+ return { status: 'failed', error: "The source couldn't be synced. Try again later." };+}++/** A sign-in the saved credentials no longer open. */+const NO_SIGN_IN = {+ status: 'auth_failed',+ error: "The store's saved sign-in can't be read. Update it in Settings › Integrations.",+};++/** A message with a store's credential values taken out, for a log. */+export function redact(message, credentials) {+ let text = String(message ?? '');+ for (const value of Object.values(credentials ?? {})) {+ if (typeof value === 'string' && value.length >= 4) text = text.split(value).join('[redacted]');+ }+ return text;}async function record(db, id, at, outcome) {@@ -60,48 +111,98 @@ async function record(db, id, at, outcome) {}/**- * Syncs one source and records how it went. A feed is read through its- * connector; an upload's text is passed in. Never throws for the source's- * own failure: it's recorded and returned.+ * Syncs one source and records how it went. A feed or a store is read+ * through its connector, a store with its saved sign-in; an upload's text is+ * passed in. Never throws for the source's own failure: it's recorded and+ * returned.*- * @param {{ id: string, brand_id: string, kind: string, format: string, url?: string|null }} source- * @param {{ text?: string, db?: object, now?: Date, fetch?: Function }} [options]+ * A source being read already isn't read again at once: the answer says so,+ * and nothing is recorded. A store's calls stop after `readMs` (STORE_READ_MS).+ *+ * @param {{ id: string, brand_id: string, kind: string, format: string, url?: string|null, account?: string|null }} source+ * @param {{ text?: string, db?: object, now?: Date, fetch?: Function, request?: Function, connectors?: object, readMs?: number }} [options]+ * `fetch` (a feed's), `request` (a store's HTTP client), `connectors` and+ * `readMs` are the tests' stand-ins*/-export async function syncSource(+export async function syncSource(source, options = {}) {+ if (reading.has(source.id)) {+ return {+ status: 'failed',+ error: 'This source is being read right now. Try again in a minute.',+ };+ }+ reading.add(source.id);+ try {+ return await syncOnce(source, options);+ } finally {+ reading.delete(source.id);+ }+}++async function syncOnce(source,- { text, db = supabaseAdmin, now = new Date(), fetch } = {},+ {+ text,+ db = supabaseAdmin,+ now = new Date(),+ fetch,+ request = requestJson,+ connectors,+ readMs = STORE_READ_MS,+ },) {const at = now.toISOString();let outcome;+ let credentials = null;try {let parsed;if (text !== undefined) {parsed = parseFeed(text, source.format);} else {- const connector = connectorFor(source.kind);- if (!connector) throw new Error(`No connector reads ${source.kind}`);- parsed = await connector.readCatalog(source, { fetch });+ const connector = connectorFor(source.kind, connectors);+ if (!connector) {+ outcome = { status: 'failed', error: "This server can't read this kind of source." };+ } else if (isStore(connector)) {+ credentials = await readCredentials(source.id, { db });+ // Every call the read makes stops at its deadline.+ const deadline = AbortSignal.timeout(readMs);+ const bounded = (address, opts = {}) =>+ request(address, {+ ...opts,+ signal: opts.signal ? AbortSignal.any([opts.signal, deadline]) : deadline,+ });+ if (!credentials) outcome = NO_SIGN_IN;+ else parsed = await connector.readCatalog(source, { request: bounded, credentials });+ } else {+ parsed = await connector.readCatalog(source, { fetch, request });+ }+ }+ if (!outcome) {+ const counts = await syncCatalog(source.brand_id, sourceKey(source.id), parsed.items, {+ db,+ now,+ });+ outcome = {+ status: 'ok',+ counts: {+ format: parsed.format,+ items: parsed.items.length,+ skipped: parsed.errors.length,+ errors: parsed.errors.slice(0, ERRORS_KEPT).map((e) => e.message),+ products: counts.products,+ variants: counts.variants,+ },+ };}- const counts = await syncCatalog(source.brand_id, sourceKey(source.id), parsed.items, {- db,- now,- });- outcome = {- status: 'ok',- counts: {- format: parsed.format,- items: parsed.items.length,- skipped: parsed.errors.length,- errors: parsed.errors.slice(0, ERRORS_KEPT).map((e) => e.message),- products: counts.products,- variants: counts.variants,- },- };} catch (err) {outcome = failureOf(err);- if (outcome.status === 'failed' && !(err instanceof PageFetchError)) {+ const told = err instanceof PageFetchError || err instanceof ConnectorHttpError;+ if (outcome.status === 'failed' && !told) {logger.error(- { err: { name: err?.name, message: err?.message }, sourceId: source.id },+ {+ err: { name: err?.name, message: redact(err?.message, credentials) },+ sourceId: source.id,+ },'[catalog] sync failed',);}@@ -111,19 +212,25 @@ export async function syncSource(}/**- * The daily run: feed sources not read for SYNC_EVERY_MS, the longest+ * The daily run: feeds and stores not read for SYNC_EVERY_MS, the longest* waiting first, SWEEP_LIMIT a run, one at a time, until SWEEP_DEADLINE_MS.* On the managed cloud, only workspaces with an active subscription.*/-export async function sweepCatalogSources({ db = supabaseAdmin, now = new Date(), fetch } = {}) {+export async function sweepCatalogSources({+ db = supabaseAdmin,+ now = new Date(),+ fetch,+ request,+ connectors,+} = {}) {const started = Date.now();const due = new Date(now.getTime() - SYNC_EVERY_MS).toISOString();const { data, error } = await db.from('catalog_sources').select(- 'id, brand_id, kind, format, url, last_synced_at, brands!inner(organizations!inner(subscription_status))',+ 'id, brand_id, kind, format, url, account, last_synced_at, brands!inner(organizations!inner(subscription_status))',)- .eq('kind', 'feed_url')+ .neq('kind', 'upload').or(`last_synced_at.is.null,last_synced_at.lt.${due}`).order('last_synced_at', { ascending: true, nullsFirst: true }).limit(SWEEP_LIMIT);@@ -133,7 +240,7 @@ export async function sweepCatalogSources({ db = supabaseAdmin, now = new Date()if (Date.now() - started > SWEEP_DEADLINE_MS) break;if (!isSubscriptionActive(source.brands?.organizations?.subscription_status)) continue;try {- await syncSource(source, { db, now, fetch });+ await syncSource(source, { db, now, fetch, request, connectors });synced++;} catch (err) {logger.error(diff --git a/server/src/lib/catalog/sources.test.js b/server/src/lib/catalog/sources.test.jsindex ff3e66d6e..309ac3da0 100644--- a/server/src/lib/catalog/sources.test.js+++ b/server/src/lib/catalog/sources.test.js@@ -1,3 +1,4 @@+import { randomBytes } from 'node:crypto';import { readFileSync } from 'node:fs';import { afterEach, beforeEach, describe, expect, it, vi } from 'vitest';@@ -5,10 +6,13 @@ import { afterEach, beforeEach, describe, expect, it, vi } from 'vitest';// hard-exits when SUPABASE_* env is absent (as in CI).vi.mock('../../config/supabase.js', () => ({ default: {} }));+import { ConnectorHttpError } from '../../connectors/http.js';+import { logger } from '../logger.js';import { PageFetchError } from '../safe-fetch.js';import { fakeDb } from './__fixtures__/fake-db.js';-import { FEED_MAX_BYTES, FEED_TYPES } from '../../connectors/feed-url/index.js';-import { sourceKey, sweepCatalogSources, syncSource } from './sources.js';+import { saveCredentials } from './credentials.js';+import feedUrl, { FEED_MAX_BYTES, FEED_TYPES } from '../../connectors/feed-url/index.js';+import { redact, sourceKey, sweepCatalogSources, syncSource } from './sources.js';const fixture = (name) =>readFileSync(new URL(`../feeds/__fixtures__/${name}`, import.meta.url), 'utf8');@@ -31,6 +35,7 @@ beforeEach(() => {});afterEach(() => {delete process.env.IS_CLOUD;+ vi.restoreAllMocks();});const kept = () => fake.rows('catalog_sources').find((s) => s.id === SOURCE_ID);@@ -116,6 +121,209 @@ describe('syncSource', () => {});});+describe('syncSource for a store (00282)', () => {+ const STORE_ID = '2c8d6a1e-2a3c-4d5e-8f90-123456789abc';+ const KEY = 'sk_live_store_secret';+ let store;+ let saved;+ const item = {+ id: 'v1',+ itemGroupId: 'p1',+ title: 'Trail Boot',+ description: null,+ link: 'https://shop.example/boot',+ imageLink: null,+ additionalImageLinks: [],+ brand: null,+ gtin: null,+ mpn: null,+ price: { amount: 129, currency: 'USD' },+ salePrice: null,+ availability: 'in_stock',+ condition: null,+ color: null,+ size: null,+ material: null,+ productType: null,+ };+ const connector = (readCatalog) => ({+ kind: 'acme_store',+ label: 'Acme Store',+ fields: [],+ connect: vi.fn(),+ readCatalog: vi.fn(readCatalog),+ });++ beforeEach(() => {+ saved = process.env.CONNECTOR_CREDENTIALS_KEY;+ process.env.CONNECTOR_CREDENTIALS_KEY = randomBytes(32).toString('base64');+ store = { id: STORE_ID, brand_id: 'brand-1', kind: 'acme_store', format: 'auto', url: null };+ fake.rows('catalog_sources').push({ ...store, last_synced_at: null });+ });+ afterEach(() => {+ if (saved === undefined) delete process.env.CONNECTOR_CREDENTIALS_KEY;+ else process.env.CONNECTOR_CREDENTIALS_KEY = saved;+ });+ const row = () => fake.rows('catalog_sources').find((s) => s.id === STORE_ID);++ it('reads a store with its saved sign-in', async () => {+ await saveCredentials(STORE_ID, { apiKey: KEY }, { db: fake.db });+ const acme = connector(async (_source, ctx) => {+ await ctx.request('https://shop.example/api/products');+ return { items: [item], errors: [], format: 'acme_store' };+ });+ const request = vi.fn().mockResolvedValue({ status: 200, headers: {}, data: null });+ const outcome = await syncSource(store, {+ db: fake.db,+ now: NOW,+ request,+ connectors: { acme_store: acme },+ });+ expect(acme.readCatalog).toHaveBeenCalledWith(store, {+ request: expect.any(Function),+ credentials: { apiKey: KEY },+ });+ // Every call the read makes carries its deadline.+ expect(request).toHaveBeenCalledWith('https://shop.example/api/products', {+ signal: expect.any(AbortSignal),+ });+ expect(outcome).toMatchObject({ status: 'ok', counts: { format: 'acme_store', items: 1 } });+ expect(fake.rows('catalog_products')).toHaveLength(1);+ });++ it("stops a store's calls at the read's deadline", async () => {+ await saveCredentials(STORE_ID, { apiKey: KEY }, { db: fake.db });+ let signal;+ const acme = connector(async (_source, ctx) => {+ await ctx.request('https://shop.example/api/products');+ return { items: [], errors: [], format: 'acme_store' };+ });+ const request = vi.fn(async (_address, opts) => {+ signal = opts.signal;+ await new Promise((resolve) => setTimeout(resolve, 30));+ if (opts.signal.aborted) {+ throw new ConnectorHttpError('timeout', 'GET shop.example/api/products was stopped');+ }+ return { status: 200, headers: {}, data: null };+ });+ const outcome = await syncSource(store, {+ db: fake.db,+ now: NOW,+ request,+ readMs: 5,+ connectors: { acme_store: acme },+ });+ expect(signal.aborted).toBe(true);+ expect(outcome).toEqual({ status: 'failed', error: 'The store took too long to answer.' });+ });++ it("doesn't read a source twice at once", async () => {+ await saveCredentials(STORE_ID, { apiKey: KEY }, { db: fake.db });+ let release;+ const acme = connector(+ () =>+ new Promise((resolve) => {+ release = () => resolve({ items: [], errors: [], format: 'acme_store' });+ }),+ );+ const options = { db: fake.db, now: NOW, connectors: { acme_store: acme } };+ const first = syncSource(store, options);+ await vi.waitFor(() => expect(release).toBeDefined());+ await expect(syncSource(store, options)).resolves.toEqual({+ status: 'failed',+ error: 'This source is being read right now. Try again in a minute.',+ });+ release();+ await expect(first).resolves.toMatchObject({ status: 'ok' });+ expect(acme.readCatalog).toHaveBeenCalledTimes(1);+ });++ it("asks for the sign-in again when there's none it can open, and reads nothing", async () => {+ const acme = connector(async () => ({ items: [item], errors: [], format: 'acme_store' }));+ const outcome = await syncSource(store, {+ db: fake.db,+ now: NOW,+ connectors: { acme_store: acme },+ });+ expect(outcome).toEqual({+ status: 'auth_failed',+ error: "The store's saved sign-in can't be read. Update it in Settings › Integrations.",+ });+ expect(acme.readCatalog).not.toHaveBeenCalled();+ expect(row()).toMatchObject({ last_status: 'auth_failed' });+ });++ it("records the store's refusal, its wait and its errors in plain words", async () => {+ await saveCredentials(STORE_ID, { apiKey: KEY }, { db: fake.db });+ const cases = [+ [+ new ConnectorHttpError('auth', 'GET x/api refused the sign-in (401)', { status: 401 }),+ 'auth_failed',+ /refused the saved sign-in/,+ ],+ [+ new ConnectorHttpError('rate_limited', 'GET x/api asked to wait', { status: 429 }),+ 'failed',+ /asked to wait/,+ ],+ [+ new ConnectorHttpError('http_error', 'GET x/api answered 500', { status: 500 }),+ 'failed',+ /answered 500/,+ ],+ ];+ for (const [err, status, words] of cases) {+ const acme = connector(async () => {+ throw err;+ });+ const outcome = await syncSource(store, {+ db: fake.db,+ now: NOW,+ connectors: { acme_store: acme },+ });+ expect(outcome.status).toBe(status);+ expect(outcome.error).toMatch(words);+ }+ });++ it('keeps the sign-in out of the log when a connector fails on its own', async () => {+ await saveCredentials(STORE_ID, { apiKey: KEY }, { db: fake.db });+ const error = vi.spyOn(logger, 'error').mockImplementation(() => {});+ const acme = connector(async () => {+ throw new Error(`unexpected answer for key ${KEY}`);+ });+ const outcome = await syncSource(store, {+ db: fake.db,+ now: NOW,+ connectors: { acme_store: acme },+ });+ expect(outcome).toEqual({+ status: 'failed',+ error: "The source couldn't be synced. Try again later.",+ });+ expect(error).toHaveBeenCalledTimes(1);+ expect(JSON.stringify(error.mock.calls)).not.toContain(KEY);+ expect(JSON.stringify(error.mock.calls)).toContain('[redacted]');+ });++ it('records a kind this server has no connector for', async () => {+ const outcome = await syncSource(store, { db: fake.db, now: NOW, connectors: {} });+ expect(outcome).toEqual({+ status: 'failed',+ error: "This server can't read this kind of source.",+ });+ });+});++describe('redact', () => {+ it('takes every credential value out of a message, but not short ones', () => {+ expect(+ redact('key abcd1234 and secret s3cr3t!', { a: 'abcd1234', b: 's3cr3t!', c: 'ab' }),+ ).toBe('key [redacted] and secret [redacted]');+ expect(redact(undefined, null)).toBe('');+ });+});+describe('sweepCatalogSources', () => {it('syncs the due feeds, skipping workspaces without an active subscription on the cloud', async () => {process.env.IS_CLOUD = 'true';@@ -136,4 +344,39 @@ describe('sweepCatalogSources', () => {expect(fetch).toHaveBeenCalledTimes(1);expect(fetch.mock.calls[0][0]).toBe('https://shop.example/feed.xml');});++ it('reads stores too, and never an upload', async () => {+ fake.rows('catalog_sources').push(+ {+ id: '3b8d6a1e-2a3c-4d5e-8f90-123456789abc',+ brand_id: 'brand-1',+ kind: 'upload',+ format: 'auto',+ url: null,+ last_synced_at: null,+ },+ {+ id: '4b8d6a1e-2a3c-4d5e-8f90-123456789abc',+ brand_id: 'brand-1',+ kind: 'acme_store',+ format: 'auto',+ url: null,+ last_synced_at: null,+ },+ );+ const fetch = vi.fn().mockResolvedValue({ html: fixture('google-feed.xml') });+ const result = await sweepCatalogSources({+ db: fake.db,+ now: NOW,+ fetch,+ connectors: { feed_url: feedUrl },+ });+ expect(result).toEqual({ due: 2, synced: 2 });+ expect(fetch).toHaveBeenCalledTimes(1);+ const tried = fake.rows('catalog_sources').filter((s) => s.last_status);+ expect(tried.map((s) => [s.kind, s.last_status]).sort()).toEqual([+ ['acme_store', 'failed'],+ ['feed_url', 'ok'],+ ]);+ });});diff --git a/server/src/lib/catalog/stores.js b/server/src/lib/catalog/stores.jsnew file mode 100644index 000000000..d019b3d21--- /dev/null+++ b/server/src/lib/catalog/stores.js@@ -0,0 +1,226 @@+import supabaseAdmin from '../../config/supabase.js';+import { ConnectError } from '../../connectors/errors.js';+import { ConnectorHttpError, requestJson } from '../../connectors/http.js';+import { connectorFor, isStore } from '../../connectors/index.js';+import { logger } from '../logger.js';+import { credentialsConfigured, readCredentials, saveCredentials } from './credentials.js';++/**+ * Connecting a brand's store as a catalog source (#1263, 00282): the+ * connector checks what the person typed with the store, then the source is+ * made and its sign-in kept, encrypted. Replacing the sign-in and removing+ * the store go through here too, so the sign-in is never handled anywhere+ * else.+ *+ * What the person typed never goes in a log or an error: a StoreError's+ * message is the words they see.+ */++export class StoreError extends Error {+ /**+ * @param {number} status the HTTP status the route answers+ * @param {'invalid'|'not_found'|'not_configured'|'connect_failed'|'auth_failed'|'unreachable'|'duplicate'|'too_many'|'other_account'} code+ * @param {string} message+ */+ constructor(status, code, message) {+ super(message);+ this.name = 'StoreError';+ this.status = status;+ this.code = code;+ }+}++/** A field's value at most this long. */+export const FIELD_MAX = 2000;++const SOURCE_COLUMNS = 'id, brand_id, kind, format, url, account, label';++function storeConnector(kind, connectors) {+ const connector = connectorFor(kind, connectors);+ if (!isStore(connector))+ throw new StoreError(404, 'not_found', 'There is no such store to connect.');+ return connector;+}++function ready() {+ if (!credentialsConfigured()) {+ throw new StoreError(+ 503,+ 'not_configured',+ "Store connections aren't set up on this server yet.",+ );+ }+}++/** What the person typed, by the connector's fields: strings, each FIELD_MAX at most. */+export function readInput(connector, input) {+ if (!input || typeof input !== 'object' || Array.isArray(input)) {+ throw new StoreError(400, 'invalid', 'Fill in the fields to connect the store.');+ }+ const out = {};+ for (const field of connector.fields ?? []) {+ const value = input[field.name] ?? '';+ if (typeof value !== 'string' || value.length > FIELD_MAX) {+ throw new StoreError(400, 'invalid', `${field.label} isn't valid.`);+ }+ out[field.name] = value;+ }+ return out;+}++const text = (value, max) =>+ typeof value === 'string' && value.trim() ? value.trim().slice(0, max) : null;++/** What `connect` returned, checked: the plumbing never keeps a malformed one. */+function checkResult(result) {+ const account = text(result?.account, 200);+ const credentials = result?.credentials;+ const url = text(result?.url, 2048);+ if (+ !account ||+ !credentials ||+ typeof credentials !== 'object' ||+ !Object.values(credentials).every((v) => typeof v === 'string') ||+ (url && !/^https?:\/\//i.test(url))+ ) {+ throw new Error('connect returned no account, credentials of strings, or an http(s) url');+ }+ return { account, url, label: text(result.label, 200), credentials };+}++/** The connector's `connect`, its failures as the person should read them. */+async function checkWithStore(connector, input, request) {+ try {+ return checkResult(await connector.connect(input, { request }));+ } catch (err) {+ if (err instanceof ConnectError) throw new StoreError(400, 'connect_failed', err.message);+ if (err instanceof ConnectorHttpError) {+ if (err.code === 'auth') {+ throw new StoreError(+ 400,+ 'auth_failed',+ 'The store refused this sign-in. Check it and try again.',+ );+ }+ if (err.code === 'blocked' || err.code === 'invalid_url') {+ throw new StoreError(+ 400,+ 'connect_failed',+ "The store's address isn't a public https address.",+ );+ }+ throw new StoreError(+ 502,+ 'unreachable',+ "The store couldn't be reached. Try again in a moment.",+ );+ }+ // Its message may hold what the person typed: only its name is logged.+ logger.error(+ { err: { name: err?.name }, kind: connector.kind },+ '[catalog] store connect failed',+ );+ throw new StoreError(+ 502,+ 'unreachable',+ "The store couldn't be checked. Try again in a moment.",+ );+ }+}++/**+ * Connects a store to a brand: checked with the store, made a source, its+ * sign-in kept. The caller has checked the person may.+ *+ * @returns {Promise<{ id: string, brand_id: string, kind: string, format: string, url: string|null, account: string, label: string|null }>}+ */+export async function connectStore(+ { brandId, kind, input, userId },+ { db = supabaseAdmin, connectors, request = requestJson } = {},+) {+ const connector = storeConnector(kind, connectors);+ ready();+ const result = await checkWithStore(connector, readInput(connector, input), request);+ const { data: existing } = await db+ .from('catalog_sources')+ .select('id')+ .eq('brand_id', brandId)+ .eq('kind', kind)+ .eq('account', result.account)+ .maybeSingle();+ if (existing) throw new StoreError(409, 'duplicate', 'This store is connected already.');+ const { data: source, error } = await db+ .from('catalog_sources')+ .insert({+ brand_id: brandId,+ kind,+ account: result.account,+ url: result.url,+ label: result.label,+ created_by: userId,+ })+ .select(SOURCE_COLUMNS)+ .single();+ if (error?.code === '23514') {+ throw new StoreError(409, 'too_many', 'A brand has 10 catalog sources at most.');+ }+ if (error?.code === '23505')+ throw new StoreError(409, 'duplicate', 'This store is connected already.');+ if (error || !source) throw new Error(`catalog_sources write: ${error?.code ?? 'no row'}`);+ try {+ await saveCredentials(source.id, result.credentials, { setBy: userId, db });+ } catch (err) {+ // A store source never stays without its sign-in.+ await db.from('catalog_sources').delete().eq('id', source.id);+ throw err;+ }+ return source;+}++/** Replaces a store's sign-in, for the same store's account only. */+export async function reconnectStore(+ { source, input, userId },+ { db = supabaseAdmin, connectors, request = requestJson } = {},+) {+ const connector = storeConnector(source.kind, connectors);+ ready();+ const result = await checkWithStore(connector, readInput(connector, input), request);+ if (source.account && result.account !== source.account) {+ throw new StoreError(+ 409,+ 'other_account',+ 'This sign-in is for another store. Add that store as a new source.',+ );+ }+ await saveCredentials(source.id, result.credentials, { setBy: userId, db });+ const { error } = await db+ .from('catalog_sources')+ .update({ url: result.url ?? source.url, last_status: null, last_error: null })+ .eq('id', source.id);+ if (error) throw new Error(`catalog_sources write: ${error.code ?? 'error'}`);+}++/**+ * Removes a store: its sign-in revoked at the store where the connector can+ * (best effort), then the source, which takes its sign-in and its products+ * with it (00279, 00282).+ */+export async function disconnectStore(+ source,+ { db = supabaseAdmin, connectors, request = requestJson } = {},+) {+ const connector = connectorFor(source.kind, connectors);+ if (isStore(connector) && typeof connector.disconnect === 'function') {+ try {+ const credentials = await readCredentials(source.id, { db });+ if (credentials) await connector.disconnect(source, { request, credentials });+ } catch (err) {+ logger.warn(+ { err: { name: err?.name }, sourceId: source.id },+ '[catalog] store disconnect failed',+ );+ }+ }+ const { error } = await db.from('catalog_sources').delete().eq('id', source.id);+ if (error) throw new Error(`catalog_sources delete: ${error.code ?? 'error'}`);+}diff --git a/server/src/lib/catalog/stores.test.js b/server/src/lib/catalog/stores.test.jsnew file mode 100644index 000000000..341bd1a0b--- /dev/null+++ b/server/src/lib/catalog/stores.test.js@@ -0,0 +1,173 @@+import { randomBytes } from 'node:crypto';+import { afterEach, beforeEach, describe, expect, it, vi } from 'vitest';++vi.mock('../../config/supabase.js', () => ({ default: {} }));++import example from '../../connectors/__example__/index.js';+import { startFakeApi } from '../../connectors/testing.js';+import { logger } from '../logger.js';+import { fakeDb } from './__fixtures__/fake-db.js';+import { readCredentials } from './credentials.js';+import { StoreError, connectStore, disconnectStore, readInput, reconnectStore } from './stores.js';++/** Connecting a store, replacing its sign-in and removing it (#1263, 00282). */++const BRAND = 'brand-1';+const KEY = 'ex_live_0123456789abcdef';+const connectors = { example_store: example };++let api;+let fake;+let saved;+beforeEach(async () => {+ saved = process.env.CONNECTOR_CREDENTIALS_KEY;+ process.env.CONNECTOR_CREDENTIALS_KEY = randomBytes(32).toString('base64');+ fake = fakeDb();+ api = await startFakeApi({+ 'GET /api/shop': (req) =>+ req.headers.authorization === `Bearer ${KEY}` ||+ req.headers.authorization === 'Bearer ex_new_key'+ ? { json: { domain: 'shop.example', name: 'Acme Outdoor' } }+ : { status: 401 },+ });+});+afterEach(async () => {+ await api.close();+ vi.restoreAllMocks();+ if (saved === undefined) delete process.env.CONNECTOR_CREDENTIALS_KEY;+ else process.env.CONNECTOR_CREDENTIALS_KEY = saved;+});++const opts = () => ({ db: fake.db, connectors, request: api.request });+const connect = (input = { url: api.origin, apiKey: KEY }) =>+ connectStore({ brandId: BRAND, kind: 'example_store', input, userId: 'user-1' }, opts());++describe('readInput', () => {+ it("keeps the connector's fields only, as strings", () => {+ expect(readInput(example, { url: 'https://s.example', apiKey: 'k', other: 'x' })).toEqual({+ url: 'https://s.example',+ apiKey: 'k',+ });+ expect(readInput(example, {})).toEqual({ url: '', apiKey: '' });+ expect(() => readInput(example, { apiKey: 'x'.repeat(2001) })).toThrow(StoreError);+ expect(() => readInput(example, null)).toThrow(StoreError);+ });+});++describe('connectStore', () => {+ it('makes the source with its account, and keeps the sign-in sealed', async () => {+ const source = await connect();+ expect(source).toMatchObject({+ brand_id: BRAND,+ kind: 'example_store',+ account: 'shop.example',+ url: api.origin,+ label: 'Acme Outdoor',+ });+ expect(fake.rows('catalog_sources')[0]).toMatchObject({ created_by: 'user-1' });+ const [row] = fake.rows('catalog_source_credentials');+ expect(row).toMatchObject({ source_id: source.id, set_by: 'user-1' });+ expect(row.secret).not.toContain(KEY);+ await expect(readCredentials(source.id, { db: fake.db })).resolves.toEqual({ apiKey: KEY });+ });++ it('says why it connected nothing', async () => {+ const cases = [+ [() => connect({ url: 'nope', apiKey: KEY }), 400, 'connect_failed'],+ [() => connect({ url: api.origin, apiKey: 'wrong' }), 400, 'auth_failed'],+ [() => connect({ url: 'http://127.0.0.1/', apiKey: KEY }), 400, 'connect_failed'],+ [+ () =>+ connectStore({ brandId: BRAND, kind: 'feed_url', input: {}, userId: 'user-1' }, opts()),+ 404,+ 'not_found',+ ],+ ];+ for (const [run, status, code] of cases) {+ await expect(run()).rejects.toMatchObject({ status, code });+ }+ expect(fake.rows('catalog_sources')).toEqual([]);+ expect(fake.rows('catalog_source_credentials')).toEqual([]);+ });++ it('connects nothing on a server that has no key for sign-ins', async () => {+ delete process.env.CONNECTOR_CREDENTIALS_KEY;+ await expect(connect()).rejects.toMatchObject({ status: 503, code: 'not_configured' });+ expect(api.calls).toEqual([]);+ });++ it('connects a store once a brand', async () => {+ await connect();+ await expect(connect()).rejects.toMatchObject({ status: 409, code: 'duplicate' });+ expect(fake.rows('catalog_sources')).toHaveLength(1);+ });++ it("logs only a broken connector's error name, never what the person typed", async () => {+ const error = vi.spyOn(logger, 'error').mockImplementation(() => {});+ const broken = {+ ...example,+ connect: async (input) => {+ throw new TypeError(`cannot read ${input.apiKey}`);+ },+ };+ await expect(+ connectStore(+ {+ brandId: BRAND,+ kind: 'example_store',+ input: { url: api.origin, apiKey: KEY },+ userId: 'u',+ },+ { ...opts(), connectors: { example_store: broken } },+ ),+ ).rejects.toMatchObject({ status: 502, code: 'unreachable' });+ expect(JSON.stringify(error.mock.calls)).not.toContain(KEY);+ expect(JSON.stringify(error.mock.calls)).toContain('TypeError');+ });+});++describe('reconnectStore', () => {+ it('replaces the sign-in of the same store, and clears the last refusal', async () => {+ const source = await connect();+ fake.rows('catalog_sources')[0].last_status = 'auth_failed';+ await reconnectStore(+ { source, input: { url: api.origin, apiKey: 'ex_new_key' }, userId: 'user-2' },+ opts(),+ );+ await expect(readCredentials(source.id, { db: fake.db })).resolves.toEqual({+ apiKey: 'ex_new_key',+ });+ expect(fake.rows('catalog_sources')[0]).toMatchObject({ last_status: null, last_error: null });+ expect(fake.rows('catalog_source_credentials')[0]).toMatchObject({ set_by: 'user-2' });+ });++ it("won't swap one store for another under the same source", async () => {+ const source = await connect();+ await expect(+ reconnectStore(+ {+ source: { ...source, account: 'other.example' },+ input: { url: api.origin, apiKey: KEY },+ },+ opts(),+ ),+ ).rejects.toMatchObject({ status: 409, code: 'other_account' });+ });+});++describe('disconnectStore', () => {+ it('revokes the sign-in at the store, best effort, then removes the source', async () => {+ const source = await connect();+ const disconnect = vi.fn().mockRejectedValue(new Error('store down'));+ vi.spyOn(logger, 'warn').mockImplementation(() => {});+ await disconnectStore(source, {+ ...opts(),+ connectors: { example_store: { ...example, disconnect } },+ });+ expect(disconnect).toHaveBeenCalledWith(source, {+ request: api.request,+ credentials: { apiKey: KEY },+ });+ expect(fake.rows('catalog_sources')).toEqual([]);+ });+});diff --git a/server/src/lib/logger.js b/server/src/lib/logger.jsindex 4f1e8e9e3..f56039362 100644--- a/server/src/lib/logger.js+++ b/server/src/lib/logger.js@@ -2,9 +2,17 @@ import pino from 'pino';const level = process.env.LOG_LEVEL || (process.env.NODE_ENV === 'development' ? 'debug' : 'info');+/**+ * Never logged: a request's sign-in, and the raw body an error carries. The+ * body parser puts a malformed request's whole body on its error (`err.body`),+ * which the global error handler logs: it may hold a store's sign-in or a+ * site token (#1263).+ */+export const REDACT_PATHS = ['req.headers.authorization', 'req.headers.cookie', 'err.body'];+export const logger = pino({level,- redact: ['req.headers.authorization', 'req.headers.cookie'],+ redact: REDACT_PATHS,});export default logger;diff --git a/server/src/lib/logger.test.js b/server/src/lib/logger.test.jsnew file mode 100644index 000000000..8ab381b38--- /dev/null+++ b/server/src/lib/logger.test.js@@ -0,0 +1,32 @@+import pino from 'pino';+import { describe, expect, it } from 'vitest';+import { REDACT_PATHS } from './logger.js';++/** What the server's log never holds (#1263). */++function logged(object) {+ const lines = [];+ const log = pino({ redact: REDACT_PATHS }, { write: (line) => lines.push(line) });+ log.error(object, 'unhandled error');+ return lines.join('');+}++describe('the logger', () => {+ it("keeps a malformed request's body, which may hold a sign-in, out of the log", () => {+ const err = Object.assign(new SyntaxError('Unexpected end of JSON input'), {+ body: '{"input":{"apiKey":"sk_live_secret"',+ type: 'entity.parse.failed',+ });+ const line = logged({ err });+ expect(line).not.toContain('sk_live_secret');+ expect(line).toContain('Unexpected end of JSON input');+ });++ it("keeps a request's sign-in headers out of the log", () => {+ const line = logged({+ req: { headers: { authorization: 'Bearer tok_secret', cookie: 'c=1' } },+ });+ expect(line).not.toContain('tok_secret');+ expect(line).not.toContain('c=1');+ });+});diff --git a/server/src/routes/catalog.js b/server/src/routes/catalog.jsindex 8cb18d8e0..807d8cc8b 100644--- a/server/src/routes/catalog.js+++ b/server/src/routes/catalog.js@@ -1,16 +1,33 @@import express, { Router } from 'express';import { z } from 'zod';import supabaseAdmin from '../config/supabase.js';+import { storeConnectors } from '../connectors/index.js';import { assertBrandAccess } from '../lib/access.js';+import { credentialsConfigured } from '../lib/catalog/credentials.js';import { syncSource } from '../lib/catalog/sources.js';+import {+ StoreError,+ connectStore,+ disconnectStore,+ reconnectStore,+} from '../lib/catalog/stores.js';import { catalogLimiter } from '../middleware/rate-limiter.js';/**- * A brand's catalog sources (#1263, 00279). Members see them through RLS;- * admins and managers add and remove feed URLs there too. The server syncs:- * - POST /api/catalog/sources/:id/sync: a feed read now;+ * A brand's catalog sources (#1263, 00279, 00282). Members see them through+ * RLS; admins and managers add and remove feed URLs there too. The server+ * syncs, and holds the stores' sign-ins:+ * - POST /api/catalog/sources/:id/sync: a feed or a store read now;* - POST /api/catalog/upload?brandId=…&format=…: a CSV (or XML) file as- * the request's body, kept as the brand's upload source.+ * the request's body, kept as the brand's upload source;+ * - GET /api/catalog/connectors: the stores a person can connect, and+ * their fields;+ * - POST /api/catalog/sources { brandId, kind, input }: a store connected,+ * its sign-in checked with it and kept encrypted, then read;+ * - PUT /api/catalog/sources/:id/credentials { input }: its sign-in+ * replaced;+ * - DELETE /api/catalog/sources/:id: a store removed, with its sign-in and+ * its products.*/const router = Router();@@ -37,6 +54,9 @@ async function assertWriter(brandId, userId) {}function fail(req, res, err, what) {+ if (err instanceof StoreError) {+ return res.status(err.status).json({ error: err.message, code: err.code });+ }if (err?.status === 403 || err?.status === 404) {return res.status(err.status).json({error: err.message,@@ -58,17 +78,94 @@ const outcomeBody = (outcome) => ({counts: outcome.counts ?? null,});-// POST /api/catalog/sources/:id/sync — read a feed now.+const SOURCE_COLUMNS = 'id, brand_id, kind, format, url, account, label, last_synced_at';+/** The kinds members add themselves; every other kind is a store's. */+const OWN_KINDS = ['feed_url', 'upload'];++async function sourceById(id) {+ const parsed = z.guid().safeParse(id);+ if (!parsed.success) return null;+ const { data } = await supabaseAdmin+ .from('catalog_sources')+ .select(SOURCE_COLUMNS)+ .eq('id', parsed.data)+ .maybeSingle();+ return data ?? null;+}++/** A store's first read, or its read after a new sign-in: not waited on, recorded on its row. */+function syncLater(req, source) {+ syncSource(source).catch((err) =>+ req.log.error({ err: { name: err?.name }, sourceId: source.id }, 'catalog store sync error'),+ );+}++// GET /api/catalog/connectors — the stores a person can connect.+router.get('/connectors', (req, res) => {+ res.json({ connectors: storeConnectors(), ready: credentialsConfigured() });+});++const connectBody = z.object({+ brandId: z.guid(),+ kind: z.string().regex(/^[a-z][a-z0-9_]{1,39}$/),+ input: z.record(z.string(), z.string()),+});++// POST /api/catalog/sources — connect a store.+router.post('/sources', catalogLimiter, async (req, res) => {+ const parsed = connectBody.safeParse(req.body);+ if (!parsed.success) return res.status(400).json({ error: 'invalid', code: 'invalid' });+ const { brandId, kind, input } = parsed.data;+ try {+ await assertWriter(brandId, req.user.id);+ const source = await connectStore({ brandId, kind, input, userId: req.user.id });+ syncLater(req, source);+ return res.status(201).json({ id: source.id, account: source.account });+ } catch (err) {+ return fail(req, res, err, 'catalog store connect error');+ }+});++// PUT /api/catalog/sources/:id/credentials — replace a store's sign-in.+router.put('/sources/:id/credentials', catalogLimiter, async (req, res) => {+ const parsed = z.object({ input: z.record(z.string(), z.string()) }).safeParse(req.body);+ if (!parsed.success) return res.status(400).json({ error: 'invalid', code: 'invalid' });+ try {+ const source = await sourceById(req.params.id);+ if (!source || OWN_KINDS.includes(source.kind)) {+ return res.status(404).json({ error: 'Not found', code: 'not_found' });+ }+ await assertWriter(source.brand_id, req.user.id);+ await reconnectStore({ source, input: parsed.data.input, userId: req.user.id });+ syncLater(req, source);+ return res.json({ ok: true });+ } catch (err) {+ return fail(req, res, err, 'catalog store reconnect error');+ }+});++// DELETE /api/catalog/sources/:id — remove a store (feeds and uploads are removed through RLS).+router.delete('/sources/:id', catalogLimiter, async (req, res) => {+ try {+ const source = await sourceById(req.params.id);+ if (!source || OWN_KINDS.includes(source.kind)) {+ return res.status(404).json({ error: 'Not found', code: 'not_found' });+ }+ await assertWriter(source.brand_id, req.user.id);+ await disconnectStore(source);+ return res.json({ ok: true });+ } catch (err) {+ return fail(req, res, err, 'catalog store disconnect error');+ }+});++// POST /api/catalog/sources/:id/sync — read a feed or a store now.router.post('/sources/:id/sync', catalogLimiter, async (req, res) => {const parsed = z.guid().safeParse(req.params.id);if (!parsed.success) return res.status(400).json({ error: 'invalid', code: 'invalid' });try {- const { data: source } = await supabaseAdmin- .from('catalog_sources')- .select('id, brand_id, kind, format, url, last_synced_at')- .eq('id', parsed.data)- .maybeSingle();- if (!source || source.kind !== 'feed_url') {+ const source = await sourceById(parsed.data);+ if (!source || source.kind === 'upload') {return res.status(404).json({ error: 'Not found', code: 'not_found' });}await assertWriter(source.brand_id, req.user.id);diff --git a/server/src/routes/catalog.test.js b/server/src/routes/catalog.test.jsindex 588d00b3a..8cdaf7678 100644--- a/server/src/routes/catalog.test.js+++ b/server/src/routes/catalog.test.js@@ -2,7 +2,8 @@ import { afterEach, beforeEach, describe, expect, it, vi } from 'vitest';import express from 'express';/**- * A brand's catalog sources (#1263): a feed synced now, a file uploaded;+ * A brand's catalog sources (#1263): a feed or a store synced now, a file+ * uploaded, a store connected, its sign-in replaced, a store removed;* admins and managers only, a brand of their own workspace only.*/@@ -13,6 +14,10 @@ let sources;let inserted;const assertBrandAccess = vi.fn();const syncSource = vi.fn();+const connectStore = vi.fn();+const reconnectStore = vi.fn();+const disconnectStore = vi.fn();+let configured;vi.mock('../config/supabase.js', () => ({default: {@@ -56,8 +61,38 @@ vi.mock('../lib/access.js', () => ({vi.mock('../lib/catalog/sources.js', () => ({syncSource: (...args) => syncSource(...args),}));+// The limiter's 20 a quarter-hour would end this file's run part way.+vi.mock('../middleware/rate-limiter.js', () => ({ catalogLimiter: (_req, _res, next) => next() }));+vi.mock('../lib/catalog/credentials.js', () => ({+ credentialsConfigured: () => configured,+}));+vi.mock('../lib/catalog/stores.js', async () => {+ class StoreError extends Error {+ constructor(status, code, message) {+ super(message);+ this.status = status;+ this.code = code;+ }+ }+ return {+ StoreError,+ connectStore: (...args) => connectStore(...args),+ reconnectStore: (...args) => reconnectStore(...args),+ disconnectStore: (...args) => disconnectStore(...args),+ };+});+vi.mock('../connectors/index.js', () => ({+ storeConnectors: () => [+ {+ kind: 'acme_store',+ label: 'Acme Store',+ fields: [{ name: 'apiKey', label: 'API key', secret: true }],+ },+ ],+}));const { default: router } = await import('./catalog.js');+const { StoreError } = await import('../lib/catalog/stores.js');let server;let base;@@ -76,6 +111,15 @@ beforeEach(async () => {];assertBrandAccess.mockReset().mockResolvedValue({ orgId: 'org-1' });syncSource.mockReset().mockResolvedValue({ status: 'ok', counts: { items: 4 } });+ connectStore.mockReset().mockResolvedValue({+ id: 'store-1',+ brand_id: BRAND,+ kind: 'acme_store',+ account: 'shop.example',+ });+ reconnectStore.mockReset().mockResolvedValue(undefined);+ disconnectStore.mockReset().mockResolvedValue(undefined);+ configured = true;const app = express();app.use(express.json());app.use((req, _res, next) => {@@ -92,6 +136,23 @@ beforeEach(async () => {afterEach(() => new Promise((resolve) => server.close(resolve)));const sync = (id = SOURCE) => fetch(`${base}/sources/${id}/sync`, { method: 'POST' });+const send = (method, path, body) =>+ fetch(`${base}${path}`, {+ method,+ headers: { 'content-type': 'application/json' },+ body: body === undefined ? undefined : JSON.stringify(body),+ });+const STORE = '5c8d6a1e-2a3c-4d5e-8f90-123456789abc';+const addStore = () =>+ sources.push({+ id: STORE,+ brand_id: BRAND,+ kind: 'acme_store',+ format: 'auto',+ url: null,+ account: 'shop.example',+ last_synced_at: null,+ });const upload = (body, query = `brandId=${BRAND}`) =>fetch(`${base}/upload?${query}`, {method: 'POST',@@ -108,6 +169,14 @@ describe('POST /api/catalog/sources/:id/sync', () => {expect(syncSource).toHaveBeenCalledWith(expect.objectContaining({ id: SOURCE }));});+ it('syncs a store too', async () => {+ addStore();+ expect((await sync(STORE)).status).toBe(200);+ expect(syncSource).toHaveBeenCalledWith(+ expect.objectContaining({ id: STORE, kind: 'acme_store' }),+ );+ });+it('refuses an analyst, another workspace, an upload source and a bad id', async () => {role = 'analyst';expect((await sync()).status).toBe(403);@@ -175,3 +244,96 @@ describe('POST /api/catalog/upload', () => {expect(syncSource).not.toHaveBeenCalled();});});++describe('GET /api/catalog/connectors', () => {+ it('lists the stores a person can connect, and whether the server can keep a sign-in', async () => {+ configured = false;+ const res = await fetch(`${base}/connectors`);+ expect(await res.json()).toEqual({+ connectors: [+ {+ kind: 'acme_store',+ label: 'Acme Store',+ fields: [{ name: 'apiKey', label: 'API key', secret: true }],+ },+ ],+ ready: false,+ });+ });+});++describe('POST /api/catalog/sources', () => {+ const body = { brandId: BRAND, kind: 'acme_store', input: { apiKey: 'sk_secret' } };++ it('connects a store for an admin or a manager, then reads it without waiting', async () => {+ role = 'manager';+ const res = await send('POST', '/sources', body);+ expect(res.status).toBe(201);+ expect(await res.json()).toEqual({ id: 'store-1', account: 'shop.example' });+ expect(assertBrandAccess).toHaveBeenCalledWith(BRAND, 'user-1');+ expect(connectStore).toHaveBeenCalledWith({+ brandId: BRAND,+ kind: 'acme_store',+ input: { apiKey: 'sk_secret' },+ userId: 'user-1',+ });+ expect(syncSource).toHaveBeenCalledWith(expect.objectContaining({ id: 'store-1' }));+ });++ it("answers the store's refusal in the person's words, and connects nothing for an analyst", async () => {+ connectStore.mockRejectedValueOnce(+ new StoreError(400, 'auth_failed', 'The store refused this sign-in. Check it and try again.'),+ );+ const refused = await send('POST', '/sources', body);+ expect(refused.status).toBe(400);+ expect(await refused.json()).toEqual({+ error: 'The store refused this sign-in. Check it and try again.',+ code: 'auth_failed',+ });+ role = 'analyst';+ expect((await send('POST', '/sources', body)).status).toBe(403);+ expect((await send('POST', '/sources', { ...body, kind: 'Acme Store' })).status).toBe(400);+ expect((await send('POST', '/sources', { ...body, input: { apiKey: 42 } })).status).toBe(400);+ expect(connectStore).toHaveBeenCalledTimes(1);+ expect(syncSource).not.toHaveBeenCalled();+ });+});++describe('PUT /api/catalog/sources/:id/credentials', () => {+ it("replaces a store's sign-in, then reads it again", async () => {+ addStore();+ const res = await send('PUT', `/sources/${STORE}/credentials`, { input: { apiKey: 'sk_new' } });+ expect(res.status).toBe(200);+ expect(reconnectStore).toHaveBeenCalledWith({+ source: expect.objectContaining({ id: STORE, account: 'shop.example' }),+ input: { apiKey: 'sk_new' },+ userId: 'user-1',+ });+ expect(syncSource).toHaveBeenCalledTimes(1);+ });++ it('has no sign-in for a feed, and none to change for an analyst', async () => {+ expect((await send('PUT', `/sources/${SOURCE}/credentials`, { input: {} })).status).toBe(404);+ addStore();+ role = 'analyst';+ expect((await send('PUT', `/sources/${STORE}/credentials`, { input: {} })).status).toBe(403);+ expect(reconnectStore).not.toHaveBeenCalled();+ });+});++describe('DELETE /api/catalog/sources/:id', () => {+ it('removes a store for an admin or a manager', async () => {+ addStore();+ expect((await send('DELETE', `/sources/${STORE}`)).status).toBe(200);+ expect(disconnectStore).toHaveBeenCalledWith(expect.objectContaining({ id: STORE }));+ });++ it("leaves feeds to the person's own removal, and stores to their admins and managers", async () => {+ expect((await send('DELETE', `/sources/${SOURCE}`)).status).toBe(404);+ addStore();+ role = 'analyst';+ expect((await send('DELETE', `/sources/${STORE}`)).status).toBe(403);+ expect((await send('DELETE', '/sources/nope')).status).toBe(404);+ expect(disconnectStore).not.toHaveBeenCalled();+ });+});diff --git a/supabase/migrations/00282_catalog_store_connections.sql b/supabase/migrations/00282_catalog_store_connections.sqlnew file mode 100644index 000000000..3e93f4076--- /dev/null+++ b/supabase/migrations/00282_catalog_store_connections.sql@@ -0,0 +1,102 @@+-- ============================================================================+-- 00282: store connections for the catalog (#1263, part 3)+-- ============================================================================+-- A catalog source (00279) can now be a store or marketplace a connector+-- reads with the brand's own sign-in (WooCommerce, Etsy, …), next to a feed+-- URL and an upload.+--+-- * A source's kind is its connector's: any short slug. The server only+-- syncs kinds it has a connector for.+-- * Members still add only feed URLs and uploads themselves. A store is+-- connected through the API, which checks the sign-in with the store,+-- makes the source and keeps its credentials, so a store source never+-- exists without them.+-- * `account` names the store's account to the person: a shop's domain,+-- a seller id. A store's `url` is optional; a feed's is required, and an+-- upload has none.+-- * A brand connects each store account once.+-- * Members remove feed URLs and uploads themselves. A store is removed+-- through the API, which revokes its sign-in at the store first.+-- * `auth_failed`: the store refused the saved sign-in.+-- * The daily run reads every kind but uploads.+--+-- Credentials live in `catalog_source_credentials`, one row per source,+-- encrypted by the API with a key only it holds (AES-256-GCM, bound to the+-- source's id). Only the service role reads or writes them: members can't+-- even see that a row exists. They go with their source.++alter table public.catalog_sources drop constraint if exists catalog_sources_kind_check;+alter table public.catalog_sources+ add constraint catalog_sources_kind_check check (kind ~ '^[a-z][a-z0-9_]{1,39}$');++alter table public.catalog_sources drop constraint if exists catalog_sources_url_kind;+alter table public.catalog_sources+ add constraint catalog_sources_url_kind check (+ (kind <> 'feed_url' or url is not null) and (kind <> 'upload' or url is null)+ );++alter table public.catalog_sources+ add column if not exists account text+ check (account is null or char_length(account) between 1 and 200);++alter table public.catalog_sources drop constraint if exists catalog_sources_last_status_check;+alter table public.catalog_sources+ add constraint catalog_sources_last_status_check check (+ last_status is null or last_status in ('ok', 'failed', 'format_error', 'auth_failed')+ );++-- Members add feed URLs and uploads; stores come through the API.+drop policy if exists "catalog_sources: admin/manager insert" on public.catalog_sources;+create policy "catalog_sources: admin/manager insert"+ on public.catalog_sources+ for insert+ with check (+ kind in ('feed_url', 'upload')+ and brand_id in (+ select b.id from public.brands b+ join public.profiles p on p.organization_id = b.organization_id+ where p.id = auth.uid() and p.role = any (array['admin'::public.user_role, 'manager'::public.user_role])+ )+ and created_by = auth.uid()+ );++-- Members remove feed URLs and uploads; stores go through the API.+drop policy if exists "catalog_sources: admin/manager delete" on public.catalog_sources;+create policy "catalog_sources: admin/manager delete"+ on public.catalog_sources+ for delete+ using (+ kind in ('feed_url', 'upload')+ and brand_id in (+ select b.id from public.brands b+ join public.profiles p on p.organization_id = b.organization_id+ where p.id = auth.uid() and p.role = any (array['admin'::public.user_role, 'manager'::public.user_role])+ )+ );++create unique index if not exists catalog_sources_one_account+ on public.catalog_sources (brand_id, kind, account)+ where account is not null;++drop index if exists public.catalog_sources_feed_due_idx;+create index if not exists catalog_sources_due_idx+ on public.catalog_sources (last_synced_at nulls first)+ where kind <> 'upload';++create table if not exists public.catalog_source_credentials (+ source_id uuid primary key references public.catalog_sources(id) on delete cascade,+ secret text not null check (char_length(secret) between 1 and 16384),+ set_at timestamptz not null default now(),+ set_by uuid references public.profiles(id) on delete set null+);++alter table public.catalog_source_credentials enable row level security;+revoke all on table public.catalog_source_credentials from public, anon, authenticated;+grant all on table public.catalog_source_credentials to service_role;++comment on table public.catalog_source_credentials is+ 'Catalog (#1263, 00282): a store source''s sign-in, encrypted by the API (AES-256-GCM, bound to the source id). Service role only; removed with its source.';+comment on column public.catalog_sources.account is+ 'The store''s account as the person knows it (a shop domain, a seller id), set by the API when the store is connected.';++notify pgrst, 'reload schema';diff --git a/supabase/schema.sql b/supabase/schema.sqlindex fc279af0d..5ffe2d93c 100644--- a/supabase/schema.sql+++ b/supabase/schema.sql@@ -39893,3 +39893,109 @@ create index if not exists catalog_variants_brand_live_idxon public.catalog_variants (brand_id)where removed_at is null;+-- ─────────────────────────────────────────────────────────────────────────+-- migrations/00282_catalog_store_connections.sql+-- ─────────────────────────────────────────────────────────────────────────+-- ============================================================================+-- 00282: store connections for the catalog (#1263, part 3)+-- ============================================================================+-- A catalog source (00279) can now be a store or marketplace a connector+-- reads with the brand's own sign-in (WooCommerce, Etsy, …), next to a feed+-- URL and an upload.+--+-- * A source's kind is its connector's: any short slug. The server only+-- syncs kinds it has a connector for.+-- * Members still add only feed URLs and uploads themselves. A store is+-- connected through the API, which checks the sign-in with the store,+-- makes the source and keeps its credentials, so a store source never+-- exists without them.+-- * `account` names the store's account to the person: a shop's domain,+-- a seller id. A store's `url` is optional; a feed's is required, and an+-- upload has none.+-- * A brand connects each store account once.+-- * Members remove feed URLs and uploads themselves. A store is removed+-- through the API, which revokes its sign-in at the store first.+-- * `auth_failed`: the store refused the saved sign-in.+-- * The daily run reads every kind but uploads.+--+-- Credentials live in `catalog_source_credentials`, one row per source,+-- encrypted by the API with a key only it holds (AES-256-GCM, bound to the+-- source's id). Only the service role reads or writes them: members can't+-- even see that a row exists. They go with their source.++alter table public.catalog_sources drop constraint if exists catalog_sources_kind_check;+alter table public.catalog_sources+ add constraint catalog_sources_kind_check check (kind ~ '^[a-z][a-z0-9_]{1,39}$');++alter table public.catalog_sources drop constraint if exists catalog_sources_url_kind;+alter table public.catalog_sources+ add constraint catalog_sources_url_kind check (+ (kind <> 'feed_url' or url is not null) and (kind <> 'upload' or url is null)+ );++alter table public.catalog_sources+ add column if not exists account text+ check (account is null or char_length(account) between 1 and 200);++alter table public.catalog_sources drop constraint if exists catalog_sources_last_status_check;+alter table public.catalog_sources+ add constraint catalog_sources_last_status_check check (+ last_status is null or last_status in ('ok', 'failed', 'format_error', 'auth_failed')+ );++-- Members add feed URLs and uploads; stores come through the API.+drop policy if exists "catalog_sources: admin/manager insert" on public.catalog_sources;+create policy "catalog_sources: admin/manager insert"+ on public.catalog_sources+ for insert+ with check (+ kind in ('feed_url', 'upload')+ and brand_id in (+ select b.id from public.brands b+ join public.profiles p on p.organization_id = b.organization_id+ where p.id = auth.uid() and p.role = any (array['admin'::public.user_role, 'manager'::public.user_role])+ )+ and created_by = auth.uid()+ );++-- Members remove feed URLs and uploads; stores go through the API.+drop policy if exists "catalog_sources: admin/manager delete" on public.catalog_sources;+create policy "catalog_sources: admin/manager delete"+ on public.catalog_sources+ for delete+ using (+ kind in ('feed_url', 'upload')+ and brand_id in (+ select b.id from public.brands b+ join public.profiles p on p.organization_id = b.organization_id+ where p.id = auth.uid() and p.role = any (array['admin'::public.user_role, 'manager'::public.user_role])+ )+ );++create unique index if not exists catalog_sources_one_account+ on public.catalog_sources (brand_id, kind, account)+ where account is not null;++drop index if exists public.catalog_sources_feed_due_idx;+create index if not exists catalog_sources_due_idx+ on public.catalog_sources (last_synced_at nulls first)+ where kind <> 'upload';++create table if not exists public.catalog_source_credentials (+ source_id uuid primary key references public.catalog_sources(id) on delete cascade,+ secret text not null check (char_length(secret) between 1 and 16384),+ set_at timestamptz not null default now(),+ set_by uuid references public.profiles(id) on delete set null+);++alter table public.catalog_source_credentials enable row level security;+revoke all on table public.catalog_source_credentials from public, anon, authenticated;+grant all on table public.catalog_source_credentials to service_role;++comment on table public.catalog_source_credentials is+ 'Catalog (#1263, 00282): a store source''s sign-in, encrypted by the API (AES-256-GCM, bound to the source id). Service role only; removed with its source.';+comment on column public.catalog_sources.account is+ 'The store''s account as the person knows it (a shop domain, a seller id), set by the API when the store is connected.';++notify pgrst, 'reload schema';+diff --git a/web/messages/en.json b/web/messages/en.jsonindex f555ec5b7..1bebd7376 100644--- a/web/messages/en.json+++ b/web/messages/en.json@@ -7572,6 +7572,7 @@"ok": "Synced","failed": "Couldn't sync","format_error": "Not a feed",+ "auth_failed": "Sign-in refused","never": "Not synced yet"},"products": "{count, plural, one {# product} other {# products}}",@@ -7599,6 +7600,35 @@"workspace": "This brand isn't in your workspace.","busy": "Too many syncs. Wait a few minutes.","failed": "Couldn't sync the catalog. Try again later."+ },+ "stores": {+ "title": "Connect a store",+ "connect": "Connect {store}",+ "formLabel": "{store} sign-in",+ "submit": "Connect",+ "update": "Update sign-in",+ "save": "Save sign-in",+ "cancel": "Cancel",+ "kept": "The sign-in is checked with the store, then kept encrypted on the server and used only to read the catalog. It isn't shown again.",+ "notConfigured": "Store connections aren't set up on this server yet.",+ "connected": "{store} connected. Its products are being read.",+ "updated": "Sign-in saved. The products are being read again.",+ "errors": {+ "connect_failed": "Check what you entered and try again.",+ "auth_failed": "The store refused this sign-in. Check it and try again.",+ "unreachable": "The store couldn't be reached. Try again in a moment.",+ "duplicate": "This store is connected already.",+ "too_many": "A brand can have 10 sources at most.",+ "other_account": "This sign-in is for another store. Add that store as a new source.",+ "not_configured": "Store connections aren't set up on this server yet.",+ "forbidden": "Only admins and managers can change the catalog.",+ "not_found": "This store no longer exists.",+ "invalid": "Fill in every field.",+ "session": "Your session expired. Sign in again.",+ "workspace": "This brand isn't in your workspace.",+ "busy": "Too many requests. Wait a few minutes.",+ "failed": "Couldn't reach the catalog. Try again later."+ }}},"integrations_connected": "Connected",diff --git a/web/src/components/settings/catalog-sources-card.test.tsx b/web/src/components/settings/catalog-sources-card.test.tsxindex 232aee34a..4b5dce7a0 100644--- a/web/src/components/settings/catalog-sources-card.test.tsx+++ b/web/src/components/settings/catalog-sources-card.test.tsx@@ -15,9 +15,17 @@ vi.mock('@/lib/actions/catalog', () => ({}));const syncCatalogSource = vi.fn();const uploadCatalogFile = vi.fn();+const listStoreConnectors = vi.fn();+const connectStore = vi.fn();+const updateStoreSignIn = vi.fn();+const removeStore = vi.fn();vi.mock('@/lib/catalog-client', () => ({syncCatalogSource: (...args: unknown[]) => syncCatalogSource(...args),uploadCatalogFile: (...args: unknown[]) => uploadCatalogFile(...args),+ listStoreConnectors: (...args: unknown[]) => listStoreConnectors(...args),+ connectStore: (...args: unknown[]) => connectStore(...args),+ updateStoreSignIn: (...args: unknown[]) => updateStoreSignIn(...args),+ removeStore: (...args: unknown[]) => removeStore(...args),}));const role = vi.hoisted(() => ({ canManage: true }));vi.mock('@/hooks/use-user-role', () => ({ useUserRole: () => role }));@@ -32,7 +40,7 @@ const { CatalogSourcesCard } = await import('./catalog-sources-card');(globalThis as typeof globalThis & { IS_REACT_ACT_ENVIRONMENT: boolean }).IS_REACT_ACT_ENVIRONMENT =true;-/** Settings › Integrations › Product catalog (#1263): a brand's feed URLs and uploads. */+/** Settings › Integrations › Product catalog (#1263): a brand's feed URLs, uploads and stores. */let container: HTMLDivElement;let root: Root;@@ -42,6 +50,8 @@ const FEED: CatalogSource = {kind: 'feed_url',format: 'auto',url: 'https://shop.example/feed.xml',+ account: null,+ label: null,lastSyncedAt: '2026-10-25T08:00:00Z',lastStatus: 'ok',lastError: null,@@ -56,6 +66,33 @@ const FEED: CatalogSource = {products: 2,};+const STORE: CatalogSource = {+ ...FEED,+ id: 'st1',+ kind: 'acme_store',+ url: 'https://acme-shop.example',+ account: 'acme-shop.example',+ label: 'Acme Outdoor',+ lastStatus: 'auth_failed',+ lastError: 'The store refused the saved sign-in. Update it in Settings › Integrations.',+ lastCounts: null,+};++const ACME = {+ kind: 'acme_store',+ label: 'Acme Store',+ fields: [+ {+ name: 'url',+ label: 'Store address',+ secret: false,+ placeholder: 'https://shop.example',+ help: null,+ },+ { name: 'apiKey', label: 'API key', secret: true, placeholder: null, help: 'Read-only key.' },+ ],+};+async function render() {await act(async () => {root.render(@@ -69,6 +106,23 @@ const button = (label: string) =>[...container.querySelectorAll('button')].find((b) => b.textContent === label || b.getAttribute('aria-label') === label,);+async function type(input: HTMLInputElement, value: string) {+ await act(async () => {+ const setter = Object.getOwnPropertyDescriptor(HTMLInputElement.prototype, 'value')!.set!;+ setter.call(input, value);+ input.dispatchEvent(new Event('input', { bubbles: true }));+ });+}+const storeForm = () =>+ container.querySelector<HTMLFormElement>('form[aria-label="Acme Store sign-in"]');+async function fillAndSubmit(form: HTMLFormElement) {+ const [address, key] = [...form.querySelectorAll<HTMLInputElement>('input')];+ await type(address, 'https://acme-shop.example');+ await type(key, 'ak_secret');+ await act(async () => {+ form.dispatchEvent(new Event('submit', { bubbles: true, cancelable: true }));+ });+}beforeEach(() => {role.canManage = true;@@ -82,6 +136,10 @@ beforeEach(() => {counts: { items: 12 },});uploadCatalogFile.mockReset();+ listStoreConnectors.mockReset().mockResolvedValue({ connectors: [], ready: true });+ connectStore.mockReset().mockResolvedValue({ ok: true });+ updateStoreSignIn.mockReset().mockResolvedValue({ ok: true });+ removeStore.mockReset().mockResolvedValue({ ok: true });toast.error.mockReset();toast.success.mockReset();container = document.createElement('div');@@ -163,4 +221,75 @@ describe('CatalogSourcesCard', () => {expect(button('Sync now')).toBeUndefined();expect(button('Remove')).toBeUndefined();});++ it('offers no store when the API has no store connectors', async () => {+ await render();+ expect(container.textContent).not.toContain('Connect a store');+ });++ it('connects a store with the fields its connector asks for, the secret one hidden', async () => {+ listStoreConnectors.mockResolvedValue({ connectors: [ACME], ready: true });+ await render();+ expect(container.textContent).toContain('Connect a store');+ await act(async () => button('Connect Acme Store')!.click());+ const form = storeForm()!;+ const inputs = [...form.querySelectorAll<HTMLInputElement>('input')];+ expect(inputs.map((i) => i.type)).toEqual(['text', 'password']);+ expect(form.textContent).toContain('Read-only key.');+ await fillAndSubmit(form);+ expect(connectStore).toHaveBeenCalledWith('b1', 'acme_store', {+ url: 'https://acme-shop.example',+ apiKey: 'ak_secret',+ });+ expect(toast.success).toHaveBeenCalledWith(+ 'Acme Store connected. Its products are being read.',+ );+ expect(storeForm()).toBeNull();+ expect(listCatalogSources).toHaveBeenCalledTimes(2);+ });++ it("shows the store's own words when it couldn't connect, and keeps the form", async () => {+ listStoreConnectors.mockResolvedValue({ connectors: [ACME], ready: true });+ connectStore.mockResolvedValue({+ ok: false,+ code: 'connect_failed',+ message: "That isn't a web address.",+ });+ await render();+ await act(async () => button('Connect Acme Store')!.click());+ await fillAndSubmit(storeForm()!);+ expect(toast.error).toHaveBeenCalledWith("That isn't a web address.");+ expect(storeForm()).not.toBeNull();+ });++ it("says the server can't keep a sign-in yet, instead of offering the stores", async () => {+ listStoreConnectors.mockResolvedValue({ connectors: [ACME], ready: false });+ await render();+ expect(container.textContent).toContain("Store connections aren't set up on this server yet.");+ expect(button('Connect Acme Store')).toBeUndefined();+ });++ it('names a store by its platform and account, replaces its sign-in, and removes it through the API', async () => {+ listStoreConnectors.mockResolvedValue({ connectors: [ACME], ready: true });+ listCatalogSources.mockResolvedValue([STORE]);+ await render();+ const text = container.textContent ?? '';+ expect(text).toContain('Acme Store · acme-shop.example');+ expect(text).toContain('Sign-in refused');+ expect(text).toContain('The store refused the saved sign-in.');+ expect(button('Sync now')).toBeDefined();++ await act(async () => button('Update sign-in')!.click());+ await fillAndSubmit(storeForm()!);+ expect(updateStoreSignIn).toHaveBeenCalledWith('st1', {+ url: 'https://acme-shop.example',+ apiKey: 'ak_secret',+ });+ expect(toast.success).toHaveBeenCalledWith('Sign-in saved. The products are being read again.');++ vi.spyOn(window, 'confirm').mockReturnValue(true);+ await act(async () => button('Remove')!.click());+ expect(removeStore).toHaveBeenCalledWith('st1');+ expect(removeCatalogSource).not.toHaveBeenCalled();+ });});diff --git a/web/src/components/settings/catalog-sources-card.tsx b/web/src/components/settings/catalog-sources-card.tsxindex 8049adaf1..4eb6a4d8a 100644--- a/web/src/components/settings/catalog-sources-card.tsx+++ b/web/src/components/settings/catalog-sources-card.tsx@@ -3,21 +3,34 @@import { useCallback, useEffect, useRef, useState } from 'react';import { useFormatter, useTranslations } from 'next-intl';import { toast } from 'sonner';-import { Loader2, Package, RefreshCw, Trash2, Upload } from 'lucide-react';+import { KeyRound, Loader2, Package, RefreshCw, Store, Trash2, Upload } from 'lucide-react';import { Badge } from '@/components/ui/badge';import { Button } from '@/components/ui/button';import { Input } from '@/components/ui/input';import { Skeleton } from '@/components/ui/skeleton';import { useUserRole } from '@/hooks/use-user-role';import { addFeedSource, listCatalogSources, removeCatalogSource } from '@/lib/actions/catalog';-import type { CatalogSource } from '@/lib/catalog';-import { syncCatalogSource, uploadCatalogFile, type CatalogSyncResult } from '@/lib/catalog-client';+import { isStoreSource, type CatalogSource } from '@/lib/catalog';+import {+ connectStore,+ listStoreConnectors,+ removeStore,+ syncCatalogSource,+ updateStoreSignIn,+ uploadCatalogFile,+ type CatalogSyncResult,+ type StoreConnector,+ type StoreResult,+} from '@/lib/catalog-client';import { useBrandStore } from '@/stores/use-brand-store';+import { StoreSignInForm } from './catalog-store-form';/*** Product catalog (#1263): where the active brand's catalog comes from, a- * product feed URL read daily, or a CSV / feed file uploaded. Admins and- * managers add, sync and remove; members see the sources and how they went.+ * product feed URL read daily, a CSV / feed file uploaded, or a store+ * connected with its sign-in (00282), as the API's connectors offer them.+ * Admins and managers add, sync and remove; members see the sources and how+ * they went.*/export function CatalogSourcesCard() {const t = useTranslations('settings.catalog');@@ -36,6 +49,24 @@ export function CatalogSourcesCard() {const [url, setUrl] = useState('');const [busy, setBusy] = useState<string | null>(null);const fileInput = useRef<HTMLInputElement>(null);+ // The stores the API can connect; null until read, or when it can't be.+ const [stores, setStores] = useState<{ connectors: StoreConnector[]; ready: boolean } | null>(+ null,+ );+ // The store being connected (its kind), or the store whose sign-in is being replaced (its id).+ const [connecting, setConnecting] = useState<string | null>(null);+ const [signingIn, setSigningIn] = useState<string | null>(null);++ useEffect(() => {+ let cancelled = false;+ void listStoreConnectors().then((list) => {+ if (!cancelled) setStores(list);+ });+ return () => {+ cancelled = true;+ };+ }, []);+ const connectorOf = (kind: string) => stores?.connectors.find((c) => c.kind === kind) ?? null;// Bumped to read the sources again, after a sync, an add or a removal.const [reads, setReads] = useState(0);@@ -93,18 +124,59 @@ export function CatalogSourcesCard() {setBusy(null);};- const remove = async (id: string) => {+ const remove = async (source: CatalogSource) => {if (!window.confirm(t('removeConfirm'))) return;- setBusy(id);- try {- await removeCatalogSource(id);+ setBusy(source.id);+ if (isStoreSource(source.kind)) {+ // A store goes through the API, which holds its sign-in.+ const removed = await removeStore(source.id);+ if (!removed.ok) toast.error(t(`stores.errors.${removed.code}`));load();- } catch {- toast.error(t('errors.forbidden'));+ } else {+ try {+ await removeCatalogSource(source.id);+ load();+ } catch {+ toast.error(t('errors.forbidden'));+ }}setBusy(null);};+ /** Says how a store call went, then reads the sources again. */+ const reportStore = (result: StoreResult, success: string) => {+ if (!result.ok) toast.error(result.message ?? t(`stores.errors.${result.code}`));+ else toast.success(success);+ load();+ return result.ok;+ };++ const connect = async (connector: StoreConnector, input: Record<string, string>) => {+ if (!activeBrandId) return;+ setBusy('connect');+ const ok = reportStore(+ await connectStore(activeBrandId, connector.kind, input),+ t('stores.connected', { store: connector.label }),+ );+ if (ok) setConnecting(null);+ setBusy(null);+ };++ const signIn = async (source: CatalogSource, input: Record<string, string>) => {+ setBusy(source.id);+ const ok = reportStore(await updateStoreSignIn(source.id, input), t('stores.updated'));+ if (ok) setSigningIn(null);+ setBusy(null);+ };++ /** How a source is named in its row: a feed by its address, a store by its name and account. */+ const nameOf = (source: CatalogSource) => {+ if (source.kind === 'feed_url') return source.url;+ if (source.kind === 'upload') return t('upload');+ const platform = connectorOf(source.kind)?.label ?? source.label ?? source.kind;+ return source.account ? `${platform} · ${source.account}` : platform;+ };+const upload = async (file: File | undefined) => {if (!file || !activeBrandId) return;setBusy('upload');@@ -145,13 +217,25 @@ export function CatalogSourcesCard() {className="min-w-0 truncate font-medium"title={source.url ?? undefined}>- {source.kind === 'feed_url' ? source.url : t('upload')}+ {nameOf(source)}</span><div className="flex items-center gap-2"><Badge variant={source.lastStatus === 'ok' ? 'secondary' : 'outline'}>{source.lastStatus ? t(`status.${source.lastStatus}`) : t('status.never')}</Badge>- {canManage && source.kind === 'feed_url' && (+ {canManage && isStoreSource(source.kind) && connectorOf(source.kind) && (+ <Button+ size="sm"+ variant="ghost"+ disabled={busy !== null}+ onClick={() => setSigningIn(signingIn === source.id ? null : source.id)}+ aria-label={t('stores.update')}+ aria-expanded={signingIn === source.id}+ >+ <KeyRound className="h-4 w-4" />+ </Button>+ )}+ {canManage && source.kind !== 'upload' && (<Buttonsize="sm"variant="ghost"@@ -171,7 +255,7 @@ export function CatalogSourcesCard() {size="sm"variant="ghost"disabled={busy !== null}- onClick={() => remove(source.id)}+ onClick={() => remove(source)}aria-label={t('remove')}><Trash2 className="h-4 w-4" />@@ -199,6 +283,15 @@ export function CatalogSourcesCard() {</ul></details>)}+ {signingIn === source.id && connectorOf(source.kind) && (+ <StoreSignInForm+ connector={connectorOf(source.kind)!}+ submitLabel={t('stores.save')}+ busy={busy === source.id}+ onSubmit={(input) => void signIn(source, input)}+ onCancel={() => setSigningIn(null)}+ />+ )}</li>))}</ul>@@ -251,6 +344,45 @@ export function CatalogSourcesCard() {</Button><span className="text-xs text-muted-foreground">{t('uploadHint')}</span></div>+ {stores && stores.connectors.length > 0 && (+ <div className="space-y-2 border-t pt-3">+ <p className="text-sm font-medium">{t('stores.title')}</p>+ {stores.ready ? (+ <>+ <div className="flex flex-wrap gap-2">+ {stores.connectors.map((connector) => (+ <Button+ key={connector.kind}+ type="button"+ variant={connecting === connector.kind ? 'secondary' : 'outline'}+ size="sm"+ disabled={busy !== null}+ aria-expanded={connecting === connector.kind}+ onClick={() =>+ setConnecting(connecting === connector.kind ? null : connector.kind)+ }+ >+ <Store className="mr-2 h-4 w-4" />+ {t('stores.connect', { store: connector.label })}+ </Button>+ ))}+ </div>+ {connecting && connectorOf(connecting) && (+ <StoreSignInForm+ key={connecting}+ connector={connectorOf(connecting)!}+ submitLabel={t('stores.submit')}+ busy={busy === 'connect'}+ onSubmit={(input) => void connect(connectorOf(connecting)!, input)}+ onCancel={() => setConnecting(null)}+ />+ )}+ </>+ ) : (+ <p className="text-xs text-muted-foreground">{t('stores.notConfigured')}</p>+ )}+ </div>+ )}</div>) : (<p className="text-xs text-muted-foreground">{t('managersOnly')}</p>diff --git a/web/src/components/settings/catalog-store-form.tsx b/web/src/components/settings/catalog-store-form.tsxnew file mode 100644index 000000000..175d817ce--- /dev/null+++ b/web/src/components/settings/catalog-store-form.tsx@@ -0,0 +1,73 @@+'use client';++import { useState } from 'react';+import { useTranslations } from 'next-intl';+import { Loader2 } from 'lucide-react';+import { Button } from '@/components/ui/button';+import { Input } from '@/components/ui/input';+import { Label } from '@/components/ui/label';+import type { StoreConnector } from '@/lib/catalog-client';++/**+ * A store's sign-in, as its connector asks for it (00282): to connect the+ * store, or to replace its sign-in. A secret field is typed like a password+ * and never shown again; what's typed goes to the API only.+ */+export function StoreSignInForm({+ connector,+ submitLabel,+ busy,+ onSubmit,+ onCancel,+}: {+ connector: StoreConnector;+ submitLabel: string;+ busy: boolean;+ onSubmit: (input: Record<string, string>) => void;+ onCancel: () => void;+}) {+ const t = useTranslations('settings.catalog.stores');+ const [values, setValues] = useState<Record<string, string>>({});+ const complete = connector.fields.every((field) => (values[field.name] ?? '').trim());++ return (+ <form+ className="space-y-3 rounded-md border bg-muted/30 p-3"+ aria-label={t('formLabel', { store: connector.label })}+ onSubmit={(e) => {+ e.preventDefault();+ if (complete && !busy) onSubmit(values);+ }}+ >+ {connector.fields.map((field) => {+ const id = `store-${connector.kind}-${field.name}`;+ return (+ <div key={field.name} className="space-y-1">+ <Label htmlFor={id}>{field.label}</Label>+ <Input+ id={id}+ type={field.secret ? 'password' : 'text'}+ autoComplete={field.secret ? 'new-password' : 'off'}+ spellCheck={false}+ placeholder={field.placeholder ?? undefined}+ maxLength={2000}+ value={values[field.name] ?? ''}+ onChange={(e) => setValues((v) => ({ ...v, [field.name]: e.target.value }))}+ />+ {field.help && <p className="text-xs text-muted-foreground">{field.help}</p>}+ </div>+ );+ })}+ <p className="text-xs text-muted-foreground">{t('kept')}</p>+ <div className="flex gap-2">+ <Button type="submit" size="sm" disabled={!complete || busy}>+ {busy && <Loader2 className="mr-2 h-4 w-4 animate-spin" />}+ {submitLabel}+ </Button>+ <Button type="button" size="sm" variant="ghost" disabled={busy} onClick={onCancel}>+ {t('cancel')}+ </Button>+ </div>+ </form>+ );+}diff --git a/web/src/lib/actions/catalog.ts b/web/src/lib/actions/catalog.tsindex 8c312dfff..16ded2a36 100644--- a/web/src/lib/actions/catalog.ts+++ b/web/src/lib/actions/catalog.ts@@ -12,9 +12,10 @@ import {} from '@/lib/catalog';/**- * A brand's catalog sources (#1263, 00279), read and changed through RLS:- * members read them; admins and managers add and remove feed URLs. The API- * reads the feeds and imports uploads (lib/catalog-client.ts).+ * A brand's catalog sources (#1263, 00279, 00282), read and changed through+ * RLS: members read them; admins and managers add and remove feed URLs. The+ * API reads the feeds and stores, imports uploads, and connects and removes+ * stores (lib/catalog-client.ts).*/const FORMATS = ['auto', 'google_xml', 'csv', 'meta_csv'] as const;@@ -25,7 +26,9 @@ export async function listCatalogSources(brandId: string): Promise<CatalogSourceconst supabase = await createClient();const { data, error } = await supabase.from('catalog_sources')- .select('id, kind, format, url, last_synced_at, last_status, last_error, last_counts')+ .select(+ 'id, kind, format, url, account, label, last_synced_at, last_status, last_error, last_counts',+ ).eq('brand_id', brandId).order('created_at', { ascending: true });if (error) throw new Error(error.message);@@ -42,6 +45,8 @@ export async function listCatalogSources(brandId: string): Promise<CatalogSourcekind: row.kind as CatalogSourceKind,format: row.format as CatalogFormat,url: row.url,+ account: row.account,+ label: row.label,lastSyncedAt: row.last_synced_at,lastStatus: row.last_status as CatalogSyncStatus | null,lastError: row.last_error,@@ -92,7 +97,7 @@ export async function addFeedSource(return { ok: true, id: data.id };}-/** Removes a source, and with it its products: admins and managers. */+/** Removes a feed or an upload, and with it its products: admins and managers. A store goes through the API. */export async function removeCatalogSource(id: string): Promise<void> {if (!z.guid().safeParse(id).success) throw new Error('invalid');const supabase = await createClient();diff --git a/web/src/lib/catalog-client.test.ts b/web/src/lib/catalog-client.test.tsindex cd2c0d569..45f50c267 100644--- a/web/src/lib/catalog-client.test.ts+++ b/web/src/lib/catalog-client.test.ts@@ -8,7 +8,14 @@ vi.mock('@/lib/supabase/client', () => ({}),}));-const { syncCatalogSource, uploadCatalogFile } = await import('./catalog-client');+const {+ connectStore,+ listStoreConnectors,+ removeStore,+ syncCatalogSource,+ updateStoreSignIn,+ uploadCatalogFile,+} = await import('./catalog-client');const fetchMock = vi.fn();beforeEach(() => {@@ -62,3 +69,60 @@ describe('uploadCatalogFile', () => {expect(fetchMock).not.toHaveBeenCalled();});});++describe('stores (00282)', () => {+ it("lists the connectors, or nothing when the API can't be read", async () => {+ answer(200, {+ connectors: [{ kind: 'acme_store', label: 'Acme Store', fields: [] }],+ ready: true,+ });+ expect(await listStoreConnectors()).toEqual({+ connectors: [{ kind: 'acme_store', label: 'Acme Store', fields: [] }],+ ready: true,+ });+ expect(fetchMock.mock.calls[0][0]).toBe('/api/aeo/api/catalog/connectors');+ answer(500, {});+ expect(await listStoreConnectors()).toBeNull();+ });++ it('sends what the person typed to the API only, as JSON with the session', async () => {+ answer(201, { id: 'st1', account: 'shop.example' });+ expect(await connectStore('b1', 'acme_store', { apiKey: 'ak_secret' })).toEqual({ ok: true });+ const [url, init] = fetchMock.mock.calls[0];+ expect(url).toBe('/api/aeo/api/catalog/sources');+ expect(init).toMatchObject({+ method: 'POST',+ headers: { 'Content-Type': 'application/json', Authorization: 'Bearer tok' },+ });+ expect(JSON.parse(init.body)).toEqual({+ brandId: 'b1',+ kind: 'acme_store',+ input: { apiKey: 'ak_secret' },+ });++ answer(200, { ok: true });+ await updateStoreSignIn('st1', { apiKey: 'ak_new' });+ expect(fetchMock.mock.calls[1][0]).toBe('/api/aeo/api/catalog/sources/st1/credentials');+ expect(fetchMock.mock.calls[1][1].method).toBe('PUT');+ await removeStore('st1');+ expect(fetchMock.mock.calls[2][0]).toBe('/api/aeo/api/catalog/sources/st1');+ expect(fetchMock.mock.calls[2][1].method).toBe('DELETE');+ });++ it("keeps the store's own words only for what to fix", async () => {+ answer(400, { code: 'connect_failed', error: "That isn't a web address." });+ expect(await connectStore('b1', 'acme_store', {})).toEqual({+ ok: false,+ code: 'connect_failed',+ message: "That isn't a web address.",+ });+ answer(400, { code: 'auth_failed', error: 'The store refused this sign-in.' });+ expect(await connectStore('b1', 'acme_store', {})).toEqual({+ ok: false,+ code: 'auth_failed',+ message: null,+ });+ answer(429, { message: 'Too many requests' });+ expect(await updateStoreSignIn('st1', {})).toEqual({ ok: false, code: 'busy', message: null });+ });+});diff --git a/web/src/lib/catalog-client.ts b/web/src/lib/catalog-client.tsindex eebf2fd0c..3b5345c11 100644--- a/web/src/lib/catalog-client.ts+++ b/web/src/lib/catalog-client.ts@@ -10,8 +10,9 @@ import {/*** A catalog sync from the browser (#1263), through the API's same-origin- * proxy: a feed read now, or a file uploaded. Not a server action, whose- * body is limited to 1 MB.+ * proxy: a feed or a store read now, or a file uploaded. Not a server+ * action, whose body is limited to 1 MB. Stores are connected here too+ * (00282): their sign-in goes to the API, which alone keeps it.*/export type CatalogSyncError =@@ -86,7 +87,7 @@ async function call(path: string, init: RequestInit): Promise<CatalogSyncResult>}}-/** Reads a feed source now. */+/** Reads a feed or a store now. */export function syncCatalogSource(id: string): Promise<CatalogSyncResult> {return call(`/sources/${encodeURIComponent(id)}/sync`, { method: 'POST' });}@@ -106,3 +107,133 @@ export async function uploadCatalogFile(body: text,});}++/** A field a store's connector asks for, as the API lists it. */+export interface StoreField {+ name: string;+ label: string;+ secret: boolean;+ placeholder: string | null;+ help: string | null;+}++/** A store or marketplace a person can connect (00282): its connector on the API. */+export interface StoreConnector {+ kind: string;+ label: string;+ fields: StoreField[];+}++export type StoreError =+ | 'connect_failed'+ | 'auth_failed'+ | 'unreachable'+ | 'duplicate'+ | 'too_many'+ | 'other_account'+ | 'not_configured'+ | 'forbidden'+ | 'not_found'+ | 'invalid'+ | 'session'+ | 'workspace'+ | 'busy'+ | 'failed';++/** `message`: the store's own words for what to fix, when it has them. */+export type StoreResult = { ok: true } | { ok: false; code: StoreError; message: string | null };++const STORE_KNOWN: readonly StoreError[] = [+ 'connect_failed',+ 'auth_failed',+ 'unreachable',+ 'duplicate',+ 'too_many',+ 'other_account',+ 'not_configured',+ 'forbidden',+ 'not_found',+ 'invalid',+];++/** A store call: the sign-in the person typed goes to the API only, over its proxy. */+async function storeCall(path: string, init: RequestInit): Promise<StoreResult> {+ try {+ const {+ data: { session },+ } = await createClient().auth.getSession();+ if (!session) return { ok: false, code: 'session', message: null };+ const res = await fetch(`${BROWSER_API_BASE_URL}/api/catalog${path}`, {+ ...init,+ headers: {+ ...init.headers,+ 'Content-Type': 'application/json',+ Authorization: `Bearer ${session.access_token}`,+ },+ });+ if (res.ok) return { ok: true };+ const body = (await res.json().catch(() => ({}))) as { code?: string; error?: string };+ const known = STORE_KNOWN.find((c) => c === body.code);+ if (known) {+ return {+ ok: false,+ code: known,+ message: known === 'connect_failed' ? (body.error ?? null) : null,+ };+ }+ const general = apiFailure(res.status, body);+ return {+ ok: false,+ code:+ general === 'session' || general === 'workspace' || general === 'busy' ? general : 'failed',+ message: null,+ };+ } catch (err) {+ console.error('Failed to reach the catalog API:', err);+ return { ok: false, code: 'failed', message: null };+ }+}++/** The stores a person can connect, and whether the API can keep a sign-in; null when unreadable. */+export async function listStoreConnectors(): Promise<{+ connectors: StoreConnector[];+ ready: boolean;+} | null> {+ try {+ const {+ data: { session },+ } = await createClient().auth.getSession();+ if (!session) return null;+ const res = await fetch(`${BROWSER_API_BASE_URL}/api/catalog/connectors`, {+ headers: { Authorization: `Bearer ${session.access_token}` },+ });+ if (!res.ok) return null;+ const body = (await res.json()) as { connectors?: StoreConnector[]; ready?: boolean };+ return { connectors: body.connectors ?? [], ready: !!body.ready };+ } catch (err) {+ console.error('Failed to read the store connectors:', err);+ return null;+ }+}++/** Connects a store to a brand with what the person typed; the API reads it right after. */+export function connectStore(+ brandId: string,+ kind: string,+ input: Record<string, string>,+): Promise<StoreResult> {+ return storeCall('/sources', { method: 'POST', body: JSON.stringify({ brandId, kind, input }) });+}++/** Replaces a store's sign-in, for the same store. */+export function updateStoreSignIn(id: string, input: Record<string, string>): Promise<StoreResult> {+ return storeCall(`/sources/${encodeURIComponent(id)}/credentials`, {+ method: 'PUT',+ body: JSON.stringify({ input }),+ });+}++/** Removes a store, its sign-in and its products. */+export function removeStore(id: string): Promise<StoreResult> {+ return storeCall(`/sources/${encodeURIComponent(id)}`, { method: 'DELETE' });+}diff --git a/web/src/lib/catalog.ts b/web/src/lib/catalog.tsindex a0994aaeb..cfb937fb0 100644--- a/web/src/lib/catalog.ts+++ b/web/src/lib/catalog.ts@@ -1,15 +1,23 @@/**- * A brand's catalog sources (#1263, 00279), as Settings › Integrations shows- * them: a feed URL the API reads daily, or the file a person uploads.+ * A brand's catalog sources (#1263, 00279, 00282), as Settings › Integrations+ * shows them: a feed URL the API reads daily, the file a person uploads, or+ * a store connected with its sign-in, whose kind is its connector's.*/-export type CatalogSourceKind = 'feed_url' | 'upload';+export type CatalogSourceKind = 'feed_url' | 'upload' | (string & {});export type CatalogFormat = 'auto' | 'google_xml' | 'csv' | 'meta_csv';-export type CatalogSyncStatus = 'ok' | 'failed' | 'format_error';+/** `auth_failed`: the store refused the saved sign-in. */+export type CatalogSyncStatus = 'ok' | 'failed' | 'format_error' | 'auth_failed';++/** Whether a source is a store, read with its sign-in, rather than a feed or an upload. */+export function isStoreSource(kind: CatalogSourceKind): boolean {+ return kind !== 'feed_url' && kind !== 'upload';+}/** What a sync did, as the API records it on the source (`last_counts`). */export interface CatalogCounts {- format: Exclude<CatalogFormat, 'auto'>;+ /** A feed's format, or a store's kind. */+ format: string;items: number;skipped: number;/** The first products skipped, and why ("Row 3: price is missing."). */@@ -23,6 +31,10 @@ export interface CatalogSource {kind: CatalogSourceKind;format: CatalogFormat;url: string | null;+ /** A store's account as the person knows it (a shop's domain); null for feeds and uploads. */+ account: string | null;+ /** A store's name, as the store gave it when connected. */+ label: string | null;lastSyncedAt: string | null;lastStatus: CatalogSyncStatus | null;lastError: string | null;diff --git a/web/src/types/supabase.ts b/web/src/types/supabase.tsindex 445c5d96b..3910f247b 100644--- a/web/src/types/supabase.ts+++ b/web/src/types/supabase.ts@@ -877,8 +877,45 @@ export type Database = {},]}+ catalog_source_credentials: {+ Row: {+ secret: string+ set_at: string+ set_by: string | null+ source_id: string+ }+ Insert: {+ secret: string+ set_at?: string+ set_by?: string | null+ source_id: string+ }+ Update: {+ secret?: string+ set_at?: string+ set_by?: string | null+ source_id?: string+ }+ Relationships: [+ {+ foreignKeyName: "catalog_source_credentials_set_by_fkey"+ columns: ["set_by"]+ isOneToOne: false+ referencedRelation: "profiles"+ referencedColumns: ["id"]+ },+ {+ foreignKeyName: "catalog_source_credentials_source_id_fkey"+ columns: ["source_id"]+ isOneToOne: true+ referencedRelation: "catalog_sources"+ referencedColumns: ["id"]+ },+ ]+ }catalog_sources: {Row: {+ account: string | nullbrand_id: stringcreated_at: stringcreated_by: string | null@@ -893,6 +930,7 @@ export type Database = {url: string | null}Insert: {+ account?: string | nullbrand_id: stringcreated_at?: stringcreated_by?: string | null@@ -907,6 +945,7 @@ export type Database = {url?: string | null}Update: {+ account?: string | nullbrand_id?: stringcreated_at?: stringcreated_by?: string | null