Open-source agentic AI Search Execution Platform. Track AI visibility, find opportunities, and let agents turn signals into prioritized actions and drafted fixes across ChatGPT, Claude, Gemini, Google AI Overviews, Google AI Mode, Perplexity, Grok, and Microsoft Copilot.
Building a catalog connector
Ansvisor keeps each brand's product catalog in one shape: products, variants, images and categories. A connector reads one kind of source into it. There are two kinds of connector:
- A feed: a public address read without a sign-in. The product feed URL connector (
server/src/connectors/feed-url/) reads Google product feeds, Meta catalog CSVs and store CSV exports. - A store or marketplace: read with the brand's own sign-in, for example WooCommerce, Walmart, Trendyol or Hepsiburada.
This guide is for writing a store connector. You can build and test one with no production keys and no real account.
How a connector fits
A connector is one folder under server/src/connectors/<kind>/, exporting one object. You register it in server/src/connectors/index.js (CONNECTORS). Everything else is already there:
- Settings › Integrations › Product catalog shows a "Connect {your store}" button and a form built from your
fields. You write no UI. - Connecting:
- Your
connectchecks what the person typed with the store. - The API makes the source and keeps the sign-in encrypted.
- It then reads the catalog.
- Your
- Syncing:
- The daily run reads every store again after 20 hours, and admins can press Sync now.
- Each run's outcome is shown on the source: synced, couldn't sync, or sign-in refused.
- The catalog:
- Whatever your
readCatalogreturns is written to the catalog. - A re-sync changes only what changed.
- A product that's gone from the store is marked removed.
- Whatever your
- Removing: removing the store deletes its sign-in and its products.
You write no migration, no route and no UI.
The interface
| Member | What it is |
|---|---|
kind |
The source's kind, a short slug: woocommerce, trendyol. Lowercase letters, digits and _, 2 to 40 characters. |
label |
The platform's name as people know it: WooCommerce. |
fields |
What the person types to connect, in order: { name, label, secret?, placeholder?, help? }. A secret field is typed like a password and never shown again. Use help to say where in the store to find it, and which access to give it. |
connect(input, ctx) |
Checks input (the fields' values, as strings) with the store. Returns { account, url?, label?, credentials } (see below). |
readCatalog(source, ctx) |
Reads the store's products. Returns { items, errors, format }, where format is your kind. |
disconnect(source, ctx) |
Optional. Revokes the sign-in at the store when the person removes it, where the platform allows. It's best effort: the source is removed anyway. |
What connect returns:
accountnames the store to the person, such as the shop's domain or the seller id. A brand can connect each account once.urlis the store's address, when it has one.labelis the store's own name, when it gives one.credentialsis an object of strings, kept encrypted forreadCatalog.
What ctx holds:
ctx.requestis the HTTP client, and the only way to call the platform.- In
readCataloganddisconnect,ctx.credentialsis whatconnectreturned ascredentials. sourcehasid,kind,url,accountandformat.
What readCatalog returns
Build each item with toItem(attributes, n, 'Item') from server/src/lib/feeds/item.js.
- Attributes: use Google product feed attribute names:
id,item_group_id,title,description,link,image_link,additional_image_link,brand,gtin,mpn,price,sale_price,availability,color,size,material,product_type. - Required: an item needs
id,title,link,price(for example"129.00 USD") andavailability. An item without them comes back as an error, which you add toerrors. The person sees the first five. - One item per variant. Put the product's own id in
item_group_id, and its own name intitle. The variant's color and size go incolorandsize.
The example connector (server/src/connectors/__example__/index.js) shows all of this in about 100 lines.
Calling the platform
Call the platform only through ctx.request(url, { method, headers, query, body }) (server/src/connectors/http.js). It returns { status, headers, data }, with the JSON body in data. It:
- reaches only public https addresses, on the standard web ports, and doesn't follow redirects. Only the test harness lets it call its fake API over http;
- tries again on 429. A call that changes nothing (GET, HEAD, PUT, DELETE) is also tried again on 502, 503 and 504 and on dropped connections; a POST isn't, since the store may have taken it. It waits as long as the store's
Retry-Afterheader says, or 1, 2 then 4 seconds. A wait longer than a minute ends the run, and the next daily run tries again; - stops every call of one sync after 10 minutes, so one slow store can't hold the daily run;
- never puts the query, the headers or the body in an error message.
Send keys in a header (Authorization) where the platform allows it, not in the address.
For paging, nextLink(headers) reads a Link: <…>; rel="next" header. Cap the pages you read, as the example does (MAX_PAGES).
Errors
| Throw | When | The person sees |
|---|---|---|
ConnectError (./errors.js) |
In connect, when what they typed is wrong: "That isn't a WooCommerce site's address." |
Your words. Never repeat a secret in them. |
Errors from ctx.request (let them through) |
401 or 403 | "Sign-in refused", and a prompt to update it. |
Errors from ctx.request (let them through) |
Anything else | The store couldn't be reached, asked to wait, or answered an error. |
FeedFormatError (server/src/lib/feeds/item.js) |
The store's data can't be read at all. | Your message. |
Anything else is logged, with every credential value taken out of its message, and shown as "couldn't be synced".
Credentials
- Where they're kept:
- The API encrypts the sign-in with AES-256-GCM, under a key only it holds (
CONNECTOR_CREDENTIALS_KEY). - It's bound to its source: copied onto another source's row, it doesn't open.
- It's kept in
catalog_source_credentials, which only the service role can read. Members can't even see that a row exists.
- The API encrypts the sign-in with AES-256-GCM, under a key only it holds (
- What to ask for: the least access the feature needs. Today that's read-only access to products (and, later, orders). Say which permission to tick in your field's
help. - Self-hosting: set
CONNECTOR_CREDENTIALS_KEYinserver/.env(openssl rand -base64 32). Without it, Settings says store connections aren't set up. If the key changes, saved sign-ins can't be opened: those stores show "Sign-in refused" until they're updated.
A platform read with the workspace's Google sign-in
Some platforms sign in with OAuth through a provider Composio already handles for the workspace, like Search Console and Analytics. Google Merchant Center (#1270, server/src/connectors/merchant-center/) is one. Such a connector differs from a store:
- No
fieldsand noconnect. It hassignIn, the Composio provider (server/src/lib/composio.jsPROVIDERS). The workspace connects that provider once in Settings › Integrations, and Composio keeps the token: it never reaches us, and the source has no row incatalog_source_credentials. - Picking the account:
listAccounts(ctx)returns the accounts the sign-in opens, as{ id, name }.- The person picks one.
POST /api/catalog/sourceswithinput: { account }then callscheckAccount(value, ctx), which returns{ account, label?, url? }or throws a ConnectError.
- Calls:
ctx.request(url, { query, signal })goes through Composio's proxy (server/src/lib/catalog/sign-in.js), to the hosts your connector names inhostsonly (Merchant Center's ismerchantapi.googleapis.com). It answers like the HTTP client and throws the sameConnectorHttpErrorcodes, and one more:auth: the sign-in is gone or refused. The source shows "Reconnect it in Settings › Integrations".not_set_up: the server's own setup is refused, not the sign-in: Composio's API key, or Google's 403 (the Google Cloud project not registered with Merchant Center, a scope missing). Reconnecting wouldn't help, so the source says the server isn't set up correctly. Both are logged as warnings, with Google's reason, for whoever runs the server.- Listing accounts and checking one stop after 30 seconds in all; a sync's calls after 10 minutes.
- Issues:
readCatalogmay also returnissues, the platform's problems per item. The plumbing replaces the source's rows incatalog_item_issueswith them (00296) and counts them on the source (last_counts.issues). - The same item listed again: Merchant Center lists an offer once per language and feed label. The catalog keeps one (a listing that isn't disapproved first, then by feed label and language), keeps the issues of all of them, and counts the others in
repeated(last_counts.repeated), not as skipped. - Testing: pass
runSyncasignInRequeststand-in.merchant-center.test.jsbuilds one from the realsignInRequestwith a fake proxy.
Testing
Connector tests run in npm test (in server/), against a fake of the platform's API, with no network and no account. The harness is server/src/connectors/testing.js:
startFakeApi(routes):- It serves your routes,
{ 'GET /api/products': (req) => ({ status, headers, json }) }, ashttp://store.test:<port>. - It gives you
requestto pass asctx.request: the real HTTP client, with its waits recorded inwaitsinstead of slept. - Every call is recorded in
calls.
- It serves your routes,
runSync(connector, source, { credentials, request })syncs through the real plumbing into an in-memory catalog. It returns the outcome and the rows written (products,variants,images,categories).
Start from the example's test (server/src/connectors/__example__/example.test.js):
import { afterEach, beforeEach, expect, it, vi } from 'vitest';
// The plumbing reaches the Supabase admin client at import time.
vi.mock('../../config/supabase.js', () => ({ default: {} }));
import { runSync, startFakeApi } from '../testing.js';
import woocommerce from './index.js';
let api;
beforeEach(async () => {
api = await startFakeApi({
'GET /wp-json/wc/v3/products': () => ({ json: [/* products, from the platform's docs */] }),
});
});
afterEach(() => api.close());
it('reads every product into the catalog', async () => {
const { outcome, products } = await runSync(woocommerce, { url: api.origin }, {
credentials: { consumerKey: 'ck_test', consumerSecret: 'cs_test' },
request: api.request,
});
expect(outcome.status).toBe('ok');
});
Cover at least these:
- connecting, with a right and a wrong sign-in;
- paging;
- a 429;
- a product that can't be kept;
- one variant per item.
Fixtures are the platform's answers as its docs show them, or as your own sandbox returns them, with made-up products. Never a real store's data, a real customer or a real key.
Review checklist
Copy this into your pull request. Every connector is reviewed against it.
- Least access: it asks only for what the feature needs (read-only products), and
helpsays which permission to give. - Credentials live only in the credential store. None is in code, logs, error messages, fixtures, screenshots or the PR. Keys go in headers, not addresses, where the platform allows.
- No buyer or customer personal data is read or kept. Orders, when they come, keep only the id, date, line items, revenue and channel.
- Removal:
disconnectrevokes the sign-in where the platform allows. The platform's data deletion requests are handled where it sends them. - Rate limits: every call goes through
ctx.request, pages are capped, and the platform's documented limits are respected. - The platform's data rules are respected (for example, restricted data on marketplaces).
- Tests run with the harness, with no network and no account. Contributors build against their own sandbox or test accounts. Production keys live only with the core team.
- Words: errors a person sees say what to do, in plain words.
Not here yet
- Orders and webhooks (
syncOrders(since),handleWebhook(event)) come with the first store that needs them (Shopify, #1265). - OAuth sign-in: platforms that sign in through a redirect (Etsy, eBay, Amazon) need a callback the plumbing doesn't have yet, unless Composio handles their sign-in, as it does Google's (see above). Open an issue before you start one.
- API keys: stores that sign in with keys (WooCommerce, Walmart, Trendyol, Hepsiburada) work today.