:hedgehog: PostHog is the leading platform for building self-driving products. Our developer tools – AI observability, analytics, session replay, flags, experiments, error tracking, logs, and more – capture all the context agents need to diagnose problems, uncover opportunities, and ship fixes. Steer it all from Slack, web, desktop, or the MCP.
Python ab-testingai-analyticsanalyticscdpdata-warehouseexperimentsfeature-flagsjavascriptproduct-analyticspythonreactsession-replaysurveystypescriptweb-analytics
refactor(hog): page through PostHog lists instead of reading them whole (#130234)
Rafael Audibert7037ce014065
diff --git a/hog/AGENTS.md b/hog/AGENTS.mdindex ada22b38c6af..84b72cad9abe 100644--- a/hog/AGENTS.md+++ b/hog/AGENTS.md@@ -8,7 +8,7 @@| --------------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- || `src/commands/` | One file per command. `index.ts` is the tree of every command; `legacy.ts` holds the ones posthog-cli still owns || `src/core/` | The shared parts every command uses: command definitions, context and global options, output, agent detection, errors, runtime, the manifest |-| `src/posthog/` | The PostHog API client and credentials |+| `src/posthog/` | The PostHog API client, the pagination helpers every list uses, and credentials || `src/error-tracking/` | Symbol upload: discovery, artifact types under `kinds/`, the container and content hash, releases, git, the upload protocol, and `run.ts`, the inject and upload steps the commands share || `src/endpoints/` | The endpoint YAML files, how they compare with PostHog, and the push and pull plans || `test/parity/` | Goldens recorded from posthog-cli and the tests that hold hog to them. Read its README before changing anything that reaches an upload |@@ -27,23 +27,38 @@ Import from another folder with `@/`, which `tsconfig.json` maps to `src/`, as iexport default defineCommand({description: 'List the feature flags in your project',args: { search: { description: 'Only show flags whose key contains this text', optional: true } },- options: { active: { type: 'boolean', description: 'Only show active flags' } },+ options: {+ limit: {+ type: 'string',+ placeholder: 'count',+ parse: parseLimit,+ description: 'How many flags to show, up to 1000',+ },+ offset: { type: 'string', placeholder: 'count', parse: parseOffset, description: 'How many flags to skip' },+ },scopes: ['feature_flag:read'],examples: ['hog flags list checkout'],async run({ args, options, ctx }) {- const api = await ctx.posthog()- const flags = await api.get<{ results: Flag[] }>(`feature_flags/?search=${encodeURIComponent(args.search ?? '')}`)- return options.active ? flags.results.filter((flag) => flag.active) : flags.results+ const query = new URLSearchParams({+ search: args.search ?? '',+ limit: String(options.limit ?? 100),+ offset: String(options.offset ?? 0),+ })+ const page = await getPage(await ctx.posthog(), `feature_flags/?${query}`, Flag)+ return { flags: page.results, nextOffset: nextOffset(page) }},human({ result, out }) {out.table(['key', 'active'],- result.map((flag) => [flag.key, flag.active])+ result.flags.map((flag) => [flag.key, flag.active]))+ if (result.nextOffset !== null) {+ out.line(`Run ${out.style(`hog flags list --offset ${result.nextOffset}`, 'bold')} for the next page.`)+ }},agent({ result, out }) {- out.jsonLines(result)+ out.json(result)},})```@@ -51,7 +66,8 @@ export default defineCommand({## Rules- `run` returns data and never prints. `human` decides what a person at a terminal sees, and `agent` decides what agents, pipes and `-o json` see. Without them, people get indented JSON and agents get one line of JSON.-- Agent output is JSON only, on stdout, with no color and no prose. Prefer `out.jsonLines` for lists, so an agent can stream them.+- Agent output is JSON only, on stdout, with no color and no prose. Prefer `out.jsonLines` for rows a command reads in one go, like query results, so an agent can stream them. A paged list prints one JSON object, so it can carry the next offset.+- A command never reads a whole PostHog list. A list command reads one page per run with `getPage` from `src/posthog/pagination.ts`. It takes `--limit` and the option that picks a page in that endpoint's DRF paginator, such as `--offset`. Its human output names the command for the next page, and its agent output has the next offset as a field. A command that needs particular items from a list uses `findNamed`, which stops when it has them. Only a command whose purpose is the whole list, like `hog endpoints pull --all`, uses `readAll`. Both stop at `MAX_PAGES`.- `out.table` colors each column by kind (number, date, boolean, JSON) and fits the terminal width. Pass a `kind` when the API declares column types, as `hog query run` does with ClickHouse types. Otherwise the kind comes from the values.- A failure throws a `HogError` with a stable `code` and a `hint` that names the next step. `cli.ts` prints it as text or as JSON on stderr. Do not catch errors to print them yourself.- Ask questions only through `ctx.prompts`. It is undefined for agents, pipes and `-o json`, so a command that needs an answer must take it as an option there, or fail with a hint that names the option. Never block an agent on a person: split the work into a call that returns at once and a call that finishes it, like `hog auth login` and `hog auth login --wait`. A command that waits for a person to approve something in a browser uses `runApproval` and `resumeApproval` from `src/core/approval.ts`, so every such command starts, waits and resumes the same way.diff --git a/hog/MIGRATING.md b/hog/MIGRATING.mdindex 9d81c7f489e3..4cff6163d772 100644--- a/hog/MIGRATING.md+++ b/hog/MIGRATING.md@@ -34,31 +34,31 @@ In CI, you can set both sets of names while some jobs still run posthog-cli.## Commands-| posthog-cli | hog |-| -------------------------------------------------- | ---------------------------------------------------------------------------- |-| `login` | `auth login`, plus `auth status` and `auth logout` |-| `sourcemap inject --directory <dir>` | `error-tracking inject <dir>` |-| `sourcemap upload --directory <dir>` | `error-tracking upload <dir>` |-| `sourcemap process --directory <dir>` | `error-tracking process <dir>` |-| `hermes inject --directory <dir>` | `error-tracking inject <dir>` |-| `hermes clone -m <file1> -c <file2>` | `error-tracking clone <file1> <file2>` |-| `hermes upload --directory <dir>` | `error-tracking upload <dir>` |-| `dsym upload --directory <dir>` | `error-tracking upload <dir>` |-| `proguard upload --path <file> --map-id <id>` | `error-tracking upload <file> --map-id <id>` |-| `symbol-sets upload --directory <dir>` | `error-tracking upload <dir>` |-| `symbol-sets download --id <id> -o <dir>` | `error-tracking download <id> --output-dir <dir>` |-| `symbol-sets download --ref <ref>` | `error-tracking download <ref>`, with `--ref` when the ref looks like an ID |-| `symbol-sets extract <file> -o <dir>` | `error-tracking extract <file> --output-dir <dir>` |-| `release resolve` | `error-tracking release -o text` |-| `release resolve --json` | `error-tracking release -o json` |-| `exp query run <query>` | `query run <query>` |-| `exp query check <query>` | `query check <query>` |-| `exp endpoints list`, `get`, `open`, `run`, `diff` | `endpoints list`, `get`, `open`, `run`, `diff` |-| `exp endpoints pull -o <dir>` | `endpoints pull --to <dir>` |-| `exp endpoints push` | `endpoints push` |-| `exp schema pull -o <file>` | `schema pull --file <file>` |-| `exp schema status` | `schema status` |-| `api tools`, `search`, `info`, `schema`, `call` | `api tools`, `search`, `info`, `schema`, `call`, with `-o json` for `--json` |+| posthog-cli | hog |+| -------------------------------------------------- | ---------------------------------------------------------------------------------------------------- |+| `login` | `auth login`, plus `auth status` and `auth logout` |+| `sourcemap inject --directory <dir>` | `error-tracking inject <dir>` |+| `sourcemap upload --directory <dir>` | `error-tracking upload <dir>` |+| `sourcemap process --directory <dir>` | `error-tracking process <dir>` |+| `hermes inject --directory <dir>` | `error-tracking inject <dir>` |+| `hermes clone -m <file1> -c <file2>` | `error-tracking clone <file1> <file2>` |+| `hermes upload --directory <dir>` | `error-tracking upload <dir>` |+| `dsym upload --directory <dir>` | `error-tracking upload <dir>` |+| `proguard upload --path <file> --map-id <id>` | `error-tracking upload <file> --map-id <id>` |+| `symbol-sets upload --directory <dir>` | `error-tracking upload <dir>` |+| `symbol-sets download --id <id> -o <dir>` | `error-tracking download <id> --output-dir <dir>` |+| `symbol-sets download --ref <ref>` | `error-tracking download <ref>`, with `--ref` when the ref looks like an ID |+| `symbol-sets extract <file> -o <dir>` | `error-tracking extract <file> --output-dir <dir>` |+| `release resolve` | `error-tracking release -o text` |+| `release resolve --json` | `error-tracking release -o json` |+| `exp query run <query>` | `query run <query>` |+| `exp query check <query>` | `query check <query>` |+| `exp endpoints list`, `get`, `open`, `run`, `diff` | `endpoints list`, `get`, `open`, `run`, `diff`. `list` shows one page, and `--offset` picks the next |+| `exp endpoints pull -o <dir>` | `endpoints pull --to <dir>` |+| `exp endpoints push` | `endpoints push` |+| `exp schema pull -o <file>` | `schema pull --file <file>` |+| `exp schema status` | `schema status` |+| `api tools`, `search`, `info`, `schema`, `call` | `api tools`, `search`, `info`, `schema`, `call`, with `-o json` for `--json` |`hog et` is short for `hog error-tracking`.`hog error-tracking upload` detects what each path holds, so one command uploads JavaScript, Hermes, ProGuard, dSYM and ELF symbols, and `--kind` skips detection.diff --git a/hog/README.md b/hog/README.mdindex dce19fd3e1e2..d00fde251c62 100644--- a/hog/README.md+++ b/hog/README.md@@ -30,7 +30,7 @@ The commands `posthog-cli` keeps under `exp` (`query`, `endpoints`, `logs`, `tas| `hog error-tracking release` | Finds or creates the release for this build. `-o text` prints just its ID. || `hog error-tracking download <id/ref>` | Downloads a symbol set from PostHog and unpacks it. || `hog error-tracking extract <file>` | Unpacks a symbol set file. |-| `hog endpoints list`, `get`, `open` | List your endpoints, show one, or open it in PostHog. |+| `hog endpoints list`, `get`, `open` | List your endpoints a page at a time, show one, or open it in PostHog. || `hog endpoints run [name]` | Runs an endpoint, or the query in a local file with `--file`. `--var name=value` sets variables. || `hog endpoints pull`, `diff`, `push` | Keep endpoints as YAML files: pull them, see how they differ, and push changes. `diff` exits with code 1 on drift. || `hog schema pull` | Generates a typed SDK (TypeScript, Go or Python) from your event definitions, and records it in `posthog.json`. |diff --git a/hog/src/commands/endpoints/diff.ts b/hog/src/commands/endpoints/diff.tsindex 9ceeb5cb0943..3fc5cfdcdb69 100644--- a/hog/src/commands/endpoints/diff.ts+++ b/hog/src/commands/endpoints/diff.ts@@ -1,6 +1,7 @@import { defineCommand } from '@/core/command'-import { listEndpoints } from '@/endpoints/api'+import { findEndpoints } from '@/endpoints/api'import { readEndpointFiles } from '@/endpoints/files'+import type { Endpoint } from '@/endpoints/model'import { planPush } from '@/endpoints/plan'import { pushPlanLines } from './push'@@ -27,9 +28,10 @@ export default defineCommand({async run({ args, ctx }) {const { endpoints, problems } = await readEndpointFiles(ctx.runtime, await endpointPaths(ctx, args.paths))- const remotes = endpoints.length > 0 ? await listEndpoints(await ctx.posthog()) : []+ const names = endpoints.map((local) => local.file.name)+ const remotes = names.length > 0 ? await findEndpoints(await ctx.posthog(), names) : new Map<string, Endpoint>()// Variable types live on the project, so a diff only compares which variables each query uses.- const plan = planPush(endpoints, remotes, []).map(({ file: _file, ...item }) => item)+ const plan = planPush(endpoints, [...remotes.values()], []).map(({ file: _file, ...item }) => item)return { plan, problems }},diff --git a/hog/src/commands/endpoints/endpoints.test.ts b/hog/src/commands/endpoints/endpoints.test.tsindex 27463ef29e7a..89d88360f109 100644--- a/hog/src/commands/endpoints/endpoints.test.ts+++ b/hog/src/commands/endpoints/endpoints.test.ts@@ -1,7 +1,7 @@import { describe, expect, test } from 'bun:test'import { z } from 'zod'-import { readJson, runHog, TEST_CREDENTIALS } from '@/testing'+import { runHog, TEST_CREDENTIALS } from '@/testing'import { pushPlanLines } from './push'@@ -54,7 +54,7 @@ function server(endpoints: unknown[], extra: (url: string, body: unknown) => unkif (reply) {return reply as { status: number; body: unknown }}- if (url === `${BASE}endpoints/`) {+ if (url.replace(/\?.*/, '') === `${BASE}endpoints/`) {return { status: 200, body: { results: endpoints } }}if (url === `${BASE}insight_variables/`) {@@ -83,28 +83,69 @@ describe('hog endpoints', () => {test.each([['', BASE],['under a path', 'https://ph.example.com/posthog/api/environments/7/'],- ])('list follows pages until PostHog has no more, for a host %s', async (_name, base) => {- const result = await runHog(['endpoints', 'list'], {+ ])('list reads one page and gives agents the next offset, for a host %s', async (_name, base) => {+ const result = await runHog(['endpoints', 'list', '--limit', '1', '--offset', '1'], {env: { ...AGENT, HOG_HOST: new URL('../../../', base).toString() },- respond: (url) => {- if (url === `${base}endpoints/`) {- return { status: 200, body: { results: [TRENDS], next: `${base}endpoints/?offset=1` } }- }- return url === `${base}endpoints/?offset=1`- ? { status: 200, body: { results: [HOGQL], next: null } }- : { status: 404, body: {} }- },+ respond: (url) =>+ url === `${base}endpoints/?limit=1&offset=1`+ ? { status: 200, body: { count: 3, results: [HOGQL], next: `${base}endpoints/?limit=1&offset=2` } }+ : { status: 404, body: {} },})- expect(- result.stdout- .trim()- .split('\n')- .map((line) => readJson(line, z.object({ name: z.string() })).name)- ).toEqual(['weekly', 'by-type'])++ expect(result.requests).toHaveLength(1)+ expect(result.json()).toMatchObject({ endpoints: [{ name: 'by-type' }], count: 3, nextOffset: 2 })+ })++ test('list tells a person the command for the next page', async () => {+ const result = await runHog(['endpoints', 'list', '--limit', '1'], {+ stdoutIsTTY: true,+ env: TEST_CREDENTIALS,+ respond: () => ({+ status: 200,+ body: { count: 2, results: [TRENDS], next: `${BASE}endpoints/?limit=1&offset=1` },+ }),+ })++ expect(result.stdout).toContain('Endpoints 1-1 of 2 in project 7')+ expect(result.stdout).toContain('Run hog endpoints list --offset 1 --limit 1 for the next page.')+ })++ test.each([+ ['stops at the page that holds every local name', false, [`${BASE}endpoints/`]],+ [+ 'looks names up one by one in a project too large to read',+ true,+ [...Array.from({ length: 20 }, () => `${BASE}endpoints/`), `${BASE}endpoints/weekly/`],+ ],+ ])('diff %s', async (_name, endless, expectedUrls) => {+ const result = await runHog(['endpoints', 'diff', 'weekly.yaml'], {+ env: AGENT,+ files: { '/work/weekly.yaml': PULLED_WEEKLY },+ respond: (url) =>+ url === `${BASE}endpoints/weekly/`+ ? { status: 200, body: TRENDS }+ : {+ status: 200,+ body: { results: endless ? [HOGQL] : [TRENDS], next: `${BASE}endpoints/?offset=1` },+ },+ })++ expect(result.exitCode).toBe(0)+ expect(result.requests.map((request) => request.url.replace(/\?.*/, ''))).toEqual(expectedUrls)+ })++ test('pull --all fails instead of pulling part of a project too large to read', async () => {+ const result = await runHog(['endpoints', 'pull', '--all', '--yes'], {+ env: AGENT,+ respond: () => ({ status: 200, body: { results: [TRENDS], next: `${BASE}endpoints/?offset=1` } }),+ })++ expect(result.error()).toMatchObject({ code: 'list_too_long' })+ expect(result.files).toEqual({})})- test('list refuses a pagination link to another host, so the token stays on the host', async () => {- const result = await runHog(['endpoints', 'list'], {+ test('pull refuses a pagination link to another host, so the token stays on the host', async () => {+ const result = await runHog(['endpoints', 'pull', '--all', '--dry-run'], {env: AGENT,respond: () => ({status: 200,@@ -143,6 +184,34 @@ describe('hog endpoints', () => {expect(result.files['/work/by-type.yaml']).toBe(PULLED_BY_TYPE.replace('type: String', 'type: string'))})+ test.each([+ ['finds a variable on a late page by paging, without a request per variable', 25, 200, 'String', false],+ ['fetches a variable past the page cap by its ID', null, 200, 'String', true],+ ['takes the type from the value when the variable was deleted', null, 404, 'string', true],+ ])('pull %s', async (_name, foundOnPage, detailStatus, type, fetchesById) => {+ const result = await runHog(['endpoints', 'pull', 'by-type', '--yes'], {+ env: AGENT,+ respond: server([HOGQL], (url) => {+ if (url === `${BASE}insight_variables/var-1/`) {+ return {+ status: detailStatus,+ body: detailStatus === 200 ? VARIABLES[0] : { detail: 'Not found.' },+ }+ }+ if (!url.startsWith(`${BASE}insight_variables/`)) {+ return undefined+ }+ const page = Number(new URL(url).searchParams.get('page') ?? '1')+ const results = page === foundOnPage ? [VARIABLES[0]] : []+ return { status: 200, body: { results, next: `${BASE}insight_variables/?page=${page + 1}` } }+ }),+ })++ expect(result.exitCode).toBe(0)+ expect(result.files['/work/by-type.yaml']).toBe(PULLED_BY_TYPE.replace('type: String', `type: ${type}`))+ expect(result.requests.some((request) => request.url.endsWith('/insight_variables/var-1/'))).toBe(fetchesById)+ })+test.each([['a file that defines another endpoint',diff --git a/hog/src/commands/endpoints/list.ts b/hog/src/commands/endpoints/list.tsindex 931606158224..f4209f324846 100644--- a/hog/src/commands/endpoints/list.ts+++ b/hog/src/commands/endpoints/list.ts@@ -1,21 +1,48 @@import { defineCommand } from '@/core/command'-import { plural } from '@/core/text'import { listEndpoints } from '@/endpoints/api'+import { MAX_LIMIT, nextOffset, parseLimit, parseOffset } from '@/posthog/pagination'import { materializationLabel, SCOPES_READ, summary } from './shared'+const DEFAULT_LIMIT = 100+export default defineCommand({- description: 'List the endpoints in your project',+ description: 'List the endpoints in your project, one page at a time',+ options: {+ limit: {+ type: 'string',+ placeholder: 'count',+ parse: parseLimit,+ description: `How many endpoints to show, up to ${MAX_LIMIT}. Defaults to ${DEFAULT_LIMIT}`,+ },+ offset: {+ type: 'string',+ placeholder: 'count',+ parse: parseOffset,+ description: 'How many endpoints to skip. Each page ends with the offset of the next one',+ },+ },scopes: SCOPES_READ,- examples: ['hog endpoints list'],+ examples: ['hog endpoints list', 'hog endpoints list --limit 20 --offset 40'],- async run({ ctx }) {- const api = await ctx.posthog()- return { projectId: (await ctx.credentials()).key.projectId, endpoints: await listEndpoints(api) }+ async run({ options, ctx }) {+ const offset = options.offset ?? 0+ const page = await listEndpoints(await ctx.posthog(), options.limit ?? DEFAULT_LIMIT, offset)+ return {+ projectId: (await ctx.credentials()).key.projectId,+ endpoints: page.results,+ count: page.count ?? null,+ offset,+ nextOffset: nextOffset(page),+ }},- human({ result, out }) {+ human({ result, options, out }) {if (result.endpoints.length === 0) {+ if (result.offset > 0) {+ out.line(`No endpoints after the first ${result.offset} in project ${result.projectId}.`)+ return+ }out.line(`No endpoints in project ${result.projectId} yet.`)out.line(`Create one from a YAML file with ${out.style('hog endpoints push my-endpoint.yaml', 'bold')}.`)return@@ -29,12 +56,19 @@ export default defineCommand({endpoint.description ?? '',]))- out.line(out.style(`${plural(result.endpoints.length, 'endpoint')} in project ${result.projectId}`, 'dim'))+ const shown = `${result.offset + 1}-${result.offset + result.endpoints.length}`+ const total = result.count === null ? '' : ` of ${result.count}`+ out.line(out.style(`Endpoints ${shown}${total} in project ${result.projectId}`, 'dim'))+ if (result.nextOffset !== null) {+ const limit = options.limit === undefined ? '' : ` --limit ${options.limit}`+ const command = `hog endpoints list --offset ${result.nextOffset}${limit}`+ out.line(`Run ${out.style(command, 'bold')} for the next page.`)+ }},agent({ result, out }) {- out.jsonLines(result.endpoints.map(summary))+ out.json({ endpoints: result.endpoints.map(summary), count: result.count, nextOffset: result.nextOffset })},- telemetry: ({ result }) => ({ endpoint_count: result.endpoints.length }),+ telemetry: ({ result }) => ({ endpoint_count: result.endpoints.length, has_next: result.nextOffset !== null }),})diff --git a/hog/src/commands/endpoints/pull.ts b/hog/src/commands/endpoints/pull.tsindex 8530d512e5ef..2e3d2291f011 100644--- a/hog/src/commands/endpoints/pull.ts+++ b/hog/src/commands/endpoints/pull.ts@@ -6,9 +6,9 @@ import { EXIT_USAGE, HogError } from '@/core/errors'import { Output } from '@/core/output'import { absolutePath } from '@/core/paths'import { plural } from '@/core/text'-import { listEndpoints, listInsightVariables } from '@/endpoints/api'+import { findEndpoints, findInsightVariables, getInsightVariable, readAllEndpoints } from '@/endpoints/api'import { readEndpointFiles, type FileProblem } from '@/endpoints/files'-import { fileFromEndpoint, remoteVariableNames } from '@/endpoints/model'+import { fileFromEndpoint, remoteVariableIds, type Endpoint } from '@/endpoints/model'import { planPull, type PullItem } from '@/endpoints/plan'import { ApiError, type PostHogApi } from '@/posthog/api'@@ -36,9 +36,22 @@ function pullPlanLines(plan: PlannedItem[], style: Styler): string[] {// posthog-cli pulls with only endpoint:read. A key that can't read the project's variables gets// each type from the variable's value instead, as posthog-cli does.-async function variableTypes(api: PostHogApi): Promise<Map<string, string>> {+// A variable past the lookup's page cap is fetched by the ID the endpoint's query holds.+async function variableTypes(api: PostHogApi, ids: ReadonlyMap<string, string>): Promise<Map<string, string>> {try {- return new Map((await listInsightVariables(api)).map((variable) => [variable.code_name, variable.type]))+ const { found, complete } = await findInsightVariables(api, [...ids.keys()])+ const types = new Map([...found].map(([name, variable]) => [name, variable.type]))+ if (!complete) {+ for (const [name, id] of ids) {+ if (!types.has(name)) {+ const variable = await getInsightVariable(api, id).catch(missingAsUndefined)+ if (variable) {+ types.set(name, variable.type)+ }+ }+ }+ }+ return types} catch (error) {if (error instanceof ApiError && error.status === 403) {return new Map()@@ -47,6 +60,13 @@ async function variableTypes(api: PostHogApi): Promise<Map<string, string>> {}}+function missingAsUndefined(error: unknown): undefined {+ if (error instanceof ApiError && error.status === 404) {+ return undefined+ }+ throw error+}+export function endpointYaml(path: string, item: PullItem, types: ReadonlyMap<string, string>): string {const body = stringify(fileFromEndpoint(item.endpoint, types), { lineWidth: 0, blockQuote: 'literal' })return `# ${basename(path)}\n${body}`@@ -92,12 +112,14 @@ export default defineCommand({: ctx.runtime.cwd()const fileTarget = isYamlFile(target)- const remotes = await listEndpoints(await ctx.posthog())+ const api = await ctx.posthog()const problems: FileProblem[] = []- let selected = remotes- if (!options.all) {- const known = new Set(remotes.map((endpoint) => endpoint.name))- const missing = args.names.filter((name) => !known.has(name))+ let selected: Endpoint[]+ if (options.all) {+ selected = await readAllEndpoints(api)+ } else {+ const found = await findEndpoints(api, args.names)+ const missing = args.names.filter((name) => !found.has(name))if (missing.length === args.names.length) {throw new HogError('endpoint_not_found',@@ -106,7 +128,7 @@ export default defineCommand({)}problems.push(...missing.map((name) => ({ path: name, message: 'PostHog has no endpoint with this name' })))- selected = remotes.filter((endpoint) => args.names.includes(endpoint.name))+ selected = [...found.values()]}if (fileTarget && selected.length > 1) {throw new HogError(@@ -163,9 +185,7 @@ export default defineCommand({if (!(await confirm(ctx, options, pullPlanLines(plan, style), `Write ${plural(pending.length, 'file')}?`))) {return { status: 'cancelled', plan, written: [], problems }}- const types = pending.some((item) => remoteVariableNames(item.endpoint).length > 0)- ? await variableTypes(await ctx.posthog())- : new Map<string, string>()+ const types = await variableTypes(api, new Map(pending.flatMap((item) => remoteVariableIds(item.endpoint))))for (const item of pending) {await ctx.runtime.writeFile(item.path, endpointYaml(item.path, item, types))}diff --git a/hog/src/commands/endpoints/push.ts b/hog/src/commands/endpoints/push.tsindex 2616195743be..c24fb247d61c 100644--- a/hog/src/commands/endpoints/push.ts+++ b/hog/src/commands/endpoints/push.ts@@ -1,9 +1,8 @@import { defineCommand } from '@/core/command'import { escapeControls, Output } from '@/core/output'import { plural } from '@/core/text'-import { listEndpoints, listInsightVariables } from '@/endpoints/api'+import { findEndpoints, findInsightVariables } from '@/endpoints/api'import { readEndpointFiles, type FileProblem } from '@/endpoints/files'-import type { InsightVariable } from '@/endpoints/model'import { applyPush, planPush, type PushItem, type PushOutcome } from '@/endpoints/plan'import {@@ -65,12 +64,16 @@ export default defineCommand({return { status: 'nothing_to_do', plan: [], outcomes: [], problems }}const api = await ctx.posthog()- const usesVariables = endpoints.some((local) => (local.file.variables ?? []).length > 0)- const [remotes, variables] = await Promise.all([- listEndpoints(api),- usesVariables ? listInsightVariables(api) : Promise.resolve([] as InsightVariable[]),+ const variableNames = endpoints.flatMap((local) => (local.file.variables ?? []).map(({ name }) => name))+ const [remotes, known] = await Promise.all([+ findEndpoints(+ api,+ endpoints.map((local) => local.file.name)+ ),+ findInsightVariables(api, variableNames),])- const items = planPush(endpoints, remotes, variables)+ const variables = [...known.found.values()]+ const items = planPush(endpoints, [...remotes.values()], variables)const plan = items.map(({ file: _file, ...item }) => item)const pending = plan.filter((item) => item.action !== 'skip')if (pending.length === 0) {diff --git a/hog/src/commands/endpoints/run.ts b/hog/src/commands/endpoints/run.tsindex 91ad81bdacb1..8f5a715b3c48 100644--- a/hog/src/commands/endpoints/run.ts+++ b/hog/src/commands/endpoints/run.ts@@ -3,9 +3,10 @@ import { z } from 'zod'import { defineCommand } from '@/core/command'import { EXIT_USAGE, HogError } from '@/core/errors'import { plural } from '@/core/text'-import { listInsightVariables } from '@/endpoints/api'+import { findInsightVariables } from '@/endpoints/api'import { readEndpointFiles } from '@/endpoints/files'import { ApiError, type PostHogApi } from '@/posthog/api'+import { listTooLong } from '@/posthog/pagination'const RunResponse = z.looseObject({columns: z.array(z.string()).nullish(),@@ -18,8 +19,11 @@ type RunResponse = z.infer<typeof RunResponse>// HogQL finds a variable by its project ID, so a file's variables must exist in the project. A// value left out uses the variable's default.async function queryVariables(api: PostHogApi, values: Map<string, unknown>): Promise<Record<string, unknown>> {- const known = new Map((await listInsightVariables(api)).map((variable) => [variable.code_name, variable]))+ const { found: known, complete } = await findInsightVariables(api, [...values.keys()])const missing = [...values.keys()].filter((name) => !known.has(name))+ if (missing.length > 0 && !complete) {+ throw listTooLong('insight variables', `hog couldn't find ${missing.join(', ')} in the pages it read.`)+ }if (missing.length > 0) {throw new HogError('unknown_variable',diff --git a/hog/src/core/globals.test.ts b/hog/src/core/globals.test.tsindex 935f013b7ea1..5ea812a66f26 100644--- a/hog/src/core/globals.test.ts+++ b/hog/src/core/globals.test.ts@@ -55,7 +55,7 @@ describe('global options', () => {['an env var', { HOG_RATE_LIMIT: '2' }, {}],['posthog.json', {}, { '/work/posthog.json': JSON.stringify({ cli: { rateLimit: 2 } }) }],])('a rate limit from %s holds the third request in a minute', async (_name, env, files) => {- const result = await runHog(['endpoints', 'list'], {+ const result = await runHog(['endpoints', 'pull', '--all', '--dry-run'], {env: { ...AGENT, ...TEST_CREDENTIALS, ...env },files,respond: threePages,@@ -65,7 +65,7 @@ describe('global options', () => {})test('a flag beats the env var, which beats posthog.json', async () => {- const result = await runHog(['endpoints', 'list', '--rate-limit', '100'], {+ const result = await runHog(['endpoints', 'pull', '--all', '--dry-run', '--rate-limit', '100'], {env: { ...AGENT, ...TEST_CREDENTIALS, HOG_RATE_LIMIT: '1' },files: { '/work/posthog.json': JSON.stringify({ cli: { rateLimit: 1 } }) },respond: threePages,diff --git a/hog/src/endpoints/api.ts b/hog/src/endpoints/api.tsindex 6a6b93b0dc3c..d49a720b0aaa 100644--- a/hog/src/endpoints/api.ts+++ b/hog/src/endpoints/api.ts@@ -1,37 +1,68 @@import { z } from 'zod'-import type { PostHogApi } from '@/posthog/api'+import { ApiError, type PostHogApi } from '@/posthog/api'+import { findNamed, getPage, listTooLong, MAX_LIMIT, readAll, type Page } from '@/posthog/pagination'import { Endpoint, InsightVariable } from './model'-// Follows `next` links-async function all<S extends z.ZodType>(api: PostHogApi, path: string, item: S): Promise<z.output<S>[]> {- const page = z.looseObject({ results: z.array(item), next: z.string().nullish() })- const results: z.output<S>[] = []- let next: string | null | undefined = path- while (next) {- const current: { results: z.output<S>[]; next?: string | null } = await api.get(next, page)- results.push(...current.results)- next = current.next ? new URL(current.next).pathname + new URL(current.next).search : undefined- }- return results+// Lookups can read many pages, so they ask for the largest page PostHog sends.+const ENDPOINT_PAGES = `endpoints/?limit=${MAX_LIMIT}`++export function listEndpoints(api: PostHogApi, limit: number, offset: number): Promise<Page<Endpoint>> {+ return getPage(api, `endpoints/?limit=${limit}${offset > 0 ? `&offset=${offset}` : ''}`, Endpoint)}-export function listEndpoints(api: PostHogApi): Promise<Endpoint[]> {- return all(api, 'endpoints/', Endpoint)+export function readAllEndpoints(api: PostHogApi): Promise<Endpoint[]> {+ return readAll(api, ENDPOINT_PAGES, Endpoint, listTooLong('endpoints', 'Pull the endpoints you need by name.'))}export function getEndpoint(api: PostHogApi, name: string): Promise<Endpoint> {return api.get(`endpoints/${encodeURIComponent(name)}/`, Endpoint)}-// PostHog allows a variable without a code name, which no query can reference, so the list leaves-// it out.+// Reads the list until it has every name. In a project too large for that, it looks up each name+// it hasn't found yet, so the cost grows with the names instead of the project.+export async function findEndpoints(api: PostHogApi, names: readonly string[]): Promise<Map<string, Endpoint>> {+ const { found, complete } = await findNamed(api, ENDPOINT_PAGES, Endpoint, names, (endpoint) => endpoint.name)+ if (!complete) {+ for (const name of names.filter((name) => !found.has(name))) {+ try {+ found.set(name, await getEndpoint(api, name))+ } catch (error) {+ if (!(error instanceof ApiError && error.status === 404)) {+ throw error+ }+ }+ }+ }+ return found+}++// PostHog can't filter variables by name or ID, so a variable lookup pages further.+const VARIABLE_MAX_PAGES = 200++// PostHog allows a variable without a code name, which no query can reference, so a lookup skips it.const ListedVariable = InsightVariable.extend({ code_name: z.string().nullable() })-export async function listInsightVariables(api: PostHogApi): Promise<InsightVariable[]> {- const variables = await all(api, 'insight_variables/', ListedVariable)- return variables.flatMap(({ code_name, ...variable }) => (code_name === null ? [] : [{ ...variable, code_name }]))+// When `complete` is false, a name missing from `found` can still be in the project. A push then+// tries to create that variable, and PostHog rejects the duplicate for that endpoint alone.+export async function findInsightVariables(+ api: PostHogApi,+ names: readonly string[]+): Promise<{ found: Map<string, InsightVariable>; complete: boolean }> {+ const { found, complete } = await findNamed(+ api,+ 'insight_variables/',+ ListedVariable,+ names,+ (variable) => variable.code_name,+ VARIABLE_MAX_PAGES+ )+ return { found: new Map([...found].map(([name, variable]) => [name, { ...variable, code_name: name }])), complete }+}++export function getInsightVariable(api: PostHogApi, id: string): Promise<InsightVariable> {+ return api.get(`insight_variables/${encodeURIComponent(id)}/`, InsightVariable)}export function createInsightVariable(diff --git a/hog/src/endpoints/model.ts b/hog/src/endpoints/model.tsindex 95b81d0c67f3..175d4a198e79 100644--- a/hog/src/endpoints/model.ts+++ b/hog/src/endpoints/model.ts@@ -156,13 +156,18 @@ export function isPlainHogql(query: Record<string, unknown>): boolean {}export function remoteVariableNames(endpoint: Endpoint | undefined): string[] {+ return remoteVariableIds(endpoint).map(([name]) => name)+}++// Each variable the query uses, as [code name, project variable ID]. The query keys variables by ID.+export function remoteVariableIds(endpoint: Endpoint | undefined): [string, string][] {const variables = endpoint?.query.variablesif (!variables || typeof variables !== 'object') {return []}- return Object.values(variables as Record<string, { code_name?: unknown }>)- .map((variable) => variable.code_name)- .filter((name): name is string => typeof name === 'string')+ return Object.entries(variables as Record<string, { code_name?: unknown }>).flatMap(([id, variable]) =>+ typeof variable.code_name === 'string' ? [[variable.code_name, id] as [string, string]] : []+ )}function inferType(value: unknown): string {diff --git a/hog/src/posthog/pagination.ts b/hog/src/posthog/pagination.tsnew file mode 100644index 000000000000..d7d971a81d0c--- /dev/null+++ b/hog/src/posthog/pagination.ts@@ -0,0 +1,127 @@+import { z } from 'zod'++import { HogError } from '@/core/errors'+import { parseOrThrow, type ParseFailure } from '@/core/schema'++import type { PostHogApi } from './api'++// A command that reads page after page stops here, so it never reads the whole of a large list.+export const MAX_PAGES = 20++export interface Page<T> {+ results: T[]+ count?: number+ next?: URL+}++export async function getPage<S extends z.ZodType>(api: PostHogApi, path: string, item: S): Promise<Page<z.output<S>>> {+ const body = await api.get(+ path,+ z.object({ results: z.array(item), count: z.number().nullish(), next: z.string().nullish() })+ )+ return {+ results: body.results,+ ...(typeof body.count === 'number' ? { count: body.count } : {}),+ ...(body.next ? { next: new URL(body.next) } : {}),+ }+}++// False when it stops at MAX_PAGES first. Keeping only a link's path and query sends every page to+// the configured host, whatever host the link names.+async function eachPage<S extends z.ZodType>(+ api: PostHogApi,+ path: string,+ item: S,+ visit: (results: z.output<S>[]) => boolean,+ maxPages = MAX_PAGES+): Promise<boolean> {+ let next: string | undefined = path+ for (let read = 0; next !== undefined; read++) {+ if (read === maxPages) {+ return false+ }+ const page: Page<z.output<S>> = await getPage(api, next, item)+ if (visit(page.results)) {+ return true+ }+ next = page.next ? page.next.pathname + page.next.search : undefined+ }+ return true+}++export function listTooLong(noun: string, hint: string): HogError {+ return new HogError('list_too_long', `The project has more ${noun} than hog reads in one command.`, hint)+}++export async function readAll<S extends z.ZodType>(+ api: PostHogApi,+ path: string,+ item: S,+ tooLong: HogError+): Promise<z.output<S>[]> {+ const results: z.output<S>[] = []+ const complete = await eachPage(api, path, item, (page) => {+ results.push(...page)+ return false+ })+ if (!complete) {+ throw tooLong+ }+ return results+}++// Reads pages until every name has turned up or the list ends. When `complete` is false, hog+// stopped at `maxPages` and a name missing from `found` can still be in the list.+export async function findNamed<S extends z.ZodType>(+ api: PostHogApi,+ path: string,+ item: S,+ names: Iterable<string>,+ nameOf: (item: z.output<S>) => string | null,+ maxPages = MAX_PAGES+): Promise<{ found: Map<string, z.output<S>>; complete: boolean }> {+ const wanted = new Set(names)+ const found = new Map<string, z.output<S>>()+ if (wanted.size === 0) {+ return { found, complete: true }+ }+ const complete = await eachPage(+ api,+ path,+ item,+ (page) => {+ for (const result of page) {+ const name = nameOf(result)+ if (name !== null && wanted.has(name)) {+ found.set(name, result)+ }+ }+ return found.size === wanted.size+ },+ maxPages+ )+ return { found, complete }+}++function wholeNumber(min: number, max?: number): z.ZodType<number, string> {+ const error = `It must be a whole number ${max === undefined ? `of ${min} or more` : `from ${min} to ${max}`}.`+ const number = z.int({ error }).min(min, { error })+ return z+ .string()+ .regex(/^\d+$/, { error })+ .transform(Number)+ .pipe(max === undefined ? number : number.max(max, { error }))+}++// PostHog answers at most 1000 rows a page from a limit/offset list (BoundedLimitOffsetPagination).+export const MAX_LIMIT = 1000+const LimitSchema = wholeNumber(1, MAX_LIMIT)+const OffsetSchema = wholeNumber(0)++export const parseLimit = (raw: string, failure: ParseFailure): number => parseOrThrow(LimitSchema, raw, failure)+export const parseOffset = (raw: string, failure: ParseFailure): number => parseOrThrow(OffsetSchema, raw, failure)++export function nextOffset(page: Page<unknown>): number | null {+ const offset = Number(page.next?.searchParams.get('offset') ?? Number.NaN)+ return Number.isSafeInteger(offset) ? offset : null+}