Skip to content

:hedgehog: PostHog is the leading platform for building self-driving products. Our developer tools – AI observability, analytics, session replay, flags, experiments, error tracking, logs, and more – capture all the context agents need to diagnose problems, uncover opportunities, and ship fixes. Steer it all from Slack, web, desktop, or the MCP.

refactor(hog): page through PostHog lists instead of reading them whole (#130234)

Rafael Audibert7037ce014065
diff --git a/hog/AGENTS.md b/hog/AGENTS.md
index ada22b38c6af..84b72cad9abe 100644
--- a/hog/AGENTS.md
+++ b/hog/AGENTS.md
@@ -8,7 +8,7 @@
| --------------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `src/commands/` | One file per command. `index.ts` is the tree of every command; `legacy.ts` holds the ones posthog-cli still owns |
| `src/core/` | The shared parts every command uses: command definitions, context and global options, output, agent detection, errors, runtime, the manifest |
-| `src/posthog/` | The PostHog API client and credentials |
+| `src/posthog/` | The PostHog API client, the pagination helpers every list uses, and credentials |
| `src/error-tracking/` | Symbol upload: discovery, artifact types under `kinds/`, the container and content hash, releases, git, the upload protocol, and `run.ts`, the inject and upload steps the commands share |
| `src/endpoints/` | The endpoint YAML files, how they compare with PostHog, and the push and pull plans |
| `test/parity/` | Goldens recorded from posthog-cli and the tests that hold hog to them. Read its README before changing anything that reaches an upload |
@@ -27,23 +27,38 @@ Import from another folder with `@/`, which `tsconfig.json` maps to `src/`, as i
export default defineCommand({
description: 'List the feature flags in your project',
args: { search: { description: 'Only show flags whose key contains this text', optional: true } },
- options: { active: { type: 'boolean', description: 'Only show active flags' } },
+ options: {
+ limit: {
+ type: 'string',
+ placeholder: 'count',
+ parse: parseLimit,
+ description: 'How many flags to show, up to 1000',
+ },
+ offset: { type: 'string', placeholder: 'count', parse: parseOffset, description: 'How many flags to skip' },
+ },
scopes: ['feature_flag:read'],
examples: ['hog flags list checkout'],
async run({ args, options, ctx }) {
- const api = await ctx.posthog()
- const flags = await api.get<{ results: Flag[] }>(`feature_flags/?search=${encodeURIComponent(args.search ?? '')}`)
- return options.active ? flags.results.filter((flag) => flag.active) : flags.results
+ const query = new URLSearchParams({
+ search: args.search ?? '',
+ limit: String(options.limit ?? 100),
+ offset: String(options.offset ?? 0),
+ })
+ const page = await getPage(await ctx.posthog(), `feature_flags/?${query}`, Flag)
+ return { flags: page.results, nextOffset: nextOffset(page) }
},
human({ result, out }) {
out.table(
['key', 'active'],
- result.map((flag) => [flag.key, flag.active])
+ result.flags.map((flag) => [flag.key, flag.active])
)
+ if (result.nextOffset !== null) {
+ out.line(`Run ${out.style(`hog flags list --offset ${result.nextOffset}`, 'bold')} for the next page.`)
+ }
},
agent({ result, out }) {
- out.jsonLines(result)
+ out.json(result)
},
})
```
@@ -51,7 +66,8 @@ export default defineCommand({
## Rules
- `run` returns data and never prints. `human` decides what a person at a terminal sees, and `agent` decides what agents, pipes and `-o json` see. Without them, people get indented JSON and agents get one line of JSON.
-- Agent output is JSON only, on stdout, with no color and no prose. Prefer `out.jsonLines` for lists, so an agent can stream them.
+- Agent output is JSON only, on stdout, with no color and no prose. Prefer `out.jsonLines` for rows a command reads in one go, like query results, so an agent can stream them. A paged list prints one JSON object, so it can carry the next offset.
+- A command never reads a whole PostHog list. A list command reads one page per run with `getPage` from `src/posthog/pagination.ts`. It takes `--limit` and the option that picks a page in that endpoint's DRF paginator, such as `--offset`. Its human output names the command for the next page, and its agent output has the next offset as a field. A command that needs particular items from a list uses `findNamed`, which stops when it has them. Only a command whose purpose is the whole list, like `hog endpoints pull --all`, uses `readAll`. Both stop at `MAX_PAGES`.
- `out.table` colors each column by kind (number, date, boolean, JSON) and fits the terminal width. Pass a `kind` when the API declares column types, as `hog query run` does with ClickHouse types. Otherwise the kind comes from the values.
- A failure throws a `HogError` with a stable `code` and a `hint` that names the next step. `cli.ts` prints it as text or as JSON on stderr. Do not catch errors to print them yourself.
- Ask questions only through `ctx.prompts`. It is undefined for agents, pipes and `-o json`, so a command that needs an answer must take it as an option there, or fail with a hint that names the option. Never block an agent on a person: split the work into a call that returns at once and a call that finishes it, like `hog auth login` and `hog auth login --wait`. A command that waits for a person to approve something in a browser uses `runApproval` and `resumeApproval` from `src/core/approval.ts`, so every such command starts, waits and resumes the same way.
diff --git a/hog/MIGRATING.md b/hog/MIGRATING.md
index 9d81c7f489e3..4cff6163d772 100644
--- a/hog/MIGRATING.md
+++ b/hog/MIGRATING.md
@@ -34,31 +34,31 @@ In CI, you can set both sets of names while some jobs still run posthog-cli.
## Commands
-| posthog-cli | hog |
-| -------------------------------------------------- | ---------------------------------------------------------------------------- |
-| `login` | `auth login`, plus `auth status` and `auth logout` |
-| `sourcemap inject --directory <dir>` | `error-tracking inject <dir>` |
-| `sourcemap upload --directory <dir>` | `error-tracking upload <dir>` |
-| `sourcemap process --directory <dir>` | `error-tracking process <dir>` |
-| `hermes inject --directory <dir>` | `error-tracking inject <dir>` |
-| `hermes clone -m <file1> -c <file2>` | `error-tracking clone <file1> <file2>` |
-| `hermes upload --directory <dir>` | `error-tracking upload <dir>` |
-| `dsym upload --directory <dir>` | `error-tracking upload <dir>` |
-| `proguard upload --path <file> --map-id <id>` | `error-tracking upload <file> --map-id <id>` |
-| `symbol-sets upload --directory <dir>` | `error-tracking upload <dir>` |
-| `symbol-sets download --id <id> -o <dir>` | `error-tracking download <id> --output-dir <dir>` |
-| `symbol-sets download --ref <ref>` | `error-tracking download <ref>`, with `--ref` when the ref looks like an ID |
-| `symbol-sets extract <file> -o <dir>` | `error-tracking extract <file> --output-dir <dir>` |
-| `release resolve` | `error-tracking release -o text` |
-| `release resolve --json` | `error-tracking release -o json` |
-| `exp query run <query>` | `query run <query>` |
-| `exp query check <query>` | `query check <query>` |
-| `exp endpoints list`, `get`, `open`, `run`, `diff` | `endpoints list`, `get`, `open`, `run`, `diff` |
-| `exp endpoints pull -o <dir>` | `endpoints pull --to <dir>` |
-| `exp endpoints push` | `endpoints push` |
-| `exp schema pull -o <file>` | `schema pull --file <file>` |
-| `exp schema status` | `schema status` |
-| `api tools`, `search`, `info`, `schema`, `call` | `api tools`, `search`, `info`, `schema`, `call`, with `-o json` for `--json` |
+| posthog-cli | hog |
+| -------------------------------------------------- | ---------------------------------------------------------------------------------------------------- |
+| `login` | `auth login`, plus `auth status` and `auth logout` |
+| `sourcemap inject --directory <dir>` | `error-tracking inject <dir>` |
+| `sourcemap upload --directory <dir>` | `error-tracking upload <dir>` |
+| `sourcemap process --directory <dir>` | `error-tracking process <dir>` |
+| `hermes inject --directory <dir>` | `error-tracking inject <dir>` |
+| `hermes clone -m <file1> -c <file2>` | `error-tracking clone <file1> <file2>` |
+| `hermes upload --directory <dir>` | `error-tracking upload <dir>` |
+| `dsym upload --directory <dir>` | `error-tracking upload <dir>` |
+| `proguard upload --path <file> --map-id <id>` | `error-tracking upload <file> --map-id <id>` |
+| `symbol-sets upload --directory <dir>` | `error-tracking upload <dir>` |
+| `symbol-sets download --id <id> -o <dir>` | `error-tracking download <id> --output-dir <dir>` |
+| `symbol-sets download --ref <ref>` | `error-tracking download <ref>`, with `--ref` when the ref looks like an ID |
+| `symbol-sets extract <file> -o <dir>` | `error-tracking extract <file> --output-dir <dir>` |
+| `release resolve` | `error-tracking release -o text` |
+| `release resolve --json` | `error-tracking release -o json` |
+| `exp query run <query>` | `query run <query>` |
+| `exp query check <query>` | `query check <query>` |
+| `exp endpoints list`, `get`, `open`, `run`, `diff` | `endpoints list`, `get`, `open`, `run`, `diff`. `list` shows one page, and `--offset` picks the next |
+| `exp endpoints pull -o <dir>` | `endpoints pull --to <dir>` |
+| `exp endpoints push` | `endpoints push` |
+| `exp schema pull -o <file>` | `schema pull --file <file>` |
+| `exp schema status` | `schema status` |
+| `api tools`, `search`, `info`, `schema`, `call` | `api tools`, `search`, `info`, `schema`, `call`, with `-o json` for `--json` |
`hog et` is short for `hog error-tracking`.
`hog error-tracking upload` detects what each path holds, so one command uploads JavaScript, Hermes, ProGuard, dSYM and ELF symbols, and `--kind` skips detection.
diff --git a/hog/README.md b/hog/README.md
index dce19fd3e1e2..d00fde251c62 100644
--- a/hog/README.md
+++ b/hog/README.md
@@ -30,7 +30,7 @@ The commands `posthog-cli` keeps under `exp` (`query`, `endpoints`, `logs`, `tas
| `hog error-tracking release` | Finds or creates the release for this build. `-o text` prints just its ID. |
| `hog error-tracking download <id/ref>` | Downloads a symbol set from PostHog and unpacks it. |
| `hog error-tracking extract <file>` | Unpacks a symbol set file. |
-| `hog endpoints list`, `get`, `open` | List your endpoints, show one, or open it in PostHog. |
+| `hog endpoints list`, `get`, `open` | List your endpoints a page at a time, show one, or open it in PostHog. |
| `hog endpoints run [name]` | Runs an endpoint, or the query in a local file with `--file`. `--var name=value` sets variables. |
| `hog endpoints pull`, `diff`, `push` | Keep endpoints as YAML files: pull them, see how they differ, and push changes. `diff` exits with code 1 on drift. |
| `hog schema pull` | Generates a typed SDK (TypeScript, Go or Python) from your event definitions, and records it in `posthog.json`. |
diff --git a/hog/src/commands/endpoints/diff.ts b/hog/src/commands/endpoints/diff.ts
index 9ceeb5cb0943..3fc5cfdcdb69 100644
--- a/hog/src/commands/endpoints/diff.ts
+++ b/hog/src/commands/endpoints/diff.ts
@@ -1,6 +1,7 @@
import { defineCommand } from '@/core/command'
-import { listEndpoints } from '@/endpoints/api'
+import { findEndpoints } from '@/endpoints/api'
import { readEndpointFiles } from '@/endpoints/files'
+import type { Endpoint } from '@/endpoints/model'
import { planPush } from '@/endpoints/plan'
import { pushPlanLines } from './push'
@@ -27,9 +28,10 @@ export default defineCommand({
async run({ args, ctx }) {
const { endpoints, problems } = await readEndpointFiles(ctx.runtime, await endpointPaths(ctx, args.paths))
- const remotes = endpoints.length > 0 ? await listEndpoints(await ctx.posthog()) : []
+ const names = endpoints.map((local) => local.file.name)
+ const remotes = names.length > 0 ? await findEndpoints(await ctx.posthog(), names) : new Map<string, Endpoint>()
// Variable types live on the project, so a diff only compares which variables each query uses.
- const plan = planPush(endpoints, remotes, []).map(({ file: _file, ...item }) => item)
+ const plan = planPush(endpoints, [...remotes.values()], []).map(({ file: _file, ...item }) => item)
return { plan, problems }
},
diff --git a/hog/src/commands/endpoints/endpoints.test.ts b/hog/src/commands/endpoints/endpoints.test.ts
index 27463ef29e7a..89d88360f109 100644
--- a/hog/src/commands/endpoints/endpoints.test.ts
+++ b/hog/src/commands/endpoints/endpoints.test.ts
@@ -1,7 +1,7 @@
import { describe, expect, test } from 'bun:test'
import { z } from 'zod'
-import { readJson, runHog, TEST_CREDENTIALS } from '@/testing'
+import { runHog, TEST_CREDENTIALS } from '@/testing'
import { pushPlanLines } from './push'
@@ -54,7 +54,7 @@ function server(endpoints: unknown[], extra: (url: string, body: unknown) => unk
if (reply) {
return reply as { status: number; body: unknown }
}
- if (url === `${BASE}endpoints/`) {
+ if (url.replace(/\?.*/, '') === `${BASE}endpoints/`) {
return { status: 200, body: { results: endpoints } }
}
if (url === `${BASE}insight_variables/`) {
@@ -83,28 +83,69 @@ describe('hog endpoints', () => {
test.each([
['', BASE],
['under a path', 'https://ph.example.com/posthog/api/environments/7/'],
- ])('list follows pages until PostHog has no more, for a host %s', async (_name, base) => {
- const result = await runHog(['endpoints', 'list'], {
+ ])('list reads one page and gives agents the next offset, for a host %s', async (_name, base) => {
+ const result = await runHog(['endpoints', 'list', '--limit', '1', '--offset', '1'], {
env: { ...AGENT, HOG_HOST: new URL('../../../', base).toString() },
- respond: (url) => {
- if (url === `${base}endpoints/`) {
- return { status: 200, body: { results: [TRENDS], next: `${base}endpoints/?offset=1` } }
- }
- return url === `${base}endpoints/?offset=1`
- ? { status: 200, body: { results: [HOGQL], next: null } }
- : { status: 404, body: {} }
- },
+ respond: (url) =>
+ url === `${base}endpoints/?limit=1&offset=1`
+ ? { status: 200, body: { count: 3, results: [HOGQL], next: `${base}endpoints/?limit=1&offset=2` } }
+ : { status: 404, body: {} },
})
- expect(
- result.stdout
- .trim()
- .split('\n')
- .map((line) => readJson(line, z.object({ name: z.string() })).name)
- ).toEqual(['weekly', 'by-type'])
+
+ expect(result.requests).toHaveLength(1)
+ expect(result.json()).toMatchObject({ endpoints: [{ name: 'by-type' }], count: 3, nextOffset: 2 })
+ })
+
+ test('list tells a person the command for the next page', async () => {
+ const result = await runHog(['endpoints', 'list', '--limit', '1'], {
+ stdoutIsTTY: true,
+ env: TEST_CREDENTIALS,
+ respond: () => ({
+ status: 200,
+ body: { count: 2, results: [TRENDS], next: `${BASE}endpoints/?limit=1&offset=1` },
+ }),
+ })
+
+ expect(result.stdout).toContain('Endpoints 1-1 of 2 in project 7')
+ expect(result.stdout).toContain('Run hog endpoints list --offset 1 --limit 1 for the next page.')
+ })
+
+ test.each([
+ ['stops at the page that holds every local name', false, [`${BASE}endpoints/`]],
+ [
+ 'looks names up one by one in a project too large to read',
+ true,
+ [...Array.from({ length: 20 }, () => `${BASE}endpoints/`), `${BASE}endpoints/weekly/`],
+ ],
+ ])('diff %s', async (_name, endless, expectedUrls) => {
+ const result = await runHog(['endpoints', 'diff', 'weekly.yaml'], {
+ env: AGENT,
+ files: { '/work/weekly.yaml': PULLED_WEEKLY },
+ respond: (url) =>
+ url === `${BASE}endpoints/weekly/`
+ ? { status: 200, body: TRENDS }
+ : {
+ status: 200,
+ body: { results: endless ? [HOGQL] : [TRENDS], next: `${BASE}endpoints/?offset=1` },
+ },
+ })
+
+ expect(result.exitCode).toBe(0)
+ expect(result.requests.map((request) => request.url.replace(/\?.*/, ''))).toEqual(expectedUrls)
+ })
+
+ test('pull --all fails instead of pulling part of a project too large to read', async () => {
+ const result = await runHog(['endpoints', 'pull', '--all', '--yes'], {
+ env: AGENT,
+ respond: () => ({ status: 200, body: { results: [TRENDS], next: `${BASE}endpoints/?offset=1` } }),
+ })
+
+ expect(result.error()).toMatchObject({ code: 'list_too_long' })
+ expect(result.files).toEqual({})
})
- test('list refuses a pagination link to another host, so the token stays on the host', async () => {
- const result = await runHog(['endpoints', 'list'], {
+ test('pull refuses a pagination link to another host, so the token stays on the host', async () => {
+ const result = await runHog(['endpoints', 'pull', '--all', '--dry-run'], {
env: AGENT,
respond: () => ({
status: 200,
@@ -143,6 +184,34 @@ describe('hog endpoints', () => {
expect(result.files['/work/by-type.yaml']).toBe(PULLED_BY_TYPE.replace('type: String', 'type: string'))
})
+ test.each([
+ ['finds a variable on a late page by paging, without a request per variable', 25, 200, 'String', false],
+ ['fetches a variable past the page cap by its ID', null, 200, 'String', true],
+ ['takes the type from the value when the variable was deleted', null, 404, 'string', true],
+ ])('pull %s', async (_name, foundOnPage, detailStatus, type, fetchesById) => {
+ const result = await runHog(['endpoints', 'pull', 'by-type', '--yes'], {
+ env: AGENT,
+ respond: server([HOGQL], (url) => {
+ if (url === `${BASE}insight_variables/var-1/`) {
+ return {
+ status: detailStatus,
+ body: detailStatus === 200 ? VARIABLES[0] : { detail: 'Not found.' },
+ }
+ }
+ if (!url.startsWith(`${BASE}insight_variables/`)) {
+ return undefined
+ }
+ const page = Number(new URL(url).searchParams.get('page') ?? '1')
+ const results = page === foundOnPage ? [VARIABLES[0]] : []
+ return { status: 200, body: { results, next: `${BASE}insight_variables/?page=${page + 1}` } }
+ }),
+ })
+
+ expect(result.exitCode).toBe(0)
+ expect(result.files['/work/by-type.yaml']).toBe(PULLED_BY_TYPE.replace('type: String', `type: ${type}`))
+ expect(result.requests.some((request) => request.url.endsWith('/insight_variables/var-1/'))).toBe(fetchesById)
+ })
+
test.each([
[
'a file that defines another endpoint',
diff --git a/hog/src/commands/endpoints/list.ts b/hog/src/commands/endpoints/list.ts
index 931606158224..f4209f324846 100644
--- a/hog/src/commands/endpoints/list.ts
+++ b/hog/src/commands/endpoints/list.ts
@@ -1,21 +1,48 @@
import { defineCommand } from '@/core/command'
-import { plural } from '@/core/text'
import { listEndpoints } from '@/endpoints/api'
+import { MAX_LIMIT, nextOffset, parseLimit, parseOffset } from '@/posthog/pagination'
import { materializationLabel, SCOPES_READ, summary } from './shared'
+const DEFAULT_LIMIT = 100
+
export default defineCommand({
- description: 'List the endpoints in your project',
+ description: 'List the endpoints in your project, one page at a time',
+ options: {
+ limit: {
+ type: 'string',
+ placeholder: 'count',
+ parse: parseLimit,
+ description: `How many endpoints to show, up to ${MAX_LIMIT}. Defaults to ${DEFAULT_LIMIT}`,
+ },
+ offset: {
+ type: 'string',
+ placeholder: 'count',
+ parse: parseOffset,
+ description: 'How many endpoints to skip. Each page ends with the offset of the next one',
+ },
+ },
scopes: SCOPES_READ,
- examples: ['hog endpoints list'],
+ examples: ['hog endpoints list', 'hog endpoints list --limit 20 --offset 40'],
- async run({ ctx }) {
- const api = await ctx.posthog()
- return { projectId: (await ctx.credentials()).key.projectId, endpoints: await listEndpoints(api) }
+ async run({ options, ctx }) {
+ const offset = options.offset ?? 0
+ const page = await listEndpoints(await ctx.posthog(), options.limit ?? DEFAULT_LIMIT, offset)
+ return {
+ projectId: (await ctx.credentials()).key.projectId,
+ endpoints: page.results,
+ count: page.count ?? null,
+ offset,
+ nextOffset: nextOffset(page),
+ }
},
- human({ result, out }) {
+ human({ result, options, out }) {
if (result.endpoints.length === 0) {
+ if (result.offset > 0) {
+ out.line(`No endpoints after the first ${result.offset} in project ${result.projectId}.`)
+ return
+ }
out.line(`No endpoints in project ${result.projectId} yet.`)
out.line(`Create one from a YAML file with ${out.style('hog endpoints push my-endpoint.yaml', 'bold')}.`)
return
@@ -29,12 +56,19 @@ export default defineCommand({
endpoint.description ?? '',
])
)
- out.line(out.style(`${plural(result.endpoints.length, 'endpoint')} in project ${result.projectId}`, 'dim'))
+ const shown = `${result.offset + 1}-${result.offset + result.endpoints.length}`
+ const total = result.count === null ? '' : ` of ${result.count}`
+ out.line(out.style(`Endpoints ${shown}${total} in project ${result.projectId}`, 'dim'))
+ if (result.nextOffset !== null) {
+ const limit = options.limit === undefined ? '' : ` --limit ${options.limit}`
+ const command = `hog endpoints list --offset ${result.nextOffset}${limit}`
+ out.line(`Run ${out.style(command, 'bold')} for the next page.`)
+ }
},
agent({ result, out }) {
- out.jsonLines(result.endpoints.map(summary))
+ out.json({ endpoints: result.endpoints.map(summary), count: result.count, nextOffset: result.nextOffset })
},
- telemetry: ({ result }) => ({ endpoint_count: result.endpoints.length }),
+ telemetry: ({ result }) => ({ endpoint_count: result.endpoints.length, has_next: result.nextOffset !== null }),
})
diff --git a/hog/src/commands/endpoints/pull.ts b/hog/src/commands/endpoints/pull.ts
index 8530d512e5ef..2e3d2291f011 100644
--- a/hog/src/commands/endpoints/pull.ts
+++ b/hog/src/commands/endpoints/pull.ts
@@ -6,9 +6,9 @@ import { EXIT_USAGE, HogError } from '@/core/errors'
import { Output } from '@/core/output'
import { absolutePath } from '@/core/paths'
import { plural } from '@/core/text'
-import { listEndpoints, listInsightVariables } from '@/endpoints/api'
+import { findEndpoints, findInsightVariables, getInsightVariable, readAllEndpoints } from '@/endpoints/api'
import { readEndpointFiles, type FileProblem } from '@/endpoints/files'
-import { fileFromEndpoint, remoteVariableNames } from '@/endpoints/model'
+import { fileFromEndpoint, remoteVariableIds, type Endpoint } from '@/endpoints/model'
import { planPull, type PullItem } from '@/endpoints/plan'
import { ApiError, type PostHogApi } from '@/posthog/api'
@@ -36,9 +36,22 @@ function pullPlanLines(plan: PlannedItem[], style: Styler): string[] {
// posthog-cli pulls with only endpoint:read. A key that can't read the project's variables gets
// each type from the variable's value instead, as posthog-cli does.
-async function variableTypes(api: PostHogApi): Promise<Map<string, string>> {
+// A variable past the lookup's page cap is fetched by the ID the endpoint's query holds.
+async function variableTypes(api: PostHogApi, ids: ReadonlyMap<string, string>): Promise<Map<string, string>> {
try {
- return new Map((await listInsightVariables(api)).map((variable) => [variable.code_name, variable.type]))
+ const { found, complete } = await findInsightVariables(api, [...ids.keys()])
+ const types = new Map([...found].map(([name, variable]) => [name, variable.type]))
+ if (!complete) {
+ for (const [name, id] of ids) {
+ if (!types.has(name)) {
+ const variable = await getInsightVariable(api, id).catch(missingAsUndefined)
+ if (variable) {
+ types.set(name, variable.type)
+ }
+ }
+ }
+ }
+ return types
} catch (error) {
if (error instanceof ApiError && error.status === 403) {
return new Map()
@@ -47,6 +60,13 @@ async function variableTypes(api: PostHogApi): Promise<Map<string, string>> {
}
}
+function missingAsUndefined(error: unknown): undefined {
+ if (error instanceof ApiError && error.status === 404) {
+ return undefined
+ }
+ throw error
+}
+
export function endpointYaml(path: string, item: PullItem, types: ReadonlyMap<string, string>): string {
const body = stringify(fileFromEndpoint(item.endpoint, types), { lineWidth: 0, blockQuote: 'literal' })
return `# ${basename(path)}\n${body}`
@@ -92,12 +112,14 @@ export default defineCommand({
: ctx.runtime.cwd()
const fileTarget = isYamlFile(target)
- const remotes = await listEndpoints(await ctx.posthog())
+ const api = await ctx.posthog()
const problems: FileProblem[] = []
- let selected = remotes
- if (!options.all) {
- const known = new Set(remotes.map((endpoint) => endpoint.name))
- const missing = args.names.filter((name) => !known.has(name))
+ let selected: Endpoint[]
+ if (options.all) {
+ selected = await readAllEndpoints(api)
+ } else {
+ const found = await findEndpoints(api, args.names)
+ const missing = args.names.filter((name) => !found.has(name))
if (missing.length === args.names.length) {
throw new HogError(
'endpoint_not_found',
@@ -106,7 +128,7 @@ export default defineCommand({
)
}
problems.push(...missing.map((name) => ({ path: name, message: 'PostHog has no endpoint with this name' })))
- selected = remotes.filter((endpoint) => args.names.includes(endpoint.name))
+ selected = [...found.values()]
}
if (fileTarget && selected.length > 1) {
throw new HogError(
@@ -163,9 +185,7 @@ export default defineCommand({
if (!(await confirm(ctx, options, pullPlanLines(plan, style), `Write ${plural(pending.length, 'file')}?`))) {
return { status: 'cancelled', plan, written: [], problems }
}
- const types = pending.some((item) => remoteVariableNames(item.endpoint).length > 0)
- ? await variableTypes(await ctx.posthog())
- : new Map<string, string>()
+ const types = await variableTypes(api, new Map(pending.flatMap((item) => remoteVariableIds(item.endpoint))))
for (const item of pending) {
await ctx.runtime.writeFile(item.path, endpointYaml(item.path, item, types))
}
diff --git a/hog/src/commands/endpoints/push.ts b/hog/src/commands/endpoints/push.ts
index 2616195743be..c24fb247d61c 100644
--- a/hog/src/commands/endpoints/push.ts
+++ b/hog/src/commands/endpoints/push.ts
@@ -1,9 +1,8 @@
import { defineCommand } from '@/core/command'
import { escapeControls, Output } from '@/core/output'
import { plural } from '@/core/text'
-import { listEndpoints, listInsightVariables } from '@/endpoints/api'
+import { findEndpoints, findInsightVariables } from '@/endpoints/api'
import { readEndpointFiles, type FileProblem } from '@/endpoints/files'
-import type { InsightVariable } from '@/endpoints/model'
import { applyPush, planPush, type PushItem, type PushOutcome } from '@/endpoints/plan'
import {
@@ -65,12 +64,16 @@ export default defineCommand({
return { status: 'nothing_to_do', plan: [], outcomes: [], problems }
}
const api = await ctx.posthog()
- const usesVariables = endpoints.some((local) => (local.file.variables ?? []).length > 0)
- const [remotes, variables] = await Promise.all([
- listEndpoints(api),
- usesVariables ? listInsightVariables(api) : Promise.resolve([] as InsightVariable[]),
+ const variableNames = endpoints.flatMap((local) => (local.file.variables ?? []).map(({ name }) => name))
+ const [remotes, known] = await Promise.all([
+ findEndpoints(
+ api,
+ endpoints.map((local) => local.file.name)
+ ),
+ findInsightVariables(api, variableNames),
])
- const items = planPush(endpoints, remotes, variables)
+ const variables = [...known.found.values()]
+ const items = planPush(endpoints, [...remotes.values()], variables)
const plan = items.map(({ file: _file, ...item }) => item)
const pending = plan.filter((item) => item.action !== 'skip')
if (pending.length === 0) {
diff --git a/hog/src/commands/endpoints/run.ts b/hog/src/commands/endpoints/run.ts
index 91ad81bdacb1..8f5a715b3c48 100644
--- a/hog/src/commands/endpoints/run.ts
+++ b/hog/src/commands/endpoints/run.ts
@@ -3,9 +3,10 @@ import { z } from 'zod'
import { defineCommand } from '@/core/command'
import { EXIT_USAGE, HogError } from '@/core/errors'
import { plural } from '@/core/text'
-import { listInsightVariables } from '@/endpoints/api'
+import { findInsightVariables } from '@/endpoints/api'
import { readEndpointFiles } from '@/endpoints/files'
import { ApiError, type PostHogApi } from '@/posthog/api'
+import { listTooLong } from '@/posthog/pagination'
const RunResponse = z.looseObject({
columns: z.array(z.string()).nullish(),
@@ -18,8 +19,11 @@ type RunResponse = z.infer<typeof RunResponse>
// HogQL finds a variable by its project ID, so a file's variables must exist in the project. A
// value left out uses the variable's default.
async function queryVariables(api: PostHogApi, values: Map<string, unknown>): Promise<Record<string, unknown>> {
- const known = new Map((await listInsightVariables(api)).map((variable) => [variable.code_name, variable]))
+ const { found: known, complete } = await findInsightVariables(api, [...values.keys()])
const missing = [...values.keys()].filter((name) => !known.has(name))
+ if (missing.length > 0 && !complete) {
+ throw listTooLong('insight variables', `hog couldn't find ${missing.join(', ')} in the pages it read.`)
+ }
if (missing.length > 0) {
throw new HogError(
'unknown_variable',
diff --git a/hog/src/core/globals.test.ts b/hog/src/core/globals.test.ts
index 935f013b7ea1..5ea812a66f26 100644
--- a/hog/src/core/globals.test.ts
+++ b/hog/src/core/globals.test.ts
@@ -55,7 +55,7 @@ describe('global options', () => {
['an env var', { HOG_RATE_LIMIT: '2' }, {}],
['posthog.json', {}, { '/work/posthog.json': JSON.stringify({ cli: { rateLimit: 2 } }) }],
])('a rate limit from %s holds the third request in a minute', async (_name, env, files) => {
- const result = await runHog(['endpoints', 'list'], {
+ const result = await runHog(['endpoints', 'pull', '--all', '--dry-run'], {
env: { ...AGENT, ...TEST_CREDENTIALS, ...env },
files,
respond: threePages,
@@ -65,7 +65,7 @@ describe('global options', () => {
})
test('a flag beats the env var, which beats posthog.json', async () => {
- const result = await runHog(['endpoints', 'list', '--rate-limit', '100'], {
+ const result = await runHog(['endpoints', 'pull', '--all', '--dry-run', '--rate-limit', '100'], {
env: { ...AGENT, ...TEST_CREDENTIALS, HOG_RATE_LIMIT: '1' },
files: { '/work/posthog.json': JSON.stringify({ cli: { rateLimit: 1 } }) },
respond: threePages,
diff --git a/hog/src/endpoints/api.ts b/hog/src/endpoints/api.ts
index 6a6b93b0dc3c..d49a720b0aaa 100644
--- a/hog/src/endpoints/api.ts
+++ b/hog/src/endpoints/api.ts
@@ -1,37 +1,68 @@
import { z } from 'zod'
-import type { PostHogApi } from '@/posthog/api'
+import { ApiError, type PostHogApi } from '@/posthog/api'
+import { findNamed, getPage, listTooLong, MAX_LIMIT, readAll, type Page } from '@/posthog/pagination'
import { Endpoint, InsightVariable } from './model'
-// Follows `next` links
-async function all<S extends z.ZodType>(api: PostHogApi, path: string, item: S): Promise<z.output<S>[]> {
- const page = z.looseObject({ results: z.array(item), next: z.string().nullish() })
- const results: z.output<S>[] = []
- let next: string | null | undefined = path
- while (next) {
- const current: { results: z.output<S>[]; next?: string | null } = await api.get(next, page)
- results.push(...current.results)
- next = current.next ? new URL(current.next).pathname + new URL(current.next).search : undefined
- }
- return results
+// Lookups can read many pages, so they ask for the largest page PostHog sends.
+const ENDPOINT_PAGES = `endpoints/?limit=${MAX_LIMIT}`
+
+export function listEndpoints(api: PostHogApi, limit: number, offset: number): Promise<Page<Endpoint>> {
+ return getPage(api, `endpoints/?limit=${limit}${offset > 0 ? `&offset=${offset}` : ''}`, Endpoint)
}
-export function listEndpoints(api: PostHogApi): Promise<Endpoint[]> {
- return all(api, 'endpoints/', Endpoint)
+export function readAllEndpoints(api: PostHogApi): Promise<Endpoint[]> {
+ return readAll(api, ENDPOINT_PAGES, Endpoint, listTooLong('endpoints', 'Pull the endpoints you need by name.'))
}
export function getEndpoint(api: PostHogApi, name: string): Promise<Endpoint> {
return api.get(`endpoints/${encodeURIComponent(name)}/`, Endpoint)
}
-// PostHog allows a variable without a code name, which no query can reference, so the list leaves
-// it out.
+// Reads the list until it has every name. In a project too large for that, it looks up each name
+// it hasn't found yet, so the cost grows with the names instead of the project.
+export async function findEndpoints(api: PostHogApi, names: readonly string[]): Promise<Map<string, Endpoint>> {
+ const { found, complete } = await findNamed(api, ENDPOINT_PAGES, Endpoint, names, (endpoint) => endpoint.name)
+ if (!complete) {
+ for (const name of names.filter((name) => !found.has(name))) {
+ try {
+ found.set(name, await getEndpoint(api, name))
+ } catch (error) {
+ if (!(error instanceof ApiError && error.status === 404)) {
+ throw error
+ }
+ }
+ }
+ }
+ return found
+}
+
+// PostHog can't filter variables by name or ID, so a variable lookup pages further.
+const VARIABLE_MAX_PAGES = 200
+
+// PostHog allows a variable without a code name, which no query can reference, so a lookup skips it.
const ListedVariable = InsightVariable.extend({ code_name: z.string().nullable() })
-export async function listInsightVariables(api: PostHogApi): Promise<InsightVariable[]> {
- const variables = await all(api, 'insight_variables/', ListedVariable)
- return variables.flatMap(({ code_name, ...variable }) => (code_name === null ? [] : [{ ...variable, code_name }]))
+// When `complete` is false, a name missing from `found` can still be in the project. A push then
+// tries to create that variable, and PostHog rejects the duplicate for that endpoint alone.
+export async function findInsightVariables(
+ api: PostHogApi,
+ names: readonly string[]
+): Promise<{ found: Map<string, InsightVariable>; complete: boolean }> {
+ const { found, complete } = await findNamed(
+ api,
+ 'insight_variables/',
+ ListedVariable,
+ names,
+ (variable) => variable.code_name,
+ VARIABLE_MAX_PAGES
+ )
+ return { found: new Map([...found].map(([name, variable]) => [name, { ...variable, code_name: name }])), complete }
+}
+
+export function getInsightVariable(api: PostHogApi, id: string): Promise<InsightVariable> {
+ return api.get(`insight_variables/${encodeURIComponent(id)}/`, InsightVariable)
}
export function createInsightVariable(
diff --git a/hog/src/endpoints/model.ts b/hog/src/endpoints/model.ts
index 95b81d0c67f3..175d4a198e79 100644
--- a/hog/src/endpoints/model.ts
+++ b/hog/src/endpoints/model.ts
@@ -156,13 +156,18 @@ export function isPlainHogql(query: Record<string, unknown>): boolean {
}
export function remoteVariableNames(endpoint: Endpoint | undefined): string[] {
+ return remoteVariableIds(endpoint).map(([name]) => name)
+}
+
+// Each variable the query uses, as [code name, project variable ID]. The query keys variables by ID.
+export function remoteVariableIds(endpoint: Endpoint | undefined): [string, string][] {
const variables = endpoint?.query.variables
if (!variables || typeof variables !== 'object') {
return []
}
- return Object.values(variables as Record<string, { code_name?: unknown }>)
- .map((variable) => variable.code_name)
- .filter((name): name is string => typeof name === 'string')
+ return Object.entries(variables as Record<string, { code_name?: unknown }>).flatMap(([id, variable]) =>
+ typeof variable.code_name === 'string' ? [[variable.code_name, id] as [string, string]] : []
+ )
}
function inferType(value: unknown): string {
diff --git a/hog/src/posthog/pagination.ts b/hog/src/posthog/pagination.ts
new file mode 100644
index 000000000000..d7d971a81d0c
--- /dev/null
+++ b/hog/src/posthog/pagination.ts
@@ -0,0 +1,127 @@
+import { z } from 'zod'
+
+import { HogError } from '@/core/errors'
+import { parseOrThrow, type ParseFailure } from '@/core/schema'
+
+import type { PostHogApi } from './api'
+
+// A command that reads page after page stops here, so it never reads the whole of a large list.
+export const MAX_PAGES = 20
+
+export interface Page<T> {
+ results: T[]
+ count?: number
+ next?: URL
+}
+
+export async function getPage<S extends z.ZodType>(api: PostHogApi, path: string, item: S): Promise<Page<z.output<S>>> {
+ const body = await api.get(
+ path,
+ z.object({ results: z.array(item), count: z.number().nullish(), next: z.string().nullish() })
+ )
+ return {
+ results: body.results,
+ ...(typeof body.count === 'number' ? { count: body.count } : {}),
+ ...(body.next ? { next: new URL(body.next) } : {}),
+ }
+}
+
+// False when it stops at MAX_PAGES first. Keeping only a link's path and query sends every page to
+// the configured host, whatever host the link names.
+async function eachPage<S extends z.ZodType>(
+ api: PostHogApi,
+ path: string,
+ item: S,
+ visit: (results: z.output<S>[]) => boolean,
+ maxPages = MAX_PAGES
+): Promise<boolean> {
+ let next: string | undefined = path
+ for (let read = 0; next !== undefined; read++) {
+ if (read === maxPages) {
+ return false
+ }
+ const page: Page<z.output<S>> = await getPage(api, next, item)
+ if (visit(page.results)) {
+ return true
+ }
+ next = page.next ? page.next.pathname + page.next.search : undefined
+ }
+ return true
+}
+
+export function listTooLong(noun: string, hint: string): HogError {
+ return new HogError('list_too_long', `The project has more ${noun} than hog reads in one command.`, hint)
+}
+
+export async function readAll<S extends z.ZodType>(
+ api: PostHogApi,
+ path: string,
+ item: S,
+ tooLong: HogError
+): Promise<z.output<S>[]> {
+ const results: z.output<S>[] = []
+ const complete = await eachPage(api, path, item, (page) => {
+ results.push(...page)
+ return false
+ })
+ if (!complete) {
+ throw tooLong
+ }
+ return results
+}
+
+// Reads pages until every name has turned up or the list ends. When `complete` is false, hog
+// stopped at `maxPages` and a name missing from `found` can still be in the list.
+export async function findNamed<S extends z.ZodType>(
+ api: PostHogApi,
+ path: string,
+ item: S,
+ names: Iterable<string>,
+ nameOf: (item: z.output<S>) => string | null,
+ maxPages = MAX_PAGES
+): Promise<{ found: Map<string, z.output<S>>; complete: boolean }> {
+ const wanted = new Set(names)
+ const found = new Map<string, z.output<S>>()
+ if (wanted.size === 0) {
+ return { found, complete: true }
+ }
+ const complete = await eachPage(
+ api,
+ path,
+ item,
+ (page) => {
+ for (const result of page) {
+ const name = nameOf(result)
+ if (name !== null && wanted.has(name)) {
+ found.set(name, result)
+ }
+ }
+ return found.size === wanted.size
+ },
+ maxPages
+ )
+ return { found, complete }
+}
+
+function wholeNumber(min: number, max?: number): z.ZodType<number, string> {
+ const error = `It must be a whole number ${max === undefined ? `of ${min} or more` : `from ${min} to ${max}`}.`
+ const number = z.int({ error }).min(min, { error })
+ return z
+ .string()
+ .regex(/^\d+$/, { error })
+ .transform(Number)
+ .pipe(max === undefined ? number : number.max(max, { error }))
+}
+
+// PostHog answers at most 1000 rows a page from a limit/offset list (BoundedLimitOffsetPagination).
+export const MAX_LIMIT = 1000
+const LimitSchema = wholeNumber(1, MAX_LIMIT)
+const OffsetSchema = wholeNumber(0)
+
+export const parseLimit = (raw: string, failure: ParseFailure): number => parseOrThrow(LimitSchema, raw, failure)
+export const parseOffset = (raw: string, failure: ParseFailure): number => parseOrThrow(OffsetSchema, raw, failure)
+
+export function nextOffset(page: Page<unknown>): number | null {
+ const offset = Number(page.next?.searchParams.get('offset') ?? Number.NaN)
+ return Number.isSafeInteger(offset) ? offset : null
+}