:hedgehog: PostHog is the leading platform for building self-driving products. Our developer tools – AI observability, analytics, session replay, flags, experiments, error tracking, logs, and more – capture all the context agents need to diagnose problems, uncover opportunities, and ship fixes. Steer it all from Slack, web, desktop, or the MCP.
Python ab-testingai-analyticsanalyticscdpdata-warehouseexperimentsfeature-flagsjavascriptproduct-analyticspythonreactsession-replaysurveystypescriptweb-analytics
chore(docs): record three rejected ci runner and sharding experiments (#129314)
Georges-Antoine Assied05e94a4ae3
diff --git a/docs/internal/ci-things-already-tried.md b/docs/internal/ci-things-already-tried.mdindex ad25393baec1..198c04b06443 100644--- a/docs/internal/ci-things-already-tried.md+++ b/docs/internal/ci-things-already-tried.md@@ -137,6 +137,46 @@ A shard retry also repeats the 7.5 minutes of setup. Thus a retry of one shard i_Also asked as:_ split the E2E tests across runners, parallelize Playwright, reduce flaky retry scope by sharding+### Run the Storybook WebKit shards on 4-CPU runners++**Verdict: rejected** · Nov 2026 · [#128516](https://github.com/PostHog/posthog/pull/128516), [run](https://github.com/PostHog/posthog/actions/runs/41694856025)++The 8 WebKit shards run on `depot-ubuntu-24.04` (2 CPUs) with 2 Jest workers. Depot metrics show the test runner step at 84% mean CPU and above 90% for a third of the time.+On `depot-ubuntu-24.04-4` the test runner step took 228s median, against about 362s on 2 CPUs.++Each shard also has about 190s of setup, and more CPUs do not make the setup faster.+Thus the slowest shard finished only about 50s sooner (477s against about 526s), and the 4-CPU runner costs twice as much per minute. WebKit cost increased by about 75%.+WebKit runs only on master pushes and in the merge queue, so 50s gives little value. Fewer shards on 4 CPUs are slower than today, because each shard then has more tests.++The 2-CPU shards peaked at 86% memory. If WebKit starts to fail with out-of-memory errors, more memory is the reason to try a larger runner again.++_Also asked as:_ bigger runners for WebKit, WebKit visual regression is slow, give Storybook more CPU++### Run the Node.js serial tests in 2 duration-balanced shards++**Verdict: rejected** · Nov 2026 · [#128491](https://github.com/PostHog/posthog/pull/128491), [2-shard run](https://github.com/PostHog/posthog/actions/runs/41702931651), [3-shard run](https://github.com/PostHog/posthog/actions/runs/41718453769)++The serial lane runs about 270s of Jest in `--runInBand`, and each shard has about 230s of setup. A sequencer that splits the files by measured duration (`nodejs/jest.serial.sequencer.js`) can make 2 shards almost equal.+The proposal was to use 2 shards and remove one runner from each run.++With 2 shards the Jest steps took 153s and 186s. The serial lane then finished about 30s later than the old worst hash shard, and it was the last Node.js job.+The JUnit times in the duration map do not include the start-up cost of each test file, so the measured shards were 15–40% heavier than the map predicted.+With 3 shards the Jest steps took 119s, 106s and 126s, and the serial lane was no longer the slowest job.++_Also asked as:_ fewer Node.js serial shards, merge the serial shards, save a runner on the Node.js tests++### Run the Rust ingestion consumer e2e job on 2-CPU runners++**Verdict: rejected** · Nov 2026 · [#129158](https://github.com/PostHog/posthog/pull/129158), [run](https://github.com/PostHog/posthog/actions/runs/41834630979)++Depot metrics showed the job at 21% mean CPU on `depot-ubuntu-24.04-4`, so a 2-CPU runner looked sufficient.+On `depot-ubuntu-24.04` the job took 675s, against a 358s median. The low average hides the setup steps, which use all the CPUs: `pnpm install` took 123s against about 16s, the Rust cache restore 47s against about 9s, and the personhog pre-build 47s against about 19s.+The job then was the slowest Node.js job by about 250s.++Average CPU does not show which steps are CPU-bound. Examine the time of each setup step before you move a job to a smaller runner.++_Also asked as:_ smaller runner for the Rust e2e job, downsize the ingestion e2e runner, cheaper Node.js CI+### Use Bazel to scope product tests**Verdict: abandoned** · Dec 2025 to Mar 2026 · [#43397](https://github.com/PostHog/posthog/pull/43397)