:hedgehog: PostHog is the leading platform for building self-driving products. Our developer tools – AI observability, analytics, session replay, flags, experiments, error tracking, logs, and more – capture all the context agents need to diagnose problems, uncover opportunities, and ship fixes. Steer it all from Slack, web, desktop, or the MCP.
CI: things already tried
This file answers one question. You have an idea for CI or the dev environment. Did someone try it before?
Most of the ideas here are good ideas. People tried them because they sounded correct. This file records the part that costs the most to find again. It records what happened when someone built the idea, and why the result did not agree with the proposal.
How to use this file
Search before you build.
rg -i "xdist|parallel" docs/internal/ci-things-already-tried.md
Each entry has the title of the proposal. The title uses the words that a person uses to propose the idea. The title does not use the symptom that appeared later. Each entry ends with Also asked as. This line gives other words for the same idea, so that a different search finds the entry.
A verdict is not a prohibition. Each entry gives the date and the specific reason for the result. Read the reason. Then examine if the reason is still correct. Runner sizes, prices, and tools change. If the reason is no longer correct, write this in the PR and try the idea again.
Verdicts
| Verdict | Meaning |
|---|---|
rejected |
Someone built the idea and measured it. The result was too bad. |
reverted |
The change went to master. Then someone removed it. |
superseded |
The problem was real. A different solution replaced this one. |
abandoned |
Someone started the work and stopped. There is no verdict. |
open |
The idea is good. The work is incomplete. You can continue it. |
landed |
The change is on master. An earlier attempt or alternative is still a trap. |
Add an entry
Add an entry when you close a PR and do not merge it. Add an entry when you revert a change. Give the entry the title of the idea. Then give the verdict, the date, and the measurement. An entry of five lines has more value than a design document that nobody opens.
Remove an entry
Age alone is not a reason to remove an entry. The pytest-xdist entry is the oldest here, and people still propose that idea.
Remove an entry when one of these is true:
The system that it describes is gone. A person cannot propose the idea any more, so the verdict guides nobody.
The idea shipped later. The entry is history, not prior art. Keep it only when the first failure is still a trap.
The entry gives a general lesson and no specific trap. "Measure before you optimize" does not need an entry.
Give the reason when you remove an entry. Do not remove an entry because it looks old.
Test parallelism and sharding
Run pytest-xdist inside the backend CI shards
Verdict: rejected · Oct 2025 · #38927
The test used 53 shards and -n 4.
Wall time decreased from approximately 15 minutes to approximately 9 minutes. The PR reports the difference as statistically strong.
CPU cost increased from 1,572 to 3,908 core-minutes. This is a factor of approximately 2.5.
The speed increase is real. The cost is the problem. A factor of 2.5 in compute is too much for 6 minutes of wall time.
pytest-xdist is still a development dependency, and it operates correctly on a local machine. CI does not use it in the shards.
Re-measured Sep 2026 · #93810
The 2025 test moved the runners from 2 cores to 8 at the same time. The 2.5x compute came from the larger runners, and not from xdist.
Two workers on the current 2-core runner was never measured. It measures -21.9% of wall time over ten Core shards, and the runner size does not change, so the cost falls with the wall time. A shard uses about half of its runner, because the test phase waits on the service stack.
That PR did not merge either. Product databases are never created for a worker.
posthog/conftest.py points each product alias at test_posthog_gwN_<product>, and nothing creates that database.
A single-worker run hides this, because both naming schemes then produce the same string.
Fixing it means provisioning the test databases before pytest starts.
Four other single-process assumptions surfaced first, and each one was deterministic rather than flaky. Read the PR before you start again.
Also asked as: parallelize tests within a shard, -n auto, use the idle cores on the runner, why is each shard single-process
Change the django_db_setup fixture from package scope to session scope
Verdict: rejected · Apr 2026 · #57030
Package scope builds the test database one time for each package directory. Session scope builds it one time for the whole run, which looks strictly faster.
The PR did not merge. posthog/conftest.py still declares @pytest.fixture(scope="package").
Read #57227 with this one. It makes the cost of django_db_setup visible in the pytest output. Today that cost hides in the setup phase of the first test that pytest collects, and makes that test look slow for no reason.
Measure the setup cost first. Then you know what a scope change can win.
Also asked as: session-scoped database fixture, build the test database once, why is the first test so slow
Collapse the warehouse-sources parametrized tests
Verdict: rejected · Sep 2026 · measured, not built (PR run, master run)
warehouse-sources collects about 58,000 tests. About 56,000 of them take less than 20 ms, and pytest-split puts almost all of them in one shard.
Their cost was not the count. They took 89 s on master and 202 s on PR runs, because turbo dropped COVERAGE_CORE from the product jobs and coverage used its slow tracer.
Four functions that run once per source give 5,392 tests and 15 s. Collapsing them saves little.
The larger fixed cost is collection: each shard of the product collects all 58,000 tests, which takes about 90 s, before pytest-split selects its group.
Also asked as: too many parametrized tests, trim the warehouse-sources suite, delete per-source tests, why is the last warehouse-sources shard slow
Lower the backend shard wall target from 12 to 10 minutes
Verdict: rejected · Sep 2026 · one run each: 12 minutes, 10 minutes
TARGET_WALL_SECONDS in turbo-discover.js sizes every backend test shard.
At 10 minutes a full PR run used 23% more shard minutes, in 85 test jobs instead of 58.
It did not finish sooner (13.4 minutes against 12.9, inside run-to-run noise). Each extra shard pays the full setup and collection cost again.
Also asked as: more shards, smaller shards, lower the shard target, split the slowest shard
Shard the Playwright E2E suite
Verdict: reverted · Feb 2026 · #46774, reverted by #46853
The change added four shards. The purpose was a smaller retry set for unreliable tests.
The setup cost of each shard is the item that the proposal did not include. Each shard needs approximately 7.5 minutes of setup. The migrations alone need 3 minutes.
All 110 tests complete in approximately 4 minutes with 6 workers. Thus four shards used approximately 22 more minutes of CPU in each run, and decreased wall time by approximately 3 minutes.
A shard retry also repeats the 7.5 minutes of setup. Thus a retry of one shard is not much faster than a retry of all the tests.
Also asked as: split the E2E tests across runners, parallelize Playwright, reduce flaky retry scope by sharding
Run the Storybook WebKit shards on 4-CPU runners
Verdict: rejected · Nov 2026 · #128516, run
The 8 WebKit shards run on depot-ubuntu-24.04 (2 CPUs) with 2 Jest workers. Depot metrics show the test runner step at 84% mean CPU and above 90% for a third of the time.
On depot-ubuntu-24.04-4 the test runner step took 228s median, against about 362s on 2 CPUs.
Each shard also has about 190s of setup, and more CPUs do not make the setup faster. Thus the slowest shard finished only about 50s sooner (477s against about 526s), and the 4-CPU runner costs twice as much per minute. WebKit cost increased by about 75%. WebKit runs only on master pushes and in the merge queue, so 50s gives little value. Fewer shards on 4 CPUs are slower than today, because each shard then has more tests.
The 2-CPU shards peaked at 86% memory. If WebKit starts to fail with out-of-memory errors, more memory is the reason to try a larger runner again.
Also asked as: bigger runners for WebKit, WebKit visual regression is slow, give Storybook more CPU
Run the Node.js serial tests in 2 duration-balanced shards
Verdict: rejected · Nov 2026 · #128491, 2-shard run, 3-shard run
The serial lane runs about 270s of Jest in --runInBand, and each shard has about 230s of setup. A sequencer that splits the files by measured duration (nodejs/jest.serial.sequencer.js) can make 2 shards almost equal.
The proposal was to use 2 shards and remove one runner from each run.
With 2 shards the Jest steps took 153s and 186s. The serial lane then finished about 30s later than the old worst hash shard, and it was the last Node.js job. The JUnit times in the duration map do not include the start-up cost of each test file, so the measured shards were 15–40% heavier than the map predicted. With 3 shards the Jest steps took 119s, 106s and 126s, and the serial lane was no longer the slowest job.
Also asked as: fewer Node.js serial shards, merge the serial shards, save a runner on the Node.js tests
Run the Rust ingestion consumer e2e job on 2-CPU runners
Verdict: rejected · Nov 2026 · #129158, run
Depot metrics showed the job at 21% mean CPU on depot-ubuntu-24.04-4, so a 2-CPU runner looked sufficient.
On depot-ubuntu-24.04 the job took 675s, against a 358s median. The low average hides the setup steps, which use all the CPUs: pnpm install took 123s against about 16s, the Rust cache restore 47s against about 9s, and the personhog pre-build 47s against about 19s.
The job then was the slowest Node.js job by about 250s.
Average CPU does not show which steps are CPU-bound. Examine the time of each setup step before you move a job to a smaller runner.
Also asked as: smaller runner for the Rust e2e job, downsize the ingestion e2e runner, cheaper Node.js CI
Use Bazel to scope product tests
Verdict: abandoned · Dec 2025 to Mar 2026 · #43397
Each product could select Bazel targets. Then a change to one product could skip the legacy pytest jobs.
The branch became inactive, and the stale bot closed it. Nobody measured the result. Thus this entry is not evidence against Bazel.
A different solution achieved the same goal. The product tests moved to Turborepo in #46971. File-level backend selection came later.
Also asked as: Bazel, build graph for test selection, only run tests for the product I changed
Test selection
Exclude products/**/backend/** from the backend paths filter
Verdict: reverted · Mar 2026 · #50137, reverted by #50181
The filter gave the decision to contract-check. contract-check decides if the Django tests are necessary for a change in a product.
This assumes that the products are isolated. Most products were not isolated.
Core code imports product views, serializers, and models directly. It imports them through posthog/api/__init__.py, posthog/tasks/, and the migrations.
contract-check examines only the facade files. Thus it cannot see these imports.
The rule is general. A path filter that skips tests makes a statement about the import graph. Examine the import graph before you make the statement.
Also asked as: narrow the backend paths filter, skip Django tests for product-only changes, trust contract-check
Certify a facade with its __all__
Verdict: superseded · Jul 2026 · #71127, replaced by #71486
The detector read __all__ to prove that a facade does not re-export internal names.
Two problems stopped it. Each review round found a new gap in the detector.
Also, most facade modules do not declare __all__. Thus the detector could certify only the part of the surface that the module declares.
The replacement puts the rule in products/architecture.md. It also makes the contract-check inputs narrow or absent, instead of a list of file globs.
Also asked as: detect facade leaks, check __all__, verify a product is really isolated
Select tests from pytest-testmon coverage data
Verdict: superseded · Apr 2026 · #56370
This PR collected the data. It did not connect the selection to CI. The merge of 52 shard artifacts gave a map of 28,322 tests over 3,691 production files. The map has approximately 1.3 million entries.
The selectivity was high. One changed file caused a median of 45 tests. This is a skip rate of 99.8%.
Two results have more importance than these numbers. First, there were no stale tests with high confidence. Thus a cleanup from this data had nothing to delete. Second, 1,020 tests appeared to touch no production code. Almost all of these results are false. They come from code with many mocks, from property-based tests, and from tests of migration rules. Testmon cannot trace these paths.
Backend test selection came later from a different mechanism. It uses the Snob import graph. Read the next entry.
Also asked as: coverage-based test selection, testmon, find stale tests from coverage, only run affected tests
Snob is in CI, but CI does not use it
Verdict: CI does use it. The scope was narrow on purpose.
tools/snob_backend_test_selection_shadow.py selects the Django test subset for a PR. It combines the Snob import graph with Django-aware heuristics.
Two things make this look inactive:
pytest-snob is an inline PEP 723 dependency of that script. It is not in pyproject.toml. Thus a search of the dependency file finds nothing.
The selection was also active for draft PRs only for some time. The team wanted a stable merge queue first.
#85530 then extended the selection to PRs that are ready for review. #88265 put the Django selection and the product selection in one job.
The comment under pull_request: in .depot/workflows/ci-backend.yml explains which tests run on each type of pull request.
The jest and Playwright suites made the same move later, so ready_for_review no longer buys a full matrix anywhere.
Selection now runs on drafts and ready PRs alike in ci-backend.yml, ci-frontend.yml, ci-storybook.yml, and ci-e2e-playwright.yml, and the merge queue's trunk-merge/** run is the only full gate.
What still differs by draft state is the fallback when a selection cannot be trusted: a draft skips the suite and defers to its ready run, a ready PR takes the full matrix because no later run on that PR would cover it.
Also asked as: snob, is test selection on, why does CI run all the tests, do we select tests on PRs
Give the jest PR matrix a fixed shard count
Verdict: superseded · Oct 2026 · 262 sampled PR runs and a probe run
Jest selection on PRs ran three shards whatever it selected, and the selection was all or nothing: 60% of PR runs executed 2,171 of 2,720 test files and 39% executed under 50.
The import graph that --findRelatedTests walks held one cycle of 2,851 files, and 92% of its edges were imports the transformer removes. Reading imports from the transformer output shrinks it to 218 files and 997 tests.
The 997 tests that load the cycle cost 3.65 seconds each against 1.07 for the rest, so a changed-file count sizes nothing. .github/scripts/jest-select.js counts the related tests and assigns one shard per 171 files, capped at eight. Shard durations measured after that first threshold put a shard at about 102s of fixed setup plus 31s and 1.38s per test file, so the original 342 files produced a ten-minute shard where 171 gives a six-minute one. The selector classifies known fallback cases before installing dependencies. Pricing each shard from the hourly run's JUnit was built and removed: two probe runs sized by count landed within 1.4x of the estimate, and the per-file map cost a recorder job and a cache.
Measured on the way and not worth repeating: --maxWorkers=100% on the selective path gained nothing on the 4 vCPU runners; persisting jest's cache made a second run on the same runner 5% (12 seconds) faster; the V8 leak fix (--retain-maps-for-n-gc=0, #114618) was worth 15 to 20% of a shard and about 5% on the merge queue; a different fixed shard count gained nothing.
Also asked as: jest shard timeout, jest selects everything, more jest shards, jest --maxWorkers, cache jest between runs
Narrow the product test matrix when the backend selector returns full
Verdict: rejected · Aug 2026 · read off the Turbo task graph, not measured
On a legacy diff turbo-discover.js sets products = allProducts, roughly 30 matrix groups and the larger half of a full backend run.
It already computes mustRunProducts (Turbo-affected products, their tach dependents, and the schema-affected set), but applies it only when the selection verdict is selected.
Applying it on a full verdict too looks like free money.
It is not. backend:test in turbo.json declares product-local inputs only — backend/**/*.py, stats/, skills/, scripts/, plus ../../pyproject.toml, ../../uv.lock, and ../../common/**/*.py.
No product task reads posthog/** or ee/**.
So Turbo cannot see the core-to-product edge at all, and on a diff that touches only posthog/ the affected set is empty, the tach dependents of an empty set are empty, and mustRunProducts comes out empty.
Narrowing to it would run zero product tests on a core change that every product imports.
The selected guard is what makes the existing narrowing safe: narrowedProducts unions mustRunProducts with the products Snob's import graph actually reached, and that union is the only thing covering the core-to-product edge.
A full verdict means Snob produced no trustworthy set, so there is nothing to union.
To go after those minutes, reduce how often the verdict is full — the MAX_CHANGED_FILES = 50 bail and the FULL_RUN_PATTERNS list in tools/snob_backend_test_selection_shadow.py are the levers, and both change recall, so measure with the shadow first.
Also asked as: skip product tests on a full run, narrow the product matrix, why do all products run when I only changed core
Disable the pytest unraisableexception and threadexception plugins
Verdict: landed · Jul 2026 to Sep 2026 · #70886, landed by #89057; the hard exit below landed by #104782
Each pytest session runs several full-heap gc.collect() passes at cleanup.
These plugins run the passes only to report __del__ exceptions and thread exceptions as warnings.
addopts already sets -p no:warnings. Thus these warnings cannot become failures, and the passes give no value.
A fixed benchmark of 320 tests decreased from 24.7 seconds to 21.8 seconds.
A reviewer approved the PR. The branch then became inactive, and the stale bot closed it.
#89057 landed the same change, and pytest.ini now disables both plugins.
Read this entry before you try a different solution for the pytest cleanup cost.
#88759 tried a different solution. It deleted the gc.unfreeze() in pytest_unconfigure.
That call is necessary. #62707 added it after the Temporal shards stopped with a segmentation fault and exit code 139. CI made the same crash again on #88759.
Frozen objects do not get the final cyclic collections of Py_FinalizeEx. Thus their finalizers run late in the teardown, after Python removes the extension modules.
Backend CI skips Py_FinalizeEx on green shards. With POSTHOG_PYTEST_HARD_EXIT=1, the root conftest runs the atexit handlers and calls os._exit after pytest_unconfigure, so the gc.unfreeze() still runs first.
One run each: each shard saved 8 to 29 seconds (before, after).
A failing shard, the hourly scheduled run and local runs keep the normal exit, so a teardown crash still shows there. Do not delete the gc.unfreeze().
Also asked as: pytest teardown is slow, reduce gc.collect at session end, speed up pytest cleanup, why does the shard hang after the tests pass, os._exit after pytest
Python and pytest runtime
Tune the Python garbage collector to make the backend tests faster
Verdict: rejected · Sep 2026 · #93565
The collector is already tuned.
conftest.py freezes the heap after boot and sets the thresholds to (50000, 20, 20).
That tuning is worth about 18% of test-phase wall time: the pre-June settings measure +21.6% against it.
Nothing is left beyond it. Higher thresholds (200000, 50, 50) measure +0.5%. A collector disabled for the test phase measures +3.1%. Both are worse than the current setting.
The suite does not wait on Python. CPU is 40% of test-phase wall time on the GitHub Actions runners and 43% on Depot CI. The rest waits on Postgres and ClickHouse.
The PR holds the method and the full tables. Every variant ran back to back on one runner, and each block ran the baseline twice. Two identical baselines on one runner differ by up to 11.7%. Any result below that is noise.
Also asked as: gc.freeze, gc.set_threshold, disable GC during tests, tune Python GC, reduce GC frequency, Python GC cache contention
Set PYTHONHASHSEED to make pytest runtimes consistent
Verdict: rejected · Sep 2026 · #93565
PYTHONHASHSEED=0 measures +1.1% on the GitHub Actions runners and -2.9% on Depot CI.
It also does not reduce the spread, which is the property the proposal asks for.
A constant seed does fix one real problem, and that problem is not a timing one.
parameterized.expand over a set bakes the iteration order into the test ids, so the ids change per process.
Fix that at the test with sorted(). A global seed hides the next one instead.
Also asked as: PYTHONHASHSEED, stabilize pytest runtimes, why do test times vary, make CI timings deterministic
Use jemalloc or cap MALLOC_ARENA_MAX for the backend tests
Verdict: rejected for speed. The memory result holds. · Sep 2026 · #93565
Both cut peak RSS by about 13% on both runner platforms. That part reproduces. Neither moves wall time outside the noise.
A backend shard peaks at about 1.6 GB on a 7.6 GB runner, so 13% releases memory that nothing needs. Try this again only for a job that runs near its memory limit.
Also asked as: jemalloc, LD_PRELOAD libjemalloc, MALLOC_ARENA_MAX, glibc allocator, reduce pytest memory
Run the CI Postgres without fsync
Verdict: rejected · Sep 2026 · #93565
fsync=off, synchronous_commit=off and full_page_writes=off measure -0.0% and +2.4% on the two runner platforms.
The experiment printed show fsync from the container, so the settings did apply.
#70891 proposed the same settings for docker-compose.dev.yml in Jul 2026.
A reviewer rejected that one for a different reason: an unclean shutdown can corrupt a developer's local database.
Also asked as: postgres fsync off, synchronous_commit off, full_page_writes, disable Postgres durability in CI
The Depot CI runners are more cache-contended than the GitHub Actions runners
Verdict: rejected · Sep 2026 · #93565
lscpu inside a Depot CI sandbox reports 2 vCPU, one thread per core, an AMD EPYC 9R45, and 32 MiB of L3.
The depot-ubuntu-24.04 GitHub Actions runners report the same.
Measure this before you build on it. An earlier internal note claimed two threads per core on Depot CI, and that reading is wrong.
depot ci dispatch needs the workflow on the default branch, the same as GitHub.
Use depot ci run --workflow <path> for a workflow that lives on a branch.
Depot Cache is separate from the GitHub cache, so the schema cache and .test_durations both miss and the shard runs a full migrate.
Measure the test phase, not the job wall time.
Also asked as: Depot CI is slower, Depot CI hyperthreading, thread to core ratio, is Depot CI a fair comparison
Docker and image builds
Apply BuildKit cache mounts to the Dockerfile
Verdict: rejected · Oct 2025 · #39700
The change added cache mounts for apt, pip, uv, node, and Playwright. It followed the documented guidance from Depot.
A measurement against master with a warm cache showed a slower build. A backend change increased from 52.7 to 57.5 seconds. This is approximately 9% slower. A frontend change increased from 55.5 to 62.2 seconds. This is approximately 12% slower.
The workload is the reason. A cache mount adds 5 to 7 seconds of overhead, even when the cache has the data. A cache mount gives a benefit only when the dependencies change. Approximately 95% of PRs change code and do not change dependencies. Thus the change made the frequent case slower to make the rare case faster.
A narrow version came later and stayed. #42124 added a uv cache mount, and the Dockerfile also has pnpm and npm mounts today.
That narrow version then caused its own failure. The uv cache kept wheels that were compiled against a different libxmlsec1 version, and the build failed with a version mismatch.
#43066 proposed to remove the mount again. #43091 gave the better fix: it puts the libxmlsec1 version in the cache ID, so a change of the system library invalidates the cache.
Read the id=uv-libxmlsec1... mount in the Dockerfile.
The rule: add a cache mount for one expensive step that you measured. Do not add cache mounts everywhere. Put the version of any system library that the cached artifacts compile against in the cache ID.
Also asked as: --mount=type=cache, speed up Docker builds, follow Depot cache best practices, xmlsec version mismatch in the image build
Move the source COPY to the end of the Dockerfile
Verdict: rejected · Oct 2025 · #39695
The source COPY was already at the end. The layer order in the Dockerfile was already correct. Thus the change did nothing.
Remember this type of proposal. Examine the current state before you optimize it.
Also asked as: improve Docker layer caching, reorder Dockerfile layers
Examine the Docker Hub credentials when CI reports a pull rate limit
Verdict: the cause was different · Aug 2026 · #81963
CI failed with this message: toomanyrequests: You have reached your unauthenticated pull rate limit.
At the maximum, this failure occurred in approximately 45% of the backend CI jobs. The usual rate is zero.
The message indicates an authentication problem. This is why the diagnosis took a long time.
docker login was successful during all of this period, and the message Login Succeeded was correct.
The true cause was a lapse in the Docker Hub subscription. A lapsed plan removes the entitlement. Docker then issues anonymous tokens for the pulls, but it continues to accept the login.
If you see this message again, examine the subscription status before you change the secrets.
Also asked as: Docker Hub rate limit in CI, unauthenticated pull limit, DOCKERHUB secret is wrong
CI orchestration
Move CI from the Depot runners to Blacksmith
Verdict: rejected · Apr 2026 to May 2026 · #54559, removed by #57991
The trial did not do a direct exchange. It ran a Blacksmith shadow of most compute jobs on the same commit, behind the BLACKSMITH_SHADOW_ENABLED variable. Each shadow used continue-on-error, and no shadow was a required check.
The team kept Depot. #57991 removed the shadow workflow and the matrix branches.
The measurements are not in the PRs, so this entry cannot show them.
It kept .github/scripts/compare-ci-runners.py and marked the file as legacy. That script produced the numbers of the trial.
If you propose this again, equalize the caches of the two providers first. A runner that keeps a warm cache between jobs measures the cache, not the compute. Run the trial for several days. A short window cannot separate the jobs whose times are close.
Also asked as: change CI provider, Blacksmith, cheaper runners, are the Depot runners slow
Put the setup actions in a parallel: block
Verdict: reverted · Sep 2026 · added by #76651
pnpm-install, setup-python-cached, and dtolnay/rust-toolchain each write $GITHUB_PATH.
The parallel: block here is GitHub's native step parallelism, shipped in June 2026.
It runs every step in the group as a background step and merges their environment changes at the implicit wait.
The implementation in the GitHub Actions runner is not thread-safe. Two branches that write at the same time crash the runner.
The failing jobs ran on Depot GHA runners, but the bug is not Depot's.
Depot confirmed that its own parallel: construct exists only in Depot CI, which parses .depot/workflows/, and that jobs under .github/workflows/ use GitHub's implementation.
The two share a keyword and nothing else.
The crash gives one of three messages. None of them names a step:
##[error]Collection was modified; enumeration operation may not execute.
##[error]The given key '<guid>' was not present in the dictionary.
SyntaxError: Unexpected end of JSON input # setup-node parsing GITHUB_EVENT_PATH
The runner then fails the step that it was running, and the whole job.
The tool itself succeeds. One failing job logs 1.91.1-x86_64-unknown-linux-gnu installed inside the step that the runner reports as failed.
The crash lands in whichever branch loses the race, so the same bug shows up as Install Rust, Install pnpm dependencies, or Set up Python.
Four product test jobs died this way between 09:05 and 11:52 on 4 Sep 2026. The same crash is in the runs of 3 Sep 2026, so it is not a single bad day. The three setup steps take 151s, 4s, and 12s in the product test job, so the block saves about 16s of a 12-minute job.
The steps are sequential today. Keep an action that writes $GITHUB_PATH or $GITHUB_ENV out of a parallel: block.
A block of run: steps is safe, and ci-backend.yml, ci-python.yml, ci-frontend.yml, and ci-nodejs.yml still use one.
Also asked as: parallel steps, run the setup steps at the same time, Collection was modified, key was not present in the dictionary, Install Rust fails in setup
Use sparse-checkout on the large CI workflows
Verdict: abandoned · Oct 2025 · #39239
The description covers the backend, frontend, and Rust workflows. The diff changes only ci-backend.yml and ci-rust.yml.
The PR did not merge, and no person reviewed it. The stale bot closed it. Thus there is no recorded reason for the result.
Sparse-checkout is correct for small jobs, and ci-storybook.yml, ci-security.yaml, and pr-resolve-outdated-bot-comments.yml use it today.
The backend timing collector keeps the full products/ tree because its optimizer uses product directory presence to retain timings for skipped products.
The bot IP updater needs only bin/update-bots-list and nodejs/assets/bot-ips.txt; its sparse checkout retains the tracked output so Git can detect and commit updates.
ci-depot-action-mirrors.yml checks out .depot/, .github/actions/, and depot_mirrors.py; its Git diff still covers all changed paths.
The Dagster path validator also uses a sparse checkout of every internal Python source file, its workflow, and the local uv action. Its import resolver needs the complete internal Python tree, including namespace packages.
ci-rust.yml also uses it on the build and test jobs, but that came before this PR.
The large Python and frontend test jobs do not use it. Their checkout is complete.
Desktop agent release verification keeps the standalone packages/agent/ workspace, its build action, .nvmrc, and the desktop lockfile used for caching.
Validate its frozen install, workspace build, and agent tests from the sparse tree when changing those inputs.
The HCL schema guard checks out posthog/clickhouse/hcl/, which contains its scripts, manifest, layers, golden dumps, and generated SQL.
Keep the full directory so every environment and role remains available to validation and drift checks.
OpenAPI codegen tests keep root files, every package manifest, patches, the pnpm setup actions, Husky hooks, the Linux-safe postinstall script, and tools/openapi-codegen/. The package manifests preserve workspace discovery during the frozen filtered install. Check both cold and warm installs, test discovery, and failure detection when changing these inputs.
The MCP live canary uses the same workspace manifest pattern, plus services/mcp/ and packages/ for its dependency tree.
It keeps the frozen filtered install and live upstream test.
If you propose this again, name the jobs and prove that each one reads only the included paths. A test job can read more of the tree than an exclusion list expects.
Storybook's select-stories job checks out only .nvmrc from the PR head. It keeps the bounded blobless history for changed-path detection, downloads the module graph, and runs the selector from a separate trusted base checkout, which also carries common/storybook/storybook-timings.json. Source files do not need to exist in the PR worktree for this job. The manifest is there because it sizes the shard count, and a file count cannot size that: the stories a frontend diff affects carry about twice the render time per story that the average story does. It is read from the base checkout so the nightly refresh applies to branches behind master and a fork cannot steer the shard count.
Also asked as: sparse-checkout, partial clone, do not check out the whole repo, speed up the checkout step
Drop filter: blob:none from a deep checkout
Verdict: rejected · Sep 2026 · run
Backend test discovery needs 1000 commits of history.
With fetch-depth: 1000 and filter: blob:none, the checkout took 42 to 46 s, because it fetches every file of the tree in a second request.
Without the filter it took 39 s, because the server then packs the file contents of all 1000 commits.
A depth-1 checkout followed by git fetch --filter=blob:none --depth=1000 takes about 7 s, and the job uses that now.
--deepen=999 in its place fetched nothing when GitHub had already replaced the PR's merge ref, which failed the run.
The test-selection-verdict and backend-coverage-report jobs still use deep blobless checkouts. They run after the required gate, so no PR waits on them.
The Agent Skills build uses a plain depth-1 checkout on master pushes because its schema cache keys read only HEAD.
Its PR checkout keeps bounded blobless history for the merge-base schema key.
Also asked as: blobless clone is slow, partial clone checkout, speed up the discovery checkout, fetch-depth 1000
Check out the PR head by SHA in the Django test shards
Verdict: rejected · Sep 2026 · before, after
The Django test shards check out pull_request.head.ref and spend about 22 s. The product shards check out the merge commit and spend about 6 s.
The difference is in the git fetch --depth=1, and on master pushes the same Django checkout takes about 7 s.
A checkout of pull_request.head.sha also took 21 to 22 s, so the refspec is not the cause. The cause is not known.
Also asked as: slow checkout in the Django shards, head.ref or head.sha, why is the Django checkout slower than the product checkout
Keep the pre-built wheels in the uv cache with prune-cache: false
Verdict: rejected · Oct 2026 · Depot run, #121208
setup-uv prunes the cache, so each backend shard downloads its wheels from PyPI in uv sync.
Between 00:14 and 03:07 UTC on 27 Oct 2026, PyPI downloads from the Depot CI sandboxes were very slow.
The Install Python dependencies step has timeout-minutes: 5 and retry: 1.
Both attempts stalled, and Depot killed the step with exec-step child exited with signal SIGTERM and produced no result.
Each shard failed about 10.5 minutes after it started. About 25 Backend CI runs failed. From 22:00 UTC the evening before to 12:00 UTC that day, no other run failed this way.
The proposal was to stop the pruning, so that the shards get the wheels from Depot Cache and not from PyPI. The cost was estimated, not measured, because no job runs with an unpruned cache:
- The unpruned cache is about 1 GB. Depot Cache storage costs $0.20 per GB-month, so the storage cost is small.
- Each shard restores the 1 GB in about 20 s (from #121208).
uv syncfrom PyPI takes 11 to 23 s in a healthy shard. The estimated net cost is 5 to 20 s for each shard, on every run. - At $0.006 per minute for a 2 vCPU sandbox, the compute of the whole outage, with the killed shards and the full reruns, is equal to about 2 to 8 days of that extra restore time.
The pruning stays because one outage at night costs less than the restore time. Try this again if PyPI outages occur during working hours or more than a few times each month. Then the developer time and the merge queue kicks change the result. A longer step timeout does not help: the 5-minute bound and the retry let the shard fail fast, and this outage was 3 hours long.
setup-uv v9 changed the prune-cache default to false. Each cache-enabled call sets prune-cache: true so that the cache stays at about 3 MB.
Also asked as: uv sync timeout, uv sync SIGTERM, PyPI is slow in CI, cache the Python wheels, why is prune-cache true, exec-step child exited with signal SIGTERM
Jest reports the Rust snapshots as obsolete
Verdict: superseded · Jan 2026 · #46008
Jest found the .snap files under rust/cymbal/tests/snapshots/ during the Storybook visual regression job. It marked them as obsolete, and all 19 jobs failed.
The PR records the attempts that did not work. modulePathIgnorePatterns changes only the module resolution. It does not change which snapshot files Jest finds.
The PR proposed haste.blockList. The repository does not use that option today, so a different change solved this.
Keep the record: the snapshot scan and the module resolution use different configuration.
Also asked as: obsolete snapshots in CI, Jest finds rust snapshots, modulePathIgnorePatterns
Skip Storybook and E2E for snapshot-only commits from the bot
Verdict: reverted · Mar 2026 · #49997, reverted by #51212
The change went to master. One week later, a revert removed it.
Also asked as: skip CI for snapshot commits, ignore bot commits in CI, don't rerun visual tests for the snapshot bot
Force-cancel the backend CI run when pytest does not stop
Verdict: reverted · Apr 2026 · #54261, reverted by #54685
The change added a watchdog job. The job cancels a run when pytest does not exit. The change stayed in master for one day.
Also asked as: cancel watchdog, kill hung CI jobs, pytest ignores SIGTERM
Add a CI step that asks a person to take a bot PR
Verdict: rejected · Jun 2026 · #62111
CI cannot assign a PR from a bot automatically, because the bot account is not a member of the team. The CI step asked a person to take the PR.
A different solution replaced it. A CI step must guess which person controls a bot PR. The agent that opens the PR already knows this person. Thus the instruction moved to the PR template.
Also asked as: auto-assign bot PRs, find the human behind an agent PR, nudge for ownership
CI failure triage
Use a small classifier model to route a failed CI job to the next investigation
Verdict: rejected · Oct 2026 · measured, not built
Jev 1.13 (TypeSafe) read a log excerpt of a failed job and picked the next investigation from five routes. A list of regular expressions over the same excerpt did at least as well. On 24 held-out jobs the rules routed 20 and got 18 right. The model routed 18 and got 14 right. The labels came from agents, and the gap is inside the noise.
The failed step name and the lines above the last ##[error] carry most of the signal, and a rule reads both.
Intermittent test failures were wrong for both. The rerun result separates them, and the log does not.
Also asked as: classify CI failures with an LLM, AI triage for red CI, Jev, TypeSafe, System One model, flaky or real failure from the log, route CI failures automatically
Dev environment
Share the dev environment and the Docker containers across worktrees
Verdict: three attempts, none merged · Oct 2025 to Apr 2026 · #40634, #45984, #51100
Each attempt used a different mechanism.
#40634 changed the compose setup so a worktree uses the containers of the main checkout.
#45984 set COMPOSE_PROJECT_NAME in the flox variables. Docker Compose uses the directory name when this variable is absent, so each worktree makes its own containers.
#51100 shared the flox environment, the Python virtual environment, and node_modules. It reports approximately 5 GB of disk for each worktree.
None of the three merged. bin/wait-for-docker gives the compose project the default name posthog today, which gives the shared containers that #45984 wanted.
Read #40634 first if you propose this again. It asks the question that stopped all three: does any person need separate databases for each worktree?
Also asked as: worktrees start their own containers, share node_modules between worktrees, worktree disk usage, COMPOSE_PROJECT_NAME
Run a dmypy daemon for fast local type checks
Verdict: rejected · Oct 2025 · #39319
A pre-commit hook used the daemon if the daemon was already active. A warm daemon gives a check of approximately 0.6 to 1.7 seconds.
The warm-up time did not decrease. Thus the first check still has the full cost. A start of the daemon from mprocs was also tested. The commits then stopped and waited, because the daemon was still warm.
Also asked as: dmypy, speed up mypy locally, type-check on commit, mypy daemon
Run uv sync at each flox re-activation
Verdict: rejected · Feb 2026 · #49183
The shell profiles synchronize the dependencies at each start of a shell. This needs approximately 660 ms when the dependencies are current.
The reason for the rejection is the frequency. The profiles also run in subshells. Thus the cost occurs much more frequently than the problem.
Also asked as: auto-sync deps, keep the venv current automatically, uv sync in the shell profile
Upgrade Python to a version that the flox uv cannot install
Verdict: reverted · Oct 2025 · #40286, reverted by #40290
Python 3.12.12 needs uv 0.9.2 or later. Flox pinned uv 0.8.23, and that version can get Python 3.12.10 at the maximum. Thus a person without Python 3.12.12 on the local machine could not build the environment.
Remember the constraint, not the versions. The repository now uses Python 3.14.7, and this pin is obsolete. Before you increase the Python version, examine which versions the pinned flox uv can get.
Also asked as: bump Python, upgrade the interpreter, why is Python pinned to an exact version
Django performance
docs/internal/django-startup-time.md is the deep source for this area. It has a Traps section that records the failure modes of each mechanism. The entries below give the proposals that people repeat.
Squash the Django migration history
Verdict: landed on the third attempt · Feb 2026 to Aug 2026 · #48267 · #60518
Two designs failed first. #48267 used Django's optimizer per app; the effect was small because RunPython/RunSQL block the optimizer, and state.clone() runs per operation, so a squash only helps in proportion to the operation count it removes. #60518 rebuilt the final project state at a cutoff date from CreateModel operations; the approach was right but the trial went stale on CI blockers and a replaces= lineage flaw (it claimed names from its own never-merged predecessor, which no real database ever recorded).
The third attempt reran the #60518 design with django-nextgensquash (started as tools/nextgensquash in this repo) after fixing the lineage rule and a set of structural gaps: idempotent finalize operations so existing databases can apply the squash tail as no-ops, a stub replaces= claim so check_consistent_history passes on live databases without --skip-checks, and forwarding for raw-SQL indexes and composite foreign keys. A fresh database migrate dropped from roughly 20 minutes to about 4, existing databases stamp the squashes on their next migrate, and makemigrations --check reports zero drift against the squashed state.
To repeat with a newer cutoff, reset every migration directory to master first (the tool re-squashes from a clean tree, not on top of its own output), then run it from the repo root through uv; the project config lives in NEXTGENSQUASH in posthog/settings/nextgensquash.py:
uv run --with git+https://github.com/PostHog/django-nextgensquash python -m nextgensquash emit --settings posthog.settings --cutoff 2026-09-07 --output-dir /tmp/squash
uv run --with git+https://github.com/PostHog/django-nextgensquash python -m nextgensquash install --settings posthog.settings --input-dir /tmp/squash
The generated finalize files import posthog/migration_helpers/squash_idempotent.py, so the package is a dev-only tool and not a dependency of this repo. The emit gate refuses young migrations that touch deferred foreign-key fields; bump the cutoff past them. Keep the window between cutoff and merge short: every migration that lands on master in that window sits before finalize_fks on a fresh database, and one that touches a deferred column forces a re-squash. A dedicated migration test (TestMigrations with migrate_from) that targets a folded migration fails with "not a valid node", because the loader drops replaced nodes; delete those tests, since a folded migration has been applied everywhere by definition. A schema_addons file must not depend on an app routed to another database (products/db_routing.yaml): the CI schema restore forgets those apps' django_migrations rows so each environment applies them under its own routing, and a dependant of a forgotten row makes Django refuse to migrate at all. The tool skips apps on another database, and posthog/test/repo_invariants/test_migration_dependencies_share_a_database.py blocks the edge in review; forgetting more rows on restore is the wrong fix, because the caller's migrate then re-applies DDL the dump already holds.
Also asked as: squash the migrations, compress the migration history, why are there so many migrations, speed up the migration replay, nextgensquash
Build the generated pydantic schema lazily with defer_build
Verdict: reverted · docs/internal/django-startup-time.md
This removed approximately 400 ms of core-schema construction from each django.setup(). The round-trip tests were all successful.
Two problems stopped it. First, the deferred builds move to the first /query of each web worker after a deploy. A warm-up loop for those builds measured approximately 2.5 times more expensive than the eager construction.
Second, the query runners construct the response models directly. This does no validation, so it does not start the lazy build. model_dump() then sends a mock serializer into pydantic-core and raises TypeError: 'MockValSer' object cannot be converted to 'SchemaSerializer'. This is a 500 error in any process.
A different solution removed the cost. django.setup() no longer imports posthog.schema at all.
Also asked as: defer_build, make the schema import lazy, pydantic model build is slow at startup
Replace pydantic in the generated schema with plain dataclasses
Verdict: not viable today
posthog/schema.py has more than 1,000 generated classes. hogli build:schema generates the file from the TypeScript types with pydantic tooling.
Approximately 220 files call model_validate, and more call the model_validate_json and model_validate_python variants. The API layer depends on this validation. Thus a change of the model library is not a local change.
The import cost is solved. posthog.schema costs approximately 2 seconds to import, but django.setup() no longer loads it. The enums also moved to posthog.schema_enums, which imports in approximately 20 ms.
Import a model from posthog.schema inside the method that uses it. Take the enums from posthog.schema_enums.
Also asked as: remove pydantic, use dataclasses for the schema, the schema import is slow, why is posthog.schema so big
Database migrations
Switch the Person model to the partitioned table with a Django setting
Verdict: eight attempts closed unmerged · Nov 2025 · #41436, #41513, #41522, #41600, #41604, #41669, #41698, #41813
The goal is to move the Person model from posthog_person to a table that is partitioned by team_id. Eight PRs tried five mechanisms.
A PERSON_TABLE_NAME setting that gives db_table to the model. A dual manager that reads both tables and prefers the new one. A swap of the two table names in the database. A wrapper that rejects any query to a partitioned table without team_id in the WHERE clause. A separate test database for the person tables.
None of them merged. The author wrote this on #41513:
I'm still not sure if I got on the wrong track here by wanting to bend all test setup to use the person_new table and other sqlx migrated stuff. It seems I overlooked something fundamental since things are failing so much.
One narrow PR from the same window did merge. #41620 put team_id into the Person queries that lacked it, and it carried the PERSON_TABLE_NAME setting to master.
The setting is in posthog/settings/data_stores.py, and Person.Meta.db_table reads it. It defaults to posthog_person, so the cutover is off.
The switch exists. The eight PRs above failed at what surrounds it: the test setup, the dual reads, and the partition guard.
posthog_person_new comes from the sqlx migrations in rust/persons_migrations/. Three Dagster jobs read it, and one carries a comment about a future name swap.
Person and group data now goes through the gRPC client in posthog/personhog_client/. .agents/architecture.md makes that client the required interface and prohibits new ORM queries against the person tables.
Read this entry before you propose a Django-level cutover. The setting is already there. What is missing is everything that must be true before a person changes its value.
Also asked as: partition the person table, PERSON_TABLE_NAME, posthog_person_new, dual-table reads, cut over the Person model
API contracts
Validate the API responses against the generated OpenAPI schema
Verdict: six attempts, none merged · Mar 2026 to Jun 2026
The idea returns in two shapes.
End-to-end traffic validation, in the Playwright run. Most of these use a Django middleware, and #49895 uses Spectral and a Prism proxy instead: #49898, #49932, #49940. Response validation inside the pytest run, with a report as a CI artifact: #56804, #56810.
Each PR made the validation optional and non-blocking, to avoid noise. None of them merged. Read this history before you start a seventh attempt. Six PRs that all stop before the merge is a signal about the design, not about the effort.
The generated types have a different guard today. The serializers produce the OpenAPI schema, and hogli build:openapi generates the TypeScript from it. CI fails when the committed output does not match.
Also asked as: contract testing, validate responses against the schema, spectral, prism, schema drift in CI
Product isolation
Move ee/ into products/enterprise/backend/
Verdict: rejected · Nov 2025 · #41025
The PR moved 613 files and kept the git history. It kept the app label ee, so the database did not change. Django validated, and no migration was necessary.
The PR did not merge, and ee/ is still a top-level directory.
A mechanically correct move is not sufficient for a directory of this size. If you propose this again, say who reviews 613 moved files, and what breaks for each open PR that touches ee/.
Also asked as: move ee to products, get rid of the ee folder, enterprise product folder
Use logs as the first product behind a facade
Verdict: superseded · Jun 2026 · #63184
logs was the first candidate for the field test. Core imports its models, its query runner, its celery task, and its temporal wiring. Thus logs is a difficult example.
The field test moved to web_analytics in #63535. #63193 collected the tools and the skill.
The note at the closure says that the tools can isolate logs quickly, if the team still wants this.
The doctrine from this work came later, in #71486.
Also asked as: which product should we isolate first, facade migration example, isolate logs
PR structure
Put two closely coupled layers in two stacked PRs
Verdict: rejected for this change · Aug 2026 · #87643, merged into #87644
The two layers read well as two stories. They did not work as two diffs. Both layers changed the same four modules, and sometimes the same lines. The upper layer deleted a block that the lower layer corrected. It also changed the signature of a function that the lower layer divided.
Each correction in the lower layer needed a repeat through these collisions. The repeat occurs again at each restack.
Divide a change by its diff surface, not by its story. If two layers touch the same lines, one PR needs less review effort than two.
Also asked as: should I stack these, split this PR, break the change into reviewable layers