Build AI methods with your coding agent, then run them anywhere: as an MCP for chatbots, as a webapp for people, or via API for your software.
feature/Test-runs-no-telemetry · L-260924-d570e6 — test runs feed prod PostHog (#1245)
Louis Choquel1a21e820917a
diff --git a/CHANGELOG.md b/CHANGELOG.mdindex a21dcbff5..84aabbb26 100644--- a/CHANGELOG.md+++ b/CHANGELOG.md@@ -8,6 +8,10 @@- **The documentation site leads with the hosted route**: its getting-started opens on a new **Quick Start** page — sign up at app.pipelex.com, the Pipelex plugin in Claude Code or Codex, the Pipelex MCP in ChatGPT or Claude, then a webapp from the method-app template or the hosted API over HTTP, TypeScript or Python — and the runtime's own install moves to **Run It Yourself**. **Build with Claude Code** and its `npm install -g mthds` route are retired, and **Claude Code Skills Plugin** becomes **Pipelex Plugin**, which documents the Pipelex plugin instead of the MTHDS plugin; the old addresses redirect. The front page's buttons and the site's description lead with what Pipelex lets you do.- **`mthds` version**: The pin moves to `mthds==0.16.0`, the release adding a `User-Agent` on every `MthdsAPIClient` request and the `mthds.version` module. Nothing in pipelex uses either, and the MTHDS standard version stays at `2.0.0`, but the pin is exact, so everyone downstream inherits `mthds` 0.16.0 and a consumer depending on `mthds` directly must move in step.+### Fixed++- **Test runs no longer send the Pipelex Gateway telemetry stream**: a runtime booted in the `pytest` or `ci` integration mode, or in a session that loads the shared pytest plugin, which sets a test run mode when pytest configures, keeps the Gateway stream off even with `pipelex_gateway` enabled, so a project's test suite no longer reports its runs and its fixture user ids to Pipelex. In a test run a boot no longer needs `PIPELEX_GATEWAY_API_KEY` for telemetry, `DO_NOT_TRACK` no longer conflicts with the Gateway, and the Gateway's Portkey debug and tracing flags have no effect, even where the custom stream is allowed; `IntegrationMode.is_test_harness` names the two modes. The plugin's session fixture `set_run_mode` is gone: the run mode is set in its `pytest_configure`, so a boot at test-module import or during collection is covered too.+## [v0.64.0] - 2026-09-24### Addeddiff --git a/docs/configuration/config-practical/telemetry-config.md b/docs/configuration/config-practical/telemetry-config.mdindex 7db76f1aa..04806a1b7 100644--- a/docs/configuration/config-practical/telemetry-config.md+++ b/docs/configuration/config-practical/telemetry-config.md@@ -249,7 +249,7 @@ set DO_NOT_TRACK=1When set, this disables:- All custom telemetry (PostHog, Langfuse, OTLP)-- Gateway telemetry (note: Gateway won't work without telemetry)+- Gateway telemetry (note: outside a test run, Gateway won't work without telemetry — see [Telemetry Setup](../../setup/telemetry.md#1-gateway-telemetry-pipelex-controlled))---diff --git a/docs/features/telemetry.md b/docs/features/telemetry.mdindex de5605723..d6a26d6c8 100644--- a/docs/features/telemetry.md+++ b/docs/features/telemetry.md@@ -29,7 +29,7 @@ A span or an event produced during a run is attributed to that run's own caller,## Gateway Telemetry-When using Pipelex Gateway as your inference backend, privacy-respecting metrics are automatically collected: models used, token counts, latency, and error rates. This data is tied to your Gateway API key (hashed for security) and requires no additional configuration.+When using Pipelex Gateway as your inference backend, privacy-respecting metrics are automatically collected: models used, token counts, latency, and error rates. This data is tied to your Gateway API key (hashed for security) and requires no additional configuration. Test runs are not collected: when the runtime boots in the `pytest` or `ci` integration mode, or in a session that loads Pipelex's shared pytest plugin, the Gateway stream stays off.## Privacy Controlsdiff --git a/docs/setup/telemetry.md b/docs/setup/telemetry.mdindex 65e8aabc3..688100983 100644--- a/docs/setup/telemetry.md+++ b/docs/setup/telemetry.md@@ -12,6 +12,8 @@ Pipelex supports two independent telemetry streams that serve different purposesWhen you use **Pipelex Gateway** as your inference backend, identified telemetry is **automatically enabled**. This telemetry is tied to your Gateway API key (hashed for security) and operates independently from your `telemetry.toml` settings.+Test runs are the exception, and either of two signals marks one. The first is the `pytest` or `ci` integration mode. The second is a test run mode, which Pipelex's shared pytest plugin (`pipelex.test_extras.shared_pytest_plugins`) sets for every session that loads it, when pytest configures and before it collects anything, so a suite booting with `Pipelex.make()` in the default mode is covered too, whether it boots in a fixture, at test-module import or in a pytest hook that runs after `pytest_configure`. What it cannot reach is anything pytest runs before it configures: the body of every `conftest.py` loaded at startup, which means the ones along the paths given on the command line or in `testpaths` and not only the one that loads the plugin, and the hooks that run before `pytest_configure`, such as `pytest_addoption` or `pytest_cmdline_main`. Boot in a fixture there. In a test run the Gateway stream stays off, so a project's test suite sends nothing to Pipelex even with the Gateway enabled: the boot does not look for `PIPELEX_GATEWAY_API_KEY` on the stream's behalf, and `DO_NOT_TRACK` does not conflict with the Gateway. The Gateway's Portkey request logging and trace correlation follow the stream, so the `debug` setting on the `pipelex_gateway` backend and the `[pipelex_gateway.portkey]` force flags in `telemetry.toml` have no effect in a test run either, even where your own custom stream is allowed in that mode.+A run that names a caller of its own is attributed to that caller on this stream too: the run's `user_id` is sent as the `distinct_id` exactly as you supplied it, and its `extras` ride the capture as groups. Anything that names no caller reports under your key's hash.**What we collect:**@@ -98,11 +100,11 @@ export DO_NOT_TRACK=1When set, this disables:-- Gateway telemetry (but note: Gateway won't work without telemetry)+- Gateway telemetry (but note: outside a test run, Gateway won't work without telemetry)- All custom telemetry destinations!!! warning "Gateway Requires Telemetry"- If you set `DO_NOT_TRACK=1` while using Pipelex Gateway, the Gateway will not function. Use direct provider backends instead if you need to disable all telemetry.+ If you set `DO_NOT_TRACK=1` while using Pipelex Gateway, the Gateway will not function, except in a test run, where the Gateway stream is off anyway. Use direct provider backends instead if you need to disable all telemetry.## Attaching your own labels to a rundiff --git a/pipelex/runtime_boot.py b/pipelex/runtime_boot.pyindex aa0b5c0c3..048d87921 100644--- a/pipelex/runtime_boot.py+++ b/pipelex/runtime_boot.py@@ -340,6 +340,56 @@ Note that this command resets all config files to their default values.If you need help, drop by our Discord: we're happy to assist: {URLs.discord}."""+ @classmethod+ def should_enable_pipelex_telemetry(+ cls,+ *,+ integration_mode: IntegrationMode,+ is_unit_testing: bool,+ is_gateway_enabled: bool,+ needs_inference: bool,+ is_gateway_config_cached: bool,+ ) -> bool:+ """Decide whether this boot sends the Pipelex Gateway telemetry stream.++ The stream is disabled when:++ - the legacy gateway backend is not enabled, OR+ - inference is not needed (no live runs to track), OR+ - the gateway config came from the cache (stale specs imply potentially stale model+ identities; phoning home about pipe runs in that state would pollute metrics), OR+ - the runtime is booted by a test harness: its runs are not usage, and its fixture user ids must+ not become persons in the production analytics project. Two signals say so, and either one is+ enough. The integration mode (`CI` or `PYTEST`) is what a harness booting without our pytest+ plugin states, such as the pipelex-js conformance scripts. The run mode is what the shared+ pytest plugin sets for every session that loads it, which covers the suites that boot in the+ default `PYTHON` mode, as the plugin's own recipe does.++ **The first condition asks about `pipelex_gateway` specifically, not about managed backends+ in general**, and the distinct id is why: it is derived from `PIPELEX_GATEWAY_API_KEY`, which+ a manifold backend neither has nor can stand in for — its own key is, for the private beta,+ one token shared by every participant, so keying on it would produce a single indistinguishable+ user rather than an identity. Asked the general way, a manifold-only installation would be+ required to hold a gateway key it has no other use for and would fail to boot without one.+ The common beta case is unaffected: a participant who keeps `pipelex_gateway` enabled has a+ real gateway key, and their manifold runs are tracked under it like everything else.++ The per-mode `telemetry_allowed_modes` table does not take part: it governs only the+ operator's custom stream, which the telemetry factory gates by itself.++ Args:+ integration_mode: The mode the runtime is booted in.+ is_unit_testing: Whether the run mode is a test mode, as `RuntimeManager.is_unit_testing` says.+ is_gateway_enabled: Whether the `pipelex_gateway` backend is enabled.+ needs_inference: Whether this boot runs live inference.+ is_gateway_config_cached: Whether the gateway config came from the cache rather than a fresh fetch.++ Returns:+ True when the Gateway telemetry stream should be sent.+ """+ is_test_harness = integration_mode.is_test_harness or is_unit_testing+ return is_gateway_enabled and needs_inference and not is_gateway_config_cached and not is_test_harness+def setup(self,*,@@ -506,28 +556,22 @@ If you need help, drop by our Discord: we're happy to assist: {URLs.discord}.secrets_config = get_config().runtime.secretssecrets_provider = secrets_provider_registry.get_required(method=secrets_config.method)(secrets_config)- # Disable Pipelex telemetry when:- # - the legacy gateway backend is not enabled, OR- # - inference is not needed (no live runs to track), OR- # - the gateway config came from the cache (stale specs imply potentially stale model- # identities; phoning home about pipe runs in that state would pollute metrics).+ # Whether the Pipelex Gateway telemetry stream is sent: the conditions, and why each one is+ # there, are in `should_enable_pipelex_telemetry`.#- # **The first condition asks about `pipelex_gateway` specifically, not about managed backends- # in general**, and the distinct id is why: it is derived from `PIPELEX_GATEWAY_API_KEY`, which- # a manifold backend neither has nor can stand in for — its own key is, for the private beta,- # one token shared by every participant, so keying on it would produce a single indistinguishable- # user rather than an identity. Asked the general way, a manifold-only installation would be- # required to hold a gateway key it has no other use for and would fail to boot without one.- # The common beta case is unaffected: a participant who keeps `pipelex_gateway` enabled has a- # real gateway key, and their manifold runs are tracked under it like everything else.- #- # Read off the mapping computed above rather than by asking the document a second time. The- # two answers are the same one — the gateway resolves to a section whenever it is enabled —- # and a second read would be a second chance for an unparseable file to escape the clause- # that frames it as a backend-library refusal.+ # The gateway's enablement is read off the mapping computed above rather than by asking the+ # document a second time. The two answers are the same one — the gateway resolves to a section+ # whenever it is enabled — and a second read would be a second chance for an unparseable file+ # to escape the clause that frames it as a backend-library refusal.gateway_source_is_cached = gateway_config_source is not None and gateway_config_source.is_cachedis_gateway_enabled = PipelexBackend.GATEWAY in managed_gateway_sections- is_pipelex_telemetry_enabled = is_gateway_enabled and needs_inference and not gateway_source_is_cached+ is_pipelex_telemetry_enabled = self.should_enable_pipelex_telemetry(+ integration_mode=integration_mode,+ is_unit_testing=runtime_manager.is_unit_testing,+ is_gateway_enabled=is_gateway_enabled,+ needs_inference=needs_inference,+ is_gateway_config_cached=gateway_source_is_cached,+ )self.telemetry_manager = TelemetryFactory.make_telemetry_manager(secrets_provider=secrets_provider,integration_mode=integration_mode,diff --git a/pipelex/system/runtime.py b/pipelex/system/runtime.pyindex d8840080a..bb7a8f63a 100644--- a/pipelex/system/runtime.py+++ b/pipelex/system/runtime.py@@ -39,6 +39,26 @@ class IntegrationMode(StrEnum):case IntegrationMode.PYTHON:return True+ @property+ def is_test_harness(self) -> bool:+ """Whether this is a mode only test suites and conformance scripts boot in.++ The boot keeps the Pipelex Gateway telemetry stream off in these modes, so a test run never+ reports to the production analytics project.+ """+ match self:+ case IntegrationMode.CI | IntegrationMode.PYTEST:+ return True+ case (+ IntegrationMode.CLI+ | IntegrationMode.DOCKER+ | IntegrationMode.FASTAPI+ | IntegrationMode.MCP+ | IntegrationMode.N8N+ | IntegrationMode.PYTHON+ ):+ return False+class RunMode(StrEnum):NORMAL = "normal"diff --git a/pipelex/system/telemetry/telemetry_manager.py b/pipelex/system/telemetry/telemetry_manager.pyindex 78f3a6222..da96e6580 100644--- a/pipelex/system/telemetry/telemetry_manager.py+++ b/pipelex/system/telemetry/telemetry_manager.py@@ -560,6 +560,10 @@ class TelemetryManager(TelemetryManagerAbstract):@overridedef is_pipelex_gateway_portkey_logging_enabled(self, *, is_debug_configured: bool) -> bool:+ # The Gateway's Portkey logging follows the Gateway stream: where the stream is off, a test run+ # above all, neither the backend's `debug` nor the force flag may turn it on.+ if not self._pipelex_telemetry_enabled:+ return Falseis_debug: bool = is_debug_configuredif (not is_debug@@ -581,6 +585,9 @@ class TelemetryManager(TelemetryManagerAbstract):@overridedef is_pipelex_gateway_portkey_tracing_enabled(self) -> bool:+ # Trace correlation follows the Gateway stream for the same reason as the logging above.+ if not self._pipelex_telemetry_enabled:+ return Falseif (self.telemetry_config.pipelex_gatewayand self.telemetry_config.pipelex_gateway.portkeydiff --git a/pipelex/test_extras/shared_pytest_plugins.py b/pipelex/test_extras/shared_pytest_plugins.pyindex cf9e8b5c3..03ee1f58d 100644--- a/pipelex/test_extras/shared_pytest_plugins.py+++ b/pipelex/test_extras/shared_pytest_plugins.py@@ -55,12 +55,11 @@ ENV_VAR_KEYS_WHICH_MAY_NEED_PLACEHOLDERS_IN_CI = [][email protected](scope="session", autouse=True)-def set_run_mode():+def _set_test_run_mode() -> None:if is_env_var_set(key="GITHUB_ACTIONS") or is_env_var_set(key="CI"):runtime_manager.set_run_mode(run_mode=RunMode.CI_TEST)elif is_env_var_truthy(key=CODEX_CLOUD_ENV_VAR_KEY):- # we're in codex cloud and this fixture is called by pytest, so we are testing in codex cloud+ # we're in codex cloud and pytest is running, so we are testing in codex cloudruntime_manager.set_run_mode(run_mode=RunMode.CODEX_CLOUD_TEST)else:runtime_manager.set_run_mode(run_mode=RunMode.UNIT_TEST)@@ -90,12 +89,20 @@ def pytest_addoption(parser: Parser):)[email protected](tryfirst=True)def pytest_configure(config: Config) -> None:- """Check prerequisites before test collection starts.+ """Set the test run mode, then check prerequisites before test collection starts.++ The run mode is set here, first among the configure hooks, rather than in a fixture, because+ it is what keeps the Pipelex Gateway telemetry stream off in a test run: a boot at test-module+ import, during collection or in a later pytest hook happens before any fixture runs, and would+ otherwise see the normal run mode and send the stream.- Validates that Pipelex Gateway terms are accepted when gateway is enabled.+ Then validates that Pipelex Gateway terms are accepted when gateway is enabled.This runs early to provide clear feedback before wasting time on test collection."""+ _set_test_run_mode()+# Skip check when inference is disabled via CLI optionif config.getoption("--disable-inference", default=False):returndiff --git a/tests/integration/pipelex/system/pipelex_service/conftest.py b/tests/integration/pipelex/system/pipelex_service/conftest.pynew file mode 100644index 000000000..349e31e4d--- /dev/null+++ b/tests/integration/pipelex/system/pipelex_service/conftest.py@@ -0,0 +1,23 @@+import os++import pytest+from pytest_mock import MockerFixture++from pipelex.system.pipelex_service.pipelex_details import PipelexDetails+from pipelex.system.telemetry.otel_constants import OTelConstants+++def gateway_telemetry_reachable(mocker: MockerFixture) -> None:+ """Clear what would stop a boot from building the Gateway telemetry stream, and send nothing.++ A placeholder gateway key and no `DO_NOT_TRACK`, so a boot that turns the stream on builds the real+ manager instead of refusing, which makes a test's assertion on the stream the thing that fails. PostHog+ is a mock, so that manager reaches no network, and the PostHog module globals it sets are restored+ after the test.+ """+ mocker.patch.dict(os.environ, {PipelexDetails.PIPELEX_GATEWAY_API_KEY_VAR: "placeholder-gateway-key"})+ os.environ.pop(OTelConstants.DO_NOT_TRACK_ENV_VAR_KEY, None)+ mocker.patch("pipelex.system.telemetry.telemetry_manager.Posthog")+ mocker.patch("posthog.default_client", None)+ mocker.patch("posthog.privacy_mode", False)diff --git a/tests/integration/pipelex/system/pipelex_service/test_setup_gateway_telemetry.py b/tests/integration/pipelex/system/pipelex_service/test_setup_gateway_telemetry.pynew file mode 100644index 000000000..207e6ca28--- /dev/null+++ b/tests/integration/pipelex/system/pipelex_service/test_setup_gateway_telemetry.py@@ -0,0 +1,85 @@+from __future__ import annotations++from typing import TYPE_CHECKING++import pytest++from pipelex import log+from pipelex.cogt.model_backends.backend import LEGACY_GATEWAY_MODEL_SPECS_SECTION, PipelexBackend+from pipelex.pipelex import Pipelex+from pipelex.system.pipelex_service.pipelex_service_agreement import (+ PipelexServiceAgreement,+ PipelexServiceOnboarding,+)+from pipelex.system.pipelex_service.pipelex_service_config import PipelexServiceConfig+from pipelex.system.runtime import IntegrationMode, runtime_manager++if TYPE_CHECKING:+ from collections.abc import Generator++ from pytest_mock import MockerFixture++RUNTIME_BOOT_MODULE = "pipelex.runtime_boot"++[email protected](scope="module", autouse=True)+def reset_pipelex_config_fixture() -> Generator[None, None, None]:+ """Override the global module fixture: each test boots with its own ``Pipelex.make``."""+ yield+ Pipelex.teardown_if_needed()+++def gateway_enabled_boot(mocker: MockerFixture) -> None:+ """A boot with `pipelex_gateway` enabled, the terms accepted, and the session's fresh remote config."""+ mocker.patch(+ f"{RUNTIME_BOOT_MODULE}.enabled_managed_gateway_sections",+ return_value={PipelexBackend.GATEWAY: LEGACY_GATEWAY_MODEL_SPECS_SECTION},+ )+ mocker.patch(+ f"{RUNTIME_BOOT_MODULE}.load_pipelex_service_config_if_exists",+ return_value=PipelexServiceConfig(+ agreement=PipelexServiceAgreement(terms_accepted=True),+ onboarding=PipelexServiceOnboarding(inference_setup_completed=True),+ ),+ )+ mocker.patch(+ "pipelex.system.runtime.RuntimeManager.is_in_codex_cloud",+ new_callable=mocker.PropertyMock,+ return_value=False,+ )+++def _boot_and_read_the_stream(*, integration_mode: IntegrationMode) -> bool:+ Pipelex.teardown_if_needed()+ try:+ telemetry_manager = Pipelex.make(integration_mode=integration_mode, needs_inference=True).telemetry_manager+ assert telemetry_manager is not None+ return telemetry_manager.is_pipelex_telemetry_enabled+ finally:+ Pipelex.teardown_if_needed()+ log.reset()++[email protected]("gateway_enabled_boot", "gateway_telemetry_reachable")+class TestSetupGatewayTelemetry:+ def test_a_deployment_boot_sends_the_stream(self, mocker: MockerFixture) -> None:+ """The production path, end to end: the one boot that must build the real manager with the Gateway stream on.++ The run mode the shared pytest plugin set is neutralised here, as a deployment has none. Nothing+ else in the suite builds this manager, so a crash in the factory's key lookup or in the+ manager's Gateway branch would otherwise reach every production boot with the Gateway enabled.+ """+ mocker.patch(f"{RUNTIME_BOOT_MODULE}.runtime_manager", is_unit_testing=False)+ assert _boot_and_read_the_stream(integration_mode=IntegrationMode.PYTHON)++ @pytest.mark.parametrize("integration_mode", [IntegrationMode.CI, IntegrationMode.PYTEST])+ def test_a_test_harness_mode_boot_keeps_the_stream_off(self, mocker: MockerFixture, integration_mode: IntegrationMode) -> None:+ """The integration mode alone turns the stream off, as for a harness that boots without the shared pytest plugin."""+ mocker.patch(f"{RUNTIME_BOOT_MODULE}.runtime_manager", is_unit_testing=False)+ assert not _boot_and_read_the_stream(integration_mode=integration_mode)++ def test_a_unit_testing_run_mode_boot_keeps_the_stream_off(self) -> None:+ """The run mode alone turns the stream off: a suite booting in the default `PYTHON` mode, as the shared plugin's recipe does."""+ assert runtime_manager.is_unit_testing, "the shared pytest plugin sets a test run mode for every session that loads it"+ assert not _boot_and_read_the_stream(integration_mode=IntegrationMode.PYTHON)diff --git a/tests/integration/pipelex/system/pipelex_service/test_setup_with_cache.py b/tests/integration/pipelex/system/pipelex_service/test_setup_with_cache.pyindex fe6726da3..b91122014 100644--- a/tests/integration/pipelex/system/pipelex_service/test_setup_with_cache.py+++ b/tests/integration/pipelex/system/pipelex_service/test_setup_with_cache.py@@ -43,7 +43,6 @@ from pipelex.system.pipelex_service.remote_config_fetcher import ()from pipelex.system.pipelex_service.types import RemoteConfigSourcefrom pipelex.system.runtime import IntegrationMode-from pipelex.system.telemetry.telemetry_manager_abstract import TelemetryManagerNoOpif TYPE_CHECKING:from collections.abc import Generator@@ -177,11 +176,16 @@ class TestSetupWithCache:Pipelex.teardown_if_needed()log.reset()+ @pytest.mark.usefixtures("gateway_telemetry_reachable")def test_telemetry_disabled_when_source_cached(self, mocker: MockerFixture) -> None:- """``Pipelex.make(needs_inference=True)`` with a cached gateway config → telemetry- manager is the no-op variant. Stale specs imply stale model identities; phoning home+ """``Pipelex.make(needs_inference=True)`` with a cached gateway config → the Gateway+ telemetry stream is off. Stale specs imply stale model identities; phoning homeabout pipe runs in that state would pollute metrics, so the guard is stricter thanthe plain ``needs_inference and gateway_enabled`` check.++ The boot is a deployment's: ``PYTHON`` mode with the test run mode neutralised, because+ either test-harness signal would turn the stream off by itself and leave the cache+ condition unguarded."""Pipelex.teardown_if_needed()@@ -208,14 +212,16 @@ class TestSetupWithCache:return_value=False,)mocker.patch.object(RemoteConfigFetcher, "fetch_remote_config", return_value=cached_result)+ mocker.patch(f"{RUNTIME_BOOT_MODULE}.runtime_manager", is_unit_testing=False)try:- pipelex_instance = Pipelex.make(- integration_mode=IntegrationMode.PYTEST,+ telemetry_manager = Pipelex.make(+ integration_mode=IntegrationMode.PYTHON,needs_inference=True,- )- assert isinstance(pipelex_instance.telemetry_manager, TelemetryManagerNoOp), (- "cached gateway config must downgrade telemetry to no-op, even when needs_inference=True"+ ).telemetry_manager+ assert telemetry_manager is not None+ assert not telemetry_manager.is_pipelex_telemetry_enabled, (+ "cached gateway config must turn the Gateway telemetry stream off, even when needs_inference=True")finally:Pipelex.teardown_if_needed()diff --git a/tests/unit/pipelex/system/telemetry/test_gateway_portkey_follows_stream.py b/tests/unit/pipelex/system/telemetry/test_gateway_portkey_follows_stream.pynew file mode 100644index 000000000..59ee2803a--- /dev/null+++ b/tests/unit/pipelex/system/telemetry/test_gateway_portkey_follows_stream.py@@ -0,0 +1,64 @@+import pytest++from pipelex.system.telemetry.otel_constants import OTelConstants+from pipelex.system.telemetry.telemetry_config import (+ PipelexGatewayTelemetryConfig,+ PortkeyConfig,+ PostHogConfig,+ PostHogMode,+ TelemetryConfig,+)+from pipelex.system.telemetry.telemetry_manager import TelemetryManager+++def _make_manager(*, is_stream_enabled: bool, force_flags: bool) -> TelemetryManager:+ """Assemble a manager whose custom stream is on, as a test run allows, bypassing the constructor's global side effects."""+ telemetry_config = TelemetryConfig(+ custom_posthog=PostHogConfig(mode=PostHogMode.ANONYMOUS, api_key="phc_test"),+ pipelex_gateway=PipelexGatewayTelemetryConfig(+ posthog=PostHogConfig(mode=PostHogMode.OFF),+ portkey=PortkeyConfig(force_debug_enabled=force_flags, force_tracing_enabled=force_flags),+ ),+ )+ manager = TelemetryManager.__new__(TelemetryManager)+ manager.telemetry_config = telemetry_config+ manager._pipelex_telemetry_enabled = is_stream_enabled # ruff: ignore[private-member-access] # pyright: ignore[reportPrivateUsage]+ return manager+++def without_do_not_track(monkeypatch: pytest.MonkeyPatch) -> None:+ """`DO_NOT_TRACK` turns both answers off by itself, which would hide what these tests are about."""+ monkeypatch.delenv(OTelConstants.DO_NOT_TRACK_ENV_VAR_KEY, raising=False)++[email protected]("without_do_not_track")+class TestGatewayPortkeyFollowsStream:+ @pytest.mark.parametrize("force_flags", [True, False])+ def test_backend_debug_does_not_log_while_the_stream_is_off(self, force_flags: bool) -> None:+ """A test run keeps the Gateway stream off, and the backend's `debug` must not reopen Portkey logging behind it."""+ manager = _make_manager(is_stream_enabled=False, force_flags=force_flags)++ assert manager.is_pipelex_gateway_portkey_logging_enabled(is_debug_configured=True) is False++ def test_force_debug_flag_does_not_log_while_the_stream_is_off(self) -> None:+ manager = _make_manager(is_stream_enabled=False, force_flags=True)++ assert manager.is_pipelex_gateway_portkey_logging_enabled(is_debug_configured=False) is False++ def test_force_tracing_flag_does_not_trace_while_the_stream_is_off(self) -> None:+ manager = _make_manager(is_stream_enabled=False, force_flags=True)++ assert manager.is_pipelex_gateway_portkey_tracing_enabled() is False++ def test_the_flags_still_apply_while_the_stream_is_on(self) -> None:+ manager = _make_manager(is_stream_enabled=True, force_flags=True)++ assert manager.is_pipelex_gateway_portkey_logging_enabled(is_debug_configured=False) is True+ assert manager.is_pipelex_gateway_portkey_tracing_enabled() is True++ def test_backend_debug_still_applies_while_the_stream_is_on(self) -> None:+ manager = _make_manager(is_stream_enabled=True, force_flags=False)++ assert manager.is_pipelex_gateway_portkey_logging_enabled(is_debug_configured=True) is True+ assert manager.is_pipelex_gateway_portkey_tracing_enabled() is Falsediff --git a/tests/unit/pipelex/system/test_integration_mode.py b/tests/unit/pipelex/system/test_integration_mode.pynew file mode 100644index 000000000..5602584b6--- /dev/null+++ b/tests/unit/pipelex/system/test_integration_mode.py@@ -0,0 +1,29 @@+import pytest++from pipelex.system.runtime import IntegrationMode++IS_TEST_HARNESS_BY_MODE: dict[IntegrationMode, bool] = {+ IntegrationMode.CI: True,+ IntegrationMode.CLI: False,+ IntegrationMode.DOCKER: False,+ IntegrationMode.FASTAPI: False,+ IntegrationMode.MCP: False,+ IntegrationMode.N8N: False,+ IntegrationMode.PYTEST: True,+ IntegrationMode.PYTHON: False,+}+++class TestIntegrationModeIsTestHarness:+ @pytest.mark.parametrize(("integration_mode", "expected"), list(IS_TEST_HARNESS_BY_MODE.items()))+ def test_is_test_harness(self, integration_mode: IntegrationMode, expected: bool) -> None:+ """The boot keeps the Gateway telemetry stream off in a harness mode, so each mode's answer is a telemetry decision.++ A deployment mode classified as a harness would silence real usage; a harness mode classified as+ a deployment would send test runs to the production analytics project.+ """+ assert integration_mode.is_test_harness is expected++ def test_the_table_covers_every_mode(self) -> None:+ """A mode added later fails here until someone decides whether it is a test harness."""+ assert set(IS_TEST_HARNESS_BY_MODE) == set(IntegrationMode)diff --git a/tests/unit/pipelex/test_runtime_boot_pipelex_telemetry.py b/tests/unit/pipelex/test_runtime_boot_pipelex_telemetry.pynew file mode 100644index 000000000..cf15a0bec--- /dev/null+++ b/tests/unit/pipelex/test_runtime_boot_pipelex_telemetry.py@@ -0,0 +1,76 @@+import pytest++from pipelex.runtime_boot import RuntimeBoot+from pipelex.runtime_hub import get_telemetry_manager+from pipelex.system.runtime import IntegrationMode++# Spelled out rather than derived from `is_test_harness`, so this table does not lean on the property it+# depends on; `test_integration_mode.py` pins the property itself.+HARNESS_MODES = [IntegrationMode.CI, IntegrationMode.PYTEST]+DEPLOYMENT_MODES = [mode for mode in IntegrationMode if mode not in HARNESS_MODES]+++class TestRuntimeBootPipelexTelemetry:+ @pytest.mark.parametrize("integration_mode", DEPLOYMENT_MODES)+ def test_a_deployment_mode_with_every_condition_met_enables_the_stream(self, integration_mode: IntegrationMode) -> None:+ assert RuntimeBoot.should_enable_pipelex_telemetry(+ integration_mode=integration_mode,+ is_unit_testing=False,+ is_gateway_enabled=True,+ needs_inference=True,+ is_gateway_config_cached=False,+ )++ @pytest.mark.parametrize(+ ("is_unit_testing", "is_gateway_enabled", "needs_inference", "is_gateway_config_cached"),+ [+ pytest.param(True, True, True, False, id="unit-testing-run-mode"),+ pytest.param(False, False, True, False, id="gateway-disabled"),+ pytest.param(False, True, False, False, id="no-inference"),+ pytest.param(False, True, True, True, id="cached-config"),+ ],+ )+ def test_each_condition_alone_disables_the_stream(+ self,+ is_unit_testing: bool,+ is_gateway_enabled: bool,+ needs_inference: bool,+ is_gateway_config_cached: bool,+ ) -> None:+ """The `unit-testing-run-mode` row is the suite that boots in the default `PYTHON` mode, as the shared plugin's recipe does."""+ assert not RuntimeBoot.should_enable_pipelex_telemetry(+ integration_mode=IntegrationMode.PYTHON,+ is_unit_testing=is_unit_testing,+ is_gateway_enabled=is_gateway_enabled,+ needs_inference=needs_inference,+ is_gateway_config_cached=is_gateway_config_cached,+ )++ @pytest.mark.parametrize("integration_mode", HARNESS_MODES)+ def test_a_test_harness_mode_disables_the_stream_even_with_every_other_condition_met(self, integration_mode: IntegrationMode) -> None:+ """Test runs are not usage, and their fixture user ids must not become persons in the production project.++ Before this condition, every pytest run with `pipelex_gateway` enabled and a fresh remote config+ sent its pipe runs to the Gateway's production analytics project, because the stream's enabling+ condition never read the integration mode. The run mode is left false here: a harness that boots+ without the shared pytest plugin, such as the pipelex-js conformance scripts, has only its mode.+ """+ assert not RuntimeBoot.should_enable_pipelex_telemetry(+ integration_mode=integration_mode,+ is_unit_testing=False,+ is_gateway_enabled=True,+ needs_inference=True,+ is_gateway_config_cached=False,+ )++ def test_the_suite_boot_has_the_gateway_stream_off(self) -> None:+ """The boot this suite runs under must not send the Gateway stream: the bug's symptom, stated directly.++ This passes trivially on a machine where `pipelex_gateway` is disabled, or where the remote+ config came from the cache, because the stream is off there for those reasons already; the+ table tests above and the setup-level tests in `test_setup_gateway_telemetry.py` are the precise+ guards. It asserts on the Gateway stream alone, not on the manager being a no-op, because a+ developer may deliberately allow the custom stream under `pytest` in their own+ `telemetry_allowed_modes`.+ """+ assert not get_telemetry_manager().is_pipelex_telemetry_enableddiff --git a/tests/unit/pipelex/test_shared_pytest_plugin_run_mode.py b/tests/unit/pipelex/test_shared_pytest_plugin_run_mode.pynew file mode 100644index 000000000..8a6349467--- /dev/null+++ b/tests/unit/pipelex/test_shared_pytest_plugin_run_mode.py@@ -0,0 +1,11 @@+from pipelex.system.runtime import runtime_manager++# Read while pytest imports this module to collect it, before any fixture has run: a boot here, such as a+# top-level `Pipelex.make()`, is exactly what the run mode must already cover.+IS_UNIT_TESTING_AT_COLLECTION = runtime_manager.is_unit_testing+++class TestSharedPytestPluginRunMode:+ def test_the_run_mode_is_a_test_mode_before_any_fixture_runs(self) -> None:+ """The shared plugin sets the run mode at configure time, so a boot during collection keeps the Gateway stream off."""+ assert IS_UNIT_TESTING_AT_COLLECTION is True