Skip to content

Build AI methods with your coding agent, then run them anywhere: as an MCP for chatbots, as a webapp for people, or via API for your software.

feature/Test-runs-no-telemetry · L-260924-d570e6 — test runs feed prod PostHog (#1245)

Louis Choquel1a21e820917a
diff --git a/CHANGELOG.md b/CHANGELOG.md
index a21dcbff5..84aabbb26 100644
--- a/CHANGELOG.md
+++ b/CHANGELOG.md
@@ -8,6 +8,10 @@
- **The documentation site leads with the hosted route**: its getting-started opens on a new **Quick Start** page — sign up at app.pipelex.com, the Pipelex plugin in Claude Code or Codex, the Pipelex MCP in ChatGPT or Claude, then a webapp from the method-app template or the hosted API over HTTP, TypeScript or Python — and the runtime's own install moves to **Run It Yourself**. **Build with Claude Code** and its `npm install -g mthds` route are retired, and **Claude Code Skills Plugin** becomes **Pipelex Plugin**, which documents the Pipelex plugin instead of the MTHDS plugin; the old addresses redirect. The front page's buttons and the site's description lead with what Pipelex lets you do.
- **`mthds` version**: The pin moves to `mthds==0.16.0`, the release adding a `User-Agent` on every `MthdsAPIClient` request and the `mthds.version` module. Nothing in pipelex uses either, and the MTHDS standard version stays at `2.0.0`, but the pin is exact, so everyone downstream inherits `mthds` 0.16.0 and a consumer depending on `mthds` directly must move in step.
+### Fixed
+
+- **Test runs no longer send the Pipelex Gateway telemetry stream**: a runtime booted in the `pytest` or `ci` integration mode, or in a session that loads the shared pytest plugin, which sets a test run mode when pytest configures, keeps the Gateway stream off even with `pipelex_gateway` enabled, so a project's test suite no longer reports its runs and its fixture user ids to Pipelex. In a test run a boot no longer needs `PIPELEX_GATEWAY_API_KEY` for telemetry, `DO_NOT_TRACK` no longer conflicts with the Gateway, and the Gateway's Portkey debug and tracing flags have no effect, even where the custom stream is allowed; `IntegrationMode.is_test_harness` names the two modes. The plugin's session fixture `set_run_mode` is gone: the run mode is set in its `pytest_configure`, so a boot at test-module import or during collection is covered too.
+
## [v0.64.0] - 2026-09-24
### Added
diff --git a/docs/configuration/config-practical/telemetry-config.md b/docs/configuration/config-practical/telemetry-config.md
index 7db76f1aa..04806a1b7 100644
--- a/docs/configuration/config-practical/telemetry-config.md
+++ b/docs/configuration/config-practical/telemetry-config.md
@@ -249,7 +249,7 @@ set DO_NOT_TRACK=1
When set, this disables:
- All custom telemetry (PostHog, Langfuse, OTLP)
-- Gateway telemetry (note: Gateway won't work without telemetry)
+- Gateway telemetry (note: outside a test run, Gateway won't work without telemetry — see [Telemetry Setup](../../setup/telemetry.md#1-gateway-telemetry-pipelex-controlled))
---
diff --git a/docs/features/telemetry.md b/docs/features/telemetry.md
index de5605723..d6a26d6c8 100644
--- a/docs/features/telemetry.md
+++ b/docs/features/telemetry.md
@@ -29,7 +29,7 @@ A span or an event produced during a run is attributed to that run's own caller,
## Gateway Telemetry
-When using Pipelex Gateway as your inference backend, privacy-respecting metrics are automatically collected: models used, token counts, latency, and error rates. This data is tied to your Gateway API key (hashed for security) and requires no additional configuration.
+When using Pipelex Gateway as your inference backend, privacy-respecting metrics are automatically collected: models used, token counts, latency, and error rates. This data is tied to your Gateway API key (hashed for security) and requires no additional configuration. Test runs are not collected: when the runtime boots in the `pytest` or `ci` integration mode, or in a session that loads Pipelex's shared pytest plugin, the Gateway stream stays off.
## Privacy Controls
diff --git a/docs/setup/telemetry.md b/docs/setup/telemetry.md
index 65e8aabc3..688100983 100644
--- a/docs/setup/telemetry.md
+++ b/docs/setup/telemetry.md
@@ -12,6 +12,8 @@ Pipelex supports two independent telemetry streams that serve different purposes
When you use **Pipelex Gateway** as your inference backend, identified telemetry is **automatically enabled**. This telemetry is tied to your Gateway API key (hashed for security) and operates independently from your `telemetry.toml` settings.
+Test runs are the exception, and either of two signals marks one. The first is the `pytest` or `ci` integration mode. The second is a test run mode, which Pipelex's shared pytest plugin (`pipelex.test_extras.shared_pytest_plugins`) sets for every session that loads it, when pytest configures and before it collects anything, so a suite booting with `Pipelex.make()` in the default mode is covered too, whether it boots in a fixture, at test-module import or in a pytest hook that runs after `pytest_configure`. What it cannot reach is anything pytest runs before it configures: the body of every `conftest.py` loaded at startup, which means the ones along the paths given on the command line or in `testpaths` and not only the one that loads the plugin, and the hooks that run before `pytest_configure`, such as `pytest_addoption` or `pytest_cmdline_main`. Boot in a fixture there. In a test run the Gateway stream stays off, so a project's test suite sends nothing to Pipelex even with the Gateway enabled: the boot does not look for `PIPELEX_GATEWAY_API_KEY` on the stream's behalf, and `DO_NOT_TRACK` does not conflict with the Gateway. The Gateway's Portkey request logging and trace correlation follow the stream, so the `debug` setting on the `pipelex_gateway` backend and the `[pipelex_gateway.portkey]` force flags in `telemetry.toml` have no effect in a test run either, even where your own custom stream is allowed in that mode.
+
A run that names a caller of its own is attributed to that caller on this stream too: the run's `user_id` is sent as the `distinct_id` exactly as you supplied it, and its `extras` ride the capture as groups. Anything that names no caller reports under your key's hash.
**What we collect:**
@@ -98,11 +100,11 @@ export DO_NOT_TRACK=1
When set, this disables:
-- Gateway telemetry (but note: Gateway won't work without telemetry)
+- Gateway telemetry (but note: outside a test run, Gateway won't work without telemetry)
- All custom telemetry destinations
!!! warning "Gateway Requires Telemetry"
- If you set `DO_NOT_TRACK=1` while using Pipelex Gateway, the Gateway will not function. Use direct provider backends instead if you need to disable all telemetry.
+ If you set `DO_NOT_TRACK=1` while using Pipelex Gateway, the Gateway will not function, except in a test run, where the Gateway stream is off anyway. Use direct provider backends instead if you need to disable all telemetry.
## Attaching your own labels to a run
diff --git a/pipelex/runtime_boot.py b/pipelex/runtime_boot.py
index aa0b5c0c3..048d87921 100644
--- a/pipelex/runtime_boot.py
+++ b/pipelex/runtime_boot.py
@@ -340,6 +340,56 @@ Note that this command resets all config files to their default values.
If you need help, drop by our Discord: we're happy to assist: {URLs.discord}.
"""
+ @classmethod
+ def should_enable_pipelex_telemetry(
+ cls,
+ *,
+ integration_mode: IntegrationMode,
+ is_unit_testing: bool,
+ is_gateway_enabled: bool,
+ needs_inference: bool,
+ is_gateway_config_cached: bool,
+ ) -> bool:
+ """Decide whether this boot sends the Pipelex Gateway telemetry stream.
+
+ The stream is disabled when:
+
+ - the legacy gateway backend is not enabled, OR
+ - inference is not needed (no live runs to track), OR
+ - the gateway config came from the cache (stale specs imply potentially stale model
+ identities; phoning home about pipe runs in that state would pollute metrics), OR
+ - the runtime is booted by a test harness: its runs are not usage, and its fixture user ids must
+ not become persons in the production analytics project. Two signals say so, and either one is
+ enough. The integration mode (`CI` or `PYTEST`) is what a harness booting without our pytest
+ plugin states, such as the pipelex-js conformance scripts. The run mode is what the shared
+ pytest plugin sets for every session that loads it, which covers the suites that boot in the
+ default `PYTHON` mode, as the plugin's own recipe does.
+
+ **The first condition asks about `pipelex_gateway` specifically, not about managed backends
+ in general**, and the distinct id is why: it is derived from `PIPELEX_GATEWAY_API_KEY`, which
+ a manifold backend neither has nor can stand in for — its own key is, for the private beta,
+ one token shared by every participant, so keying on it would produce a single indistinguishable
+ user rather than an identity. Asked the general way, a manifold-only installation would be
+ required to hold a gateway key it has no other use for and would fail to boot without one.
+ The common beta case is unaffected: a participant who keeps `pipelex_gateway` enabled has a
+ real gateway key, and their manifold runs are tracked under it like everything else.
+
+ The per-mode `telemetry_allowed_modes` table does not take part: it governs only the
+ operator's custom stream, which the telemetry factory gates by itself.
+
+ Args:
+ integration_mode: The mode the runtime is booted in.
+ is_unit_testing: Whether the run mode is a test mode, as `RuntimeManager.is_unit_testing` says.
+ is_gateway_enabled: Whether the `pipelex_gateway` backend is enabled.
+ needs_inference: Whether this boot runs live inference.
+ is_gateway_config_cached: Whether the gateway config came from the cache rather than a fresh fetch.
+
+ Returns:
+ True when the Gateway telemetry stream should be sent.
+ """
+ is_test_harness = integration_mode.is_test_harness or is_unit_testing
+ return is_gateway_enabled and needs_inference and not is_gateway_config_cached and not is_test_harness
+
def setup(
self,
*,
@@ -506,28 +556,22 @@ If you need help, drop by our Discord: we're happy to assist: {URLs.discord}.
secrets_config = get_config().runtime.secrets
secrets_provider = secrets_provider_registry.get_required(method=secrets_config.method)(secrets_config)
- # Disable Pipelex telemetry when:
- # - the legacy gateway backend is not enabled, OR
- # - inference is not needed (no live runs to track), OR
- # - the gateway config came from the cache (stale specs imply potentially stale model
- # identities; phoning home about pipe runs in that state would pollute metrics).
+ # Whether the Pipelex Gateway telemetry stream is sent: the conditions, and why each one is
+ # there, are in `should_enable_pipelex_telemetry`.
#
- # **The first condition asks about `pipelex_gateway` specifically, not about managed backends
- # in general**, and the distinct id is why: it is derived from `PIPELEX_GATEWAY_API_KEY`, which
- # a manifold backend neither has nor can stand in for — its own key is, for the private beta,
- # one token shared by every participant, so keying on it would produce a single indistinguishable
- # user rather than an identity. Asked the general way, a manifold-only installation would be
- # required to hold a gateway key it has no other use for and would fail to boot without one.
- # The common beta case is unaffected: a participant who keeps `pipelex_gateway` enabled has a
- # real gateway key, and their manifold runs are tracked under it like everything else.
- #
- # Read off the mapping computed above rather than by asking the document a second time. The
- # two answers are the same one — the gateway resolves to a section whenever it is enabled —
- # and a second read would be a second chance for an unparseable file to escape the clause
- # that frames it as a backend-library refusal.
+ # The gateway's enablement is read off the mapping computed above rather than by asking the
+ # document a second time. The two answers are the same one — the gateway resolves to a section
+ # whenever it is enabled — and a second read would be a second chance for an unparseable file
+ # to escape the clause that frames it as a backend-library refusal.
gateway_source_is_cached = gateway_config_source is not None and gateway_config_source.is_cached
is_gateway_enabled = PipelexBackend.GATEWAY in managed_gateway_sections
- is_pipelex_telemetry_enabled = is_gateway_enabled and needs_inference and not gateway_source_is_cached
+ is_pipelex_telemetry_enabled = self.should_enable_pipelex_telemetry(
+ integration_mode=integration_mode,
+ is_unit_testing=runtime_manager.is_unit_testing,
+ is_gateway_enabled=is_gateway_enabled,
+ needs_inference=needs_inference,
+ is_gateway_config_cached=gateway_source_is_cached,
+ )
self.telemetry_manager = TelemetryFactory.make_telemetry_manager(
secrets_provider=secrets_provider,
integration_mode=integration_mode,
diff --git a/pipelex/system/runtime.py b/pipelex/system/runtime.py
index d8840080a..bb7a8f63a 100644
--- a/pipelex/system/runtime.py
+++ b/pipelex/system/runtime.py
@@ -39,6 +39,26 @@ class IntegrationMode(StrEnum):
case IntegrationMode.PYTHON:
return True
+ @property
+ def is_test_harness(self) -> bool:
+ """Whether this is a mode only test suites and conformance scripts boot in.
+
+ The boot keeps the Pipelex Gateway telemetry stream off in these modes, so a test run never
+ reports to the production analytics project.
+ """
+ match self:
+ case IntegrationMode.CI | IntegrationMode.PYTEST:
+ return True
+ case (
+ IntegrationMode.CLI
+ | IntegrationMode.DOCKER
+ | IntegrationMode.FASTAPI
+ | IntegrationMode.MCP
+ | IntegrationMode.N8N
+ | IntegrationMode.PYTHON
+ ):
+ return False
+
class RunMode(StrEnum):
NORMAL = "normal"
diff --git a/pipelex/system/telemetry/telemetry_manager.py b/pipelex/system/telemetry/telemetry_manager.py
index 78f3a6222..da96e6580 100644
--- a/pipelex/system/telemetry/telemetry_manager.py
+++ b/pipelex/system/telemetry/telemetry_manager.py
@@ -560,6 +560,10 @@ class TelemetryManager(TelemetryManagerAbstract):
@override
def is_pipelex_gateway_portkey_logging_enabled(self, *, is_debug_configured: bool) -> bool:
+ # The Gateway's Portkey logging follows the Gateway stream: where the stream is off, a test run
+ # above all, neither the backend's `debug` nor the force flag may turn it on.
+ if not self._pipelex_telemetry_enabled:
+ return False
is_debug: bool = is_debug_configured
if (
not is_debug
@@ -581,6 +585,9 @@ class TelemetryManager(TelemetryManagerAbstract):
@override
def is_pipelex_gateway_portkey_tracing_enabled(self) -> bool:
+ # Trace correlation follows the Gateway stream for the same reason as the logging above.
+ if not self._pipelex_telemetry_enabled:
+ return False
if (
self.telemetry_config.pipelex_gateway
and self.telemetry_config.pipelex_gateway.portkey
diff --git a/pipelex/test_extras/shared_pytest_plugins.py b/pipelex/test_extras/shared_pytest_plugins.py
index cf9e8b5c3..03ee1f58d 100644
--- a/pipelex/test_extras/shared_pytest_plugins.py
+++ b/pipelex/test_extras/shared_pytest_plugins.py
@@ -55,12 +55,11 @@ ENV_VAR_KEYS_WHICH_MAY_NEED_PLACEHOLDERS_IN_CI = [
]
[email protected](scope="session", autouse=True)
-def set_run_mode():
+def _set_test_run_mode() -> None:
if is_env_var_set(key="GITHUB_ACTIONS") or is_env_var_set(key="CI"):
runtime_manager.set_run_mode(run_mode=RunMode.CI_TEST)
elif is_env_var_truthy(key=CODEX_CLOUD_ENV_VAR_KEY):
- # we're in codex cloud and this fixture is called by pytest, so we are testing in codex cloud
+ # we're in codex cloud and pytest is running, so we are testing in codex cloud
runtime_manager.set_run_mode(run_mode=RunMode.CODEX_CLOUD_TEST)
else:
runtime_manager.set_run_mode(run_mode=RunMode.UNIT_TEST)
@@ -90,12 +89,20 @@ def pytest_addoption(parser: Parser):
)
[email protected](tryfirst=True)
def pytest_configure(config: Config) -> None:
- """Check prerequisites before test collection starts.
+ """Set the test run mode, then check prerequisites before test collection starts.
+
+ The run mode is set here, first among the configure hooks, rather than in a fixture, because
+ it is what keeps the Pipelex Gateway telemetry stream off in a test run: a boot at test-module
+ import, during collection or in a later pytest hook happens before any fixture runs, and would
+ otherwise see the normal run mode and send the stream.
- Validates that Pipelex Gateway terms are accepted when gateway is enabled.
+ Then validates that Pipelex Gateway terms are accepted when gateway is enabled.
This runs early to provide clear feedback before wasting time on test collection.
"""
+ _set_test_run_mode()
+
# Skip check when inference is disabled via CLI option
if config.getoption("--disable-inference", default=False):
return
diff --git a/tests/integration/pipelex/system/pipelex_service/conftest.py b/tests/integration/pipelex/system/pipelex_service/conftest.py
new file mode 100644
index 000000000..349e31e4d
--- /dev/null
+++ b/tests/integration/pipelex/system/pipelex_service/conftest.py
@@ -0,0 +1,23 @@
+import os
+
+import pytest
+from pytest_mock import MockerFixture
+
+from pipelex.system.pipelex_service.pipelex_details import PipelexDetails
+from pipelex.system.telemetry.otel_constants import OTelConstants
+
+
+def gateway_telemetry_reachable(mocker: MockerFixture) -> None:
+ """Clear what would stop a boot from building the Gateway telemetry stream, and send nothing.
+
+ A placeholder gateway key and no `DO_NOT_TRACK`, so a boot that turns the stream on builds the real
+ manager instead of refusing, which makes a test's assertion on the stream the thing that fails. PostHog
+ is a mock, so that manager reaches no network, and the PostHog module globals it sets are restored
+ after the test.
+ """
+ mocker.patch.dict(os.environ, {PipelexDetails.PIPELEX_GATEWAY_API_KEY_VAR: "placeholder-gateway-key"})
+ os.environ.pop(OTelConstants.DO_NOT_TRACK_ENV_VAR_KEY, None)
+ mocker.patch("pipelex.system.telemetry.telemetry_manager.Posthog")
+ mocker.patch("posthog.default_client", None)
+ mocker.patch("posthog.privacy_mode", False)
diff --git a/tests/integration/pipelex/system/pipelex_service/test_setup_gateway_telemetry.py b/tests/integration/pipelex/system/pipelex_service/test_setup_gateway_telemetry.py
new file mode 100644
index 000000000..207e6ca28
--- /dev/null
+++ b/tests/integration/pipelex/system/pipelex_service/test_setup_gateway_telemetry.py
@@ -0,0 +1,85 @@
+from __future__ import annotations
+
+from typing import TYPE_CHECKING
+
+import pytest
+
+from pipelex import log
+from pipelex.cogt.model_backends.backend import LEGACY_GATEWAY_MODEL_SPECS_SECTION, PipelexBackend
+from pipelex.pipelex import Pipelex
+from pipelex.system.pipelex_service.pipelex_service_agreement import (
+ PipelexServiceAgreement,
+ PipelexServiceOnboarding,
+)
+from pipelex.system.pipelex_service.pipelex_service_config import PipelexServiceConfig
+from pipelex.system.runtime import IntegrationMode, runtime_manager
+
+if TYPE_CHECKING:
+ from collections.abc import Generator
+
+ from pytest_mock import MockerFixture
+
+RUNTIME_BOOT_MODULE = "pipelex.runtime_boot"
+
+
[email protected](scope="module", autouse=True)
+def reset_pipelex_config_fixture() -> Generator[None, None, None]:
+ """Override the global module fixture: each test boots with its own ``Pipelex.make``."""
+ yield
+ Pipelex.teardown_if_needed()
+
+
+def gateway_enabled_boot(mocker: MockerFixture) -> None:
+ """A boot with `pipelex_gateway` enabled, the terms accepted, and the session's fresh remote config."""
+ mocker.patch(
+ f"{RUNTIME_BOOT_MODULE}.enabled_managed_gateway_sections",
+ return_value={PipelexBackend.GATEWAY: LEGACY_GATEWAY_MODEL_SPECS_SECTION},
+ )
+ mocker.patch(
+ f"{RUNTIME_BOOT_MODULE}.load_pipelex_service_config_if_exists",
+ return_value=PipelexServiceConfig(
+ agreement=PipelexServiceAgreement(terms_accepted=True),
+ onboarding=PipelexServiceOnboarding(inference_setup_completed=True),
+ ),
+ )
+ mocker.patch(
+ "pipelex.system.runtime.RuntimeManager.is_in_codex_cloud",
+ new_callable=mocker.PropertyMock,
+ return_value=False,
+ )
+
+
+def _boot_and_read_the_stream(*, integration_mode: IntegrationMode) -> bool:
+ Pipelex.teardown_if_needed()
+ try:
+ telemetry_manager = Pipelex.make(integration_mode=integration_mode, needs_inference=True).telemetry_manager
+ assert telemetry_manager is not None
+ return telemetry_manager.is_pipelex_telemetry_enabled
+ finally:
+ Pipelex.teardown_if_needed()
+ log.reset()
+
+
[email protected]("gateway_enabled_boot", "gateway_telemetry_reachable")
+class TestSetupGatewayTelemetry:
+ def test_a_deployment_boot_sends_the_stream(self, mocker: MockerFixture) -> None:
+ """The production path, end to end: the one boot that must build the real manager with the Gateway stream on.
+
+ The run mode the shared pytest plugin set is neutralised here, as a deployment has none. Nothing
+ else in the suite builds this manager, so a crash in the factory's key lookup or in the
+ manager's Gateway branch would otherwise reach every production boot with the Gateway enabled.
+ """
+ mocker.patch(f"{RUNTIME_BOOT_MODULE}.runtime_manager", is_unit_testing=False)
+ assert _boot_and_read_the_stream(integration_mode=IntegrationMode.PYTHON)
+
+ @pytest.mark.parametrize("integration_mode", [IntegrationMode.CI, IntegrationMode.PYTEST])
+ def test_a_test_harness_mode_boot_keeps_the_stream_off(self, mocker: MockerFixture, integration_mode: IntegrationMode) -> None:
+ """The integration mode alone turns the stream off, as for a harness that boots without the shared pytest plugin."""
+ mocker.patch(f"{RUNTIME_BOOT_MODULE}.runtime_manager", is_unit_testing=False)
+ assert not _boot_and_read_the_stream(integration_mode=integration_mode)
+
+ def test_a_unit_testing_run_mode_boot_keeps_the_stream_off(self) -> None:
+ """The run mode alone turns the stream off: a suite booting in the default `PYTHON` mode, as the shared plugin's recipe does."""
+ assert runtime_manager.is_unit_testing, "the shared pytest plugin sets a test run mode for every session that loads it"
+ assert not _boot_and_read_the_stream(integration_mode=IntegrationMode.PYTHON)
diff --git a/tests/integration/pipelex/system/pipelex_service/test_setup_with_cache.py b/tests/integration/pipelex/system/pipelex_service/test_setup_with_cache.py
index fe6726da3..b91122014 100644
--- a/tests/integration/pipelex/system/pipelex_service/test_setup_with_cache.py
+++ b/tests/integration/pipelex/system/pipelex_service/test_setup_with_cache.py
@@ -43,7 +43,6 @@ from pipelex.system.pipelex_service.remote_config_fetcher import (
)
from pipelex.system.pipelex_service.types import RemoteConfigSource
from pipelex.system.runtime import IntegrationMode
-from pipelex.system.telemetry.telemetry_manager_abstract import TelemetryManagerNoOp
if TYPE_CHECKING:
from collections.abc import Generator
@@ -177,11 +176,16 @@ class TestSetupWithCache:
Pipelex.teardown_if_needed()
log.reset()
+ @pytest.mark.usefixtures("gateway_telemetry_reachable")
def test_telemetry_disabled_when_source_cached(self, mocker: MockerFixture) -> None:
- """``Pipelex.make(needs_inference=True)`` with a cached gateway config → telemetry
- manager is the no-op variant. Stale specs imply stale model identities; phoning home
+ """``Pipelex.make(needs_inference=True)`` with a cached gateway config → the Gateway
+ telemetry stream is off. Stale specs imply stale model identities; phoning home
about pipe runs in that state would pollute metrics, so the guard is stricter than
the plain ``needs_inference and gateway_enabled`` check.
+
+ The boot is a deployment's: ``PYTHON`` mode with the test run mode neutralised, because
+ either test-harness signal would turn the stream off by itself and leave the cache
+ condition unguarded.
"""
Pipelex.teardown_if_needed()
@@ -208,14 +212,16 @@ class TestSetupWithCache:
return_value=False,
)
mocker.patch.object(RemoteConfigFetcher, "fetch_remote_config", return_value=cached_result)
+ mocker.patch(f"{RUNTIME_BOOT_MODULE}.runtime_manager", is_unit_testing=False)
try:
- pipelex_instance = Pipelex.make(
- integration_mode=IntegrationMode.PYTEST,
+ telemetry_manager = Pipelex.make(
+ integration_mode=IntegrationMode.PYTHON,
needs_inference=True,
- )
- assert isinstance(pipelex_instance.telemetry_manager, TelemetryManagerNoOp), (
- "cached gateway config must downgrade telemetry to no-op, even when needs_inference=True"
+ ).telemetry_manager
+ assert telemetry_manager is not None
+ assert not telemetry_manager.is_pipelex_telemetry_enabled, (
+ "cached gateway config must turn the Gateway telemetry stream off, even when needs_inference=True"
)
finally:
Pipelex.teardown_if_needed()
diff --git a/tests/unit/pipelex/system/telemetry/test_gateway_portkey_follows_stream.py b/tests/unit/pipelex/system/telemetry/test_gateway_portkey_follows_stream.py
new file mode 100644
index 000000000..59ee2803a
--- /dev/null
+++ b/tests/unit/pipelex/system/telemetry/test_gateway_portkey_follows_stream.py
@@ -0,0 +1,64 @@
+import pytest
+
+from pipelex.system.telemetry.otel_constants import OTelConstants
+from pipelex.system.telemetry.telemetry_config import (
+ PipelexGatewayTelemetryConfig,
+ PortkeyConfig,
+ PostHogConfig,
+ PostHogMode,
+ TelemetryConfig,
+)
+from pipelex.system.telemetry.telemetry_manager import TelemetryManager
+
+
+def _make_manager(*, is_stream_enabled: bool, force_flags: bool) -> TelemetryManager:
+ """Assemble a manager whose custom stream is on, as a test run allows, bypassing the constructor's global side effects."""
+ telemetry_config = TelemetryConfig(
+ custom_posthog=PostHogConfig(mode=PostHogMode.ANONYMOUS, api_key="phc_test"),
+ pipelex_gateway=PipelexGatewayTelemetryConfig(
+ posthog=PostHogConfig(mode=PostHogMode.OFF),
+ portkey=PortkeyConfig(force_debug_enabled=force_flags, force_tracing_enabled=force_flags),
+ ),
+ )
+ manager = TelemetryManager.__new__(TelemetryManager)
+ manager.telemetry_config = telemetry_config
+ manager._pipelex_telemetry_enabled = is_stream_enabled # ruff: ignore[private-member-access] # pyright: ignore[reportPrivateUsage]
+ return manager
+
+
+def without_do_not_track(monkeypatch: pytest.MonkeyPatch) -> None:
+ """`DO_NOT_TRACK` turns both answers off by itself, which would hide what these tests are about."""
+ monkeypatch.delenv(OTelConstants.DO_NOT_TRACK_ENV_VAR_KEY, raising=False)
+
+
[email protected]("without_do_not_track")
+class TestGatewayPortkeyFollowsStream:
+ @pytest.mark.parametrize("force_flags", [True, False])
+ def test_backend_debug_does_not_log_while_the_stream_is_off(self, force_flags: bool) -> None:
+ """A test run keeps the Gateway stream off, and the backend's `debug` must not reopen Portkey logging behind it."""
+ manager = _make_manager(is_stream_enabled=False, force_flags=force_flags)
+
+ assert manager.is_pipelex_gateway_portkey_logging_enabled(is_debug_configured=True) is False
+
+ def test_force_debug_flag_does_not_log_while_the_stream_is_off(self) -> None:
+ manager = _make_manager(is_stream_enabled=False, force_flags=True)
+
+ assert manager.is_pipelex_gateway_portkey_logging_enabled(is_debug_configured=False) is False
+
+ def test_force_tracing_flag_does_not_trace_while_the_stream_is_off(self) -> None:
+ manager = _make_manager(is_stream_enabled=False, force_flags=True)
+
+ assert manager.is_pipelex_gateway_portkey_tracing_enabled() is False
+
+ def test_the_flags_still_apply_while_the_stream_is_on(self) -> None:
+ manager = _make_manager(is_stream_enabled=True, force_flags=True)
+
+ assert manager.is_pipelex_gateway_portkey_logging_enabled(is_debug_configured=False) is True
+ assert manager.is_pipelex_gateway_portkey_tracing_enabled() is True
+
+ def test_backend_debug_still_applies_while_the_stream_is_on(self) -> None:
+ manager = _make_manager(is_stream_enabled=True, force_flags=False)
+
+ assert manager.is_pipelex_gateway_portkey_logging_enabled(is_debug_configured=True) is True
+ assert manager.is_pipelex_gateway_portkey_tracing_enabled() is False
diff --git a/tests/unit/pipelex/system/test_integration_mode.py b/tests/unit/pipelex/system/test_integration_mode.py
new file mode 100644
index 000000000..5602584b6
--- /dev/null
+++ b/tests/unit/pipelex/system/test_integration_mode.py
@@ -0,0 +1,29 @@
+import pytest
+
+from pipelex.system.runtime import IntegrationMode
+
+IS_TEST_HARNESS_BY_MODE: dict[IntegrationMode, bool] = {
+ IntegrationMode.CI: True,
+ IntegrationMode.CLI: False,
+ IntegrationMode.DOCKER: False,
+ IntegrationMode.FASTAPI: False,
+ IntegrationMode.MCP: False,
+ IntegrationMode.N8N: False,
+ IntegrationMode.PYTEST: True,
+ IntegrationMode.PYTHON: False,
+}
+
+
+class TestIntegrationModeIsTestHarness:
+ @pytest.mark.parametrize(("integration_mode", "expected"), list(IS_TEST_HARNESS_BY_MODE.items()))
+ def test_is_test_harness(self, integration_mode: IntegrationMode, expected: bool) -> None:
+ """The boot keeps the Gateway telemetry stream off in a harness mode, so each mode's answer is a telemetry decision.
+
+ A deployment mode classified as a harness would silence real usage; a harness mode classified as
+ a deployment would send test runs to the production analytics project.
+ """
+ assert integration_mode.is_test_harness is expected
+
+ def test_the_table_covers_every_mode(self) -> None:
+ """A mode added later fails here until someone decides whether it is a test harness."""
+ assert set(IS_TEST_HARNESS_BY_MODE) == set(IntegrationMode)
diff --git a/tests/unit/pipelex/test_runtime_boot_pipelex_telemetry.py b/tests/unit/pipelex/test_runtime_boot_pipelex_telemetry.py
new file mode 100644
index 000000000..cf15a0bec
--- /dev/null
+++ b/tests/unit/pipelex/test_runtime_boot_pipelex_telemetry.py
@@ -0,0 +1,76 @@
+import pytest
+
+from pipelex.runtime_boot import RuntimeBoot
+from pipelex.runtime_hub import get_telemetry_manager
+from pipelex.system.runtime import IntegrationMode
+
+# Spelled out rather than derived from `is_test_harness`, so this table does not lean on the property it
+# depends on; `test_integration_mode.py` pins the property itself.
+HARNESS_MODES = [IntegrationMode.CI, IntegrationMode.PYTEST]
+DEPLOYMENT_MODES = [mode for mode in IntegrationMode if mode not in HARNESS_MODES]
+
+
+class TestRuntimeBootPipelexTelemetry:
+ @pytest.mark.parametrize("integration_mode", DEPLOYMENT_MODES)
+ def test_a_deployment_mode_with_every_condition_met_enables_the_stream(self, integration_mode: IntegrationMode) -> None:
+ assert RuntimeBoot.should_enable_pipelex_telemetry(
+ integration_mode=integration_mode,
+ is_unit_testing=False,
+ is_gateway_enabled=True,
+ needs_inference=True,
+ is_gateway_config_cached=False,
+ )
+
+ @pytest.mark.parametrize(
+ ("is_unit_testing", "is_gateway_enabled", "needs_inference", "is_gateway_config_cached"),
+ [
+ pytest.param(True, True, True, False, id="unit-testing-run-mode"),
+ pytest.param(False, False, True, False, id="gateway-disabled"),
+ pytest.param(False, True, False, False, id="no-inference"),
+ pytest.param(False, True, True, True, id="cached-config"),
+ ],
+ )
+ def test_each_condition_alone_disables_the_stream(
+ self,
+ is_unit_testing: bool,
+ is_gateway_enabled: bool,
+ needs_inference: bool,
+ is_gateway_config_cached: bool,
+ ) -> None:
+ """The `unit-testing-run-mode` row is the suite that boots in the default `PYTHON` mode, as the shared plugin's recipe does."""
+ assert not RuntimeBoot.should_enable_pipelex_telemetry(
+ integration_mode=IntegrationMode.PYTHON,
+ is_unit_testing=is_unit_testing,
+ is_gateway_enabled=is_gateway_enabled,
+ needs_inference=needs_inference,
+ is_gateway_config_cached=is_gateway_config_cached,
+ )
+
+ @pytest.mark.parametrize("integration_mode", HARNESS_MODES)
+ def test_a_test_harness_mode_disables_the_stream_even_with_every_other_condition_met(self, integration_mode: IntegrationMode) -> None:
+ """Test runs are not usage, and their fixture user ids must not become persons in the production project.
+
+ Before this condition, every pytest run with `pipelex_gateway` enabled and a fresh remote config
+ sent its pipe runs to the Gateway's production analytics project, because the stream's enabling
+ condition never read the integration mode. The run mode is left false here: a harness that boots
+ without the shared pytest plugin, such as the pipelex-js conformance scripts, has only its mode.
+ """
+ assert not RuntimeBoot.should_enable_pipelex_telemetry(
+ integration_mode=integration_mode,
+ is_unit_testing=False,
+ is_gateway_enabled=True,
+ needs_inference=True,
+ is_gateway_config_cached=False,
+ )
+
+ def test_the_suite_boot_has_the_gateway_stream_off(self) -> None:
+ """The boot this suite runs under must not send the Gateway stream: the bug's symptom, stated directly.
+
+ This passes trivially on a machine where `pipelex_gateway` is disabled, or where the remote
+ config came from the cache, because the stream is off there for those reasons already; the
+ table tests above and the setup-level tests in `test_setup_gateway_telemetry.py` are the precise
+ guards. It asserts on the Gateway stream alone, not on the manager being a no-op, because a
+ developer may deliberately allow the custom stream under `pytest` in their own
+ `telemetry_allowed_modes`.
+ """
+ assert not get_telemetry_manager().is_pipelex_telemetry_enabled
diff --git a/tests/unit/pipelex/test_shared_pytest_plugin_run_mode.py b/tests/unit/pipelex/test_shared_pytest_plugin_run_mode.py
new file mode 100644
index 000000000..8a6349467
--- /dev/null
+++ b/tests/unit/pipelex/test_shared_pytest_plugin_run_mode.py
@@ -0,0 +1,11 @@
+from pipelex.system.runtime import runtime_manager
+
+# Read while pytest imports this module to collect it, before any fixture has run: a boot here, such as a
+# top-level `Pipelex.make()`, is exactly what the run mode must already cover.
+IS_UNIT_TESTING_AT_COLLECTION = runtime_manager.is_unit_testing
+
+
+class TestSharedPytestPluginRunMode:
+ def test_the_run_mode_is_a_test_mode_before_any_fixture_runs(self) -> None:
+ """The shared plugin sets the run mode at configure time, so a boot during collection keeps the Gateway stream off."""
+ assert IS_UNIT_TESTING_AT_COLLECTION is True