Skip to content

Build AI methods with your coding agent, then run them anywhere: as an MCP for chatbots, as a webapp for people, or via API for your software.

fix/Gcp-sink-silent-dead · L-260927-e209aa (#1284)

Louis Choqueldee45c26e8df
diff --git a/CHANGELOG.md b/CHANGELOG.md
index afa030e1d..68b929613 100644
--- a/CHANGELOG.md
+++ b/CHANGELOG.md
@@ -8,8 +8,14 @@
- **`PipelexKernel.make` takes `request_id` and `pipeline_run_id`**: a kernel-driven run can now carry the inbound request id a hosted deployment filters its logs on, and name itself with an id the host chooses. A host inside a replay-based executor passes a `pipeline_run_id` or a trace context, since the default `uuid4` changes on every replay; a trace context and a `pipeline_run_id` that disagree, or an empty `pipeline_run_id`, raise `ValueError`.
- **`PipelexMTHDSProtocol.execute` takes `request_id`**: a host running `execute` passes the inbound request id, which the run's `RunMetadata` carries, so the log lines of a run dispatched to a distributed orchestrator's workers carry `request_id` as a direct run's do. A value that is not one to 128 printable ASCII characters is refused with a `ValueError` before the run is set up, where it used to surface as a `PipeExecutionError` blaming the caller's input; the constraint is stated once, as `validate_request_id` in `pipelex.system.job_metadata`, which `RunMetadata` applies too.
+### Changed
+
+- **The `gcp` log sink stops the boot when Google refuses its credentials (Breaking)**: the sink refreshes its credentials once when it is built, within five seconds, and credentials Google refuses (a revoked or expired refresh token, a deleted service-account key, a Google Cloud machine with no service account) or that cannot be loaded (no Application Default Credentials, a key file that is missing or holds no valid key) raise the new `GcpLogSinkCredentialsError`, a `config`-domain `CredentialsError` naming the credentials, Google's answer and what renews them, which `pipelex doctor` reports in its log-sink row. Such a process used to boot and lose every record in silence, since over gRPC the failed refresh is retried for a minute before anything reports it; a refresh that cannot reach the token endpoint or the metadata server, gets an answer that says to retry (a `5xx`, `408` or `429`), or does not answer in time, is said on stderr and the boot goes on.
+
### Fixed
+- **The `gcp` log sink says when its flush runs out of time, and why**: a teardown whose flush reaches its five-second deadline with records still held now prints on stderr that they are lost unless the transport sends them while it closes, followed by what a fresh refresh of the credentials answers, where only the client library's "Failed to send N pending logs." used to appear. A handler that has closed no longer flushes again when the process exits, which cost a second five-second wait.
+- **A `GcpLogSink` installed a second time raises instead of sending nothing**: installing the same sink object again after a reset built a handler on the transport the first teardown had closed, which queued every record and sent none; `make_handler` now raises `RuntimeError`, as the `otlp` sink does, and the registered `gcp` factory still builds a new sink at every boot.
- **A kernel step's log lines name its run and its step**: `run_llm_text`, `run_llm_object`, `generate_object_content`, `run_extract`, `run_search` and `run_img_gen` now bind the `job_metadata` they are handed for the whole call, so a program driving the kernel directly gets log records carrying `request_id`, `pipeline_run_id` and `pipe_run_id`, as an interpreted run's do. An interpreted run's lines are unchanged.
## [v0.67.1] - 2026-09-27
diff --git a/docs/configuration/config-practical/logging-config.md b/docs/configuration/config-practical/logging-config.md
index f80d1bccc..55c8f682b 100644
--- a/docs/configuration/config-practical/logging-config.md
+++ b/docs/configuration/config-practical/logging-config.md
@@ -242,6 +242,14 @@ credentials_file_path = "gcp_credentials.json"
- `project_id`: the project to write to. Left unset, the client library resolves it from the credentials or from the metadata server of the machine the process runs on
- `credentials_file_path`: a service-account JSON file to build the client from. Left unset, authentication is Application Default Credentials, which is what a process already running on Google Cloud has. This is a plain config value rather than a secret id read through the secrets provider, because the log sink is the first capability boot resolves — deliberately ahead of the secrets provider, so that every later line of the boot goes through the sink the configuration chose — and there is no provider to ask when this section is read
+### Credentials are checked at boot
+
+The sink refreshes its credentials once when it is built, before the transport starts, and waits no longer than five seconds for the answer. Over gRPC, the client library's default, a refresh that fails later is retried like a network error for a minute before anything reports it, so a short-lived process whose credentials no longer work would otherwise exit having lost every record, with nothing on stderr but the library's own count of what it could not send.
+
+- **Credentials Google refuses stop the boot**: a refresh token that was revoked or has expired, a service-account key that was deleted or a key file that cannot be read, a machine on Google Cloud whose metadata server has no service account to hand out, or no credentials at all raise `GcpLogSinkCredentialsError`, whose message names the credentials, quotes Google's answer and says what renews them: `gcloud auth application-default login` on a workstation, the service account the machine runs as on Google Cloud, a valid key at `credentials_file_path`, or the `json` sink instead. `pipelex doctor` reports the same message in its log-sink row
+- **A refresh that gets no answer does not stop it**: when the token endpoint or the metadata server cannot be reached, answers with a status that says to retry, or does not answer within the five seconds, nothing is known about the credentials themselves, so the sink says so on stderr and is installed, and its transport retries
+- A service-account key over gRPC signs its own token and never calls the token endpoint, so for `credentials_file_path` the check costs one token exchange the transport would not have made; for Application Default Credentials that fetch a token, the token the check earns is the one the first export sends
+
### What each entry carries
Each record becomes one Cloud Logging entry with a JSON payload:
@@ -250,9 +258,9 @@ Each record becomes one Cloud Logging entry with a JSON payload:
- The payload carries `message`, `logger` and `exception` when the record carries one, then every field and the `data` attribute flat beside them. The keys the `json` sink reserves are reserved here too, `time`, `severity`, `trace_id`, `span_id` and `trace_flags` included although the payload carries none of them — the client library takes the time and the severity out of band, and the entry's own `trace` field carries the trace — so a field named like one of them is carried under a `field_` prefix under either sink rather than under one and not the other; a value JSON cannot carry — a non-finite float, a model, a circular structure — is written as text rather than costing the line
- The run-scoped identifiers become the entry's **labels** rather than payload keys: `request_id`, `pipeline_run_id` and `pipe_run_id`, whichever of them the record carries. Cloud Logging indexes labels, so these are what a query filters a run by
- The entry's `trace` field carries the run's own OpenTelemetry trace id, project-qualified as `projects/<project>/traces/<trace-id>`. The id is derived from `pipeline_run_id` by the same hash the tracer uses, so a line and the spans of the run it belongs to agree on it and Cloud Logging files them together
-- The entries leave through the client library's background-thread transport, which batches them off the thread that logged, so no record costs an API round trip on the calling thread. The teardown flushes and closes it, and the flush carries a deadline of its own because the library's does not
-- What the export path logs never leaves through the sink: the library reports a refused batch through a logger of its own, and a report exported through the pipeline it reports on would fail with it and be reported again. The handler rejects those records, and every record emitted on the library's export thread, before its lock is taken — the same guard the `otlp` sink carries, against the same deadlock at exit
-- A refused write is still said: the library marks a refused batch done, so the flush succeeds and nothing else would report it, and the entries in it are lost. The export path's warnings and errors are printed on stderr instead, the first one in each window of time with its traceback and the ones after it counted in a single line once that window has passed — with the next failure, on the next line the process logs, or at the teardown at the latest — so a process whose credentials Cloud Logging refuses says so without printing once per line it logs
+- The entries leave through the client library's background-thread transport, which batches them off the thread that logged, so no record costs an API round trip on the calling thread. The teardown flushes and closes it, and the flush carries a deadline of its own, five seconds, because the library's does not. A flush that runs out of time says so on stderr, that the transport was still writing and that what it still holds is lost unless it sends it while it closes, followed by what a fresh refresh of the credentials answers, a refresh given five seconds of its own, so such a flush returns within ten and the close then gives the transport the client library's own grace period: refused, with Google's answer, which is why the write is held; unreachable; or still accepted, in which case the Cloud Logging API or the network is what holds it. A handler that has closed does not flush again when the process exits
+- What the export path logs never leaves through the sink: the library reports a refused batch through a logger of its own, and a report exported through the pipeline it reports on would fail with it and be reported again. The handler rejects those records, and every record emitted on the library's export thread, before its lock is taken — the same guard the `otlp` sink carries, against the same deadlock at exit. What the sink's own credentials check logs while a teardown's flush waits on it is rejected the same way and printed on stderr, so the check still answers when the process's exit holds the handler's lock
+- A refused write is still said: the library marks a refused batch done, so the flush succeeds and nothing else would report it, and the entries in it are lost. The export path's warnings and errors are printed on stderr instead, the first one in each window of time with its traceback and the ones after it counted in a single line once that window has passed — with the next failure, on the next line the process logs, or at the teardown at the latest — so a process whose writes Cloud Logging refuses says so without printing once per line it logs
## Example Configuration
diff --git a/docs/errors/gcp-log-sink-credentials-error.md b/docs/errors/gcp-log-sink-credentials-error.md
new file mode 100644
index 000000000..518a4fe3d
--- /dev/null
+++ b/docs/errors/gcp-log-sink-credentials-error.md
@@ -0,0 +1,21 @@
+---
+title: "Log sink credentials missing or refused"
+description: "Reference for the `GcpLogSinkCredentialsError` Pipelex error class."
+---
+
+<!-- pipelex:generated -->
+
+# Log sink credentials missing or refused
+
+Raised at boot when the ``gcp`` log sink's credentials cannot be found or are refused when refreshed.
+
+| Field | Value |
+|---|---|
+| `error_type` | `GcpLogSinkCredentialsError` |
+| `title` | Log sink credentials missing or refused |
+| `type_uri` | `https://docs.pipelex.com/latest/errors/gcp-log-sink-credentials-error/` |
+| `error_domain` | `config` |
+| Defined in | `pipelex.tools.log.exceptions` |
+| Parent class | [`CredentialsError`](credentials-error.md) |
+
+[Back to Error Reference](index.md)
diff --git a/docs/errors/platform-and-tooling.md b/docs/errors/platform-and-tooling.md
index 3b63207ee..a47154c2a 100644
--- a/docs/errors/platform-and-tooling.md
+++ b/docs/errors/platform-and-tooling.md
@@ -29,6 +29,7 @@ own page. Classes are grouped by subsystem.
- [`CsvFlatnessError`](csv-flatness-error.md) — CSV flatness error
- [`CsvReadError`](csv-read-error.md) — CSV read error
- [`FileTypeError`](file-type-error.md) — File type
+- [`GcpLogSinkCredentialsError`](gcp-log-sink-credentials-error.md) — Log sink credentials missing or refused
- [`Jinja2ContextError`](jinja2-context-error.md) — Jinja 2 context
- [`Jinja2DetectVariablesError`](jinja2-detect-variables-error.md) — Jinja 2 detect variables
- [`Jinja2StuffError`](jinja2-stuff-error.md) — Jinja 2 stuff
diff --git a/docs/under-the-hood/log-sink-plugins.md b/docs/under-the-hood/log-sink-plugins.md
index 7fff7a15c..60fe6d881 100644
--- a/docs/under-the-hood/log-sink-plugins.md
+++ b/docs/under-the-hood/log-sink-plugins.md
@@ -87,7 +87,7 @@ class LogSink(ABC):
def redirect_to_stderr(self) -> None: ... # a no-op unless the sink writes to a process stream
```
-- **`make_handler`** builds the stdlib handler the sink installs on the root logger, and is where a heavy import belongs. It is called once per install, so selecting a sink is what pays for its dependency and a sink whose package is missing fails there, at boot, with `MissingDependencyError` naming the extra. A sink object installed again after a `reset` builds a second handler rather than being handed the first one back, because the teardown closed it — and a close is terminal for a sink that really releases what it writes to, so a reused handler would accept every record and drop the lot with nothing raised to say so. **Open the target in `make_handler`, not in `__init__`**, or the second install writes to something already closed. A sink that cannot, because it is handed its target already built, refuses a second handler from `make_handler` instead of building one on the closed target: the built-in `otlp` sink is constructed around its processor and shuts it down with its provider at close, so installing the same `OtlpLogSink` object again raises, and its factory builds a new one at every boot.
+- **`make_handler`** builds the stdlib handler the sink installs on the root logger, and is where a heavy import belongs. It is called once per install, so selecting a sink is what pays for its dependency and a sink whose package is missing fails there, at boot, with `MissingDependencyError` naming the extra. A sink object installed again after a `reset` builds a second handler rather than being handed the first one back, because the teardown closed it — and a close is terminal for a sink that really releases what it writes to, so a reused handler would accept every record and drop the lot with nothing raised to say so. **Open the target in `make_handler`, not in `__init__`**, or the second install writes to something already closed. A sink that cannot, because it is handed its target already built, refuses a second handler from `make_handler` instead of building one on the closed target: the built-in `otlp` sink is constructed around its processor and shuts it down with its provider at close, and the built-in `gcp` sink around the client library's transport, whose worker thread its close stops for good, so installing the same `OtlpLogSink` or `GcpLogSink` object again raises, and each factory builds a new one at every boot.
- **`processors`** is the slot a cross-cutting transformation hooks into. A processor is a callable that edits a record in place, `Callable[[logging.LogRecord], None]`, and the base runs every processor over each record before the handler formats it, through a filter on the sink's own handler. Each one is guarded on its own: the stdlib runs a handler's filters outside any `try`, so a processor that raised would raise out of the `log.<level>(...)` call that emitted the record, and instead what it raises is reported on stderr by its type and the processor's name — never its text or its traceback, since a processor fails on the record's own values and its exception is where those end up — guarded in its turn since a closed stderr makes the reporter raise, the processors after it still run, and the record is handed on. A failing processor costs that record its processing, never the call and never the line; the redaction processor, which must not hand on what it failed to scrub, strips the record down to a `[REDACTION FAILED: <exception type>]` notice before it raises, and where even that stripping could not complete it leaves a mark that has the record dropped rather than emitted — the one case in which a sink does not see a record its logger accepted. **Redaction is the first processor, and the base is not what appends it**: `log.install_sink` puts it in front of the sink's own list, so every sink gets it, built-in, out-of-tree or the doctor's fallback, with none of them knowing about it — see [Redaction](../tools/logging.md#redaction) for why it edits the record every root handler shares instead of a copy.
- **`redirect_to_stderr`** exists for the agent CLI, whose stdout is reserved for its structured envelope: a sink that writes to a process stream moves it to stderr, and one that writes to a collector ignores the call.
@@ -124,7 +124,7 @@ class LogSinkPlugin:
| `json` | One JSON object per line on the configured stream: `time`, `severity`, `logger`, `message`, `exception` when there is one, `trace_id`, `span_id` and `trace_flags` in hex when the record was logged inside a span, the active Pipelex span or else the current OpenTelemetry one, under the keys OpenTelemetry specifies for trace context in a JSON log that is not OTLP, then the fields, the context identifiers and `data` flat beside them, a field named like one of those keys under a `field_` prefix on every line, a non-finite float as the string `"NaN"`, `"Infinity"` or `"-Infinity"`. The key names are the ones the CloudWatch agent, the Google Cloud Logging agent and any OTLP collector ingest without a parser, and no ANSI ever. What the hosted plane's runner and worker select. | `console_log_target` |
| `console` | The Rich handler with the emoji formatter and every `[runtime.log.rich_log]` setting, byte for byte what the console showed before sinks existed. Rich, the `cli` extra, is imported when the handler is built, and a process that selects another sink never loads it through this path. | `console_log_target`, `[runtime.log.rich_log]` |
| `otlp` | The OpenTelemetry logs signal: a `LoggerProvider` carrying the same service identity as the tracer, a `BatchLogRecordProcessor` and the OTLP HTTP log exporter. The message is the body, the level maps onto the OTel severity scale, the fields, identifiers and `data` ride as attributes (a mapping as JSON text), an exception lands under the `exception.*` semantic-convention keys, each record is filed under the span it was logged in, the active Pipelex span or else the current OpenTelemetry one, and a collector receives the logs beside the spans the runtime already exports. The keys the sink writes itself, the `code.*` source location and the `exception.*` set, are reserved on every record whether or not it carries an exception, so a field named like one of them rides under a `field_` prefix rather than being overwritten. A filter on the handler rejects the records of the sink's own export path, the SDK's by logger name and the transport's by the context value the SDK sets around an export, before the handler's lock is taken, so an export failure never re-enters the pipeline and a shutdown never waits on itself. | `[runtime.log.otlp]` |
-| `gcp` | Google Cloud Logging through the `google-cloud-logging` client library, behind the `gcp-logging` extra. Each record becomes one struct entry: the level maps onto the Cloud Logging severity scale (which has nothing below `DEBUG`, so `VERBOSE` and `DEV` both land there), the message, the logger, the exception and the fields become the JSON payload, the run-scoped identifiers become the entry's labels, which is what Cloud Logging indexes, and the entry's `trace` field carries the run's own OpenTelemetry trace id project-qualified, derived from `pipeline_run_id` by the same hash the tracer uses, so a line and the spans of its run agree on it. The entries leave through the client library's background-thread transport, which batches them off the calling thread. **Most processes on Google Cloud want `json` instead**: a platform whose logging agent reads the container's stdout, Cloud Run and GKE among them, ingests what that sink writes and needs no client at all; `gcp` is for a process with no such agent in front of it, or one writing to a log or a project that is not the ambient one. | `[runtime.log.gcp]` |
+| `gcp` | Google Cloud Logging through the `google-cloud-logging` client library, behind the `gcp-logging` extra. Each record becomes one struct entry: the level maps onto the Cloud Logging severity scale (which has nothing below `DEBUG`, so `VERBOSE` and `DEV` both land there), the message, the logger, the exception and the fields become the JSON payload, the run-scoped identifiers become the entry's labels, which is what Cloud Logging indexes, and the entry's `trace` field carries the run's own OpenTelemetry trace id project-qualified, derived from `pipeline_run_id` by the same hash the tracer uses, so a line and the spans of its run agree on it. The entries leave through the client library's background-thread transport, which batches them off the calling thread. The factory refreshes the credentials once before that transport starts, so credentials Google refuses stop the boot with `GcpLogSinkCredentialsError` rather than losing every record in silence. **Most processes on Google Cloud want `json` instead**: a platform whose logging agent reads the container's stdout, Cloud Run and GKE among them, ingests what that sink writes and needs no client at all; `gcp` is for a process with no such agent in front of it, or one writing to a log or a project that is not the ambient one. | `[runtime.log.gcp]` |
The `otlp` factory imports the OpenTelemetry logs SDK when it runs, and the `gcp` factory the Google Cloud Logging client library, never at register, so registering the built-ins imports neither, which the import-light guard pins.
diff --git a/pipelex/tools/log/exceptions.py b/pipelex/tools/log/exceptions.py
new file mode 100644
index 000000000..1ab37e5db
--- /dev/null
+++ b/pipelex/tools/log/exceptions.py
@@ -0,0 +1,14 @@
+from pipelex.base_exceptions import ErrorDomain
+from pipelex.system.exceptions import CredentialsError
+
+
+class GcpLogSinkCredentialsError(CredentialsError):
+ """Raised at boot when the ``gcp`` log sink's credentials cannot be found or are refused when refreshed.
+
+ Every record the sink would export is lost without them, so the boot stops rather than running a
+ process whose logs go nowhere. Fixed by an operator renewing the credentials or selecting another
+ sink, not by the caller correcting input.
+ """
+
+ error_domain = ErrorDomain.CONFIG
+ _declared_title = "Log sink credentials missing or refused"
diff --git a/pipelex/tools/log/gcp_log_sink.py b/pipelex/tools/log/gcp_log_sink.py
index 303abd3c4..238b457c8 100644
--- a/pipelex/tools/log/gcp_log_sink.py
+++ b/pipelex/tools/log/gcp_log_sink.py
@@ -9,6 +9,13 @@ thread that logged it. What that thread itself logs never leaves through the sin
a refused batch through a logger of its own, and a report exported through the pipeline it reports on
fails with it and is reported again, so the handler rejects the export path before its lock is taken.
+The credentials are refreshed once when the sink is built. Over gRPC, the library's default, a refresh
+that fails surfaces as a retriable ``UNAVAILABLE``, so the transport retries the batch for a minute before
+anything reports it, and a process that closes sooner loses every record without a word. The refresh at
+boot is what says it instead: credentials Google refuses stop the boot naming them, and a refresh that
+cannot get an answer is said on stderr. A flush that runs out of time at teardown says so too, naming
+what a fresh refresh of the credentials answers, since that is the likeliest reason a write is held.
+
**Most processes should not select this sink.** A process on Cloud Run, on GKE, or on any platform
whose logging agent reads the container's stdout needs no client at all: the agent ingests one JSON
object per line with a ``severity`` and a ``message``, which is exactly what the ``json`` sink emits,
@@ -29,11 +36,13 @@ import math
import threading
import time
from enum import StrEnum
-from typing import TYPE_CHECKING, Any, Protocol, cast
+from http import HTTPStatus
+from typing import TYPE_CHECKING, Any, NamedTuple, Protocol, cast
from typing_extensions import override
from pipelex.system.exceptions import MissingDependencyError
+from pipelex.tools.log.exceptions import GcpLogSinkCredentialsError
from pipelex.tools.log.json_log_sink import EXCEPTION_KEY, FIXED_KEYS, LOGGER_KEY, MESSAGE_KEY
from pipelex.tools.log.log_context import PIPE_RUN_ID_FIELD, PIPELINE_RUN_ID_FIELD, REQUEST_ID_FIELD
from pipelex.tools.log.log_fields import COLLIDING_FIELD_PREFIX, carried_attributes
@@ -41,6 +50,8 @@ from pipelex.tools.log.log_sink import LogSink, LogSinkMethod, render_json
from pipelex.tools.misc.hash_utils import hash_md5_to_int
if TYPE_CHECKING:
+ from collections.abc import Callable
+
from pipelex.tools.log.log_config import GcpLogSinkConfig
# The dependency the ``gcp-logging`` extra installs, and the extra's own name, as the install hint spells them.
@@ -84,6 +95,39 @@ EXPORT_FAILURE_REPORT_INTERVAL_SECONDS = 20.0
FLUSH_TIMEOUT_SECONDS = 5.0
_FLUSH_THREAD_NAME = "pipelex-gcp-log-flush"
+# The deadline one refresh of the sink's credentials is given, at boot and again at teardown when a flush
+# has run out of time. A refresh is one round trip to a token endpoint or to the metadata server, but the
+# auth library's own request timeout is two minutes and its metadata client retries, so the bound is
+# imposed the way the flush's is: the refresh runs on a thread of its own and the join carries it.
+CREDENTIALS_CHECK_TIMEOUT_SECONDS = 5.0
+_CREDENTIALS_CHECK_THREAD_NAME = "pipelex-gcp-log-credentials"
+
+
+def _print_record_on_stderr(*, record: logging.LogRecord) -> None:
+ """Print a record the sink rejected on stderr, through the stdlib's last-resort handler, which never takes the sink's lock."""
+ stderr_handler = logging.lastResort
+ if stderr_handler is not None:
+ stderr_handler.handle(record)
+
+
+def _say_on_stderr(*, message: str) -> None:
+ """Print one warning of the sink's own on stderr, through the stdlib's last-resort handler.
+
+ What the sink says about itself cannot go through the sink: at boot it is not installed yet, and
+ when its writes are refused, a line exported through it is lost with the rest. The last resort
+ takes its own lock and never the handler's, so saying it cannot deadlock the teardown either.
+ """
+ _print_record_on_stderr(
+ record=logging.makeLogRecord(
+ {
+ "name": __name__,
+ "levelno": logging.WARNING,
+ "levelname": logging.getLevelName(logging.WARNING),
+ "msg": message,
+ }
+ )
+ )
+
class GcpLogSeverity(StrEnum):
"""The Cloud Logging severities this sink writes. The API takes the name and uppercases it."""
@@ -136,6 +180,13 @@ class GcpExportPathFilter(logging.Filter):
it and waits for the queue to drain is a deadlock at exit, and it was reproduced here with the
installed library.
+ The sink's own credentials check is guarded the same way, by the name of its thread. At teardown it
+ runs while the flush that started it waits, and the stdlib's shutdown at exit holds this handler's
+ lock around that flush, so a warning the auth library logs during the refresh would wait on the lock
+ until the check's deadline passed and the check would answer that the refresh did not answer. What
+ that thread logs is printed on stderr through the last resort, every line of it: it is not an export
+ failure, and the check logs a handful of lines at most.
+
A rejected record at ``WARNING`` or above is printed on stderr through the stdlib's last-resort
handler rather than dropped. The library reports a refused batch only through that record and then
marks the batch done, so a flush still succeeds, and the stdlib prints a record through its last
@@ -154,6 +205,10 @@ class GcpExportPathFilter(logging.Filter):
@override
def filter(self, record: logging.LogRecord) -> bool:
+ if threading.current_thread().name == _CREDENTIALS_CHECK_THREAD_NAME:
+ if record.levelno >= logging.WARNING:
+ _print_record_on_stderr(record=record)
+ return False
if not self._is_export_path(record=record):
# Read outside the lock, which keeps an ordinary record's cost to this one read: a count it
# misses is said on a later record, or when the handler closes.
@@ -188,9 +243,7 @@ class GcpExportPathFilter(logging.Filter):
unreported_count = self._unreported_count
self._unreported_count = 0
self._print_unreported_count(unreported_count=unreported_count)
- stderr_handler = logging.lastResort
- if stderr_handler is not None:
- stderr_handler.handle(record)
+ _print_record_on_stderr(record=record)
def _report_count_once_its_window_has_passed(self) -> None:
"""Print the count an outage that has ended left behind, since no later failure comes to print it.
@@ -210,20 +263,9 @@ class GcpExportPathFilter(logging.Filter):
@classmethod
def _print_unreported_count(cls, *, unreported_count: int) -> None:
- stderr_handler = logging.lastResort
- if not unreported_count or stderr_handler is None:
+ if not unreported_count:
return
- stderr_handler.handle(
- logging.makeLogRecord(
- {
- "name": __name__,
- "levelno": logging.WARNING,
- "levelname": logging.getLevelName(logging.WARNING),
- "msg": "The gcp log sink's transport reported %d more export failures since the last one printed",
- "args": (unreported_count,),
- }
- )
- )
+ _say_on_stderr(message=f"The gcp log sink's transport reported {unreported_count} more export failures since the last one printed")
def severity_for_level(*, levelno: int) -> GcpLogSeverity:
@@ -318,13 +360,163 @@ def _entry_labels(*, record: logging.LogRecord) -> dict[str, str]:
return labels
+class GcpCredentialsOutcome(StrEnum):
+ """What one refresh of the sink's credentials came to."""
+
+ REFRESHED = "refreshed"
+ # Google answered and refused them: a revoked or expired refresh token, a deleted key, a machine
+ # whose metadata server has no service account to hand out. Retrying does not change the answer.
+ REFUSED = "refused"
+ # The token endpoint or the metadata server could not be reached, or answered with a status that
+ # says to retry: nothing is known about the credentials themselves.
+ UNREACHABLE = "unreachable"
+ # The refresh had not returned when its deadline passed.
+ UNANSWERED = "unanswered"
+
+
+class GcpCredentialsVerdict(NamedTuple):
+ """The outcome of one refresh, with the exception it raised when it raised one."""
+
+ outcome: GcpCredentialsOutcome
+ failure: Exception | None = None
+
+
+def describe_failure(*, failure: BaseException | None) -> str:
+ """The exception's class and message, as the sink's lines quote it."""
+ if failure is None:
+ return "no exception"
+ return f"{type(failure).__name__}: {failure}"
+
+
+class GcpCredentialsCheck:
+ """Refreshes the sink's credentials once, off the calling thread and within a deadline, and says what came of it.
+
+ The refresh is the auth library's, handed in as a callable so this module imports none of it and a
+ test substitutes its own. What tells an unreachable endpoint from a refusal is handed in the same way:
+ the auth library's transport error types. A failure is ``UNREACHABLE`` when it, or an exception it was
+ raised from, is one of them or declares itself retryable — the metadata-server credentials wrap a
+ transport failure in the refusal type, so the class of the outermost exception is not enough — and
+ ``REFUSED`` otherwise, an exception nobody anticipated included, because the boot it stops names it.
+ A transport failure that carries a final answer is a refusal all the same: the metadata server
+ answers a machine with no service account with a ``404``, and its client raises that as a transport
+ error with the response attached rather than retry it.
+ """
+
+ def __init__(
+ self,
+ *,
+ refresh: Callable[[], None],
+ transport_error_types: tuple[type[Exception], ...],
+ source: str,
+ ) -> None:
+ self._refresh = refresh
+ self._transport_error_types = transport_error_types
+ self._source = source
+
+ @property
+ def source(self) -> str:
+ """Which credentials these are, as a sentence names them: the Application Default Credentials, or a key file."""
+ return self._source
+
+ def run(self) -> GcpCredentialsVerdict:
+ """Refresh once and wait no longer than ``CREDENTIALS_CHECK_TIMEOUT_SECONDS`` for the answer.
+
+ A refresh still running when the deadline passes is left to finish on its daemon thread, and
+ whatever it answers then is not read: the verdict is ``UNANSWERED``.
+ """
+ verdicts: list[GcpCredentialsVerdict] = []
+
+ def refresh_once() -> None:
+ try:
+ self._refresh()
+ except Exception as exc: # ruff: ignore[blind-except]
+ # Unbounded code: the auth library's refresh, whose exceptions depend on the kind of credentials.
+ # Nothing is swallowed: the failure is classified and handed back in the verdict.
+ verdicts.append(GcpCredentialsVerdict(outcome=self._classify(failure=exc), failure=exc))
+ return
+ verdicts.append(GcpCredentialsVerdict(outcome=GcpCredentialsOutcome.REFRESHED))
+
+ refreshing = threading.Thread(target=refresh_once, name=_CREDENTIALS_CHECK_THREAD_NAME, daemon=True)
+ refreshing.start()
+ refreshing.join(timeout=CREDENTIALS_CHECK_TIMEOUT_SECONDS)
+ if not verdicts:
+ return GcpCredentialsVerdict(outcome=GcpCredentialsOutcome.UNANSWERED)
+ return verdicts[0]
+
+ def _classify(self, *, failure: Exception) -> GcpCredentialsOutcome:
+ """``UNREACHABLE`` when a transport failure is anywhere on the chain the traceback would print, ``REFUSED`` otherwise."""
+ seen: set[int] = set()
+ current: BaseException | None = failure
+ while current is not None and id(current) not in seen:
+ if isinstance(current, self._transport_error_types):
+ if _carries_a_final_answer(failure=current):
+ return GcpCredentialsOutcome.REFUSED
+ return GcpCredentialsOutcome.UNREACHABLE
+ if getattr(current, "retryable", False) is True:
+ return GcpCredentialsOutcome.UNREACHABLE
+ seen.add(id(current))
+ if current.__cause__ is not None:
+ current = current.__cause__
+ elif current.__suppress_context__:
+ current = None
+ else:
+ current = current.__context__
+ return GcpCredentialsOutcome.REFUSED
+
+
+def _carries_a_final_answer(*, failure: BaseException) -> bool:
+ """Whether a transport failure carries an HTTP answer that retrying does not change: a ``4xx`` other than a timeout or a rate limit.
+
+ The auth library's metadata client raises its transport error with the response as the second
+ argument when the server answers with a status it does not retry, and with none when the server
+ could not be reached or its retries ran out; ``5xx``, ``408`` and ``429`` are the ones it retries.
+ """
+ if len(failure.args) < 2:
+ return False
+ status = getattr(failure.args[1], "status", None)
+ if not isinstance(status, int):
+ return False
+ return HTTPStatus.BAD_REQUEST <= status < HTTPStatus.INTERNAL_SERVER_ERROR and not says_to_retry(status=status)
+
+
+def says_to_retry(*, status: int) -> bool:
+ """Whether an HTTP status from a credential endpoint says to retry: a ``5xx``, a timeout or a rate limit, the ones the auth library retries."""
+ return status >= HTTPStatus.INTERNAL_SERVER_ERROR or status in {HTTPStatus.REQUEST_TIMEOUT, HTTPStatus.TOO_MANY_REQUESTS}
+
+
+def _undelivered_message(*, cause: str | None) -> str:
+ """What the sink says when its flush ran out of time, followed by the likeliest cause when one was found."""
+ held = (
+ f"The '{LogSinkMethod.GCP}' log sink's transport was still writing to Cloud Logging when the "
+ f"{FLUSH_TIMEOUT_SECONDS:g}-second flush deadline passed, and the records it still holds are lost unless it sends them while it closes."
+ )
+ if cause is None:
+ return held
+ return f"{held} {cause}"
+
+
+def _cause_from_refresh(*, source: str, verdict: GcpCredentialsVerdict) -> str:
+ """What a refresh made once the flush ran out of time says about why the write is held."""
+ match verdict.outcome:
+ case GcpCredentialsOutcome.REFRESHED:
+ return f"A refresh of {source} still succeeds, so the Cloud Logging API or the network is what holds the write."
+ case GcpCredentialsOutcome.REFUSED:
+ return f"A refresh of {source} is now refused, which is why: {describe_failure(failure=verdict.failure)}"
+ case GcpCredentialsOutcome.UNREACHABLE:
+ return f"A refresh of {source} cannot reach the credential endpoint either: {describe_failure(failure=verdict.failure)}"
+ case GcpCredentialsOutcome.UNANSWERED:
+ return f"A refresh of {source} did not answer within {CREDENTIALS_CHECK_TIMEOUT_SECONDS:g} seconds either."
+
+
class GcpLogHandler(logging.Handler):
"""Translates each stdlib record into one Cloud Logging entry and hands it to the transport."""
- def __init__(self, *, transport: GcpLogTransport, project: str) -> None:
+ def __init__(self, *, transport: GcpLogTransport, project: str, credentials_check: GcpCredentialsCheck | None = None) -> None:
super().__init__(level=logging.NOTSET)
self._transport = transport
self._project = project
+ self._credentials_check = credentials_check
+ self._is_closed = False
self._export_path_filter = GcpExportPathFilter()
self.addFilter(self._export_path_filter)
@@ -348,13 +540,26 @@ class GcpLogHandler(logging.Handler):
@override
def flush(self) -> None:
- """Waits for the transport's queue to drain, and never longer than ``FLUSH_TIMEOUT_SECONDS``.
+ """Waits for the transport's queue to drain for ``FLUSH_TIMEOUT_SECONDS`` at most, then says so when it did not.
The library's own flush waits on the queue with no deadline, and the teardown that calls it runs
in a ``finally``: a batch the API is refusing, or an export the network is holding, would keep
the process alive with no way out. The drain is given a thread so the join can carry the bound,
and a failure it raises is re-raised here so the teardown still reports it.
+
+ A drain still running when the deadline passes is said on stderr rather than left to the
+ library's close, whose only line counts what is still queued and names no reason: the sink
+ refreshes its credentials once more and says what that answered, since a refresh the gRPC
+ transport keeps retrying is the likeliest reason a write is held. That refresh has its own
+ deadline, ``CREDENTIALS_CHECK_TIMEOUT_SECONDS``, so a flush that runs out of time returns within
+ the sum of the two. The transport's worker goes on sending meanwhile, so the wait costs no record.
+
+ After ``close`` it does nothing. The stdlib's shutdown at exit flushes every handler still
+ alive, and a drain left running holds this one alive, so without that it would wait out a
+ second deadline on a transport already closed and say so a second time.
"""
+ if self._is_closed:
+ return
failure: list[Exception] = []
def drain() -> None:
@@ -370,33 +575,122 @@ class GcpLogHandler(logging.Handler):
flushing.join(timeout=FLUSH_TIMEOUT_SECONDS)
if failure:
raise failure[0]
+ if flushing.is_alive():
+ self._say_undelivered()
@override
def close(self) -> None:
- """Closes the transport, then prints the export failures still counted.
+ """Closes the transport, then prints the export failures still counted. Once: a second close does nothing.
In that order, because the close drains the queue and a refusal met there is counted too.
"""
+ if self._is_closed:
+ return
+ self._is_closed = True
self._transport.close()
self._export_path_filter.report_unreported()
super().close()
+ def _say_undelivered(self) -> None:
+ """Say on stderr that the flush ran out of time, with what a fresh refresh of the credentials answers."""
+ cause: str | None = None
+ if self._credentials_check is not None:
+ cause = _cause_from_refresh(source=self._credentials_check.source, verdict=self._credentials_check.run())
+ _say_on_stderr(message=_undelivered_message(cause=cause))
+
class GcpLogSink(LogSink):
"""Google Cloud Logging behind one transport, a batching background thread in production.
- The transport and the project are resolved before the sink is built, so the sink itself imports
- nothing and a test builds it around a capture.
+ The transport, the project and the credentials check are resolved before the sink is built, so the
+ sink itself imports nothing and a test builds it around a capture. Without a check, a flush that
+ runs out of time is still said, with no cause named.
+
+ It builds one handler. The handler's close closes the transport, whose worker thread nothing starts
+ again, so a handler built on it afterwards would take every record and send none.
"""
- def __init__(self, *, transport: GcpLogTransport, project: str) -> None:
+ def __init__(self, *, transport: GcpLogTransport, project: str, credentials_check: GcpCredentialsCheck | None = None) -> None:
super().__init__()
self._transport = transport
self._project = project
+ self._credentials_check = credentials_check
+ self._has_built_handler = False
@override
def make_handler(self) -> logging.Handler:
- return GcpLogHandler(transport=self._transport, project=self._project)
+ if self._has_built_handler:
+ msg = (
+ "This gcp sink was installed once already, and the teardown that closed its handler closed its transport, whose "
+ "worker thread nothing starts again, so it can send nothing to Cloud Logging. Build a new GcpLogSink for this install; "
+ "the registered 'gcp' factory builds one at every boot."
+ )
+ raise RuntimeError(msg)
+ handler = GcpLogHandler(transport=self._transport, project=self._project, credentials_check=self._credentials_check)
+ self._has_built_handler = True
+ return handler
+
+
+def _credentials_source(*, config: GcpLogSinkConfig) -> str:
+ """The credentials the sink authenticates with, as a sentence names them."""
+ if config.credentials_file_path is None:
+ return "the Application Default Credentials"
+ return f"the service-account key at '{config.credentials_file_path}'"
+
+
+def _credentials_remedy(*, config: GcpLogSinkConfig) -> str:
+ """What renews the credentials, for the error that stops the boot."""
+ if config.credentials_file_path is None:
+ return (
+ "Renew them (`gcloud auth application-default login` on a workstation, or the service account the machine runs as "
+ "on Google Cloud), or point `credentials_file_path` in [runtime.log.gcp] at a valid service-account key"
+ )
+ return "Point `credentials_file_path` in [runtime.log.gcp] at a valid service-account key"
+
+
+def _unloaded_credentials_message(*, source: str, remedy: str, failure: Exception) -> str:
+ """What stops the boot when the credentials could not be loaded at all."""
+ return (
+ f"The '{LogSinkMethod.GCP}' log sink could not load {source} to write to Google Cloud Logging with "
+ f"({describe_failure(failure=failure)}). {remedy}, or select the '{LogSinkMethod.JSON}' sink in [runtime.log]."
+ )
+
+
+def confirm_credentials_at_boot(*, credentials_check: GcpCredentialsCheck, remedy: str) -> None:
+ """Refresh the credentials once before the transport starts, and stop the boot when Google refuses them.
+
+ A refusal stops the boot because every record the sink would export is lost, and the transport
+ would say so only after retrying for a minute, which a short process never reaches. A refresh that
+ gets no answer says nothing about the credentials themselves, so it is said on stderr and the boot
+ goes on: the transport retries, and what it cannot deliver is said by the flush at the latest.
+
+ Raises:
+ GcpLogSinkCredentialsError: If Google refuses the credentials.
+
+ """
+ verdict = credentials_check.run()
+ why: str
+ match verdict.outcome:
+ case GcpCredentialsOutcome.REFRESHED:
+ return
+ case GcpCredentialsOutcome.REFUSED:
+ msg = (
+ f"The '{LogSinkMethod.GCP}' log sink cannot write to Google Cloud Logging: a refresh of {credentials_check.source} "
+ f"was refused at boot, so every record it exported would be lost ({describe_failure(failure=verdict.failure)}). "
+ f"{remedy}, or select the '{LogSinkMethod.JSON}' sink in [runtime.log]."
+ )
+ raise GcpLogSinkCredentialsError(msg) from verdict.failure
+ case GcpCredentialsOutcome.UNREACHABLE:
+ why = f"a refresh could not reach the credential endpoint ({describe_failure(failure=verdict.failure)})"
+ case GcpCredentialsOutcome.UNANSWERED:
+ why = f"a refresh did not answer within {CREDENTIALS_CHECK_TIMEOUT_SECONDS:g} seconds"
+ _say_on_stderr(
+ message=(
+ f"The '{LogSinkMethod.GCP}' log sink could not confirm at boot that Google accepts {credentials_check.source}: {why}. "
+ "The sink is installed and its transport retries; records it cannot deliver are said on stderr, "
+ "at the latest when the process tears the sink down."
+ )
+ )
def make_gcp_log_sink(*, config: GcpLogSinkConfig) -> GcpLogSink:
@@ -406,15 +700,26 @@ def make_gcp_log_sink(*, config: GcpLogSinkConfig) -> GcpLogSink:
another sink never loads it, and one that selected this sink without the extra fails here, at
boot, with the extra and the ``json`` alternative named.
+ Where the credentials are proved: a key file that cannot be read or holds no valid key, and
+ Application Default Credentials that cannot be found, stop the boot before the client exists; the
+ credentials the client then holds are refreshed once before the transport's thread starts, so a
+ boot whose credentials Google refuses stops here naming them rather than losing every record in
+ silence.
+
Raises:
MissingDependencyError: If ``google-cloud-logging`` is not installed.
+ GcpLogSinkCredentialsError: If the credentials cannot be loaded, or Google refuses them.
"""
try:
+ import requests # ruff: ignore[import-outside-top-level]
+ from google.auth import exceptions as google_auth_exceptions # ruff: ignore[import-outside-top-level]
+ from google.auth.transport.requests import Request as GoogleAuthRequest # ruff: ignore[import-outside-top-level]
from google.cloud import logging as cloud_logging # ruff: ignore[import-outside-top-level]
from google.cloud.logging_v2.handlers.transports import ( # ruff: ignore[import-outside-top-level]
BackgroundThreadTransport,
)
+ from google.oauth2 import service_account as google_service_account # ruff: ignore[import-outside-top-level]
except ImportError as exc:
msg = (
f"The '{LogSinkMethod.GCP}' log sink writes to Google Cloud Logging through the client library. "
@@ -428,13 +733,68 @@ def make_gcp_log_sink(*, config: GcpLogSinkConfig) -> GcpLogSink:
message=msg,
) from exc
+ source = _credentials_source(config=config)
+ remedy = _credentials_remedy(config=config)
client: Any
if config.credentials_file_path is not None:
- client = cloud_logging.Client.from_service_account_json( # pyright: ignore[reportUnknownMemberType]
- config.credentials_file_path,
- project=config.project_id,
- )
+ # The key is loaded apart from the client, so that the catch covers the file and nothing else: a
+ # key file that is missing or unreadable raises ``OSError``, and one that is not JSON or holds no
+ # valid key raises ``ValueError``, the auth library's ``MalformedError`` among them. The client
+ # raises ``OSError`` too when no project can be determined, which is not a credentials failure.
+ # Handed the key, it takes the project from it when ``project_id`` is unset, as the library's own
+ # ``from_service_account_json`` does.
+ try:
+ key_credentials = google_service_account.Credentials.from_service_account_file( # pyright: ignore[reportUnknownMemberType]
+ config.credentials_file_path
+ )
+ except (OSError, ValueError) as exc:
+ msg = _unloaded_credentials_message(source=source, remedy=remedy, failure=exc)
+ raise GcpLogSinkCredentialsError(msg) from exc
+ client = cloud_logging.Client(project=config.project_id, credentials=key_credentials)
else:
- client = cloud_logging.Client(project=config.project_id)
+ try:
+ client = cloud_logging.Client(project=config.project_id)
+ except google_auth_exceptions.DefaultCredentialsError as exc:
+ msg = _unloaded_credentials_message(source=source, remedy=remedy, failure=exc)
+ raise GcpLogSinkCredentialsError(msg) from exc
+
+ # The object the client authenticates every call with, after the scoping it applied: refreshing it
+ # proves the credentials the transport will use, and for credentials that fetch a token, the token
+ # it earns is the one the first export sends. A service-account key over gRPC signs its own token
+ # instead, so there the refresh costs one exchange the transport never makes, and proves the key.
+ # The library keeps the object on a private attribute, which the contract test pins.
+ credentials: Any = client._credentials # ruff: ignore[private-member-access]
+
+ def refresh_credentials() -> None:
+ # The auth library raises a 5xx or a 429 from the IAM Credentials API, from a subject-token URL or
+ # from the STS exchange as a refusal that neither declares itself retryable nor carries the status,
+ # so a transient outage of those endpoints would read as credentials Google refuses and stop the
+ # boot. The status is read instead from a response hook on the request's own session, which keeps
+ # the request the library's type: the metadata client reaches into its session. A refusal after an
+ # answer that says to retry is raised again as retryable; the token endpoint's own already is.
+ answered_statuses: list[int] = []
+
+ def record_status( # kw-only: ignore — requests calls a hook with the response positionally
+ response: requests.Response, **_hook_kwargs: Any
+ ) -> None:
+ answered_statuses.append(response.status_code)
+
+ session = requests.Session()
+ session.hooks["response"].append(record_status)
+ try:
+ credentials.refresh(GoogleAuthRequest(session=session))
+ except google_auth_exceptions.GoogleAuthError as exc:
+ last_status = answered_statuses[-1] if answered_statuses else None
+ if getattr(exc, "retryable", False) is True or last_status is None or not says_to_retry(status=last_status):
+ raise
+ msg = f"The credential endpoint answered {last_status}, a status that says to retry ({describe_failure(failure=exc)})"
+ raise google_auth_exceptions.RefreshError(msg, retryable=True) from exc
+
+ credentials_check = GcpCredentialsCheck(
+ refresh=refresh_credentials,
+ transport_error_types=(google_auth_exceptions.TransportError,),
+ source=source,
+ )
+ confirm_credentials_at_boot(credentials_check=credentials_check, remedy=remedy)
transport = cast("GcpLogTransport", BackgroundThreadTransport(client, config.log_name))
- return GcpLogSink(transport=transport, project=str(client.project))
+ return GcpLogSink(transport=transport, project=str(client.project), credentials_check=credentials_check)
diff --git a/pipelex/tools/log/log_sink.py b/pipelex/tools/log/log_sink.py
index 72d13d5a8..c50c9ab9d 100644
--- a/pipelex/tools/log/log_sink.py
+++ b/pipelex/tools/log/log_sink.py
@@ -139,7 +139,7 @@ class LogSink(ABC):
target there comes back whole; one that was handed its target built, and whose close released
it, cannot, and says so from ``make_handler`` rather than build a handler on the dead target:
the ``otlp`` sink, whose close shuts down the provider and the processor it was constructed
- with, is that case.
+ with, is that case, and so is the ``gcp`` sink, whose close stops its transport's worker.
"""
self._handler = None
diff --git a/tests/data/errors/error_identity.txt b/tests/data/errors/error_identity.txt
index 87aafe4d1..9f6e13734 100644
--- a/tests/data/errors/error_identity.txt
+++ b/tests/data/errors/error_identity.txt
@@ -108,6 +108,7 @@ GatewaySearchResponseError | Gateway search response | https://docs.pipelex.com/
GatewayTelemetryManagerInjectedError | Gateway telemetry manager injected | https://docs.pipelex.com/latest/errors/gateway-telemetry-manager-injected-error/
GatewayTermsNotAcceptedError | Gateway terms not accepted | https://docs.pipelex.com/latest/errors/gateway-terms-not-accepted-error/
GatewayUnknownModelError | Gateway unknown model | https://docs.pipelex.com/latest/errors/gateway-unknown-model-error/
+GcpLogSinkCredentialsError | Log sink credentials missing or refused | https://docs.pipelex.com/latest/errors/gcp-log-sink-credentials-error/
GeneratedImageError | Generated image | https://docs.pipelex.com/latest/errors/generated-image-error/
GoogleImgGenWorkerError | Google img gen worker | https://docs.pipelex.com/latest/errors/google-img-gen-worker-error/
GoogleLLMWorkerError | Google LLM worker | https://docs.pipelex.com/latest/errors/google-llm-worker-error/
diff --git a/tests/unit/pipelex/tools/test_log_gcp_client_library_contract.py b/tests/unit/pipelex/tools/test_log_gcp_client_library_contract.py
index 02b6e5876..deee96d2f 100644
--- a/tests/unit/pipelex/tools/test_log_gcp_client_library_contract.py
+++ b/tests/unit/pipelex/tools/test_log_gcp_client_library_contract.py
@@ -6,7 +6,8 @@ no spelling for our ``VERBOSE`` and ``DEV``, and it rejects the library's own ex
names and the thread name the library chooses. Every one of those is an assumption about a third party
whose change would break a production sink silently and no other test would see: a renamed worker
thread or a renamed transport logger would let a refused batch be reported through the pipeline that
-refused it, which is a spin that never ends and a deadlock at exit.
+refused it, which is a spin that never ends and a deadlock at exit. The sink also refreshes, at boot,
+the credentials the client keeps on a private attribute: renamed, the boot would fail on every process.
"""
from __future__ import annotations
@@ -14,6 +15,8 @@ from __future__ import annotations
import logging
from typing import Any, cast
+from google.auth.credentials import AnonymousCredentials
+from google.cloud import logging as cloud_logging
from google.cloud.logging_v2.entries import StructEntry
from google.cloud.logging_v2.handlers.transports.background_thread import (
_WORKER_THREAD_NAME, # pyright: ignore[reportPrivateUsage]
@@ -65,3 +68,9 @@ class TestTheClientLibraryContract:
assert any(transport_logger.name == prefix or transport_logger.name.startswith(f"{prefix}.") for prefix in GCP_LOGGING_LOGGER_PREFIXES)
assert transport_logger.propagate
assert transport_logger.isEnabledFor(logging.ERROR)
+
+ def test_the_client_keeps_the_credentials_it_authenticates_with_where_the_sink_refreshes_them(self) -> None:
+ credentials = AnonymousCredentials()
+ client: Any = cloud_logging.Client(project="p", credentials=credentials)
+
+ assert client._credentials is credentials # ruff: ignore[private-member-access]
diff --git a/tests/unit/pipelex/tools/test_log_gcp_sink.py b/tests/unit/pipelex/tools/test_log_gcp_sink.py
index a3992c024..9fe5c7fa9 100644
--- a/tests/unit/pipelex/tools/test_log_gcp_sink.py
+++ b/tests/unit/pipelex/tools/test_log_gcp_sink.py
@@ -26,6 +26,7 @@ from pipelex.tools.log.gcp_log_sink import (
GCP_WORKER_THREAD_NAME,
LOGGER_KEY,
MESSAGE_KEY,
+ GcpCredentialsCheck,
GcpExportPathFilter,
GcpLogSeverity,
GcpLogSink,
@@ -159,6 +160,18 @@ class CapturingHandler(logging.Handler):
self.records.append(record)
+def _refuse_the_grant() -> None:
+ """A refresh the token endpoint refuses, as it refuses a revoked refresh token."""
+ msg = "invalid_grant: Token has been expired or revoked."
+ raise PermissionError(msg)
+
+
+def _miss_the_endpoint() -> None:
+ """A refresh that cannot reach the token endpoint."""
+ msg = "connection refused"
+ raise ConnectionError(msg)
+
+
class TestGcpLogSink:
@pytest.fixture
def gcp_log(self, caplog: pytest.LogCaptureFixture) -> Iterator[tuple[Log, FakeTransport]]:
@@ -308,6 +321,24 @@ class TestGcpLogSink:
assert transport.flush_count > 0
+ def test_the_same_sink_installed_again_after_a_reset_is_refused_rather_than_sending_nothing(self) -> None:
+ """The teardown closes the transport, whose worker thread nothing starts again, so a second install must say so."""
+ transport = FakeTransport()
+ sink = GcpLogSink(transport=transport, project=PROJECT)
+ fresh = Log()
+ fresh.configure(log_config=_package_log_config())
+ fresh.install_sink(sink)
+ fresh.reset()
+ fresh.configure(log_config=_package_log_config())
+ try:
+ with pytest.raises(RuntimeError, match="installed once already"):
+ fresh.install_sink(sink)
+
+ assert fresh.sink is None
+ assert sink.processors == []
+ finally:
+ fresh.reset()
+
def test_the_client_librarys_own_export_failure_is_rejected_rather_than_exported(self, gcp_log: tuple[Log, FakeTransport]) -> None:
"""The library reports a refused batch at ``ERROR``; exported, that report fails with the batch and is reported again."""
_, transport = gcp_log
@@ -394,6 +425,122 @@ class TestGcpLogSink:
with pytest.raises(RuntimeError, match="the API refused the batch"):
handler.flush()
+ @pytest.mark.parametrize(
+ ("credentials_check", "cause"),
+ [
+ (None, None),
+ (
+ GcpCredentialsCheck(refresh=lambda: None, transport_error_types=(ConnectionError,), source="the test credentials"),
+ "A refresh of the test credentials still succeeds, so the Cloud Logging API or the network is what holds the write.",
+ ),
+ (
+ GcpCredentialsCheck(refresh=_refuse_the_grant, transport_error_types=(ConnectionError,), source="the test credentials"),
+ "A refresh of the test credentials is now refused, which is why: PermissionError: invalid_grant: Token has been expired or revoked.",
+ ),
+ (
+ GcpCredentialsCheck(refresh=_miss_the_endpoint, transport_error_types=(ConnectionError,), source="the test credentials"),
+ "A refresh of the test credentials cannot reach the credential endpoint either: ConnectionError: connection refused",
+ ),
+ ],
+ )
+ def test_a_flush_that_runs_out_of_time_says_so_with_what_a_fresh_refresh_answers(
+ self,
+ mocker: MockerFixture,
+ credentials_check: GcpCredentialsCheck | None,
+ cause: str | None,
+ ) -> None:
+ """The library's close only counts what is still queued; a process whose writes are held must be told, and why."""
+ mocker.patch("pipelex.tools.log.gcp_log_sink.FLUSH_TIMEOUT_SECONDS", 0.2)
+ stderr = CapturingHandler()
+ mocker.patch.object(logging, "lastResort", stderr)
+ transport = NeverDrainingTransport()
+ handler = GcpLogSink(transport=transport, project=PROJECT, credentials_check=credentials_check).make_handler()
+
+ try:
+ handler.flush()
+ finally:
+ transport.released.set()
+
+ (record,) = stderr.records
+ message = record.getMessage()
+ assert message.startswith(
+ "The 'gcp' log sink's transport was still writing to Cloud Logging when the 0.2-second flush deadline passed, "
+ "and the records it still holds are lost unless it sends them while it closes."
+ )
+ if cause is None:
+ assert message.endswith("while it closes.")
+ else:
+ assert message.endswith(f" {cause}")
+
+ def test_a_flush_that_drains_in_time_says_nothing(self, mocker: MockerFixture) -> None:
+ stderr = CapturingHandler()
+ mocker.patch.object(logging, "lastResort", stderr)
+ transport = FakeTransport()
+ handler = GcpLogSink(transport=transport, project=PROJECT).make_handler()
+
+ handler.flush()
+
+ assert transport.flush_count == 1
+ assert stderr.records == []
+
+ def test_a_closed_handler_neither_flushes_nor_closes_again(self, mocker: MockerFixture) -> None:
+ """The stdlib's shutdown at exit flushes every handler still alive, and a drain left running keeps this one alive."""
+ mocker.patch("pipelex.tools.log.gcp_log_sink.FLUSH_TIMEOUT_SECONDS", 0.2)
+ stderr = CapturingHandler()
+ mocker.patch.object(logging, "lastResort", stderr)
+ transport = NeverDrainingTransport()
+ handler = GcpLogSink(transport=transport, project=PROJECT).make_handler()
+ closing = FakeTransport()
+ closed_handler = GcpLogSink(transport=closing, project=PROJECT).make_handler()
+
+ handler.close()
+ started = time.monotonic()
+ handler.flush()
+ waited = time.monotonic() - started
+ closed_handler.close()
+ closed_handler.close()
+
+ assert not transport.entered.is_set(), "a closed handler reached its transport's flush"
+ assert waited < 0.2
+ assert stderr.records == []
+ assert closing.close_count == 1
+
+ def test_the_credentials_check_at_teardown_answers_while_the_shutdown_holds_the_handler_lock(self, mocker: MockerFixture) -> None:
+ """The stdlib's shutdown at exit holds this lock around the flush, and the refresh the flush starts logs through the handler it waits on."""
+ mocker.patch("pipelex.tools.log.gcp_log_sink.FLUSH_TIMEOUT_SECONDS", 0.2)
+ mocker.patch("pipelex.tools.log.gcp_log_sink.CREDENTIALS_CHECK_TIMEOUT_SECONDS", 2.0)
+ stderr = CapturingHandler()
+ mocker.patch.object(logging, "lastResort", stderr)
+ auth_logger = logging.getLogger(f"{__name__}.auth")
+ auth_logger.propagate = False
+
+ def refresh_after_a_retry() -> None:
+ auth_logger.warning("Compute Engine Metadata server unavailable on attempt 1 of 5.")
+
+ transport = NeverDrainingTransport()
+ credentials_check = GcpCredentialsCheck(
+ refresh=refresh_after_a_retry, transport_error_types=(ConnectionError,), source="the test credentials"
+ )
+ handler = GcpLogSink(transport=transport, project=PROJECT, credentials_check=credentials_check).make_handler()
+ auth_logger.addHandler(handler)
+
+ handler.acquire()
+ try:
+ handler.flush()
+ finally:
+ handler.release()
+ auth_logger.removeHandler(handler)
+ transport.released.set()
+
+ assert [record.getMessage() for record in stderr.records] == [
+ "Compute Engine Metadata server unavailable on attempt 1 of 5.",
+ (
+ "The 'gcp' log sink's transport was still writing to Cloud Logging when the 0.2-second flush deadline passed, "
+ "and the records it still holds are lost unless it sends them while it closes. "
+ "A refresh of the test credentials still succeeds, so the Cloud Logging API or the network is what holds the write."
+ ),
+ ]
+
def test_an_export_failure_is_printed_on_stderr_rather_than_exported(self, gcp_log: tuple[Log, FakeTransport], mocker: MockerFixture) -> None:
"""The library reports a refused batch only through this record, and the sink's own handler is never the place it can go."""
_, transport = gcp_log
diff --git a/tests/unit/pipelex/tools/test_log_gcp_sink_credentials.py b/tests/unit/pipelex/tools/test_log_gcp_sink_credentials.py
new file mode 100644
index 000000000..596fdc14f
--- /dev/null
+++ b/tests/unit/pipelex/tools/test_log_gcp_sink_credentials.py
@@ -0,0 +1,489 @@
+from __future__ import annotations
+
+import io
+import json
+import logging
+import threading
+import time
+from http.server import BaseHTTPRequestHandler, HTTPServer
+from typing import TYPE_CHECKING, Any, ClassVar
+
+import pytest
+import requests
+from cryptography.hazmat.primitives import serialization
+from cryptography.hazmat.primitives.asymmetric import rsa
+from google.auth import compute_engine
+from google.auth import exceptions as google_auth_exceptions
+from requests.adapters import HTTPAdapter
+from typing_extensions import override
+
+from pipelex.tools.log.exceptions import GcpLogSinkCredentialsError
+from pipelex.tools.log.gcp_log_sink import (
+ GCP_WORKER_THREAD_NAME,
+ GcpCredentialsCheck,
+ GcpCredentialsOutcome,
+ confirm_credentials_at_boot,
+ make_gcp_log_sink,
+)
+from pipelex.tools.log.log_config import GcpLogSinkConfig
+
+if TYPE_CHECKING:
+ from collections.abc import Callable, Iterator
+ from pathlib import Path
+
+ from pytest_mock import MockerFixture
+
+SOURCE = "the test credentials"
+REMEDY = "Renew the test credentials"
+SOURCE_TOKEN_URI = "https://oauth2.googleapis.com/token"
+# What `gcloud auth application-default login --impersonate-service-account` writes.
+IMPERSONATED_ADC: dict[str, Any] = {
+ "type": "impersonated_service_account",
+ "service_account_impersonation_url": (
+ "https://iamcredentials.googleapis.com/v1/projects/-/serviceAccounts/[email protected]:generateAccessToken"
+ ),
+ "source_credentials": {
+ "type": "authorized_user",
+ "client_id": "a-client-id",
+ "client_secret": "a-client-secret",
+ "refresh_token": "a-refresh-token",
+ "token_uri": SOURCE_TOKEN_URI,
+ },
+ "delegates": [],
+}
+
+
+class RefusedError(Exception):
+ """Stands in for the auth library's refusal type."""
+
+
+class UnreachableError(Exception):
+ """Stands in for the auth library's transport error type."""
+
+
+class RetryableRefusalError(RefusedError):
+ """A refusal that declares itself retryable, as the auth library's does for a 5xx from the token endpoint."""
+
+ @property
+ def retryable(self) -> bool:
+ return True
+
+
+class MetadataResponse:
+ """What the auth library's metadata client reads of an HTTP response: the status, the body and the content type."""
+
+ def __init__(self, *, status: int) -> None:
+ self.status = status
+ self.data = b"no service account"
+ self.headers = {"content-type": "text/html"}
+
+ @override
+ def __repr__(self) -> str:
+ return f"MetadataResponse(status={self.status})"
+
+
+class CapturingHandler(logging.Handler):
+ """Stands in for the stdlib's last-resort handler, keeping what it would have printed on stderr."""
+
+ def __init__(self) -> None:
+ super().__init__(level=logging.WARNING)
+ self.records: list[logging.LogRecord] = []
+
+ @override
+ def emit(self, record: logging.LogRecord) -> None:
+ self.records.append(record)
+
+
+class TokenEndpointRefusingEveryGrant(BaseHTTPRequestHandler):
+ """A token endpoint answering every grant as Google answers a revoked or deleted credential."""
+
+ grant_count: ClassVar[int] = 0
+
+ def do_POST(self) -> None:
+ TokenEndpointRefusingEveryGrant.grant_count += 1
+ self.rfile.read(int(self.headers.get("Content-Length", "0")))
+ body = json.dumps({"error": "invalid_grant", "error_description": "Invalid JWT Signature."}).encode()
+ self.send_response(400)
+ self.send_header("Content-Type", "application/json")
+ self.send_header("Content-Length", str(len(body)))
+ self.end_headers()
+ self.wfile.write(body)
+
+ @override
+ def log_message(self, format: str, *args: Any) -> None:
+ return None
+
+
+def _check(*, refresh: Callable[[], None]) -> GcpCredentialsCheck:
+ return GcpCredentialsCheck(refresh=refresh, transport_error_types=(UnreachableError,), source=SOURCE)
+
+
+def _refused() -> None:
+ msg = "invalid_grant: Bad Request"
+ raise RefusedError(msg)
+
+
+def _unreachable() -> None:
+ msg = "connection refused"
+ raise UnreachableError(msg)
+
+
+def _refused_from_unreachable() -> None:
+ """The metadata-server credentials' shape: the transport failure re-raised as a refusal."""
+ try:
+ _unreachable()
+ except UnreachableError as exc:
+ msg = "metadata server unreachable"
+ raise RefusedError(msg) from exc
+
+
+def _refused_from_none_while_unreachable() -> None:
+ try:
+ _unreachable()
+ except UnreachableError:
+ msg = "refused, the transport failure explicitly left out of the chain"
+ raise RefusedError(msg) from None
+
+
+def _retryable_refusal() -> None:
+ msg = "503 from the token endpoint"
+ raise RetryableRefusalError(msg)
+
+
+def _unanticipated() -> None:
+ msg = "a key the auth library could not parse"
+ raise ValueError(msg)
+
+
+def _refreshed() -> None:
+ return None
+
+
+def _write_service_account_key(*, path: Path, token_uri: str, project_id: str | None = "a-test-project") -> None:
+ private_key = rsa.generate_private_key(public_exponent=65537, key_size=2048)
+ private_key_pem = private_key.private_bytes(
+ encoding=serialization.Encoding.PEM,
+ format=serialization.PrivateFormat.PKCS8,
+ encryption_algorithm=serialization.NoEncryption(),
+ ).decode()
+ key_info = {
+ "type": "service_account",
+ "private_key_id": "0123456789abcdef",
+ "private_key": private_key_pem,
+ "client_email": "[email protected]",
+ "client_id": "123456789",
+ "token_uri": token_uri,
+ }
+ if project_id is not None:
+ key_info["project_id"] = project_id
+ path.write_text(json.dumps(key_info), encoding="utf-8")
+
+
+class TestGcpLogSinkCredentials:
+ @pytest.fixture
+ def stderr(self, mocker: MockerFixture) -> CapturingHandler:
+ capture = CapturingHandler()
+ mocker.patch.object(logging, "lastResort", capture)
+ return capture
+
+ @pytest.fixture
+ def refusing_token_endpoint(self) -> Iterator[str]:
+ TokenEndpointRefusingEveryGrant.grant_count = 0
+ server = HTTPServer(("127.0.0.1", 0), TokenEndpointRefusingEveryGrant)
+ serving = threading.Thread(target=server.serve_forever, daemon=True)
+ serving.start()
+ try:
+ yield f"http://127.0.0.1:{server.server_address[1]}/token"
+ finally:
+ server.shutdown()
+ server.server_close()
+ serving.join(timeout=5)
+
+ @pytest.mark.parametrize(
+ ("refresh", "outcome"),
+ [
+ (_refreshed, GcpCredentialsOutcome.REFRESHED),
+ (_refused, GcpCredentialsOutcome.REFUSED),
+ (_unreachable, GcpCredentialsOutcome.UNREACHABLE),
+ (_refused_from_unreachable, GcpCredentialsOutcome.UNREACHABLE),
+ (_retryable_refusal, GcpCredentialsOutcome.UNREACHABLE),
+ (_refused_from_none_while_unreachable, GcpCredentialsOutcome.REFUSED),
+ (_unanticipated, GcpCredentialsOutcome.REFUSED),
+ ],
+ )
+ def test_a_refresh_is_classified_by_the_whole_chain_it_raised(self, refresh: Callable[[], None], outcome: GcpCredentialsOutcome) -> None:
+ """A transport failure anywhere on the chain the traceback prints means nothing is known of the credentials themselves."""
+ verdict = _check(refresh=refresh).run()
+
+ assert verdict.outcome == outcome
+ match verdict.outcome:
+ case GcpCredentialsOutcome.REFRESHED:
+ assert verdict.failure is None
+ case GcpCredentialsOutcome.REFUSED | GcpCredentialsOutcome.UNREACHABLE | GcpCredentialsOutcome.UNANSWERED:
+ assert verdict.failure is not None
+
+ def test_a_refresh_that_outlives_its_deadline_is_unanswered_without_waiting_for_it(self, mocker: MockerFixture) -> None:
+ """The auth library's own request timeout is two minutes, and a boot must not wait on it."""
+ mocker.patch("pipelex.tools.log.gcp_log_sink.CREDENTIALS_CHECK_TIMEOUT_SECONDS", 0.2)
+ released = threading.Event()
+
+ def hang() -> None:
+ # Bounded only so a regression fails the test instead of hanging the suite.
+ released.wait(timeout=10)
+
+ started = time.monotonic()
+ try:
+ verdict = _check(refresh=hang).run()
+ waited = time.monotonic() - started
+ finally:
+ released.set()
+
+ assert verdict.outcome == GcpCredentialsOutcome.UNANSWERED
+ assert verdict.failure is None
+ assert waited < 5, f"the check waited {waited:.1f}s on a refresh that never answers"
+
+ def test_credentials_refused_at_boot_stop_it_naming_them_and_the_remedy(self, stderr: CapturingHandler) -> None:
+ with pytest.raises(GcpLogSinkCredentialsError) as exc_info:
+ confirm_credentials_at_boot(credentials_check=_check(refresh=_refused), remedy=REMEDY)
+
+ message = str(exc_info.value)
+ assert SOURCE in message
+ assert "RefusedError: invalid_grant: Bad Request" in message
+ assert REMEDY in message
+ assert "'json' sink" in message
+ assert isinstance(exc_info.value.__cause__, RefusedError)
+ assert stderr.records == []
+
+ @pytest.mark.parametrize(
+ ("refresh", "reason"),
+ [
+ (_unreachable, "could not reach the credential endpoint (UnreachableError: connection refused)"),
+ (_refused_from_unreachable, "could not reach the credential endpoint (RefusedError: metadata server unreachable)"),
+ ],
+ )
+ def test_credentials_that_cannot_be_reached_at_boot_are_said_on_stderr_and_the_boot_goes_on(
+ self,
+ stderr: CapturingHandler,
+ refresh: Callable[[], None],
+ reason: str,
+ ) -> None:
+ confirm_credentials_at_boot(credentials_check=_check(refresh=refresh), remedy=REMEDY)
+
+ (record,) = stderr.records
+ assert record.levelno == logging.WARNING
+ assert f"could not confirm at boot that Google accepts {SOURCE}: a refresh {reason}." in record.getMessage()
+ assert "at the latest when the process tears the sink down" in record.getMessage()
+
+ def test_a_refresh_that_does_not_answer_at_boot_is_said_on_stderr_and_the_boot_goes_on(
+ self, stderr: CapturingHandler, mocker: MockerFixture
+ ) -> None:
+ mocker.patch("pipelex.tools.log.gcp_log_sink.CREDENTIALS_CHECK_TIMEOUT_SECONDS", 0.2)
+ released = threading.Event()
+
+ def hang() -> None:
+ # Bounded only so a regression fails the test instead of hanging the suite.
+ released.wait(timeout=10)
+
+ try:
+ confirm_credentials_at_boot(credentials_check=_check(refresh=hang), remedy=REMEDY)
+ finally:
+ released.set()
+
+ (record,) = stderr.records
+ assert f"could not confirm at boot that Google accepts {SOURCE}: a refresh did not answer within 0.2 seconds." in record.getMessage()
+
+ def test_credentials_that_refresh_at_boot_say_nothing(self, stderr: CapturingHandler) -> None:
+ confirm_credentials_at_boot(credentials_check=_check(refresh=_refreshed), remedy=REMEDY)
+
+ assert stderr.records == []
+
+ @pytest.mark.parametrize(
+ ("response", "outcome"),
+ [
+ (None, GcpCredentialsOutcome.UNREACHABLE),
+ (MetadataResponse(status=404), GcpCredentialsOutcome.REFUSED),
+ (MetadataResponse(status=403), GcpCredentialsOutcome.REFUSED),
+ (MetadataResponse(status=408), GcpCredentialsOutcome.UNREACHABLE),
+ (MetadataResponse(status=429), GcpCredentialsOutcome.UNREACHABLE),
+ (MetadataResponse(status=502), GcpCredentialsOutcome.UNREACHABLE),
+ ],
+ )
+ def test_the_auth_librarys_metadata_server_shape_is_classified_by_the_answer_it_carries(
+ self, response: MetadataResponse | None, outcome: GcpCredentialsOutcome
+ ) -> None:
+ """The compute-engine credentials re-raise the transport failure as a ``RefreshError``, so the outer class alone would read as a refusal.
+
+ The transport failure carries the response when the server answered with a status the client does not retry.
+ """
+
+ def metadata_server_fails() -> None:
+ try:
+ msg = "Failed to retrieve the service account from the Google Compute Engine metadata service."
+ if response is None:
+ raise google_auth_exceptions.TransportError(msg)
+ raise google_auth_exceptions.TransportError(msg, response)
+ except google_auth_exceptions.TransportError as exc:
+ raise google_auth_exceptions.RefreshError(exc) from exc
+
+ check = GcpCredentialsCheck(refresh=metadata_server_fails, transport_error_types=(google_auth_exceptions.TransportError,), source=SOURCE)
+
+ assert check.run().outcome == outcome
+
+ def test_a_machine_whose_metadata_server_has_no_service_account_stops_the_boot(self) -> None:
+ """The reported shape end to end: the library's own compute-engine credentials, and a metadata server answering 404."""
+ metadata_requests: list[str] = []
+
+ def metadata_server(url: str, **kwargs: Any) -> MetadataResponse:
+ metadata_requests.append(f"{kwargs.get('method', 'GET')} {url}")
+ return MetadataResponse(status=404)
+
+ credentials: Any = compute_engine.Credentials()
+ check = GcpCredentialsCheck(
+ refresh=lambda: credentials.refresh(metadata_server),
+ transport_error_types=(google_auth_exceptions.TransportError,),
+ source=SOURCE,
+ )
+
+ with pytest.raises(GcpLogSinkCredentialsError) as exc_info:
+ confirm_credentials_at_boot(credentials_check=check, remedy=REMEDY)
+
+ assert "Status: 404" in str(exc_info.value)
+ assert isinstance(exc_info.value.__cause__, google_auth_exceptions.RefreshError)
+ assert len(metadata_requests) == 1, "a final answer is not retried"
+
+ @pytest.mark.parametrize(
+ ("iam_status", "is_refused"),
+ [
+ (503, False),
+ (502, False),
+ (429, False),
+ (403, True),
+ ],
+ )
+ def test_impersonated_credentials_are_refused_only_by_an_answer_that_does_not_say_to_retry(
+ self,
+ stderr: CapturingHandler,
+ monkeypatch: pytest.MonkeyPatch,
+ mocker: MockerFixture,
+ tmp_path: Path,
+ iam_status: int,
+ is_refused: bool,
+ ) -> None:
+ """The auth library raises the IAM Credentials API's 5xx or 429 as a refusal carrying no status, so the sink reads it off the response."""
+ adc_path = tmp_path / "application_default_credentials.json"
+ adc_path.write_text(json.dumps(IMPERSONATED_ADC), encoding="utf-8")
+ monkeypatch.setenv("GOOGLE_APPLICATION_CREDENTIALS", str(adc_path))
+ requested_urls: list[str] = []
+
+ def answer(_adapter: HTTPAdapter, request: requests.PreparedRequest, **_send_kwargs: Any) -> requests.Response:
+ url = str(request.url)
+ requested_urls.append(url)
+ response = requests.Response()
+ response.request = request
+ response.url = url
+ response.headers["content-type"] = "application/json"
+ if url == SOURCE_TOKEN_URI:
+ response.status_code = 200
+ body: dict[str, Any] = {"access_token": "a-source-token", "expires_in": 3600}
+ else:
+ response.status_code = iam_status
+ body = {"error": {"code": iam_status, "message": "The IAM Credentials API answered."}}
+ response.raw = io.BytesIO(json.dumps(body).encode())
+ return response
+
+ mocker.patch.object(HTTPAdapter, "send", autospec=True, side_effect=answer)
+ config = GcpLogSinkConfig(log_name="pipelex", project_id="a-test-project")
+
+ if is_refused:
+ with pytest.raises(GcpLogSinkCredentialsError, match="was refused at boot"):
+ make_gcp_log_sink(config=config)
+ assert stderr.records == []
+ else:
+ sink = make_gcp_log_sink(config=config)
+ sink.make_handler().close()
+ (record,) = stderr.records
+ assert (
+ "could not confirm at boot that Google accepts the Application Default Credentials: a refresh could not reach the credential "
+ f"endpoint (RefreshError: The credential endpoint answered {iam_status}, a status that says to retry"
+ ) in record.getMessage()
+ assert requested_urls[0] == SOURCE_TOKEN_URI
+ assert "iamcredentials.googleapis.com" in requested_urls[-1]
+
+ def test_application_default_credentials_that_cannot_be_found_stop_the_boot_naming_the_remedy(
+ self, monkeypatch: pytest.MonkeyPatch, tmp_path: Path
+ ) -> None:
+ monkeypatch.setenv("GOOGLE_APPLICATION_CREDENTIALS", str(tmp_path / "missing.json"))
+
+ with pytest.raises(GcpLogSinkCredentialsError) as exc_info:
+ make_gcp_log_sink(config=GcpLogSinkConfig(log_name="pipelex", project_id="a-test-project"))
+
+ message = str(exc_info.value)
+ assert "could not load the Application Default Credentials" in message
+ assert "gcloud auth application-default login" in message
+ assert isinstance(exc_info.value.__cause__, google_auth_exceptions.DefaultCredentialsError)
+
+ @pytest.mark.parametrize(
+ ("key_content", "failure_type"),
+ [
+ (None, FileNotFoundError),
+ ("", json.JSONDecodeError),
+ ("not json", json.JSONDecodeError),
+ (json.dumps({"type": "service_account"}), google_auth_exceptions.MalformedError),
+ (
+ json.dumps(
+ {
+ "type": "service_account",
+ "private_key": "-----BEGIN PRIVATE KEY-----\nnot a key\n-----END PRIVATE KEY-----\n",
+ "client_email": "[email protected]",
+ "token_uri": "https://oauth2.googleapis.com/token",
+ }
+ ),
+ ValueError,
+ ),
+ ],
+ )
+ def test_a_key_file_that_cannot_be_loaded_stops_the_boot_naming_the_file(
+ self, tmp_path: Path, key_content: str | None, failure_type: type[Exception]
+ ) -> None:
+ """A missing file, one that is not JSON, and one whose key is incomplete or not a key each raise something else from the library."""
+ key_path = tmp_path / "key.json"
+ if key_content is not None:
+ key_path.write_text(key_content, encoding="utf-8")
+
+ with pytest.raises(GcpLogSinkCredentialsError) as exc_info:
+ make_gcp_log_sink(config=GcpLogSinkConfig(log_name="pipelex", project_id="a-test-project", credentials_file_path=str(key_path)))
+
+ message = str(exc_info.value)
+ assert f"could not load the service-account key at '{key_path}'" in message
+ assert "credentials_file_path" in message
+ assert "'json' sink" in message
+ assert isinstance(exc_info.value.__cause__, failure_type)
+
+ def test_a_key_file_without_a_project_is_not_a_credentials_failure(
+ self, tmp_path: Path, monkeypatch: pytest.MonkeyPatch, mocker: MockerFixture
+ ) -> None:
+ """The client raises ``OSError`` when no project can be determined, which the key file's catch must not relabel."""
+ for variable in ("GOOGLE_CLOUD_PROJECT", "GCLOUD_PROJECT"):
+ monkeypatch.delenv(variable, raising=False)
+ mocker.patch("google.cloud.client._determine_default_project", return_value=None)
+ key_path = tmp_path / "key.json"
+ _write_service_account_key(path=key_path, token_uri="http://127.0.0.1:9/token", project_id=None)
+
+ with pytest.raises(OSError, match="Project was not passed"):
+ make_gcp_log_sink(config=GcpLogSinkConfig(log_name="pipelex", credentials_file_path=str(key_path)))
+
+ def test_a_key_google_refuses_stops_the_boot_before_the_transport_starts(self, tmp_path: Path, refusing_token_endpoint: str) -> None:
+ """The reported failure end to end: the client the factory builds, the refresh of the credentials it holds, the refusal it meets."""
+ key_path = tmp_path / "key.json"
+ _write_service_account_key(path=key_path, token_uri=refusing_token_endpoint)
+ workers_before = {thread.ident for thread in threading.enumerate() if thread.name == GCP_WORKER_THREAD_NAME}
+
+ with pytest.raises(GcpLogSinkCredentialsError) as exc_info:
+ make_gcp_log_sink(config=GcpLogSinkConfig(log_name="pipelex", project_id="a-test-project", credentials_file_path=str(key_path)))
+
+ message = str(exc_info.value)
+ assert f"a refresh of the service-account key at '{key_path}' was refused at boot" in message
+ assert "invalid_grant" in message
+ assert isinstance(exc_info.value.__cause__, google_auth_exceptions.RefreshError)
+ assert TokenEndpointRefusingEveryGrant.grant_count == 1
+ assert {thread.ident for thread in threading.enumerate() if thread.name == GCP_WORKER_THREAD_NAME} == workers_before