LLM-Driven Extraction of Unstructured Data — Built for API Deployments & ETL Pipeline Workflows
Python ai-agentsdata-engineeringdocument-aigenerative-aiidpjson-extractionllmmcp-serverocrpdf-extractionprompt-engineeringstructured-output
"""Prefork supervisor for the PG-queue consumer.
Forks ``WORKER_PG_QUEUE_CONSUMER_CONCURRENCY`` copies of the single-threaded
:class:`~queue_backend.pg_queue.consumer.PgQueueConsumer` so multiple file
batches run in parallel — the PG analogue of Celery's ``--pool=prefork
--concurrency=N``. ``SELECT … FOR UPDATE SKIP LOCKED`` distributes work across the
children (and across replicas): each child claims distinct rows, a single
execution is still capped by ``MAX_PARALLEL_FILE_BATCHES``, and total live
parallelism = ``concurrency × replicas`` (k8s HPA scales the replica count).
**Process model** (matches Celery prefork — the cloud-trusted choice): each child
is a fully isolated process with its own DB connections and thread-local
``StateStore`` — no shared mutable state, no thread-safety surface. A child crash
is isolated and re-forked (rate-limited); its in-flight message redelivers via
``vt`` (at-least-once). The **consumer code is unchanged** — concurrency is purely
a launch concern. ``CONCURRENCY = 1`` keeps the plain single-process ``main()``
path (byte-identical to before this module existed).
**Health**: the supervisor owns the single liveness port and reports the *fleet's*
freshness — the staleness of the oldest-polling child (each child publishes its
last-poll wall-time into a shared array). A child that dies is re-forked
internally (transient); a child that **crash-loops** (dies immediately N times in
a row, never reaching a real poll) forces the probe to 503 so k8s restarts the
pod rather than the supervisor masking a wedged fleet with fresh-looking re-forks.
**Child watchdog** (UN-4223, opt-in via ``CHILD_WATCHDOG``): a loaded child whose
heartbeat goes stale past ``HEALTH_STALE_SECONDS``, or a child still not loaded
that long after its fork, is SIGKILLed and re-forked on its own. Its lease-renewal
thread dies with it, so the claim lapses and the reaper redelivers the message
(the poison cap bounds a task that hangs on every attempt). The siblings keep
running, so a pod without a liveness probe on ``/health`` loses one slot for one
stale window instead of the whole fleet. ``/health`` itself still reports 503 from
the stale crossing until the replacement loads (~20s), because ``oldest_age``
counts the killed slot's heartbeat. SIGKILL rather than SIGTERM: a graceful stop
waits for the in-flight task, which is the thing that is hung.
**Readiness** (UN-4136): the same port serves ``/ready``, which answers 200 only
once EVERY child has finished its ``import worker`` bootstrap and built its
consumer. ``/health`` cannot say this — the heartbeats are seeded fresh at
construction so liveness does not trip during a slow import — which is why a pod
used to go Ready ~20s after start while its N children were still importing, and
the HPA counted that multi-core start-up burst as load. A k8s ``startupProbe`` on
``/ready`` keeps the pod NotReady until the burst is over.
**Fork safety**: the initial fleet is forked while the parent is single-threaded.
Re-forks happen after the liveness daemon thread exists; the only other thread is
that probe (idle in ``select`` between requests, and CPython 3.12 re-inits the
``logging`` locks across ``fork`` via ``os.register_at_fork``), and each child
resets inherited signal handlers + does its own ``import worker`` before touching
shared resources — so an inherited held lock or the parent's ``_on_term`` cannot
wedge or mis-signal a child.
"""
from __future__ import annotations
import contextlib
import logging
import math
import multiprocessing
import os
import signal
import threading
import time
from typing import TYPE_CHECKING
if TYPE_CHECKING:
from queue_backend.pg_queue.liveness import LivenessServer
logger = logging.getLogger(__name__)
_DEFAULT_CONCURRENCY = 1
# Fork-bomb backstop for a fat-fingered env (a single machine can't usefully run
# hundreds of heavy file-processing children anyway — scale replicas instead).
_MAX_CONCURRENCY = 64
# How often each child republishes its heartbeat, and the parent reaps + checks.
_REPORT_INTERVAL_SECONDS = 1.0
_MONITOR_INTERVAL_SECONDS = 1.0
# Re-fork backoff floor and ceiling — a crash-looping child must not fork-storm.
_RESTART_MIN_INTERVAL_SECONDS = 2.0
_RESTART_MAX_BACKOFF_SECONDS = 30.0
# A child that stays up at least this long before exiting is a normal exit, not an
# immediate crash — it resets the slot's consecutive-crash counter.
_MIN_HEALTHY_UPTIME_SECONDS = 10.0
# Consecutive immediate crashes after which the fleet probe is forced unhealthy.
_CRASH_LOOP_THRESHOLD = 3
# Least time a child gets to finish ``import worker`` before the watchdog treats it
# as hung. HEALTH_STALE is sized for task runtime and can be short (ide-callback:
# 180s), while a bootstrap under a CPU cap can take minutes.
_MIN_BOOTSTRAP_BUDGET_SECONDS = 600.0
# Fallback graceful-drain budget (s, shared across all children) on shutdown, used
# only when neither an explicit override nor the consumer VT is set — see
# shutdown_grace_from_env().
# This is NOT the live value: the live grace defaults to the consumer's visibility
# timeout so a SIGTERM (deploy / HPA scale-down) lets an in-flight batch finish
# instead of a mid-flight SIGKILL that orphans it.
_DEFAULT_SHUTDOWN_GRACE_SECONDS = 30.0
def concurrency_from_env() -> int:
"""Parse ``WORKER_PG_QUEUE_CONSUMER_CONCURRENCY`` (default 1, clamped to a sane
max). 1 → the single-process path; >1 → the prefork supervisor.
"""
raw = os.environ.get("WORKER_PG_QUEUE_CONSUMER_CONCURRENCY")
if raw is None or raw == "":
return _DEFAULT_CONCURRENCY
try:
n = int(raw)
except (TypeError, ValueError) as exc:
raise ValueError(
f"Invalid WORKER_PG_QUEUE_CONSUMER_CONCURRENCY={raw!r}: {exc}"
) from exc
if n < 1:
raise ValueError(f"WORKER_PG_QUEUE_CONSUMER_CONCURRENCY must be >= 1, got {n}")
if n > _MAX_CONCURRENCY:
logger.warning(
"WORKER_PG_QUEUE_CONSUMER_CONCURRENCY=%s exceeds the %s cap; clamping "
"(scale replicas for more parallelism, not one fat process)",
n,
_MAX_CONCURRENCY,
)
n = _MAX_CONCURRENCY
return n
def _parse_bool(raw: str) -> bool:
value = raw.strip().lower()
if value in ("1", "true", "yes", "on"):
return True
if value in ("0", "false", "no", "off"):
return False
raise ValueError(f"expected a boolean, got {raw!r}")
def child_watchdog_from_env() -> float | None:
"""Heartbeat age (seconds) past which the supervisor SIGKILLs a single child, or
``None`` when the watchdog is off.
Off unless ``WORKER_PG_QUEUE_CONSUMER_CHILD_WATCHDOG`` is true. Opt-in because
it turns ``HEALTH_STALE_SECONDS`` into a hard per-task wall-clock cap: on k8s a
liveness probe already killed at that threshold (the whole pod), but a
docker-compose install has no healthcheck, so there it has never been enforced
and may sit below a legitimately long batch. Enable it only where
``HEALTH_STALE_SECONDS`` exceeds the longest legitimate task. Enabling it
without that knob is a misconfiguration (its 60s code default would kill any
task longer than a minute), so it raises.
"""
from queue_backend.pg_queue.consumer import consumer_env
if not consumer_env("CHILD_WATCHDOG", False, _parse_bool):
return None
stale: float | None = consumer_env("HEALTH_STALE_SECONDS", None, float)
if stale is None:
raise ValueError(
"WORKER_PG_QUEUE_CONSUMER_CHILD_WATCHDOG is enabled but "
"WORKER_PG_QUEUE_CONSUMER_HEALTH_STALE_SECONDS is unset — set it above "
"the longest legitimate task"
)
if not math.isfinite(stale) or stale <= 0:
raise ValueError(
"WORKER_PG_QUEUE_CONSUMER_HEALTH_STALE_SECONDS must be a finite number "
f"> 0, got {stale!r}"
)
return stale
def shutdown_grace_from_env() -> float:
"""Graceful-drain budget (seconds) on shutdown, shared across all children,
before SIGKILL.
Defaults to the consumer's visibility timeout (``WORKER_PG_QUEUE_CONSUMER_VT_SECONDS``)
— the by-design upper bound on a single task's runtime — so a graceful SIGTERM
(deploy / HPA scale-down) lets an in-flight batch finish rather than being
SIGKILLed mid-flight and orphaned. The chart sets the pod's
``terminationGracePeriodSeconds`` to VT + a buffer, so even a genuinely-wedged
child is SIGKILLed here just BEFORE k8s reaps the pod. An explicit
``WORKER_PG_QUEUE_CONSUMER_SHUTDOWN_GRACE_SECONDS`` overrides (and is honoured
as-is, e.g. a short dev drain); otherwise the VT is floored at
``_DEFAULT_SHUTDOWN_GRACE_SECONDS`` so a tiny/unset VT still gets a sane drain.
A hardcoded 30s here previously undercut the chart's ~2.5h budget by ~300×,
SIGKILLing in-flight batches on every scale-down/rollout.
"""
from queue_backend.pg_queue.consumer import _DEFAULT_VT_SECONDS, consumer_env
override: float | None = consumer_env("SHUTDOWN_GRACE_SECONDS", None, float)
if override is not None:
# A negative / non-finite drain budget is never intentional and would
# re-introduce the exact failure this module prevents: <0 or nan → 0 → every
# child SIGKILLed with no drain; inf → shutdown hangs until k8s hard-kills the
# pod. Fail fast at startup, mirroring concurrency_from_env()'s validation.
if not math.isfinite(override) or override < 0:
raise ValueError(
"WORKER_PG_QUEUE_CONSUMER_SHUTDOWN_GRACE_SECONDS must be a finite "
f"number >= 0, got {override!r}"
)
return override
vt = consumer_env("VT_SECONDS", _DEFAULT_VT_SECONDS, int)
return max(_DEFAULT_SHUTDOWN_GRACE_SECONDS, float(vt))
class _Fleet:
"""Owns the per-slot child state — pid, last-fork, heartbeat, crash count and
pending-restart schedule — keeping them mutually consistent. Slots are
validated against ``[0, concurrency)`` so a stray key can't silently desync
the structures or ``IndexError`` the shared array.
"""
def __init__(self, concurrency: int) -> None:
self._n = concurrency
# Shared, fork-inherited heartbeat slots (one last-poll wall-time per
# child). lock=False is safe: a slot is written either by the parent
# (seed, at construction, while no child owns it) OR by that child's
# heartbeat thread — never concurrently — and only read by the parent, so
# a torn double read just yields one stale sample that self-corrects.
self._heartbeats = multiprocessing.Array("d", concurrency, lock=False)
now = time.time()
for i in range(concurrency):
self._heartbeats[i] = now
# Shared, fork-inherited "finished loading" flags (one per child), read by
# /ready. Same write discipline as the heartbeats: the owning child sets
# its slot once bootstrapped; the parent clears it in reap(), when no child
# owns the slot. Zero-initialised, so a fresh fleet starts not-ready.
self._loaded = multiprocessing.Array("b", concurrency, lock=False)
self._pids: dict[int, int] = {}
self._last_fork: dict[int, float] = {}
self._consecutive_crashes: dict[int, int] = {}
self._restart_due: dict[int, float] = {} # slot -> monotonic not-before
# Written by the main thread, read by the liveness thread for /metrics; an
# int read is atomic under the GIL.
self.watchdog_kills = 0
@property
def concurrency(self) -> int:
return self._n
@property
def heartbeats(self): # noqa: ANN201
"""The shared heartbeat array (a ctypes array, passed to forked children,
which write their own slot directly).
"""
return self._heartbeats
@property
def loaded(self): # noqa: ANN201
"""The shared loaded-flag array (a ctypes array, passed to forked children,
which set their own slot once bootstrapped).
"""
return self._loaded
def _validate(self, slot: int) -> None:
if not 0 <= slot < self._n:
raise IndexError(f"slot {slot} out of range [0, {self._n})")
def record_fork(self, slot: int, pid: int) -> None:
"""Mark ``slot`` alive under ``pid``; clears any pending restart. Note the
heartbeat is deliberately NOT reseeded here — a re-forked child must earn
freshness by actually polling, so a crash-looping slot ages instead of
looking perpetually fresh.
"""
self._validate(slot)
self._pids[slot] = pid
self._last_fork[slot] = time.monotonic()
self._restart_due.pop(slot, None)
def reap(self, slot: int) -> float:
"""Drop the slot's pid + last-fork together; return the child's uptime (s).
Also clears the slot's loaded flag: its replacement must finish its own
bootstrap before the fleet counts as loaded again.
"""
forked_at = self._last_fork.pop(slot, time.monotonic())
self._pids.pop(slot, None)
self._loaded[slot] = 0
return time.monotonic() - forked_at
def schedule_restart(self, slot: int, uptime: float) -> int:
"""Record the exit and set the re-fork not-before; return the consecutive
immediate-crash count. A child that ran healthily before exiting resets the
counter; an immediate death increments it and backs the restart off
(capped) so a crash loop can't fork-storm.
"""
if uptime < _MIN_HEALTHY_UPTIME_SECONDS:
n = self._consecutive_crashes.get(slot, 0) + 1
else:
n = 0 # ran fine, then exited — not a crash loop
self._consecutive_crashes[slot] = n
backoff = min(
_RESTART_MIN_INTERVAL_SECONDS * max(1, n), _RESTART_MAX_BACKOFF_SECONDS
)
self._restart_due[slot] = time.monotonic() + backoff
return n
def due_restarts(self) -> list[int]:
"""Slots whose re-fork backoff has elapsed (oldest schedule first)."""
now = time.monotonic()
return sorted(s for s, due in self._restart_due.items() if due <= now)
def alive_items(self) -> list[tuple[int, int]]:
return list(self._pids.items())
def alive_count(self) -> int:
return len(self._pids)
def is_crash_looping(self) -> bool:
"""True if any slot has died immediately ``_CRASH_LOOP_THRESHOLD`` times in
a row — the signal that the heartbeat alone can't be trusted fresh.
Snapshots the values first (``tuple(...)`` is atomic under the GIL): this
runs in the liveness daemon thread (via :meth:`freshness`) while the main
thread mutates ``_consecutive_crashes`` in :meth:`schedule_restart`, so a
bare ``.values()`` iteration could raise "dictionary changed size during
iteration".
"""
return any(
n >= _CRASH_LOOP_THRESHOLD for n in tuple(self._consecutive_crashes.values())
)
def loaded_count(self) -> int:
"""Children that have finished their bootstrap (``import worker`` + build)."""
return sum(self._loaded)
def all_loaded(self) -> bool:
"""Readiness verdict source: True once every slot's child has loaded."""
return self.loaded_count() == self._n
def is_loaded(self, slot: int) -> bool:
self._validate(slot)
return bool(self._loaded[slot])
def fork_age(self, slot: int) -> float:
"""Seconds since ``slot``'s current child was forked."""
self._validate(slot)
return time.monotonic() - self._last_fork.get(slot, time.monotonic())
def slot_age(self, slot: int) -> float:
"""Seconds since ``slot``'s child last polled, per its published heartbeat."""
self._validate(slot)
return time.time() - self._heartbeats[slot]
def oldest_age(self) -> float:
now = time.time()
return max((now - hb for hb in self._heartbeats), default=0.0)
def freshness(self) -> float:
"""Liveness verdict source: a crash-looping fleet is force-stale (``inf``)
so the probe trips 503 even if a just-constructed child briefly looked
fresh; otherwise the oldest child's staleness (catches a wedged-alive
child).
"""
return float("inf") if self.is_crash_looping() else self.oldest_age()
def _run_child(slot: int, heartbeats, loaded) -> None: # noqa: ANN001 (ctypes arrays)
"""Build one consumer and run it forever, publishing its heartbeat.
The worker import (and any connections it opens) happens HERE, in the child —
never inherited across the fork — so each process owns its own connections.
A *guarded* daemon thread publishes the consumer's last-poll wall-time into
``heartbeats[slot]`` for the supervisor's fleet liveness, and ``loaded[slot]``
is set once the bootstrap is done, for the supervisor's ``/ready``.
"""
started = time.monotonic()
from pg_queue_consumer._bootstrap import select_source_worker_type
select_source_worker_type() # set WORKER_TYPE before importing worker
import worker # noqa: F401 — side-effect: registers the source worker's tasks
from queue_backend.pg_queue.consumer import build_consumer_from_env
consumer = build_consumer_from_env()
def _publish_heartbeat() -> None:
# last-poll wall-time = now − (seconds since last poll). Frozen while a
# task runs (the consumer stamps its heartbeat before each claim),
# so a child stuck on a too-long task goes stale exactly as the single
# consumer does. Guarded so a transient error (e.g. teardown during
# shutdown) logs loudly and the loop continues instead of dying silently
# and false-staling a healthy child.
while True:
try:
heartbeats[slot] = time.time() - consumer.seconds_since_last_poll()
except Exception:
logger.exception(
"PG-queue consumer: heartbeat publish failed for slot=%s", slot
)
time.sleep(_REPORT_INTERVAL_SECONDS)
# Publish once before ``loaded`` so the watchdog never pairs a loaded slot with
# the previous child's stale heartbeat.
heartbeats[slot] = time.time() - consumer.seconds_since_last_poll()
threading.Thread(target=_publish_heartbeat, daemon=True, name=f"pg-hb-{slot}").start()
loaded[slot] = 1
logger.info(
"PG-queue consumer: child slot=%s loaded in %.1fs",
slot,
time.monotonic() - started,
)
# consumer.run() installs its own SIGTERM/SIGINT handlers → graceful stop.
consumer.run()
def _child_after_fork(slot: int, heartbeats, loaded) -> None: # noqa: ANN001 (ctypes arrays)
"""Child side of the fork: reset inherited state, run, hard-exit on failure.
Resets the supervisor's signal handlers to ``SIG_DFL`` *immediately* — until
``consumer.run()`` installs its own, a SIGTERM arriving in the fork→run window
(which spans the slow ``import worker`` bootstrap) must NOT fire the parent's
``_on_term`` closure in the child (it captured a stale ``children`` dict and
would signal sibling pids). ``SIG_DFL`` = terminate, the correct disposition
for a not-yet-running child.
"""
signal.signal(signal.SIGTERM, signal.SIG_DFL)
signal.signal(signal.SIGINT, signal.SIG_DFL)
try:
_run_child(slot, heartbeats, loaded)
except Exception:
# A child that can't even start must not return into the supervisor loop
# (it would fork grandchildren). Log + hard exit. Exception (not
# BaseException) is the realistic startup-failure surface here — import,
# connection, config; SystemExit/KeyboardInterrupt would exit anyway.
logger.exception("PG-queue consumer: child slot=%s failed to run", slot)
os._exit(1)
os._exit(0)
def _try_fork_child(fleet: _Fleet, slot: int) -> bool:
"""Fork one child for ``slot``. Returns False (without raising) if ``os.fork``
fails — EAGAIN (RLIMIT_NPROC) / ENOMEM are realistic under heavy-child load —
so the caller can fail fast (initial fleet) or leave the slot for the next
monitor tick (re-fork path) instead of an uncaught crash taking the fleet down.
"""
try:
pid = os.fork()
except OSError:
logger.exception(
"PG-queue consumer: os.fork() failed for slot=%s (process/memory "
"limit?) — will retry",
slot,
)
return False
if pid == 0: # child — never returns
_child_after_fork(slot, fleet.heartbeats, fleet.loaded)
fleet.record_fork(slot, pid)
logger.info("PG-queue consumer: forked child slot=%s pid=%s", slot, pid)
return True
def _reap_dead(fleet: _Fleet, stopping: threading.Event) -> None:
"""Reap exited children and schedule their re-fork (unless shutting down)."""
for slot, pid in fleet.alive_items():
try:
reaped, _status = os.waitpid(pid, os.WNOHANG)
except ChildProcessError:
reaped = pid # already reaped elsewhere — treat as gone
if reaped == 0:
continue # still alive
uptime = fleet.reap(slot)
if stopping.is_set():
continue # do not resurrect during shutdown
crashes = fleet.schedule_restart(slot, uptime)
level = logging.ERROR if crashes >= _CRASH_LOOP_THRESHOLD else logging.WARNING
logger.log(
level,
"PG-queue consumer: child slot=%s pid=%s exited after %.1fs "
"(consecutive immediate crashes=%s) — re-fork scheduled",
slot,
pid,
uptime,
crashes,
)
def _restart_due_children(fleet: _Fleet, stopping: threading.Event) -> None:
"""Re-fork the slots whose backoff has elapsed — non-blocking (the backoff is
a scheduled not-before, not an in-loop sleep), and re-checking ``stopping``
each iteration so a SIGTERM mid-cycle can't spawn a fresh child into shutdown.
"""
for slot in fleet.due_restarts():
if stopping.is_set():
return
# On success record_fork clears the pending restart; on fork failure the
# slot stays due and is retried next tick.
_try_fork_child(fleet, slot)
def _kill_stale_children(fleet: _Fleet, stale_after: float, killed: set[int]) -> None:
"""SIGKILL every child silent for longer than ``stale_after``.
A *loaded* child is judged by its heartbeat. A child not yet loaded is judged
by time since its fork, against the larger of ``stale_after`` and
``_MIN_BOOTSTRAP_BUDGET_SECONDS``: a re-forked slot keeps its predecessor's old
heartbeat until the new child bootstraps (record_fork does not reseed it), so
the heartbeat would kill every replacement during import, while a child that
hangs in ``import worker`` would otherwise never be judged at all. ``killed``
holds pids already signalled and not yet reaped, so a child is killed and
logged once; the next ``_reap_dead`` reaps it and schedules the re-fork like
any other exit.
"""
bootstrap_budget = max(stale_after, _MIN_BOOTSTRAP_BUDGET_SECONDS)
for slot, pid in fleet.alive_items():
if pid in killed:
continue
loaded = fleet.is_loaded(slot)
age = fleet.slot_age(slot) if loaded else fleet.fork_age(slot)
limit = stale_after if loaded else bootstrap_budget
if age <= limit:
continue
logger.error(
"PG-queue consumer: child slot=%s pid=%s %s for %.0fs (> %.0fs) — "
"presumed hung; SIGKILL so its message redelivers and the slot is "
"re-forked",
slot,
pid,
"has not polled" if loaded else "has not finished loading",
age,
limit,
)
killed.add(pid)
try:
os.kill(pid, signal.SIGKILL)
except ProcessLookupError:
continue # exited on its own first; the reap handles it, not a kill
fleet.watchdog_kills += 1
def _monitor_tick(
fleet: _Fleet,
stopping: threading.Event,
watchdog_after: float | None,
killed: set[int],
) -> None:
"""One supervisor iteration: reap exits, re-fork due slots, then the watchdog.
``killed`` is trimmed to live pids right after the reap, so a recycled pid is
never mistaken for one already signalled.
"""
_reap_dead(fleet, stopping)
killed.intersection_update(pid for _slot, pid in fleet.alive_items())
_restart_due_children(fleet, stopping)
if watchdog_after is not None and not stopping.is_set():
_kill_stale_children(fleet, watchdog_after, killed)
def run_supervised(concurrency: int) -> None:
"""Fork ``concurrency`` consumer children and supervise them until SIGTERM."""
logging.basicConfig(level=os.getenv("LOG_LEVEL", "INFO"))
fleet = _Fleet(concurrency)
grace_seconds = shutdown_grace_from_env()
logger.info(
"PG-queue consumer supervisor: shutdown drain grace = %.0fs (shared across "
"children)",
grace_seconds,
)
watchdog_after = child_watchdog_from_env()
if watchdog_after is None:
logger.info("PG-queue consumer supervisor: child watchdog off")
else:
logger.info(
"PG-queue consumer supervisor: child watchdog kills a child silent for "
"> %.0fs",
watchdog_after,
)
killed: set[int] = set()
stopping = threading.Event()
def _signal_children(sig: int) -> None:
for _slot, pid in fleet.alive_items():
with contextlib.suppress(ProcessLookupError):
os.kill(pid, sig)
def _on_term(signum: int, _frame: object) -> None:
logger.info(
"PG-queue consumer supervisor: signal %s — stopping %d child(ren)",
signum,
fleet.alive_count(),
)
stopping.set()
_signal_children(signal.SIGTERM)
signal.signal(signal.SIGTERM, _on_term)
signal.signal(signal.SIGINT, _on_term)
# Fork the initial fleet while single-threaded (before the liveness thread).
# A fork failure here is fatal + actionable rather than a half-started fleet.
for slot in range(concurrency):
if not _try_fork_child(fleet, slot):
stopping.set()
_signal_children(signal.SIGTERM)
_join_children(fleet, grace_seconds)
raise RuntimeError(
f"PG-queue consumer: os.fork() failed starting child {slot}/"
f"{concurrency} — reduce WORKER_PG_QUEUE_CONSUMER_CONCURRENCY or "
"raise the process/memory limit"
)
health = _maybe_start_supervisor_health(fleet)
try:
while not stopping.is_set():
_monitor_tick(fleet, stopping, watchdog_after, killed)
stopping.wait(_MONITOR_INTERVAL_SECONDS) # responsive to SIGTERM
finally:
stopping.set()
_signal_children(signal.SIGTERM)
_join_children(fleet, grace_seconds)
if health is not None:
health.stop()
logger.info("PG-queue consumer supervisor: stopped")
def _wait_for_exit(pid: int, deadline: float) -> bool:
"""Poll ``pid`` until it exits or ``deadline`` (monotonic) passes. True if it
exited (or was already reaped).
Polls ``waitpid`` at least once regardless of the deadline: with the single
SHARED shutdown deadline, a child iterated after the window has already elapsed
would otherwise be reported "did not drain" and SIGKILLed — a false hard-kill
alarm — even though it exited cleanly within the grace.
"""
while True:
try:
reaped, _status = os.waitpid(pid, os.WNOHANG)
except ChildProcessError:
return True
if reaped != 0:
return True
if time.monotonic() >= deadline:
return False
time.sleep(0.1)
def _join_children(fleet: _Fleet, grace_seconds: float) -> None:
"""Wait up to ``grace_seconds`` TOTAL for all children to drain, then SIGKILL +
reap any straggler. A SINGLE shared deadline (not per-child): the children are
all SIGTERM'd together *before* this call and drain in parallel, so one shared
window still gives every child its full grace from that common SIGTERM — while
bounding the total wait to ~``grace_seconds``. A per-child deadline would instead
sum to N×grace and, at grace≈VT (thousands of seconds), blow past the pod's
``terminationGracePeriodSeconds`` — so k8s SIGKILLs the whole pod, hard-killing
siblings that were still draining cleanly.
"""
deadline = time.monotonic() + grace_seconds
for slot, pid in fleet.alive_items():
if _wait_for_exit(pid, deadline):
continue
logger.warning(
"PG-queue consumer: child slot=%s pid=%s did not drain within the "
"shared %.0fs grace — SIGKILL",
slot,
pid,
grace_seconds,
)
with contextlib.suppress(ProcessLookupError):
os.kill(pid, signal.SIGKILL)
with contextlib.suppress(ChildProcessError):
os.waitpid(pid, 0)
def _maybe_start_supervisor_health(fleet: _Fleet) -> LivenessServer | None:
"""Start the fleet liveness server when a port is configured; else None.
Reuses the single-process consumer's env knobs (``..._HEALTH_PORT`` /
``..._HEALTH_STALE_SECONDS``) and the same HTTP contract (``/health`` →
200/503), so the k8s probe config is unchanged. The JSON body differs
(``check="pg_queue_fleet"``, age key ``oldest_child_seconds_since_poll``) since
the freshness source is the fleet's oldest child, not one poll loop.
``/ready`` → 200 only once every child has loaded (:meth:`_Fleet.all_loaded`),
for the chart's opt-in ``startupProbe``.
A bind failure does not abort the consumer (it must keep draining the queue),
but ``EADDRINUSE`` usually signals a real config bug, so it's logged at error;
either way ``liveness_probe_bound: false`` is surfaced in the status payload so
the degradation is observable.
"""
from queue_backend.pg_queue.consumer import (
_DEFAULT_HEALTH_STALE_SECONDS,
consumer_env,
)
from queue_backend.pg_queue.liveness import LivenessServer
port: int | None = consumer_env("HEALTH_PORT", None, int)
if port is None:
logger.info("PG-queue consumer supervisor: HEALTH_PORT unset — liveness disabled")
return None
stale_after = consumer_env(
"HEALTH_STALE_SECONDS", _DEFAULT_HEALTH_STALE_SECONDS, float
)
def _extra_status() -> dict[str, object]:
return {
"alive_children": fleet.alive_count(),
"loaded_children": fleet.loaded_count(),
"concurrency": fleet.concurrency,
"crash_looping": fleet.is_crash_looping(),
"liveness_probe_bound": True,
}
from queue_backend.pg_queue.metrics import ConsumerMetrics
metrics = ConsumerMetrics(
freshness_fn=fleet.freshness,
alive_children_fn=lambda: float(fleet.alive_count()),
concurrency_fn=lambda: float(fleet.concurrency),
watchdog_kills_fn=lambda: float(fleet.watchdog_kills),
)
server = LivenessServer(
freshness_fn=fleet.freshness,
stale_after=stale_after,
port=port,
check_name="pg_queue_fleet",
age_key="oldest_child_seconds_since_poll",
extra_status_fn=_extra_status,
metrics_fn=metrics.render,
ready_fn=fleet.all_loaded,
thread_name="pg-supervisor-liveness",
log_label="pg-queue supervisor",
)
try:
server.start()
except OSError as exc:
import errno
level = logging.ERROR if exc.errno == errno.EADDRINUSE else logging.WARNING
logger.log(
level,
"PG-queue consumer supervisor: liveness could not bind :%s (%s) — "
"continuing WITHOUT a probe",
port,
exc.strerror or exc,
exc_info=True,
)
return None
logger.info(
"PG-queue consumer supervisor: fleet liveness on :%s/health, readiness on "
"/ready (stale after %ss, %d children)",
server.bound_port,
stale_after,
fleet.concurrency,
)
return server