Skip to content

LLM-Driven Extraction of Unstructured Data — Built for API Deployments & ETL Pipeline Workflows

68.4 KB
Raw Blame
"""PG fan-in barrier — the ``Barrier`` implementation.

Coordinates "wait for N header tasks, then fire the callback with their
results" on a Postgres row (``pg_barrier_state``) rather than a Celery chord or
a Redis ``DECR`` counter. Both of those substrates were deleted with the Celery
transport (UN-4078); this is the only barrier.

**Why a Postgres substrate.** An execution coordinates in the *same* Postgres
that holds the PG queue, so the fan-in needs no second datastore — and, unlike a
broker-side or TTL'd-counter barrier, its state is a directly queryable row the
reaper can recover.

**Wire model.**

1. ``enqueue``: UPSERT one ``pg_barrier_state`` row (``remaining = N``,
   ``results = []``, ``expires_at = now() + ttl``, ``last_progress_at = now()``)
   — the UPSERT clears any stale state from a prior run reusing the same
   ``execution_id``. Each header task is then dispatched onto the PG queue with a
   ``_barrier_context`` kwarg carrying ``execution_id`` / ``batch_index`` /
   ``callback_descriptor``.
2. Per-task success: a PG-consumed task fires no Celery ``.link``, so it claims
   its batch and runs the decrement **in-body** (:func:`run_batch_with_barrier` →
   :func:`_barrier_pg_decrement`) — ONE atomic statement,
   ``UPDATE … SET remaining = remaining - 1, results = results ||
   jsonb_build_array(result) … RETURNING remaining, results``. The row lock
   serialises concurrent decrements, so exactly one task observes ``remaining =
   0``; that task self-chains the callback onto PG with the aggregated results,
   then deletes the row. (No Lua — a single ``UPDATE … RETURNING`` is atomic in
   Postgres. The guarantee relies on each decrement committing in its own
   transaction — do NOT batch decrements into a shared transaction, or the row
   lock would hold and the serialisation that makes exactly one see 0 breaks.)
3. Per-task failure: :func:`barrier_pg_abort`, called in-body, tears the barrier
   down with ONE atomic statement (``DELETE … RETURNING`` in a single
   transaction): the row's existence is the dedup token, so N concurrent
   failures collapse to a single cleanup, and a crash mid-abort rolls back
   (leaving the row for a sibling to retry) — there is no claimed-but-not-deleted
   window. Preserves chord's old error semantic (callback not invoked on header
   failure).
4. Stuck bound: ``last_progress_at`` is re-stamped to ``now()`` on enqueue AND on
   every decrement, tracking when a batch last completed. The
   :mod:`~queue_backend.pg_queue.reaper` marks the stranded execution ERROR once
   ``last_progress_at`` is older than ``WORKER_PG_BATCH_STUCK_TIMEOUT_SECONDS``
   (default 2.5h — in the same band as Celery's per-task
   ``FILE_PROCESSING_TASK_TIME_LIMIT``, which ships 2h–3h) — a
   per-PROGRESS window, invariant to how many batches run in parallel
   (``MAX_PARALLEL_FILE_BATCHES`` is dynamic per-org), so it never false-fails a
   legitimately long multi-batch run. ``expires_at`` (fixed at enqueue,
   ``WORKER_BARRIER_KEY_TTL_SECONDS``, default 6h) is the absolute last-resort cap
   the reaper also sweeps.

**Failure-masking guards.** The callback can only fire when ``remaining`` hits
exactly 0, which requires ALL N header tasks to have decremented — i.e. all
succeeded. A failed task runs the abort (which deletes the row) instead of a
decrement, so the count never reaches 0 and a late in-flight decrement finds no
row → abandons. A decrement that drives ``remaining`` negative (expiry/replay)
also cleans up without firing. The callback is dispatched BEFORE the row is
deleted, so a callback dispatch failure leaves the row (and its expiry) in place
rather than stranding the execution; the post-dispatch delete is best-effort
(logged, not raised) since the callback has already fired.
"""

from __future__ import annotations

import contextlib
import logging
import threading
import time
from collections.abc import Callable, Iterator
from dataclasses import dataclass
from typing import TYPE_CHECKING, Any, Final, NamedTuple

import psycopg2
import psycopg2.errors
import psycopg2.extensions

from unstract.core.data_models import LEGACY_TRANSPORT_KEY, LEGACY_TRANSPORT_VALUE
from unstract.core.jsonb import dumps_for_jsonb

from .barrier import (
    BarrierContext,
    CallbackDescriptor,
    barrier_ttl_seconds,
    callback_recovery_identity,
)
from .decorator import worker_task
from .fairness import (
    DEFAULT_PRIORITY,
    FAIRNESS_HEADER_NAME,
    FairnessKey,
    WorkloadType,
)
from .handle import BarrierHandle
from .pg_queue.connection import CONN_DEAD_ERRORS as _CONN_DEAD_ERRORS
from .pg_queue.connection import PAYLOAD_REJECTED_ERRORS as _PAYLOAD_REJECTED_ERRORS
from .pg_queue.connection import create_pg_connection, is_connection_dead
from .pg_queue.schema import qualified

if TYPE_CHECKING:
    from celery.canvas import Signature
    from psycopg2.extensions import connection as PgConnection
    from psycopg2.extensions import cursor as PgCursor

logger = logging.getLogger(__name__)


# ``_CONN_DEAD_ERRORS`` (the "is this a connection death?" test, used by
# ``_recover_after_error``/``_cursor`` to discard a stale handle and by the
# enqueue/decrement retries to decide eligibility) is imported from
# ``pg_queue.connection`` so the dispatch/result/barrier sites can't drift.


# Thread-local owned connection (prefork → one per child; thread-local keeps it
# correct under -P threads too, since a libpq connection is not concurrency-safe
# across threads). Self-recovers a dropped socket / PgBouncer recycle — same
# posture as queue_backend.dispatch / PgQueueClient.
_local = threading.local()


def _get_conn() -> PgConnection:
    conn = getattr(_local, "conn", None)
    if conn is None or conn.closed:
        conn = create_pg_connection(env_prefix="DB_")
        _local.conn = conn
    return conn


def _recover_after_error(conn: PgConnection, exc: BaseException) -> bool:
    """Roll back ``conn``; if it is dead, discard the thread-local handle so the
    next :func:`_get_conn` reconnects. Returns whether the connection was judged
    dead (a failed rollback proves it dead regardless of the original error).

    Factored out so ``_cursor`` and the decrement phase-split (which can't use
    ``_cursor`` because it must distinguish execute-phase from commit-phase
    failures) share one definition of "recover a connection after an error".
    """
    conn_dead = is_connection_dead(exc)
    try:
        conn.rollback()
    except Exception:
        # A failed rollback proves the connection is unusable regardless of the
        # original error's subclass — treat it as dead. Surface why (instead of
        # swallowing it): this also reclassifies a non-conn error (e.g. DataError)
        # whose rollback fails into the dead/retry path, so the trail must show it
        # was the rollback, not the original error, that condemned the connection.
        logger.warning(
            "PgBarrier: rollback during error recovery failed (original error %s) "
            "— treating the connection as dead and discarding it.",
            type(exc).__name__,
            exc_info=True,
        )
        conn_dead = True
    if conn_dead or conn.closed:
        with contextlib.suppress(Exception):
            conn.close()
        _local.conn = None
    return conn_dead


@contextlib.contextmanager
def _cursor() -> Iterator[Any]:
    """Yield a cursor; commit on success, roll back + recover on error."""
    conn = _get_conn()
    try:
        with conn.cursor() as cur:
            yield cur
        conn.commit()
    except Exception as exc:
        _recover_after_error(conn, exc)
        raise


# One retry for an IDEMPOTENT, pre-dispatch barrier write. The thread-local
# connection is cached across barrier ops, so it can be reaped server-side
# (PgBouncer server_idle_timeout / DB failover) while sitting idle BETWEEN ops —
# and ``_get_conn`` can't tell, since ``conn.closed`` is a client-side flag only.
# The first statement after the idle gap then fails; ``_cursor`` discards the
# dead conn, so a single retry runs against a freshly reconnected one. This turns
# a transient blip (which previously aborted the whole execution at barrier
# enqueue) into a self-heal. Kept a literal (not env-driven) so the idempotency
# bound can't be weakened operationally. Also bounds the phase-split RETURNING
# claims (``claim_batch`` / ``try_claim_orchestration`` via
# :func:`_run_returning_claim_with_reconnect`); they are non-idempotent but retry
# ONLY the provably-uncommitted execute phase, so the same one-retry bound holds
# without reopening a claim-flip.
_BARRIER_WRITE_ATTEMPTS: Final = 2  # total attempts: 1 initial + 1 retry
# Small fixed pause before the retry. The idle-reap case reconnects instantly
# regardless; this only widens the self-heal window for a brief DB failover, and
# avoids immediately re-hammering a struggling server on the rarer server-side
# errors the broad psycopg2.OperationalError catch also covers.
_BARRIER_RETRY_BACKOFF_SECONDS: Final = 0.5

# Callback kwarg the PG barrier stamps on the aggregating callback's dispatch so
# the callback can PG-gate its at-least-once duplicate guard (see
# _fire_barrier_callback). callback/tasks.py IMPORTS this symbol (not a re-typed
# literal), so the producer and consumer can't drift. Underscore-prefixed so it
# can't collide with a real task kwarg.
PG_TRANSPORT_CALLBACK_KWARG: Final = "_pg_transport"

# One retry for the NON-idempotent barrier DECREMENT — but only on an
# execute-phase failure on a reused connection (see :func:`_apply_decrement`),
# never on commit (ambiguous). Same one-shot bound as the idempotent write.
_BARRIER_DECREMENT_ATTEMPTS: Final = 2  # total attempts: 1 initial + 1 retry


def _run_idempotent_pre_dispatch_write(
    operation: Callable[[PgCursor], None], *, what: str
) -> None:
    """Run ``operation(cur)`` in a committed cursor, retrying ONCE if the cached
    connection was dead.

    The name spells out the contract because it can't be type-enforced: only call
    with an **idempotent** statement run **before** any task dispatch — the
    barrier UPSERT (``ON CONFLICT … DO UPDATE`` → same row, ``remaining``/
    ``results`` reset identically; timestamps refresh, harmlessly) + the
    per-execution dedup reset (``DELETE WHERE execution_id``). Re-running them
    after an ambiguous commit is a no-op, so a retry can neither duplicate a row
    nor double-dispatch work (no header has been enqueued yet). The ``-> None``
    op signature also blocks passing a ``RETURNING``-reading op by construction.

    Deliberately NOT used for the barrier **decrement** (``remaining =
    remaining - 1``): it is not idempotent — a re-applied decrement can fire the
    callback **prematurely with incomplete results**, or skip past 0 and
    **strand the barrier** to expiry — so it stays on the plain :func:`_cursor`
    (recover-but-don't-retry). The RETURNING claims (``claim_batch`` /
    ``try_claim_orchestration``) are likewise non-idempotent — a committed claim's
    win/lost answer flips on a naive re-run — so they retry via
    :func:`_run_returning_claim_with_reconnect`, which retries ONLY the
    provably-uncommitted execute phase.
    """
    for attempt in range(1, _BARRIER_WRITE_ATTEMPTS + 1):
        try:
            with _cursor() as cur:
                operation(cur)
            return
        except _CONN_DEAD_ERRORS as exc:
            # A payload rejection also reaches this clause (ProgramLimitExceeded
            # subclasses OperationalError). It is permanent, so let it through to
            # _barrier_pg_decrement's handler instead of re-sending it.
            if not is_connection_dead(exc):
                raise
            # _cursor already dropped the dead thread-local conn → the next
            # _get_conn() reconnects. Retry once; re-raise if it still fails
            # (a genuinely-down DB surfaces as ERROR, as before). Name the real
            # error + keep the traceback: the broad catch also covers server-side
            # conditions (statement timeout, deadlock, admin shutdown), so the
            # message must not assert a connection drop it can't be sure of.
            if attempt >= _BARRIER_WRITE_ATTEMPTS:
                raise
            logger.warning(
                "PgBarrier: %s — DB write failed (%s: %s); cached connection "
                "likely stale, reconnecting and retrying (attempt %d/%d)",
                what,
                type(exc).__name__,
                exc,
                attempt,
                _BARRIER_WRITE_ATTEMPTS,
                exc_info=True,
            )
            time.sleep(_BARRIER_RETRY_BACKOFF_SECONDS)


def _delete_barrier(execution_id: str) -> None:
    with _cursor() as cur:
        cur.execute(
            f"DELETE FROM {qualified('pg_barrier_state')} WHERE execution_id = %s",
            (execution_id,),
        )


def _run_returning_claim_with_reconnect(
    operation: Callable[[PgCursor], bool], *, what: str
) -> bool:
    """Run a single ``INSERT … ON CONFLICT DO NOTHING RETURNING`` claim
    (``operation(cur)`` → did-I-win), with the **phase-split reconnect-retry**
    shared by :func:`claim_batch` and :func:`try_claim_orchestration`.

    These are the FIRST DB write of their task, so on an idle worker they most
    often meet a PgBouncer-reaped *cached* connection. The claim is retried ONCE,
    but ONLY on an **execute-phase** failure of a cached connection: the statement
    never committed (rolled back on disconnect), so re-running lands exactly once
    and the retry's ``RETURNING`` answer is authoritative — no win→lost flip.

    NOT retried (each propagates):
    - **commit-phase** failure — AMBIGUOUS (the server may have committed). A re-run
      could flip ``True`` → ``False`` → the caller skips and the barrier/claim
      strands; and if the row *did* persist, the redelivery's claim likewise returns
      ``False``, so the reaper recovers at expiry either way.
    - **fresh-conn** failure — a real DB error; reconnecting buys nothing.

    (This is why the idempotent ``-> None`` pre-dispatch helper can't be reused: a
    RETURNING claim is non-idempotent.) ``what`` is a caller-formatted log label,
    e.g. ``[exec:<id>] claim_batch(<n>)``.
    """
    # Only a CACHED handle can be a stale idle-reap; sample before _get_conn()
    # materialises one, so a fresh-conn failure isn't misread as a reap.
    reused = getattr(_local, "conn", None) is not None and not _local.conn.closed
    for attempt in range(1, _BARRIER_WRITE_ATTEMPTS + 1):
        conn = _get_conn()
        try:
            with conn.cursor() as cur:
                won = operation(cur)
        except Exception as exc:
            conn_dead = _recover_after_error(conn, exc)
            if conn_dead and reused and attempt < _BARRIER_WRITE_ATTEMPTS:
                logger.warning(
                    "%s: execute failed on a cached connection (%s) — reconnecting "
                    "and retrying once (attempt %d/%d); the INSERT never committed.",
                    what,
                    type(exc).__name__,
                    attempt,
                    _BARRIER_WRITE_ATTEMPTS,
                    exc_info=True,
                )
                # The dead handle was discarded, so the next _get_conn() reconnects
                # — clear `reused` so a fresh-conn death on a later attempt is never
                # retried. Keeps the invariant structural, not merely masked by
                # _BARRIER_WRITE_ATTEMPTS == 2.
                reused = False
                time.sleep(_BARRIER_RETRY_BACKOFF_SECONDS)
                continue
            raise
        try:
            conn.commit()
        except Exception as exc:
            _recover_after_error(conn, exc)
            logger.warning(
                "%s: commit failed (%s) — NOT retrying (the server may already have "
                "committed; a re-run could flip the claim → the caller skips and the "
                "barrier strands; if the row persisted, the redelivery skips and the "
                "reaper recovers at expiry). Propagating.",
                what,
                type(exc).__name__,
                exc_info=True,
            )
            raise
        return won
    # Unreachable: the loop either returns or raises.
    raise AssertionError(f"{what}: claim loop fell through")


def claim_batch(execution_id: str, batch_index: int) -> bool:
    """Claim ``(execution_id, batch_index)`` for processing — the per-batch
    idempotency gate for the at-least-once PG pipeline.

    Atomically inserts the dedup marker (``pg_batch_dedup``); returns ``True``
    if THIS call inserted the row (first delivery — the caller should proceed
    and perform the single barrier decrement) and ``False`` if the row already
    existed (a redelivery — the caller should skip). This function itself only
    inserts the marker; pairing it with the caller's decrement is what keeps the
    barrier decremented exactly once per batch. ``ON CONFLICT DO NOTHING`` makes
    the check-and-set a single statement, so concurrent redeliveries of the same
    batch resolve to exactly one ``True`` (the row's existence is the token) with
    no race.

    Called on the in-body PG path from :func:`run_batch_with_barrier` (claim at
    batch start), alongside the in-body decrement. The companion
    :func:`clear_execution_batches` reclaims the markers at barrier finalise.

    Runs through the shared phase-split reconnect-retry
    (:func:`_run_returning_claim_with_reconnect`): the module's thread-local
    connection sits idle **between pipeline runs** (a scheduled pipeline can idle
    for tens of minutes, past PgBouncer's ``server_idle_timeout``), so the FIRST
    ``INSERT`` of a run can hit an idle-reaped handle; that execute-phase reap
    self-heals on one reconnect. See the helper for the full rationale.
    """

    def _op(cur: PgCursor) -> bool:
        cur.execute(
            f"INSERT INTO {qualified('pg_batch_dedup')} "
            "(execution_id, batch_index, created_at) "
            "VALUES (%s, %s, now()) "
            "ON CONFLICT (execution_id, batch_index) DO NOTHING "
            "RETURNING execution_id",
            (execution_id, batch_index),
        )
        return cur.fetchone() is not None

    return _run_returning_claim_with_reconnect(
        _op, what=f"[exec:{execution_id}] claim_batch({batch_index})"
    )


def clear_execution_batches(execution_id: str) -> int:
    """Delete all per-batch dedup markers for an execution; returns the count.

    Called at barrier teardown (``remaining`` → 0) so the markers live exactly
    as long as the execution needs them. Executions that never finalise
    currently leak their markers — the reaper sweeps the orphaned
    ``pg_barrier_state`` row but does NOT yet reclaim ``pg_batch_dedup`` (no
    cascade); a dedup-orphan sweep is intended future work (see the
    ``PgBatchDedup`` model docstring). Safe to call when no rows exist (returns
    0). Uses the ``(execution_id, batch_index)`` constraint index (``execution_id``
    leading) for the lookup.

    Called from the barrier-finalise + abort paths.
    """
    with _cursor() as cur:
        cur.execute(
            f"DELETE FROM {qualified('pg_batch_dedup')} WHERE execution_id = %s",
            (execution_id,),
        )
        return cur.rowcount


def try_claim_orchestration(execution_id: str, organization_id: str) -> bool:
    """Try to claim the orchestration slot for ``execution_id`` — the
    execution-level idempotency gate mirroring :func:`claim_batch`, one level up.
    The ``try_`` prefix signals the return is "did I win": ``True`` = claimed
    (proceed), ``False`` = already claimed (a duplicate/redelivery — no-op).

    Atomically inserts the single ``pg_orchestration_claim`` row via
    ``INSERT … ON CONFLICT DO NOTHING RETURNING`` (a single statement, so two
    replicas racing the same orchestration resolve to exactly one ``True`` — the
    row's existence is the token). ``organization_id`` is stamped on the row so the
    reaper's ``sweep_orphan_claims`` can call the org-scoped status/mark API when it
    recovers or GCs the claim (``""`` if the caller has no org).

    Claimed BEFORE the orchestration work (arm barrier / dispatch batches). The
    claim is **released on failure** (see :func:`release_orchestration_claim`,
    called from the orchestrator's error path) so a redelivery/retry can
    re-orchestrate; it persists only after a SUCCESSFUL orchestration, as a
    tombstone that blocks a redelivery AFTER the execution completes (when the
    barrier row is already gone) from re-arming and re-dispatching.

    **Recovery + GC (the reaper's ``sweep_orphan_claims``).** Unlike
    :func:`claim_batch`, whose barrier is always armed before any batch runs (so
    the reaper always has a ``pg_barrier_state`` handle), this claim is taken
    BEFORE the barrier is armed. A hard crash (SIGKILL / eviction) in the
    claim→arm window leaves a claim with no barrier row; the reaper sweeps such
    orphan claims older than the stuck-timeout — GC'ing a terminal execution's
    tombstone and marking a non-terminal (crash-window) execution ERROR.

    PG path only — the caller gates on the transport (Celery has no redelivery, so
    no double-orchestration to guard).

    **Stale-connection resilience (phase-split, like :func:`_apply_decrement`).**
    This is the FIRST DB write of the orchestration task, so on a worker that has
    been idle it is the one most likely to meet a PgBouncer-reaped connection. The
    INSERT is retried ONCE, but ONLY on an execute-phase failure of a *cached*
    connection: the statement never committed (non-autocommit → rolled back on
    disconnect), so re-running it lands exactly once and the retry's ``RETURNING``
    answer is authoritative — no won/lost flip. A *commit*-phase failure is
    AMBIGUOUS (the server may have committed) and is NEVER retried — a re-run could
    flip a real winner to a loser and strand the execution — so it propagates (the
    orchestrator marks ERROR and the redelivery re-orchestrates). This is why the
    idempotent-pre-dispatch helper (``-> None``, no RETURNING) can't be reused here.
    """

    def _op(cur: PgCursor) -> bool:
        cur.execute(
            f"INSERT INTO {qualified('pg_orchestration_claim')} "
            "(execution_id, organization_id, claimed_at) "
            "VALUES (%s, %s, now()) "
            "ON CONFLICT (execution_id) DO NOTHING "
            "RETURNING execution_id",
            (execution_id, organization_id),
        )
        return cur.fetchone() is not None

    try:
        return _run_returning_claim_with_reconnect(
            _op, what=f"[exec:{execution_id}] orchestration claim"
        )
    except (psycopg2.errors.UndefinedTable, psycopg2.errors.UndefinedColumn) as exc:
        # Fail fast with an actionable message instead of a generic per-execution
        # stack trace when the schema is behind: UndefinedTable (0012 missing) OR
        # UndefinedColumn — a 0012-but-not-0013 deploy has the table but not
        # organization_id, so the INSERT above raises UndefinedColumn (a sibling
        # SQLSTATE). NOT swallowed to a "proceed" fallback — that would reopen the
        # double-orchestration window this guard exists to close.
        raise RuntimeError(
            "pg_orchestration_claim schema is out of date — migrations "
            "0012_pgorchestrationclaim / 0013_pgorchestrationclaim_organization_id "
            "have not been fully applied. Run backend migrations — PG is the only "
            "execution transport, so this blocks all orchestration."
        ) from exc


def release_orchestration_claim(execution_id: str) -> None:
    """Release the orchestration claim so a redelivery/retry can re-orchestrate.

    Called from the orchestrator's failure path: the claim is taken before the
    work, so without this a transient first-attempt failure would leave the claim
    committed and make every redelivery permanently no-op (a silently-lost
    execution). Deleting it on failure restores the retry/redelivery path. A
    successful orchestration does NOT call this — its claim persists as the
    post-completion tombstone. Safe when no row exists (deletes nothing).

    Stale-connection resilience: the claim is taken at orchestration START and
    released only on the failure path, so the pg_barrier thread-local connection
    has idled through the entire orchestration attempt — this DELETE is a
    first-write-after-idle, the one most likely to meet a PgBouncer-reaped
    connection, in exactly the scenario the release exists for. The caller
    swallows a raise here (best-effort), so without a retry a single idle-reap
    leaves the claim committed and suppresses every redelivery. The DELETE is
    idempotent (deleting an absent row is a no-op), so it reuses the retry-once
    helper verbatim.
    """

    def _op(cur: PgCursor) -> None:
        cur.execute(
            f"DELETE FROM {qualified('pg_orchestration_claim')} WHERE execution_id = %s",
            (execution_id,),
        )

    _run_idempotent_pre_dispatch_write(
        _op, what=f"release orchestration claim [exec:{execution_id}]"
    )


def _dispatch_pg(
    task_name: str,
    *,
    args: list[Any] | None = None,
    kwargs: dict[str, Any] | None = None,
    queue: str | None,
    fairness: FairnessKey | None = None,
) -> Any:
    """Enqueue a task onto the PG queue (the one place that owns the cycle-avoiding
    local imports + the ``backend=QueueBackend.PG`` argument).

    Both the header fan-out (:meth:`PgBarrier._dispatch_header_pg`) and the
    self-chained callback (:func:`_fire_barrier_callback`) route through here.
    Returns the ``dispatch`` handle. ``queue`` is required (may be ``None`` only
    if the caller has already logged the fallback) — a ``None`` queue makes
    ``dispatch`` fall back to its default PG queue.
    """
    # Local import: dispatch/routing pull in queue plumbing that imports the
    # barrier package — importing at module load would be a cycle.
    from .dispatch import dispatch
    from .routing import QueueBackend

    return dispatch(
        task_name,
        args=args,
        kwargs=kwargs,
        queue=queue,
        fairness=fairness,
        backend=QueueBackend.PG,
    )


@dataclass(frozen=True, slots=True)
class _PgBarrierHandle:
    """Minimal ``BarrierHandle`` — ``id`` is the execution id (what call sites
    log for chord-id tracing), same as ``_RedisBarrierHandle``.
    """

    id: str


def build_callback_descriptor(
    *,
    task_name: str,
    kwargs: dict[str, Any],
    queue: str,
    fairness_headers: dict[str, Any] | None,
) -> CallbackDescriptor:
    """Build the descriptor carried to the consumer that fires the callback.

    A function rather than an inline literal so the rolling-deploy shim it stamps
    (:data:`~unstract.core.data_models.LEGACY_TRANSPORT_KEY`) is pinned by a test
    that needs no Postgres — the enqueue path's own tests are skipped wherever a
    database is unavailable, which would let the shim be dropped against a green
    required build. See ``tests/test_legacy_transport_shim.py``.
    """
    return {
        "task_name": task_name,
        "kwargs": kwargs,
        "queue": queue,
        "fairness_headers": fairness_headers,
        # Rolling-deploy shim: see LEGACY_TRANSPORT_KEY.
        LEGACY_TRANSPORT_KEY: LEGACY_TRANSPORT_VALUE,
    }


class PgBarrier:
    """``Barrier`` implementation via a Postgres ``pg_barrier_state`` row.

    The only implementation. See the module docstring for the wire model and the
    failure-masking guards.
    """

    def enqueue(
        self,
        header_tasks: list[Signature],
        *,
        callback_task_name: str,
        callback_kwargs: dict[str, Any],
        callback_queue: str,
        app_instance: Any,
        fairness: FairnessKey | None = None,
    ) -> BarrierHandle | None:
        """See :class:`queue_backend.barrier.Barrier.enqueue`.

        Empty ``header_tasks`` → ``None`` (caller owns the zero-files contract);
        any substrate failure raises. ``app_instance`` is accepted for Protocol
        parity but unused.

        Header tasks are dispatched onto the PG queue (``dispatch(backend=PG)``)
        with a ``_barrier_context`` kwarg carrying ``execution_id`` /
        ``batch_index`` / ``callback_descriptor``. A PG-consumed task fires no
        Celery ``.link``, so it claims its batch and runs the decrement in-body,
        self-chaining the callback at ``remaining`` → 0.
        """
        del app_instance  # Protocol parity; callback dispatched by the decrement.
        if not header_tasks:
            logger.info(
                f"[exec:{callback_kwargs.get('execution_id')}] "
                f"[pipeline:{callback_kwargs.get('pipeline_id')}] "
                "Zero header tasks detected — skipping barrier enqueue "
                "(parent should handle pipeline status updates directly)"
            )
            return None

        execution_id = callback_kwargs.get("execution_id")
        if not execution_id:
            raise ValueError(
                "PgBarrier requires execution_id in callback_kwargs — it's the "
                "primary key of the per-execution pg_barrier_state row"
            )
        execution_id = str(execution_id)

        try:
            fairness_headers = fairness.as_header() if fairness else None
            callback_descriptor = build_callback_descriptor(
                task_name=callback_task_name,
                kwargs=callback_kwargs,
                queue=callback_queue,
                fairness_headers=fairness_headers,
            )
            # expires_at = absolute orphan cap (6h). last_progress_at = now() (the
            # reaper's fast stuck signal, re-stamped on every decrement).
            ttl_seconds = barrier_ttl_seconds()
            # Stamp the owning org so the reaper can call the org-scoped status
            # API when recovering this barrier if it strands (it has only the
            # execution_id off the row otherwise).
            organization_id = str(callback_kwargs.get("organization_id") or "")
            if not organization_id:
                # Should never happen — every fan-out passes organization_id in
                # callback_kwargs. Surface loudly: a barrier with no org can't be
                # recovered via the org-scoped API if it strands.
                logger.error(
                    f"[exec:{execution_id}] PgBarrier enqueued with NO "
                    f"organization_id in callback_kwargs — a stranded barrier "
                    f"could not be reaper-recovered. This is a bug in the caller."
                )

            def _reset_barrier(cur: PgCursor) -> None:
                # UPSERT clears any leftover state from a prior run with this id.
                # No inline expiry sweep here — an unbounded global DELETE on the
                # enqueue hot path risks lock contention / deadlocks between
                # concurrent enqueues; orphan reclaim is a separate (future)
                # periodic sweep keyed on pg_barrier_expires_idx.
                cur.execute(
                    f"INSERT INTO {qualified('pg_barrier_state')} "
                    "(execution_id, organization_id, remaining, results, "
                    " created_at, expires_at, last_progress_at) "
                    "VALUES (%s, %s, %s, '[]'::jsonb, now(), "
                    "        now() + make_interval(secs => %s), now()) "
                    "ON CONFLICT (execution_id) DO UPDATE SET "
                    "  organization_id = EXCLUDED.organization_id, "
                    "  remaining = EXCLUDED.remaining, results = '[]'::jsonb, "
                    "  created_at = now(), expires_at = EXCLUDED.expires_at, "
                    "  last_progress_at = now()",
                    (
                        execution_id,
                        organization_id,
                        len(header_tasks),
                        ttl_seconds,
                    ),
                )
                # Reset per-batch dedup markers from a prior run reusing this
                # execution_id, ATOMICALLY with the barrier reset. Without this a
                # re-enqueue would leave stale markers → every in-body claim_batch
                # returns False → all batches skip → barrier hangs to expiry.
                # (Markers are written by claim_batch() on the in-body PG path.)
                cur.execute(
                    f"DELETE FROM {qualified('pg_batch_dedup')} WHERE execution_id = %s",
                    (execution_id,),
                )

            # Idempotent + pre-dispatch → safe to retry; see
            # _run_idempotent_pre_dispatch_write.
            _run_idempotent_pre_dispatch_write(
                _reset_barrier, what=f"enqueue exec={execution_id}"
            )

            self._dispatch_headers(
                header_tasks,
                execution_id=execution_id,
                callback_descriptor=callback_descriptor,
                fairness=fairness,
            )

            logger.info(
                f"Barrier enqueued via PgBarrier — "
                f"exec_id={execution_id}, header_tasks={len(header_tasks)}, "
                f"callback={callback_task_name}, queue={callback_queue}"
            )
            return _PgBarrierHandle(id=execution_id)

        except Exception:
            logger.exception(
                f"[exec:{execution_id}] "
                f"[pipeline:{callback_kwargs.get('pipeline_id')}] "
                f"Failed to enqueue barrier via Postgres "
                f"(callback={callback_task_name}, queue={callback_queue}, "
                f"header_tasks={len(header_tasks)})"
            )
            raise

    def _dispatch_headers(
        self,
        header_tasks: list[Signature],
        *,
        execution_id: str,
        callback_descriptor: CallbackDescriptor,
        fairness: FairnessKey | None,
    ) -> None:
        """Dispatch the N header tasks fire-and-forget onto the PG queue.

        On any mid-loop dispatch failure, ``i`` of N never reached the queue so
        the counter can't reach 0 — delete the barrier row and reclaim the dedup
        markers an earlier header may have committed (the in-flight
        ``barrier_pg_abort`` is a no-op once the row is gone) so an in-flight
        decrement finds nothing, then re-raise.
        """
        for i, task in enumerate(header_tasks):
            try:
                self._dispatch_header_pg(
                    task, i, execution_id, callback_descriptor, fairness
                )
            except Exception:
                with contextlib.suppress(Exception):
                    _delete_barrier(execution_id)
                with contextlib.suppress(Exception):
                    clear_execution_batches(execution_id)
                logger.exception(
                    f"[exec:{execution_id}] header dispatch failed at task "
                    f"{i}/{len(header_tasks)}; barrier row deleted to prevent "
                    f"spurious callback fires from the orphan tasks."
                )
                raise

    @staticmethod
    def _dispatch_header_pg(
        task: Signature,
        batch_index: int,
        execution_id: str,
        callback_descriptor: CallbackDescriptor,
        fairness: FairnessKey | None,
    ) -> None:
        """Dispatch one header task onto the PG queue (fire-and-forget mode).

        Unpacks the Celery ``Signature`` (the fan-out built it as
        ``app.signature(name, kwargs={batch_files, batch_index, total_batches},
        queue=...)`` — the batch payload is in ``kwargs``) and re-dispatches it via
        :func:`_dispatch_pg` with an added ``_barrier_context`` kwarg. The PG
        consumer runs the task; it claims ``(execution_id, batch_index)`` and runs
        the decrement in-body (no ``.link``). ``fairness`` carries org/priority
        onto the row exactly as the bare-dispatch sites do.
        """
        barrier_context: BarrierContext = {
            "execution_id": execution_id,
            "batch_index": batch_index,
            "callback_descriptor": callback_descriptor,
        }
        header_kwargs = dict(task.kwargs or {})
        header_kwargs["_barrier_context"] = barrier_context
        queue = task.options.get("queue")
        if queue is None:
            # The fan-out always sets a queue; a missing one would silently route
            # to dispatch's default PG queue (off the intended file-processing
            # queue), so surface it rather than let it slip by.
            logger.warning(
                f"[exec:{execution_id}] header task {task.task!r} (batch "
                f"{batch_index}) has no queue option — falling back to the default "
                f"PG queue; the consumer for the intended queue won't see it."
            )
        _dispatch_pg(
            task.task,
            args=list(task.args or ()),
            kwargs=header_kwargs,
            queue=queue,
            fairness=fairness,
        )


def _fairness_from_headers(
    fairness_headers: dict[str, Any] | None,
) -> FairnessKey | None:
    """Reconstruct a :class:`FairnessKey` from the stored ``x-fairness-key`` header.

    The descriptor carries the wire-shape headers (``FairnessKey.as_header()``),
    but the PG dispatch path wants the ``FairnessKey`` itself (``dispatch`` writes
    ``org_id`` + ``pipeline_priority`` onto the row). Reconstructing keeps the PG
    callback at the producer's org/priority — parity with the Celery path, which
    applies the headers directly. ``None`` when the producer attached no key.
    """
    payload = (fairness_headers or {}).get(FAIRNESS_HEADER_NAME)
    if not payload:
        return None
    return FairnessKey(
        org_id=payload.get("org_id"),
        workload_type=WorkloadType(payload["workload_type"]),
        pipeline_priority=payload.get("pipeline_priority", DEFAULT_PRIORITY),
    )


def _fire_barrier_callback(
    callback_descriptor: CallbackDescriptor, all_results: list[Any]
) -> str:
    """Dispatch the aggregating callback when the barrier completes; return its id.

    Self-chains the callback onto the PG queue via :func:`_dispatch_pg`, carrying
    the producer's fairness (reconstructed from the stored headers) so the
    callback rides the same org/priority as its header tasks did.

    The callback kwargs carry :data:`PG_TRANSPORT_CALLBACK_KWARG` so the
    aggregating callback can gate its at-least-once duplicate guard — it is
    otherwise unguarded against the redelivery its own dispatch admits.
    """
    # Copy (don't mutate the shared descriptor) + tag the transport so the
    # callback can gate its duplicate guard on it.
    pg_kwargs = {
        **callback_descriptor["kwargs"],
        PG_TRANSPORT_CALLBACK_KWARG: True,
    }
    handle = _dispatch_pg(
        callback_descriptor["task_name"],
        args=[all_results],
        kwargs=pg_kwargs,
        queue=callback_descriptor["queue"],
        fairness=_fairness_from_headers(callback_descriptor.get("fairness_headers")),
    )
    return str(handle.id)


class _DecrementRow(NamedTuple):
    """The decrement ``UPDATE … RETURNING`` row, named so the caller reads
    ``row.remaining`` / ``row.results`` instead of positional ``row[0]`` / ``row[1]``
    and the ``results`` list-shape is documented in the type.
    """

    remaining: int
    results: list[Any]


def _apply_decrement(
    execution_id: str, result_json: str, *, reused: bool
) -> _DecrementRow | None:
    """Apply the barrier decrement ``UPDATE`` and return its ``(remaining,
    results)`` row (``None`` if the barrier row is already gone).

    The decrement is NON-idempotent — re-applying it double-counts (premature
    callback fire with incomplete results, or a strand past 0) — so it is split
    into its two phases and retried in ONLY the one phase where a re-apply is
    provably safe:

    - **EXECUTE phase** (``UPDATE … RETURNING`` + ``fetchone``) — the statement
      runs but is NOT yet committed. A connection-level failure here when the
      connection was *cached* (``reused`` — sampled by the caller BEFORE the entry
      guard materialises a connection) is the PgBouncer idle-reap: the statement
      never reached the server, no commit was issued, and the open transaction is
      rolled back on disconnect. So the decrement provably never landed →
      reconnect and re-apply it exactly once. A *freshly-created* connection
      failing (``reused`` False) is a genuine DB error, not a reap, so it is NOT
      retried — a reconnect would buy nothing. **This safety relies on the
      connection being non-autocommit** (psycopg2's default; ``create_pg_connection``
      does not override it): even if the server actually ran the ``UPDATE`` before
      the socket dropped, it sits in an uncommitted transaction that is rolled back
      on disconnect — durability happens ONLY at the commit below — so re-applying
      can never double-count. (``test_create_pg_connection_is_non_autocommit`` pins
      that default at its source.)
    - **COMMIT phase** — ``conn.commit()``. A failure here is AMBIGUOUS: the
      server may have applied the commit before the socket dropped. Re-applying
      could double-count, so it is NEVER retried — it propagates. The in-body
      caller (:func:`run_batch_with_barrier`) then tears the barrier down so the
      execution fails fast, and a barrier left behind by a hard crash is
      reclaimed at ``expires_at`` by the reaper. Either way the counter is never
      corrupted.

    Any non-connection error (e.g. the NUL-byte ``psycopg2.DataError`` from the
    jsonb cast) also propagates unchanged — only the idle-reap self-heals. This is
    the decrement's counterpart to ``pg_queue.client.send``'s reused-guard
    and the idempotent pre-dispatch write's retry, but phase-split
    because the decrement, unlike those, must never replay a committed write.
    """
    # ``last_progress_at = now()`` records that a batch just completed — the
    # reaper's stuck signal (it marks the execution ERROR only once no decrement
    # has landed for stuck_timeout; see barrier.py / reaper.py). last_progress_at
    # is UNINDEXED, so — like remaining/results — this stays a heap-only-tuple (HOT)
    # update: no index churn on the decrement hot path. (expires_at, the indexed
    # absolute cap, is deliberately NOT touched here, to preserve HOT.)
    sql = (
        f"UPDATE {qualified('pg_barrier_state')} "
        "   SET remaining = remaining - 1, "
        "       results = results || jsonb_build_array(%s::jsonb), "
        "       last_progress_at = now() "
        " WHERE execution_id = %s "
        "RETURNING remaining, results"
    )
    for attempt in range(1, _BARRIER_DECREMENT_ATTEMPTS + 1):
        conn = _get_conn()
        try:
            with conn.cursor() as cur:
                cur.execute(sql, (result_json, execution_id))
                fetched = cur.fetchone()
        except Exception as exc:
            conn_dead = _recover_after_error(conn, exc)
            # Retry ONLY a reused-conn death on the execute phase: it never
            # committed (re-applying lands exactly once) and reconnecting can
            # actually help. ``reused`` reflects the connection state at entry, so
            # it is True only on the first attempt's cached handle; a reconnect
            # makes the next attempt fresh, and the attempt bound stops the loop.
            if conn_dead and reused and attempt < _BARRIER_DECREMENT_ATTEMPTS:
                logger.warning(
                    "[exec:%s] PgBarrier decrement: execute failed on a cached "
                    "connection (%s) — reconnecting and re-applying once "
                    "(attempt %d/%d); the decrement never committed, so it lands "
                    "exactly once.",
                    execution_id,
                    type(exc).__name__,
                    attempt,
                    _BARRIER_DECREMENT_ATTEMPTS,
                    exc_info=True,
                )
                time.sleep(_BARRIER_RETRY_BACKOFF_SECONDS)
                continue
            # Not retried. A non-connection error (e.g. DataError) is logged by the
            # caller's own teardown; but a conn-dead give-up (fresh-conn death, or
            # the retry budget spent) would otherwise be silent — log it so the
            # terminal "didn't self-heal" decision leaves the same breadcrumb the
            # retryable path does, then propagate.
            if conn_dead:
                logger.warning(
                    "[exec:%s] PgBarrier decrement: execute failed with a "
                    "connection error (%s) that is NOT being retried (reused=%s, "
                    "attempt %d/%d) — propagating; the barrier is not self-healed "
                    "here.",
                    execution_id,
                    type(exc).__name__,
                    reused,
                    attempt,
                    _BARRIER_DECREMENT_ATTEMPTS,
                    exc_info=True,
                )
            raise
        try:
            conn.commit()
        except Exception as exc:
            _recover_after_error(conn, exc)
            # AMBIGUOUS — the server may have committed. Re-applying could
            # double-count, so do NOT retry: propagate (the caller fails the
            # barrier fast, or it is reclaimed at expiry) rather than corrupt it.
            logger.warning(
                "[exec:%s] PgBarrier decrement: commit failed (%s) — NOT retrying "
                "(the server may already have applied it; a re-apply would "
                "double-count). Propagating.",
                execution_id,
                type(exc).__name__,
                exc_info=True,
            )
            raise
        return None if fetched is None else _DecrementRow(int(fetched[0]), fetched[1])
    # Defensive: the loop always returns or raises before here. The annotation
    # permits None, so a type checker would NOT catch a stray fall-through — this
    # guards a future edit that breaks the always-return/raise invariant.
    raise AssertionError("unreachable: _apply_decrement loop exited without return")


def _barrier_pg_decrement(
    result: Any,
    *,
    execution_id: str,
    callback_descriptor: CallbackDescriptor,
) -> dict[str, Any]:
    """Atomic barrier decrement + last-task callback fire (substrate core).

    Appends this task's result and decrements ``remaining`` in one atomic
    statement; the single caller that drives ``remaining`` to 0 dispatches the
    aggregating callback (then deletes the row). Called directly from the header
    task's body by :func:`run_batch_with_barrier` — a PG-consumed task fires no
    Celery ``.link``, so the decrement must run in-body. (It used to have a
    second entry point, a ``@worker_task`` link wrapper, deleted with the Celery
    transport in UN-4078.)

    Callers MUST run each decrement in its own committed transaction and MUST NOT
    replay a *committed* decrement: re-applying one that already landed corrupts
    the count (fires the callback early with incomplete results, or skips past 0
    and strands the barrier). This is enforced loudly, not just in prose — entry
    raises if the shared connection is already mid-transaction (see the guard
    below). The task-level replay guard that used to back this up — the Celery
    wrapper's ``max_retries=0`` — went with ``barrier_pg_decr_and_check`` in
    UN-4078; the decrement now runs in-body, so a replay would be a redelivery of
    the batch itself, which ``claim_batch`` already makes idempotent. An orphaned
    barrier is bounded by ``expires_at``.

    The decrement DOES self-heal one narrow, provably-safe case via
    :func:`_apply_decrement`: an execute-phase failure on a cached connection that
    PgBouncer idle-reaped (the statement never reached the server, so nothing
    committed) is reconnected and re-applied exactly once. A commit-phase failure
    is ambiguous and is NEVER retried — see :func:`_apply_decrement`.
    """
    # Sample whether the decrement will run on a *cached* connection BEFORE the
    # entry-guard's _get_conn() below materialises one. _apply_decrement's
    # reused-guard needs the pre-guard state: a freshly-created connection must NOT
    # be classified 'reused' (only a cached handle can be a stale idle-reap), but
    # the guard's _get_conn() would otherwise always leave _local.conn populated,
    # making 'reused' permanently True. Mirrors pg_queue.client.send, which samples
    # before any _get_conn().
    conn_was_cached = getattr(_local, "conn", None) is not None and not _local.conn.closed

    # Enforce the "own committed transaction" contract loudly. A caller that
    # invokes this inside an already-open transaction on the shared thread-local
    # connection would hold the row lock across the call and let the outer txn's
    # commit boundary replay the decrement — corrupting ``remaining`` and
    # breaking the exactly-one-sees-zero serialisation, surfacing only as a
    # barrier hung until ``expires_at`` (~6h). The Celery ``.link`` path always
    # enters idle (``_cursor`` commits after every use); the PG in-body caller
    # must too. (Tests inject an autocommit connection → always idle → no trip.)
    if (
        _get_conn().get_transaction_status()
        != psycopg2.extensions.TRANSACTION_STATUS_IDLE
    ):
        raise RuntimeError(
            f"[exec:{execution_id}] _barrier_pg_decrement entered with an open "
            f"transaction on the shared connection — each decrement MUST run in "
            f"its own committed transaction (see this function's docstring). "
            f"Refusing to proceed to avoid corrupting the barrier counter."
        )

    try:
        # No default=str — a non-JSON-safe leaf must fail loudly here (it would
        # signal a BatchExecutionResult.to_dict() typed-boundary regression).
        # dumps_for_jsonb additionally repairs the strings a jsonb cast refuses
        # (NUL, lone surrogate) so a stray control byte in a header result no
        # longer costs the whole barrier. What it cannot repair (NaN, or a
        # rejection it does not model) propagates to the caller, which owns
        # teardown — see the note on the raise below.
        result_json = dumps_for_jsonb(result)
    except (TypeError, ValueError):
        # Log and re-raise; do NOT tear the barrier down here. ``allow_nan=False``
        # moves NaN/Infinity onto this branch, so this is the encode-side twin of
        # the rejection branch below and must follow the same rule.
        logger.exception(
            f"[exec:{execution_id}] Header task result cannot be encoded for "
            f"jsonb (not JSON-serialisable, or a non-finite number) — "
            f"propagating so the caller can mark the execution terminal before "
            f"the barrier row is released."
        )
        raise

    # jsonb_build_array(...) appends exactly one element regardless of the
    # result's shape (``||`` would concatenate if the result were itself a list).
    try:
        row = _apply_decrement(execution_id, result_json, reused=conn_was_cached)
    except _PAYLOAD_REJECTED_ERRORS:
        # Reached for a rejection dumps_for_jsonb does not model. The decrement
        # never lands, so this batch can never complete the barrier.
        #
        # Catch the closed family the result backend uses, not ``DataError``
        # alone: ``ProgramLimitExceeded`` ("string too long", SQLSTATE 54)
        # subclasses ``OperationalError``, so a DataError-only net lets an
        # oversized header result through without this specific, actionable log.
        #
        # Deliberately does NOT call ``_delete_barrier``. Teardown belongs to
        # ``run_batch_with_barrier``, which marks the execution ERROR *first* and
        # releases the row only once that mark is confirmed — because the row is
        # the reaper's only recovery handle. Deleting it here pre-empts that
        # ordering: when the mark cannot be confirmed (backend unreachable, or no
        # organization_id on the descriptor) the caller's ``else`` branch
        # deliberately preserves the row, logs "leaving the barrier row intact
        # for the reaper" — and finds it already gone. The execution then sits
        # non-terminal forever with nothing for the reaper's sweep to find, which
        # is a permanent strand: strictly worse than the ~6h hang-to-expiry this
        # teardown was trying to avoid, and the exact class UN-4126 exists to
        # eliminate.
        logger.exception(
            f"[exec:{execution_id}] Header result rejected by the database even "
            f"after sanitisation — propagating so the caller can mark the "
            f"execution terminal before the barrier row is released. If this is "
            f"a content rejection it is a gap in unstract.core.jsonb — capture "
            f"the payload and extend it."
        )
        raise

    if row is None:
        # Barrier already torn down (a header failed → abort deleted the row, or
        # an expiry sweep removed it). No callback.
        logger.error(
            f"[exec:{execution_id}] PgBarrier decrement found no row — barrier "
            f"already torn down. No callback dispatched."
        )
        return {"status": "abandoned", "remaining": None}

    remaining, all_results = row.remaining, row.results
    logger.info(f"[exec:{execution_id}] PgBarrier decrement → remaining={remaining}")

    if remaining > 0:
        return {"status": "pending", "remaining": remaining}

    if remaining < 0:
        # Decremented past 0 — expiry/replay. Clean up, no fire.
        _delete_barrier(execution_id)
        logger.error(
            f"[exec:{execution_id}] PgBarrier abandoned — remaining={remaining} "
            f"(expired or torn down). No callback dispatched; execution likely "
            f"in an inconsistent terminal state and needs investigation."
        )
        return {"status": "abandoned", "remaining": remaining}

    # remaining == 0: we are the last task. psycopg2 decodes the jsonb array to
    # a Python list already — no per-element json.loads needed.
    #
    # Dispatch the callback FIRST; delete the row only after dispatch succeeds, so
    # a callback dispatch failure leaves the row (and its expiry) in place rather
    # than stranding the execution with no state and no recovery path. No
    # double-fire is possible: this is the last decrement (remaining hit 0) and
    # max_retries=0, so the task is never replayed.
    callback_id = _fire_barrier_callback(callback_descriptor, all_results)
    # Post-dispatch cleanup is best-effort: the callback already fired, so a
    # failure here is logged (not raised) — the rows linger until reclaim rather
    # than re-running the callback. The two cleanups are independent so one
    # failing doesn't skip the other and each names what failed: the barrier row
    # is reclaimed by expiry, the dedup markers by the (PR-3-blocking) dedup-orphan
    # sweep — neither re-runs anything (this is the last decrement, max_retries=0).
    try:
        _delete_barrier(execution_id)
    except Exception:
        logger.exception(
            f"[exec:{execution_id}] Barrier callback dispatched "
            f"(callback_task_id={callback_id}) but deleting the pg_barrier_state "
            f"row failed — reclaimed at expiry. Callback NOT re-run."
        )
    try:
        clear_execution_batches(execution_id)
    except Exception:
        logger.exception(
            f"[exec:{execution_id}] Barrier callback dispatched "
            f"(callback_task_id={callback_id}) but clearing pg_batch_dedup markers "
            f"failed — reclaimed by the dedup-orphan sweep. Callback NOT re-run."
        )

    logger.info(
        f"[exec:{execution_id}] Barrier complete — fired callback "
        f"{callback_descriptor['task_name']} on {callback_descriptor['queue']} "
        f"with {len(all_results)} aggregated results "
        f"(callback_task_id={callback_id})"
    )
    return {
        "status": "complete",
        "callback_task_id": callback_id,
        "aggregated_count": len(all_results),
    }


@worker_task(name="barrier_pg_abort", max_retries=0)
def barrier_pg_abort(
    request: Any = None,
    exc: Any = None,
    traceback: Any = None,
    *,
    execution_id: str,
    preserve_dedup_markers: bool = False,
) -> dict[str, Any]:
    """``link_error`` callback: a header task failed → tear down barrier state.

    Mirrors chord's default error semantic (callback not invoked on header
    failure). The claim and the teardown are a SINGLE atomic statement — a
    ``DELETE … RETURNING`` in one transaction. The row's existence is the dedup
    token: the first abort deletes it (and so "wins"); every concurrent sibling
    finds nothing to delete and short-circuits. Because it's one transaction, a
    crash/failure mid-abort rolls the whole thing back (the row survives) so a
    sibling can retry — there is no "claimed-but-not-deleted" window. A late
    in-flight successful decrement then finds no row and abandons (no partial
    fire).

    ``preserve_dedup_markers`` keeps the per-batch ``pg_batch_dedup`` markers in
    place (default ``False`` — the Celery ``link_error`` path writes no markers,
    so clearing them is a harmless no-op there). The in-body PG path passes
    ``True``: the failed message can still redeliver, and a surviving marker makes
    ``claim_batch`` return ``False`` on redelivery → the batch is skipped, not
    re-run wholesale (real LLM spend). The markers are reclaimed later by the
    barrier re-arm (same execution_id) or the dedup-orphan sweep; they can never
    re-run anything on their own.

    The ``request``/``exc``/``traceback`` defaults keep the old-style
    ``errback(task_id)`` invocation (mixed-version deploy / ``bind=True``) from
    raising a ``TypeError``. This task does NOT drive workflow terminal status —
    the outer orchestrators own that (the in-body PG path marks the execution
    ERROR before calling here; see :func:`run_batch_with_barrier`).
    """
    del request, exc, traceback  # logged by the outer task; unused here
    with _cursor() as cur:
        cur.execute(
            "WITH claimed AS ("
            f"    DELETE FROM {qualified('pg_barrier_state')} WHERE execution_id = %s "
            "    RETURNING execution_id"
            ") SELECT execution_id FROM claimed",
            (execution_id,),
        )
        claimed = cur.fetchone() is not None

    if not claimed:
        # Another failure already aborted this execution (or it's already gone).
        return {"status": "already_aborted", "execution_id": execution_id}

    if not preserve_dedup_markers:
        # We won the abort: reclaim the per-batch dedup markers too (best-effort —
        # the barrier is already torn down). A leftover marker would otherwise
        # linger until the future dedup-orphan sweep; it can never re-run anything.
        with contextlib.suppress(Exception):
            clear_execution_batches(execution_id)

    logger.error(
        f"[exec:{execution_id}] PgBarrier aborted — a header task failed; "
        f"barrier state cleaned up (aggregating callback will not fire; "
        f"dedup markers {'preserved' if preserve_dedup_markers else 'cleared'})."
    )
    return {"status": "aborted", "execution_id": execution_id}


def _abort_barrier_in_body(
    execution_id: str, *, reason: str, preserve_dedup_markers: bool = False
) -> None:
    """Tear the barrier down from the in-body PG path (no ``.link_error`` here).

    Best-effort: a batch failure and a DB failure are correlated, so the abort
    itself can fail — that's logged (not swallowed silently) so a stuck barrier
    isn't masked by a misleading "torn down" message. A failed teardown bottoms
    out at the barrier's ``expires_at`` and is reclaimed by the reaper.

    ``preserve_dedup_markers`` forwards to :func:`barrier_pg_abort` — the in-body
    path keeps the per-batch markers so a redelivered batch is skipped by
    ``claim_batch`` rather than re-run wholesale.
    """
    try:
        barrier_pg_abort(
            execution_id=execution_id, preserve_dedup_markers=preserve_dedup_markers
        )
        logger.error(
            f"[exec:{execution_id}] {reason} — barrier teardown attempted in-body "
            f"(no .link_error on the PG path); aggregating callback won't fire."
        )
    except Exception:
        logger.exception(
            f"[exec:{execution_id}] {reason} AND the in-body barrier teardown "
            f"itself failed — barrier will hang until expiry and be reclaimed by "
            f"the reaper (max_retries=0, no replay)."
        )


def _mark_execution_error_on_abort(
    barrier_context: BarrierContext, *, reason: str
) -> bool:
    """Best-effort: mark the execution ERROR (+cascade files) on an in-body PG
    batch failure, so a failed batch reaches a terminal state instead of being
    stranded ``EXECUTING`` forever with no handle the reaper can find.

    Returns ``True`` iff the execution is now terminal (the mark was confirmed) —
    only then is it safe for the caller to tear the barrier row down. On failure
    (backend unreachable, or no ``organization_id`` to scope the org API) returns
    ``False``: the caller must leave the barrier row so the reaper recovers it at
    expiry, rather than erasing the only recovery handle.

    PG path only — ``run_batch_with_barrier`` is the fire-and-forget substrate;
    the Celery path never reaches here (its outer orchestrator owns terminal
    status). The org and execution id come off the barrier's callback kwargs, the
    same values the enqueue stamped onto ``pg_barrier_state``.
    """
    execution_id = str(barrier_context["execution_id"])
    # .get(): this runs inside run_batch_with_barrier's except block, so a missing
    # callback_descriptor (a future/legacy dispatch path) must not raise a KeyError
    # that would mask the original batch exception before the re-raise.
    _, org = callback_recovery_identity(barrier_context.get("callback_descriptor") or {})
    organization_id = str(org or "")
    if not organization_id:
        logger.error(
            f"[exec:{execution_id}] {reason} but the barrier carries no "
            f"organization_id — cannot mark it ERROR via the org-scoped API; "
            f"leaving the barrier row for the reaper (not erasing the handle)."
        )
        return False
    # Lazy import: keep this module free of the HTTP/env client at import time
    # (mirrors the reaper) and avoid an import cycle via shared.api.
    from shared.api import InternalAPIClient

    from .pg_queue.recovery import mark_execution_error

    try:
        api_client = InternalAPIClient()
    except Exception:
        logger.exception(
            f"[exec:{execution_id}] {reason} and building the internal API client "
            f"to mark it ERROR failed — leaving the barrier row for the reaper."
        )
        return False
    return mark_execution_error(
        api_client,
        execution_id,
        organization_id,
        error_message=f"[pg-barrier-abort] {reason}.",
    )


# A batch whose execution is already terminal returns its result with this
# marker set. run_batch_with_barrier bypasses the barrier decrement for it: the
# reaper has by definition already torn the barrier down, so decrementing would
# only log a spurious "decrement found no row" ERROR (false alert noise), and an
# already-terminal execution must not have its aggregating callback fired.
SKIPPED_TERMINAL_EXECUTION_KEY = "skipped_terminal_execution"


def run_batch_with_barrier(
    barrier_context: BarrierContext, work_fn: Callable[[], dict[str, Any]]
) -> dict[str, Any]:
    """Run a fire-and-forget PG-path batch under the barrier protocol.

    A PG-consumed header task fires no Celery ``.link``, so the barrier
    coordination runs in-body here, given the ``_barrier_context`` that
    :meth:`PgBarrier._dispatch_header_pg` injected (``execution_id`` /
    ``batch_index`` / ``callback_descriptor``):

    1. **Claim** ``(execution_id, batch_index)``. A redelivery (at-least-once
       queue) finds the marker already set → returns a no-op result WITHOUT
       re-running the work or re-decrementing — exactly-once decrement.
    2. **Run** ``work_fn()`` (the batch).
    3. **Decrement** the barrier with the batch result; the task that drives
       ``remaining`` → 0 self-chains the aggregating callback onto PG.

    Steps 2 *and* 3 are wrapped: any catchable failure (work error, the decrement
    guard / DB error, or the last-batch callback dispatch failing) tears the
    barrier down in-body via :func:`barrier_pg_abort` and re-raises, so it fails
    fast instead of hanging to expiry — mirroring the chord path's ``.link_error``.

    **Strand windows the reaper must cover (NOT catchable here — hard merge
    dependency for enabling the flag in PR 3, see the module / PgBatchDedup docs):**

    - A hard crash / SIGKILL / visibility-timeout expiry *during* ``work_fn()``
      leaves the dedup marker committed (claim is step 1), so redelivery returns
      ``skipped_redelivery`` and never re-runs the batch — the barrier hangs to
      ``expires_at``. (Claiming before the work avoids re-processing on the common
      redelivery case at the cost of this crash window.)
    - A hard crash *between* the decrement committing (``remaining`` → 0) and the
      callback enqueue completing: the decrement is committed (so redelivery is
      blocked by the marker) but the process is gone before the callback is
      enqueued and before any in-body abort can run, so the ``pg_barrier_state``
      row survives to ``expires_at`` with no callback ever fired. (A *software*
      callback-dispatch failure here is NOT this window — that's catchable and is
      torn down by step 3's wrap above.)

    For both, the **reaper** (sweep stranded ``pg_barrier_state``, mark the
    execution ERROR / re-drive, reclaim ``pg_batch_dedup``) is the recovery net;
    it must be live before PR 3 flips ``pg_queue_execution_enabled``.

    ``work_fn`` must return the JSON-serialisable batch result (the same dict the
    Celery chord path returns), since the decrement appends it to the barrier's
    aggregated results that the callback receives.
    """
    execution_id = str(barrier_context["execution_id"])
    batch_index = int(barrier_context["batch_index"])

    if not claim_batch(execution_id, batch_index):
        logger.info(
            f"[exec:{execution_id}] batch {batch_index} already claimed — "
            f"redelivery, skipping (barrier already decremented for this batch)."
        )
        return {
            "status": "skipped_redelivery",
            "execution_id": execution_id,
            "batch_index": batch_index,
        }

    # Wrap BOTH the work and the decrement: a decrement-side failure (the
    # open-transaction guard, a json/DB error, or the last-batch callback dispatch
    # raising) must also tear the barrier down, else the marker is committed,
    # redelivery is blocked, and the barrier strands to expiry.
    try:
        result = work_fn()
        if result.get(SKIPPED_TERMINAL_EXECUTION_KEY):
            # The batch's execution is already terminal, so the reaper has by
            # definition already torn the barrier down. Bypass the decrement —
            # decrementing a gone barrier only logs a spurious "decrement found
            # no row" ERROR (false alert noise) — and do NOT abort (nothing to
            # tear down; and an already-terminal execution must not have its
            # aggregating callback re-fired).
            logger.info(
                f"[exec:{execution_id}] batch {batch_index} skipped — execution "
                f"already terminal; barrier decrement bypassed."
            )
            return result
        _barrier_pg_decrement(
            result,
            execution_id=execution_id,
            callback_descriptor=barrier_context["callback_descriptor"],
        )
    except Exception:
        reason = f"batch {batch_index} failed"
        # Mark the execution ERROR FIRST (+cascade files) so it reaches a terminal
        # state. Only if that's confirmed do we tear the barrier row down — the
        # row is the reaper's only recovery handle, so we must not delete it while
        # the execution is still non-terminal. On a confirmed mark, keep the dedup
        # markers so this message's redelivery is skipped by claim_batch (belt-and-
        # braces with the terminal-execution guard) rather than re-run wholesale.
        if _mark_execution_error_on_abort(barrier_context, reason=reason):
            _abort_barrier_in_body(
                execution_id, reason=reason, preserve_dedup_markers=True
            )
        else:
            # Mark unconfirmed (backend down / no org): leave the barrier row so
            # the reaper marks it ERROR and reclaims it at expiry. Do NOT erase the
            # handle — that's the strand this ticket fixes.
            logger.error(
                f"[exec:{execution_id}] {reason} and could not confirm the ERROR "
                f"mark — leaving the barrier row intact for the reaper (barrier "
                f"hangs to expiry rather than stranding non-terminal)."
            )
        raise
    return result