Skip to content

Use Claude Code, Codex, VSCode, Pi, and OpenCode (and 6 other harnesses) for free (1.3B+ free tokens) from your terminal, app, IDE, or phone, and now from the browser with native browser sessions (multi-harness + multi-model) like OpenClaw (voice supported + ToS friendly)

Python

Make provider executions own attempt lifecycles (#1508)

Ali Khokhar904581c5094f
diff --git a/ARCHITECTURE.md b/ARCHITECTURE.md
index adacb83a..d0125fa1 100644
--- a/ARCHITECTURE.md
+++ b/ARCHITECTURE.md
@@ -707,15 +707,32 @@ provider ID within a generation; there is no pass-through cache object, process
singleton, or second admission registry.
[providers/admission.py](src/free_claude_code/providers/admission.py) owns the
-complete shared upstream-admission lifecycle for that provider generation. A
-strict sliding window admits each real attempt before a concurrency bulkhead;
-the bulkhead is held only while an upstream operation or stream is active, never
-during retry backoff. The first retryable failure before upstream acceptance
-opens one recovery episode and elects that logical execution as leader. The
-leader alone waits and sends half-open probes. Concurrent failures coalesce into
-the same episode, later callers wait, and already active streams continue. A
-stale in-flight success or failure cannot close or extend the episode, while
-leader cancellation transfers ownership to a waiter.
+complete shared upstream-admission lifecycle for that provider generation. Its
+three scopes are explicit:
+
+- `ProviderAdmissionController` owns the provider-generation sliding window,
+ concurrency bulkhead, recovery episode, backoff, and probe election;
+- `ProviderExecution` owns one logical provider operation, its correlation ID,
+ single five-attempt budget, active-attempt identity, last raw failure, and
+ terminal state;
+- `ProviderAttempt` owns one physical provider HTTP/generation call, including
+ its opaque execution claim, admission permit, concurrency lease, acceptance
+ state, one idempotent outcome, and exact-once close.
+
+Only `ProviderExecution.open_attempt()` can enter physical provider I/O. A
+strict sliding window admits the call before the concurrency bulkhead; the
+bulkhead is held only while that call or stream is active, never during retry
+backoff. Every physical call emits one metadata-only
+`provider.attempt.started`/`provider.attempt.resolved` pair with execution ID,
+operation kind, and attempt ordinal. Prompts, credentials, bodies, and raw
+errors are not part of those events.
+
+The first retryable failure before upstream acceptance opens one recovery
+episode and elects that logical execution as leader. The leader alone waits and
+sends half-open probes. Concurrent failures coalesce into the same episode,
+later callers wait, and already active streams continue. A stale in-flight
+success or failure cannot close or extend the episode, while leader cancellation
+transfers ownership to a waiter.
A successful probe closes the episode and releases waiters through ordinary
rate and concurrency admission. A non-retryable probe response also closes it
@@ -726,6 +743,15 @@ New work fails fast during the provider-directed cooldown; once it expires,
exactly one new caller becomes the next probe. There is no background retry
worker, copied request queue, or second scheduling system.
+The one-call helper on `ProviderExecution` accepts only a callback that performs
+one quota-bearing provider call. Higher-level orchestration remains with its
+protocol owner: every model-catalog page receives a separate logical execution,
+while a Codex catalog GET followed by credential refresh and another catalog GET
+uses two attempts in one execution. OAuth refresh itself remains auth-owned and
+does not hold or consume a provider-attempt lease. Discovery and generation use
+distinct operation kinds and budgets but intentionally share the same
+provider-generation health episode.
+
Retired generations retain their own synchronization state until request leases
drain, while new generations and separate server instances never reuse it. Hot
replacement therefore begins with fresh quota and recovery state; an old and new
@@ -1085,18 +1111,20 @@ owns only the 0.75-second/65,536-byte commit holdback and the choice between
transparent replay, request-local continuation/tool salvage, and final failure.
It consumes an explicit retryability decision and does not import provider
transport SDKs or classify exceptions.
-`ProviderRetrySession` owns one five-attempt budget for the whole logical
-execution: initial opening, deterministic request-shape corrections, early
-replay, continuation, and tool repair all consume that same budget. There are no
-nested retry counters. Deterministic corrections retry immediately; transient
-failures use exponential backoff with jitter and honor `Retry-After` as a
-minimum. When partial output exists, the last available attempt is reserved for
-continuation or repair instead of replaying the full request again. Completed
-tool calls can be salvaged without an upstream attempt. The application-owned
-progress window is an outer no-progress bound, not another retry counter: opening
-a new attempt never resets it, while a real emitted provider chunk does. Fast
-transient failures can therefore still use all five attempts, but repeated
-fully-stalled operations cannot outlive the downstream harness.
+`ProviderExecution` owns one five-attempt budget for the whole logical
+operation: initial opening, deterministic request-shape corrections, early
+replay, continuation, and tool repair all consume that same budget through
+separate `ProviderAttempt` leases. There are no nested retry counters or
+controller-owned callback loop. Deterministic corrections retry immediately;
+transient failures use exponential backoff with jitter and honor `Retry-After`
+as a minimum. When partial output exists, the last available attempt is reserved
+for continuation or repair instead of replaying the full request again.
+Completed tool calls can be salvaged without an upstream attempt. The
+application-owned progress window is an outer no-progress bound, not another
+retry counter: opening a new attempt never resets it, while a real emitted
+provider chunk does. Fast transient failures can therefore still use all five
+attempts, but repeated fully-stalled operations cannot outlive the downstream
+harness.
For streams, upstream acceptance is the first received chunk. Retryable failure
before that point participates in provider-wide coordinated recovery. Failure
diff --git a/pyproject.toml b/pyproject.toml
index d8821f80..5ef2e6db 100644
--- a/pyproject.toml
+++ b/pyproject.toml
@@ -4,7 +4,7 @@ build-backend = "hatchling.build"
[project]
name = "free-claude-code"
-version = "5.13.1"
+version = "5.13.2"
description = "Local proxy connecting coding agents to OpenAI-compatible AI providers"
readme = "README.md"
requires-python = ">=3.14.0"
diff --git a/src/free_claude_code/providers/admission.py b/src/free_claude_code/providers/admission.py
index d4a73375..3bd9490a 100644
--- a/src/free_claude_code/providers/admission.py
+++ b/src/free_claude_code/providers/admission.py
@@ -4,10 +4,12 @@ import asyncio
import math
import random
import time
+import uuid
from collections.abc import Awaitable, Callable
from dataclasses import dataclass, field
from datetime import UTC, datetime
from email.utils import parsedate_to_datetime
+from enum import StrEnum
from typing import TypeVar
from loguru import logger
@@ -29,16 +31,68 @@ DEFAULT_UPSTREAM_MAX_DELAY = 60.0
DEFAULT_UPSTREAM_JITTER = 1.0
-class ProviderRetrySession:
- """One non-multiplying upstream-attempt budget for a logical execution."""
+class ProviderOperationKind(StrEnum):
+ """Safe trace category for one physical provider call."""
- def __init__(self, *, max_attempts: int, request_id: str | None = None) -> None:
- if max_attempts <= 0:
- raise ValueError("max_attempts must be > 0")
+ MODEL_DISCOVERY = "model_discovery"
+ GENERATION = "generation"
+ CONTINUATION = "continuation"
+ TOOL_REPAIR = "tool_repair"
+
+
+class ProviderExecutionState(StrEnum):
+ """Terminal state of one logical provider operation."""
+
+ ACTIVE = "active"
+ SUCCEEDED = "succeeded"
+ FAILED = "failed"
+ ABANDONED = "abandoned"
+
+
+class ProviderCorrectionAction(StrEnum):
+ """Whether one deterministic request correction may consume another attempt."""
+
+ RETRY = "retry"
+ FINAL = "final"
+
+
+@dataclass(frozen=True, slots=True)
+class ProviderFailureDecision:
+ """Immutable provider qualification and execution-budget decision."""
+
+ retryable: bool
+ retry_allowed: bool
+
+
+@dataclass(frozen=True, slots=True)
+class _AttemptClaim:
+ execution_id: str
+ ordinal: int
+ operation_kind: ProviderOperationKind
+
+
+class ProviderExecution:
+ """Own one logical provider operation and its physical-attempt budget."""
+
+ def __init__(
+ self,
+ controller: ProviderAdmissionController,
+ *,
+ max_attempts: int,
+ request_id: str | None,
+ ) -> None:
+ self._controller = controller
+ self._execution_id = str(uuid.uuid4())
self._max_attempts = max_attempts
self._request_id = request_id
self._attempts_started = 0
- self._terminal_error: Exception | None = None
+ self._active_claim: _AttemptClaim | None = None
+ self._last_failure: Exception | None = None
+ self._state = ProviderExecutionState.ACTIVE
+
+ @property
+ def execution_id(self) -> str:
+ return self._execution_id
@property
def max_attempts(self) -> int:
@@ -54,23 +108,121 @@ class ProviderRetrySession:
@property
def can_attempt(self) -> bool:
- return self._attempts_started < self._max_attempts
+ return (
+ self._state is ProviderExecutionState.ACTIVE
+ and self._attempts_started < self._max_attempts
+ )
@property
def attempts_remaining(self) -> int:
- return self._max_attempts - self._attempts_started
+ if self._state is not ProviderExecutionState.ACTIVE:
+ return 0
+ return max(0, self._max_attempts - self._attempts_started)
+
+ @property
+ def state(self) -> ProviderExecutionState:
+ return self._state
- def _claim_attempt(self) -> int:
+ @property
+ def last_failure(self) -> Exception | None:
+ return self._last_failure
+
+ async def open_attempt(
+ self,
+ operation_kind: ProviderOperationKind,
+ ) -> ProviderAttempt:
+ """Open the sole active physical call for this execution."""
+ return await self._controller._open_attempt(self, operation_kind)
+
+ async def run_call(
+ self,
+ fn: Callable[[], Awaitable[T]],
+ *,
+ operation_kind: ProviderOperationKind,
+ provider_failure_override: ProviderFailureOverride | None = None,
+ ) -> T:
+ """Run a callable that performs exactly one provider call per invocation."""
+ try:
+ while self.can_attempt:
+ attempt = await self.open_attempt(operation_kind)
+ try:
+ result = await fn()
+ except asyncio.CancelledError:
+ raise
+ except Exception as error:
+ decision = await attempt.fail(
+ error,
+ provider_failure_override=provider_failure_override,
+ )
+ if not decision.retry_allowed:
+ raise
+ else:
+ await attempt.accept()
+ self.succeed()
+ return result
+ finally:
+ await attempt.aclose()
+ except asyncio.CancelledError:
+ self.abandon()
+ raise
+ except Exception as error:
+ self.fail(error)
+ raise
+
+ if self._last_failure is not None:
+ self.fail(self._last_failure)
+ raise self._last_failure
+ self.abandon()
+ raise RuntimeError("provider execution ended without an attempt outcome")
+
+ def succeed(self) -> None:
+ """Mark the complete logical provider operation successful."""
+ if self._state is ProviderExecutionState.ACTIVE:
+ self._state = ProviderExecutionState.SUCCEEDED
+
+ def fail(self, error: Exception) -> None:
+ """Mark the complete logical provider operation failed."""
+ if self._state is ProviderExecutionState.ACTIVE:
+ self._last_failure = error
+ self._state = ProviderExecutionState.FAILED
+
+ def abandon(self) -> None:
+ """Mark an unfinished logical provider operation abandoned."""
+ if self._state is ProviderExecutionState.ACTIVE:
+ self._state = ProviderExecutionState.ABANDONED
+
+ def _claim_attempt(
+ self,
+ operation_kind: ProviderOperationKind,
+ ) -> _AttemptClaim:
if not self.can_attempt:
- raise RuntimeError("provider retry session is exhausted")
+ raise RuntimeError("provider execution is terminal or exhausted")
+ if self._active_claim is not None:
+ raise RuntimeError("provider execution already has an active attempt")
self._attempts_started += 1
- return self._attempts_started
+ claim = _AttemptClaim(
+ execution_id=self._execution_id,
+ ordinal=self._attempts_started,
+ operation_kind=operation_kind,
+ )
+ self._active_claim = claim
+ return claim
+
+ def _close_attempt(self, claim: _AttemptClaim) -> None:
+ if self._active_claim == claim:
+ self._active_claim = None
+
+ def _record_failure(self, error: Exception) -> None:
+ if self._state is ProviderExecutionState.ACTIVE:
+ self._last_failure = error
def _fail_recovery(self, error: Exception) -> None:
- self._terminal_error = error
+ self.fail(error)
def _terminal_failure(self) -> Exception | None:
- return self._terminal_error
+ if self._state is ProviderExecutionState.FAILED:
+ return self._last_failure
+ return None
@dataclass(frozen=True, slots=True)
@@ -82,10 +234,10 @@ class _GatePermit:
@dataclass(slots=True)
class _RecoveryEpisode:
generation: int
- leader: ProviderRetrySession | None
+ leader: ProviderExecution | None
ready_at: float
last_error: Exception
- waiters: set[ProviderRetrySession] = field(default_factory=set)
+ waiters: set[ProviderExecution] = field(default_factory=set)
probe_active: bool = False
terminal_until: float | None = None
@@ -96,15 +248,16 @@ class ProviderAttempt:
def __init__(
self,
controller: ProviderAdmissionController,
- session: ProviderRetrySession,
+ execution: ProviderExecution,
permit: _GatePermit,
+ claim: _AttemptClaim,
) -> None:
self._controller = controller
- self._session = session
+ self._execution = execution
self._permit = permit
+ self._claim = claim
self._resolved = False
self._accepted = False
- self._failure_retryable: bool | None = None
self._closed = False
@property
@@ -112,35 +265,49 @@ class ProviderAttempt:
"""Return whether upstream acceptance has resolved this attempt."""
return self._accepted
- @property
- def failure_retryable(self) -> bool | None:
- """Return the provider classification recorded for a failed attempt."""
- return self._failure_retryable
+ async def __aenter__(self) -> ProviderAttempt:
+ return self
- async def succeeded(self) -> None:
- """Record a valid upstream response and close a recovery episode if probing."""
- if self._resolved:
- return
- await self._controller._attempt_succeeded(self._session, self._permit)
- self._resolved = True
- self._accepted = True
+ async def __aexit__(
+ self,
+ exc_type: type[BaseException] | None,
+ exc: BaseException | None,
+ traceback: object | None,
+ ) -> None:
+ del exc_type, exc, traceback
+ await self.aclose()
- async def retry_immediately(self) -> None:
- """Keep probe ownership for a bounded request-shape correction."""
- if self._resolved:
+ async def accept(self) -> None:
+ """Record a valid upstream response and close a recovery episode if probing."""
+ if self._resolved or self._closed:
return
- await self._controller._attempt_corrected(self._session, self._permit)
- self._resolved = True
-
- async def retry(
+ await self._controller._attempt_accepted(self._execution, self._permit)
+ self._resolve("accepted", accepted=True)
+
+ async def correct(self, error: Exception) -> ProviderCorrectionAction:
+ """Resolve one deterministic request correction without provider backoff."""
+ if self._resolved or self._closed:
+ return ProviderCorrectionAction.FINAL
+ self._execution._record_failure(error)
+ status = _exception_status(error)
+ if self._execution.can_attempt:
+ await self._controller._attempt_corrected(self._execution, self._permit)
+ self._resolve("corrected", status=status, error=error)
+ return ProviderCorrectionAction.RETRY
+ await self._controller._attempt_rejected(self._execution, self._permit)
+ self._execution.fail(error)
+ self._resolve("rejected", status=status, error=error)
+ return ProviderCorrectionAction.FINAL
+
+ async def fail(
self,
error: Exception,
*,
provider_failure_override: ProviderFailureOverride | None = None,
- ) -> bool:
- """Record a retryable failure and return whether this execution may retry."""
- if self._resolved:
- return False
+ ) -> ProviderFailureDecision:
+ """Classify one failure and return its immutable retry decision."""
+ if self._resolved or self._closed:
+ return ProviderFailureDecision(retryable=False, retry_allowed=False)
effective_error = (
provider_failure_override(error)
if provider_failure_override is not None
@@ -150,19 +317,31 @@ class ProviderAttempt:
effective_error = error
status = retryable_upstream_status(effective_error)
retryable = is_retryable_provider_error(effective_error)
- self._failure_retryable = retryable
+ self._execution._record_failure(error)
if not retryable:
- await self._controller._attempt_rejected(self._session, self._permit)
- self._resolved = True
- return False
- should_retry = await self._controller._attempt_failed(
- self._session,
+ await self._controller._attempt_rejected(self._execution, self._permit)
+ self._execution.fail(error)
+ self._resolve("rejected", status=status, error=effective_error)
+ return ProviderFailureDecision(retryable=False, retry_allowed=False)
+ await self._controller._attempt_failed(
+ self._execution,
self._permit,
error=error,
status=status,
)
- self._resolved = True
- return should_retry
+ retry_allowed = self._execution.can_attempt
+ if not retry_allowed:
+ self._execution.fail(error)
+ self._resolve(
+ "retryable_failure",
+ status=status,
+ error=effective_error,
+ retry_allowed=retry_allowed,
+ )
+ return ProviderFailureDecision(
+ retryable=True,
+ retry_allowed=retry_allowed,
+ )
async def aclose(self) -> None:
"""Release attempt ownership and its concurrency slot exactly once."""
@@ -171,12 +350,50 @@ class ProviderAttempt:
self._closed = True
try:
if not self._resolved:
- await asyncio.shield(
- self._controller._attempt_abandoned(self._session, self._permit)
- )
+ try:
+ await asyncio.shield(
+ self._controller._attempt_abandoned(
+ self._execution,
+ self._permit,
+ )
+ )
+ finally:
+ self._resolve("abandoned")
finally:
+ self._execution._close_attempt(self._claim)
self._controller._release_concurrency()
+ def _resolve(
+ self,
+ outcome: str,
+ *,
+ accepted: bool = False,
+ status: int | None = None,
+ error: BaseException | None = None,
+ retry_allowed: bool | None = None,
+ ) -> None:
+ if self._resolved:
+ return
+ self._resolved = True
+ self._accepted = accepted
+ trace_event(
+ stage="provider",
+ event="provider.attempt.resolved",
+ source="provider",
+ provider=self._controller._provider_name,
+ request_id=self._execution.request_id,
+ execution_id=self._execution.execution_id,
+ operation_kind=self._claim.operation_kind.value,
+ attempt=self._claim.ordinal,
+ max_attempts=self._execution.max_attempts,
+ probe=self._permit.probe,
+ recovery_generation=self._permit.generation,
+ outcome=outcome,
+ status_code=status,
+ exc_type=(None if error is None else type(error).__name__),
+ retry_allowed=retry_allowed,
+ )
+
class ProviderAdmissionController:
"""Coordinate one provider's rate, concurrency, and recovery state.
@@ -236,78 +453,78 @@ class ProviderAdmissionController:
max_attempts,
)
- def new_retry_session(
+ def start_execution(
self,
*,
request_id: str | None = None,
- ) -> ProviderRetrySession:
- """Return a fresh logical-execution retry budget."""
- return ProviderRetrySession(
+ ) -> ProviderExecution:
+ """Create the sole lifecycle owner for one logical provider operation."""
+ return ProviderExecution(
+ self,
max_attempts=self._max_attempts,
request_id=request_id,
)
- async def open_attempt(self, session: ProviderRetrySession) -> ProviderAttempt:
+ async def _open_attempt(
+ self,
+ execution: ProviderExecution,
+ operation_kind: ProviderOperationKind,
+ ) -> ProviderAttempt:
"""Wait for provider admission and hold one active-operation slot."""
- if not session.can_attempt:
- raise RuntimeError("provider retry session is exhausted")
+ if not isinstance(operation_kind, ProviderOperationKind):
+ raise TypeError("operation_kind must be a ProviderOperationKind")
+ if (terminal_error := execution._terminal_failure()) is not None:
+ raise ProviderRecoveryExhausted(terminal_error)
+ if not execution.can_attempt:
+ raise RuntimeError("provider execution is terminal or exhausted")
+ if execution._active_claim is not None:
+ raise RuntimeError("provider execution already has an active attempt")
while True:
- permit = await self._wait_for_gate(session)
+ permit = await self._wait_for_gate(execution)
slot_acquired = False
+ claim: _AttemptClaim | None = None
try:
admitted = await self._proactive_limiter.acquire_if(
- lambda permit=permit: self._permit_is_current(session, permit)
+ lambda permit=permit: self._permit_is_current(execution, permit)
)
if not admitted:
- await self._abandon_probe_permit(session, permit)
+ await self._abandon_probe_permit(execution, permit)
continue
await self._concurrency_sem.acquire()
slot_acquired = True
- if not self._permit_is_current(session, permit):
+ if not self._permit_is_current(execution, permit):
self._concurrency_sem.release()
slot_acquired = False
- await self._abandon_probe_permit(session, permit)
+ await self._abandon_probe_permit(execution, permit)
continue
- session._claim_attempt()
- return ProviderAttempt(self, session, permit)
+ claim = execution._claim_attempt(operation_kind)
+ attempt = ProviderAttempt(self, execution, permit, claim)
+ trace_event(
+ stage="provider",
+ event="provider.attempt.started",
+ source="provider",
+ provider=self._provider_name,
+ request_id=execution.request_id,
+ execution_id=execution.execution_id,
+ operation_kind=operation_kind.value,
+ attempt=claim.ordinal,
+ max_attempts=execution.max_attempts,
+ probe=permit.probe,
+ recovery_generation=permit.generation,
+ )
+ return attempt
except BaseException:
+ if claim is not None:
+ execution._close_attempt(claim)
if slot_acquired:
self._concurrency_sem.release()
- await self._abandon_probe_permit(session, permit)
+ await self._abandon_probe_permit(execution, permit)
raise
- async def run_with_retry(
- self,
- fn: Callable[[], Awaitable[T]],
- *,
- provider_failure_override: ProviderFailureOverride | None = None,
- request_id: str | None = None,
- ) -> T:
- """Run one non-streaming provider operation through coordinated retries."""
- session = self.new_retry_session(request_id=request_id)
+ async def _wait_for_gate(self, execution: ProviderExecution) -> _GatePermit:
while True:
- attempt = await self.open_attempt(session)
- try:
- result = await fn()
- except asyncio.CancelledError:
- raise
- except Exception as error:
- should_retry = await attempt.retry(
- error,
- provider_failure_override=provider_failure_override,
- )
- if not should_retry:
- raise
- else:
- await attempt.succeeded()
- return result
- finally:
- await attempt.aclose()
-
- async def _wait_for_gate(self, session: ProviderRetrySession) -> _GatePermit:
- while True:
- if (terminal_error := session._terminal_failure()) is not None:
+ if (terminal_error := execution._terminal_failure()) is not None:
raise ProviderRecoveryExhausted(terminal_error)
sleep_delay: float | None = None
claimed_generation: int | None = None
@@ -321,15 +538,15 @@ class ProviderAdmissionController:
if now < episode.terminal_until:
raise ProviderRecoveryExhausted(episode.last_error)
episode = self._start_recovery_episode(
- leader=session,
+ leader=execution,
ready_at=now,
last_error=episode.last_error,
)
if episode.leader is None:
- episode.leader = session
- episode.waiters.discard(session)
- if episode.leader is session:
+ episode.leader = execution
+ episode.waiters.discard(execution)
+ if episode.leader is execution:
if episode.probe_active:
await self._condition.wait()
continue
@@ -337,9 +554,9 @@ class ProviderAdmissionController:
claimed_generation = episode.generation
if sleep_delay == 0:
episode.probe_active = True
- return self._probe_permit(session, episode)
+ return self._probe_permit(execution, episode)
else:
- episode.waiters.add(session)
+ episode.waiters.add(execution)
try:
await self._condition.wait()
except asyncio.CancelledError:
@@ -348,7 +565,7 @@ class ProviderAdmissionController:
current is not None
and current.generation == episode.generation
):
- current.waiters.discard(session)
+ current.waiters.discard(execution)
raise
continue
@@ -368,18 +585,18 @@ class ProviderAdmissionController:
episode is not None
and episode.generation == claimed_generation
and episode.terminal_until is None
- and episode.leader is session
+ and episode.leader is execution
and not episode.probe_active
):
episode.probe_active = True
- return self._probe_permit(session, episode)
+ return self._probe_permit(execution, episode)
except asyncio.CancelledError:
- await self._abandon_waiting_leader(session, claimed_generation)
+ await self._abandon_waiting_leader(execution, claimed_generation)
raise
def _probe_permit(
self,
- session: ProviderRetrySession,
+ execution: ProviderExecution,
episode: _RecoveryEpisode,
) -> _GatePermit:
trace_event(
@@ -387,16 +604,16 @@ class ProviderAdmissionController:
event="provider.recovery.probe",
source="provider",
provider=self._provider_name,
- request_id=session.request_id,
+ request_id=execution.request_id,
generation=episode.generation,
- attempt=session.attempts_started + 1,
- max_attempts=session.max_attempts,
+ attempt=execution.attempts_started + 1,
+ max_attempts=execution.max_attempts,
)
return _GatePermit(generation=episode.generation, probe=True)
def _permit_is_current(
self,
- session: ProviderRetrySession,
+ execution: ProviderExecution,
permit: _GatePermit,
) -> bool:
episode = self._episode
@@ -406,19 +623,19 @@ class ProviderAdmissionController:
episode is not None
and episode.generation == permit.generation
and episode.terminal_until is None
- and episode.leader is session
+ and episode.leader is execution
and episode.probe_active
)
- async def _attempt_succeeded(
+ async def _attempt_accepted(
self,
- session: ProviderRetrySession,
+ execution: ProviderExecution,
permit: _GatePermit,
) -> None:
if not permit.probe:
return
async with self._condition:
- episode = self._matching_probe(session, permit)
+ episode = self._matching_probe(execution, permit)
if episode is None:
return
self._episode = None
@@ -428,21 +645,21 @@ class ProviderAdmissionController:
event="provider.recovery.closed",
source="provider",
provider=self._provider_name,
- request_id=session.request_id,
+ request_id=execution.request_id,
generation=permit.generation,
- attempt=session.attempts_started,
+ attempt=execution.attempts_started,
outcome="success",
)
async def _attempt_corrected(
self,
- session: ProviderRetrySession,
+ execution: ProviderExecution,
permit: _GatePermit,
) -> None:
if not permit.probe:
return
async with self._condition:
- episode = self._matching_probe(session, permit)
+ episode = self._matching_probe(execution, permit)
if episode is None:
return
episode.probe_active = False
@@ -451,14 +668,14 @@ class ProviderAdmissionController:
async def _attempt_rejected(
self,
- session: ProviderRetrySession,
+ execution: ProviderExecution,
permit: _GatePermit,
) -> None:
"""Close a probe episode when upstream responds with a final rejection."""
if not permit.probe:
return
async with self._condition:
- episode = self._matching_probe(session, permit)
+ episode = self._matching_probe(execution, permit)
if episode is None:
return
self._episode = None
@@ -468,21 +685,21 @@ class ProviderAdmissionController:
event="provider.recovery.closed",
source="provider",
provider=self._provider_name,
- request_id=session.request_id,
+ request_id=execution.request_id,
generation=permit.generation,
- attempt=session.attempts_started,
+ attempt=execution.attempts_started,
outcome="rejected",
)
async def _attempt_abandoned(
self,
- session: ProviderRetrySession,
+ execution: ProviderExecution,
permit: _GatePermit,
) -> None:
if not permit.probe:
return
async with self._condition:
- episode = self._matching_probe(session, permit)
+ episode = self._matching_probe(execution, permit)
if episode is None:
return
episode.leader = None
@@ -492,24 +709,24 @@ class ProviderAdmissionController:
async def _attempt_failed(
self,
- session: ProviderRetrySession,
+ execution: ProviderExecution,
permit: _GatePermit,
*,
error: Exception,
status: int | None,
- ) -> bool:
- can_retry = session.can_attempt
- delay = self._retry_delay(error, session.attempts_started)
+ ) -> None:
+ can_retry = execution.can_attempt
+ delay = self._retry_delay(error, execution.attempts_started)
became_leader = False
exhausted_episode = False
async with self._condition:
episode = self._episode
- matching_probe = self._matching_probe(session, permit)
+ matching_probe = self._matching_probe(execution, permit)
if can_retry:
if episode is None:
episode = self._start_recovery_episode(
- leader=session,
+ leader=execution,
ready_at=time.monotonic() + delay,
last_error=error,
)
@@ -520,21 +737,21 @@ class ProviderAdmissionController:
episode.ready_at = time.monotonic() + delay
became_leader = True
elif episode.terminal_until is not None:
- session._fail_recovery(episode.last_error)
+ execution._fail_recovery(episode.last_error)
else:
- episode.waiters.add(session)
+ episode.waiters.add(execution)
self._condition.notify_all()
elif episode is None or matching_probe is not None:
terminal_delay = self._retry_delay(
error,
- session.attempts_started,
+ execution.attempts_started,
)
if episode is None:
episode = self._start_recovery_episode(
leader=None,
ready_at=time.monotonic(),
last_error=error,
- request_id=session.request_id,
+ request_id=execution.request_id,
)
episode.last_error = error
episode.leader = None
@@ -551,8 +768,8 @@ class ProviderAdmissionController:
logger.warning(
"{}, attempt {}/{} failed; one provider recovery probe in {:.1f}s",
label,
- session.attempts_started,
- session.max_attempts,
+ execution.attempts_started,
+ execution.max_attempts,
delay,
)
trace_event(
@@ -560,11 +777,11 @@ class ProviderAdmissionController:
event="provider.retry.scheduled",
source="provider",
provider=self._provider_name,
- request_id=session.request_id,
+ request_id=execution.request_id,
status_code=status,
exc_type=type(error).__name__,
- attempt=session.attempts_started,
- max_attempts=session.max_attempts,
+ attempt=execution.attempts_started,
+ max_attempts=execution.max_attempts,
delay_s=round(delay, 3),
coordinated=True,
)
@@ -574,35 +791,34 @@ class ProviderAdmissionController:
event="provider.retry.coalesced",
source="provider",
provider=self._provider_name,
- request_id=session.request_id,
+ request_id=execution.request_id,
status_code=status,
exc_type=type(error).__name__,
- attempt=session.attempts_started,
- max_attempts=session.max_attempts,
+ attempt=execution.attempts_started,
+ max_attempts=execution.max_attempts,
)
else:
logger.warning(
"{} retry exhausted (attempts={})",
label,
- session.attempts_started,
+ execution.attempts_started,
)
trace_event(
stage="provider",
event="provider.retry.exhausted",
source="provider",
provider=self._provider_name,
- request_id=session.request_id,
+ request_id=execution.request_id,
status_code=status,
exc_type=type(error).__name__,
- attempts=session.attempts_started,
+ attempts=execution.attempts_started,
episode_exhausted=exhausted_episode,
)
- return can_retry
def _start_recovery_episode(
self,
*,
- leader: ProviderRetrySession | None,
+ leader: ProviderExecution | None,
ready_at: float,
last_error: Exception,
request_id: str | None = None,
@@ -627,7 +843,7 @@ class ProviderAdmissionController:
def _matching_probe(
self,
- session: ProviderRetrySession,
+ execution: ProviderExecution,
permit: _GatePermit,
) -> _RecoveryEpisode | None:
episode = self._episode
@@ -635,7 +851,7 @@ class ProviderAdmissionController:
not permit.probe
or episode is None
or episode.generation != permit.generation
- or episode.leader is not session
+ or episode.leader is not execution
or not episode.probe_active
):
return None
@@ -643,13 +859,13 @@ class ProviderAdmissionController:
async def _abandon_probe_permit(
self,
- session: ProviderRetrySession,
+ execution: ProviderExecution,
permit: _GatePermit,
) -> None:
if not permit.probe:
return
async with self._condition:
- episode = self._matching_probe(session, permit)
+ episode = self._matching_probe(execution, permit)
if episode is None:
return
episode.leader = None
@@ -659,7 +875,7 @@ class ProviderAdmissionController:
async def _abandon_waiting_leader(
self,
- session: ProviderRetrySession,
+ execution: ProviderExecution,
generation: int,
) -> None:
async with self._condition:
@@ -667,7 +883,7 @@ class ProviderAdmissionController:
if (
episode is None
or episode.generation != generation
- or episode.leader is not session
+ or episode.leader is not execution
or episode.probe_active
):
return
@@ -715,3 +931,12 @@ def _retry_after_seconds(error: Exception) -> float | None:
if not math.isfinite(seconds):
return None
return max(0.0, seconds)
+
+
+def _exception_status(error: BaseException) -> int | None:
+ status = getattr(error, "status_code", None)
+ if isinstance(status, int):
+ return status
+ response = getattr(error, "response", None)
+ response_status = getattr(response, "status_code", None)
+ return response_status if isinstance(response_status, int) else None
diff --git a/src/free_claude_code/providers/cloudflare/client.py b/src/free_claude_code/providers/cloudflare/client.py
index a484d6f0..dafac98d 100644
--- a/src/free_claude_code/providers/cloudflare/client.py
+++ b/src/free_claude_code/providers/cloudflare/client.py
@@ -11,7 +11,10 @@ from free_claude_code.application.errors import ApplicationUnavailableError
from free_claude_code.application.model_metadata import ProviderModelInfo
from free_claude_code.config.provider_catalog import CLOUDFLARE_AI_REST_ROOT
from free_claude_code.core.anthropic import ReasoningReplayMode
-from free_claude_code.providers.admission import ProviderAdmissionController
+from free_claude_code.providers.admission import (
+ ProviderAdmissionController,
+ ProviderOperationKind,
+)
from free_claude_code.providers.base import ProviderConfig
from free_claude_code.providers.http import maybe_await_aclose
from free_claude_code.providers.model_listing import (
@@ -111,7 +114,11 @@ class CloudflareProvider(OpenAIChatProvider):
raise
return response
- response = await self._admission.run_with_retry(request)
+ execution = self._admission.start_execution()
+ response = await execution.run_call(
+ request,
+ operation_kind=ProviderOperationKind.MODEL_DISCOVERY,
+ )
try:
try:
payload = response.json()
diff --git a/src/free_claude_code/providers/github_models/client.py b/src/free_claude_code/providers/github_models/client.py
index b8c76c8f..639d3fd9 100644
--- a/src/free_claude_code/providers/github_models/client.py
+++ b/src/free_claude_code/providers/github_models/client.py
@@ -7,7 +7,10 @@ import httpx
from free_claude_code.application.model_metadata import ProviderModelInfo
from free_claude_code.core.anthropic import ReasoningReplayMode
-from free_claude_code.providers.admission import ProviderAdmissionController
+from free_claude_code.providers.admission import (
+ ProviderAdmissionController,
+ ProviderOperationKind,
+)
from free_claude_code.providers.base import ProviderConfig
from free_claude_code.providers.http import maybe_await_aclose
from free_claude_code.providers.model_listing import (
@@ -75,7 +78,11 @@ class GitHubModelsProvider(OpenAIChatProvider):
raise
return response
- response = await self._admission.run_with_retry(request)
+ execution = self._admission.start_execution()
+ response = await execution.run_call(
+ request,
+ operation_kind=ProviderOperationKind.MODEL_DISCOVERY,
+ )
try:
try:
payload = response.json()
diff --git a/src/free_claude_code/providers/openai_chat/provider.py b/src/free_claude_code/providers/openai_chat/provider.py
index 834f9ee2..17f56005 100644
--- a/src/free_claude_code/providers/openai_chat/provider.py
+++ b/src/free_claude_code/providers/openai_chat/provider.py
@@ -37,7 +37,9 @@ from free_claude_code.core.trace import provider_chat_body_snapshot, trace_event
from free_claude_code.providers.admission import (
ProviderAdmissionController,
ProviderAttempt,
- ProviderRetrySession,
+ ProviderCorrectionAction,
+ ProviderExecution,
+ ProviderOperationKind,
)
from free_claude_code.providers.base import BaseProvider, ProviderConfig
from free_claude_code.providers.failure_policy import (
@@ -208,20 +210,26 @@ class OpenAIChatProvider(BaseProvider):
async def _list_models_payload(self) -> Any:
"""Fetch one OpenAI-compatible model-list payload with shared retries."""
- payload = await self._admission.run_with_retry(
- self._fetch_models_payload,
+ return await self._fetch_models_payload()
+
+ async def _fetch_models_payload(self) -> Any:
+ """Fetch the complete profile-selected model-list payload."""
+ listing = self._profile.model_listing
+ if listing.path is not None and listing.pagination is not None:
+ return await self._fetch_paginated_models_payload(listing.path)
+ execution = self._admission.start_execution()
+ return await execution.run_call(
+ self._fetch_models_payload_once,
+ operation_kind=ProviderOperationKind.MODEL_DISCOVERY,
provider_failure_override=self._provider_failure_override,
)
- return payload
- async def _fetch_models_payload(self) -> Any:
+ async def _fetch_models_payload_once(self) -> Any:
"""Fetch the profile-selected model-list endpoint once."""
listing = self._profile.model_listing
path = listing.path
if path is None:
return await self._client.models.list()
- if listing.pagination is not None:
- return await self._fetch_paginated_models_payload(path)
if listing.query_params:
return await self._client.get(
path,
@@ -231,7 +239,7 @@ class OpenAIChatProvider(BaseProvider):
return await self._client.get(path, cast_to=object)
async def _fetch_paginated_models_payload(self, path: str) -> Any:
- """Fetch one complete bounded model catalog within a retry attempt."""
+ """Fetch a bounded model catalog with one execution per physical page."""
listing = self._profile.model_listing
pagination = listing.pagination
if pagination is None:
@@ -243,10 +251,15 @@ class OpenAIChatProvider(BaseProvider):
while total_pages is None or page < pagination.first_page + total_pages:
params = dict(listing.query_params)
params[pagination.page_param] = str(page)
- payload = await self._client.get(
- path,
- cast_to=object,
- options={"params": params},
+ execution = self._admission.start_execution()
+ payload = await execution.run_call(
+ lambda params=params: self._client.get(
+ path,
+ cast_to=object,
+ options={"params": params},
+ ),
+ operation_kind=ProviderOperationKind.MODEL_DISCOVERY,
+ provider_failure_override=self._provider_failure_override,
)
total_pages = validate_model_list_page(
payload,
@@ -323,14 +336,15 @@ class OpenAIChatProvider(BaseProvider):
async def _create_stream(
self,
body: dict,
- retry_session: ProviderRetrySession,
+ execution: ProviderExecution,
+ operation_kind: ProviderOperationKind,
) -> tuple[Any, dict, ProviderAttempt]:
"""Create a streaming chat completion with bounded request fallbacks."""
body = self._apply_learned_output_cap(body)
used_retry_kinds: set[str] = set()
- while retry_session.can_attempt:
- attempt = await self._admission.open_attempt(retry_session)
+ while execution.can_attempt:
+ attempt = await execution.open_attempt(operation_kind)
stream: Any | None = None
retain_attempt = False
try:
@@ -346,15 +360,17 @@ class OpenAIChatProvider(BaseProvider):
raise
except Exception as error:
retry_body = self._next_create_retry_body(error, body, used_retry_kinds)
- if retry_body is not None and retry_session.can_attempt:
- await attempt.retry_immediately()
- body = retry_body
- continue
- should_retry = await attempt.retry(
+ if retry_body is not None:
+ correction = await attempt.correct(error)
+ if correction is ProviderCorrectionAction.RETRY:
+ body = retry_body
+ continue
+ raise
+ decision = await attempt.fail(
error,
provider_failure_override=self._provider_failure_override,
)
- if not should_retry:
+ if not decision.retry_allowed:
raise
finally:
if not retain_attempt:
@@ -363,11 +379,13 @@ class OpenAIChatProvider(BaseProvider):
stream,
active_error=sys.exception(),
provider_name=self._provider_name,
- request_id=retry_session.request_id,
+ request_id=execution.request_id,
)
await attempt.aclose()
- raise RuntimeError("provider retry session exhausted without a final error")
+ if execution.last_failure is not None:
+ raise execution.last_failure
+ raise RuntimeError("provider execution ended without a final error")
def _normalize_stream(self, stream: Any, _body: Mapping[str, Any]) -> Any:
"""Return the provider-specific stream view consumed by the base runner."""
@@ -483,13 +501,33 @@ class _OpenAIChatStreamRunner:
async def run(self) -> AsyncIterator[str]:
"""Convert the upstream OpenAI-chat stream into Anthropic SSE."""
+ execution = self._provider._admission.start_execution(
+ request_id=self._request_id
+ )
+ provider_stream = self._run_execution(execution)
+ try:
+ async for event in provider_stream:
+ yield event
+ except asyncio.CancelledError:
+ raise
+ except Exception as error:
+ execution.fail(error)
+ raise
+ else:
+ execution.succeed()
+ finally:
+ await maybe_await_aclose(provider_stream)
+ execution.abandon()
+
+ async def _run_execution(
+ self,
+ execution: ProviderExecution,
+ ) -> AsyncIterator[str]:
+ """Run one provider execution while retaining transport-owned state."""
tag = self._provider._provider_name
req_tag = f" request_id={self._request_id}" if self._request_id else ""
ledger = self._new_ledger()
recovery = RecoveryController()
- retry_session = self._provider._admission.new_retry_session(
- request_id=self._request_id
- )
def hold_event(event: str) -> Iterator[str]:
yield from recovery.push(event)
@@ -510,6 +548,7 @@ class _OpenAIChatStreamRunner:
source="provider",
provider=tag,
request_id=self._request_id,
+ execution_id=execution.execution_id,
gateway_model=self._response_model,
downstream_model=body.get("model"),
message_count=len(body.get("messages", [])),
@@ -541,13 +580,14 @@ class _OpenAIChatStreamRunner:
try:
stream, body, attempt = await self._provider._create_stream(
body,
- retry_session,
+ execution,
+ ProviderOperationKind.GENERATION,
)
stream_opened = True
tool_argument_aliases = self._provider._tool_argument_aliases(body)
async for chunk in stream:
if not attempt.accepted:
- await attempt.succeeded()
+ await attempt.accept()
chunk_usage = getattr(chunk, "usage", None)
if chunk_usage is not None:
usage_info = chunk_usage
@@ -720,8 +760,9 @@ class _OpenAIChatStreamRunner:
except asyncio.CancelledError, GeneratorExit:
raise
except Exception as error:
+ attempt_failure = None
if attempt is not None and not attempt.accepted:
- await attempt.retry(
+ attempt_failure = await attempt.fail(
error,
provider_failure_override=(
self._provider._provider_failure_override
@@ -734,8 +775,8 @@ class _OpenAIChatStreamRunner:
and all_emitted_tools_complete(ledger, self._request)
)
retryable = (
- attempt.failure_retryable
- if attempt is not None and attempt.failure_retryable is not None
+ attempt_failure.retryable
+ if attempt_failure is not None
else is_retryable_stream_error(error)
)
decision = recovery.advance_failure(
@@ -743,7 +784,7 @@ class _OpenAIChatStreamRunner:
stream_opened=stream_opened,
generated_output=generated_output,
complete_tool_salvageable=complete_tool_salvageable,
- attempts_remaining=retry_session.attempts_remaining,
+ attempts_remaining=execution.attempts_remaining,
)
if decision.action == RecoveryFailureAction.EARLY_RETRY:
trace_event(
@@ -752,8 +793,8 @@ class _OpenAIChatStreamRunner:
source="provider",
provider=tag,
request_id=self._request_id,
- attempts_started=retry_session.attempts_started,
- max_attempts=retry_session.max_attempts,
+ attempts_started=execution.attempts_started,
+ max_attempts=execution.max_attempts,
retryable=True,
)
ledger = self._new_ledger()
@@ -768,6 +809,17 @@ class _OpenAIChatStreamRunner:
continue
if decision.action == RecoveryFailureAction.MIDSTREAM_RECOVERY:
+ if stream is not None:
+ await close_provider_stream(
+ stream,
+ active_error=error,
+ provider_name=tag,
+ request_id=self._request_id,
+ )
+ stream = None
+ if attempt is not None:
+ await attempt.aclose()
+ attempt = None
try:
recovery_events = await self._recovery_events(
body=body,
@@ -775,7 +827,7 @@ class _OpenAIChatStreamRunner:
error=error,
tool_argument_alias_buffers=tool_argument_alias_buffers,
output_reasoning=output_reasoning,
- retry_session=retry_session,
+ execution=execution,
)
except Exception as recovery_error:
trace_event(
@@ -930,17 +982,19 @@ class _OpenAIChatStreamRunner:
body: dict[str, Any],
*,
include_reasoning: bool,
- retry_session: ProviderRetrySession,
+ execution: ProviderExecution,
+ operation_kind: ProviderOperationKind,
) -> _CollectedRecoveryOutput:
"""Collect one complete buffered continuation response."""
last_error: Exception | None = None
- while retry_session.can_attempt:
+ while execution.can_attempt:
stream: Any | None = None
attempt: ProviderAttempt | None = None
try:
stream, accepted_body, attempt = await self._provider._create_stream(
body,
- retry_session,
+ execution,
+ operation_kind,
)
text_parts: list[str] = []
thinking_parts: list[str] = []
@@ -948,7 +1002,7 @@ class _OpenAIChatStreamRunner:
terminal_seen = False
async for chunk in stream:
if not attempt.accepted:
- await attempt.succeeded()
+ await attempt.accept()
if not getattr(chunk, "choices", None):
continue
choice = chunk.choices[0]
@@ -993,15 +1047,14 @@ class _OpenAIChatStreamRunner:
last_error = error
retryable = is_retryable_stream_error(error)
if attempt is not None and not attempt.accepted:
- await attempt.retry(
+ failure = await attempt.fail(
error,
provider_failure_override=(
self._provider._provider_failure_override
),
)
- if attempt.failure_retryable is not None:
- retryable = attempt.failure_retryable
- if not retryable or not retry_session.can_attempt:
+ retryable = failure.retryable
+ if not retryable or not execution.can_attempt:
raise
trace_event(
stage="provider",
@@ -1009,8 +1062,8 @@ class _OpenAIChatStreamRunner:
source="provider",
provider=self._provider._provider_name,
recovery_kind="openai_text",
- attempts_started=retry_session.attempts_started,
- max_attempts=retry_session.max_attempts,
+ attempts_started=execution.attempts_started,
+ max_attempts=execution.max_attempts,
exc_type=type(error).__name__,
)
finally:
@@ -1030,7 +1083,7 @@ class _OpenAIChatStreamRunner:
error: Exception,
tool_argument_alias_buffers: dict[int, str],
output_reasoning: bool,
- retry_session: ProviderRetrySession,
+ execution: ProviderExecution,
) -> list[str] | None:
"""Build terminal recovery events when the interrupted stream permits it."""
if ledger.has_emitted_tool_block():
@@ -1039,7 +1092,7 @@ class _OpenAIChatStreamRunner:
body=body,
ledger=ledger,
tool_argument_alias_buffers=tool_argument_alias_buffers,
- retry_session=retry_session,
+ execution=execution,
)
if repair_events is None:
return None
@@ -1083,7 +1136,8 @@ class _OpenAIChatStreamRunner:
recovered = await self._collect_recovery_output(
recovery_body,
include_reasoning=output_reasoning,
- retry_session=retry_session,
+ execution=execution,
+ operation_kind=ProviderOperationKind.CONTINUATION,
)
text_suffix = continuation_suffix(partial_text, recovered.text)
thinking_suffix = continuation_suffix(partial_thinking, recovered.thinking)
@@ -1122,7 +1176,7 @@ class _OpenAIChatStreamRunner:
body: dict[str, Any],
ledger: AnthropicStreamLedger,
tool_argument_alias_buffers: dict[int, str],
- retry_session: ProviderRetrySession,
+ execution: ProviderExecution,
) -> list[str] | None:
schemas = tool_schemas_by_name(self._request)
events: list[str] = []
@@ -1151,12 +1205,13 @@ class _OpenAIChatStreamRunner:
)
accepted_suffix: str | None = None
repair_attempt = 0
- while retry_session.can_attempt:
+ while execution.can_attempt:
repair_attempt += 1
recovered = await self._collect_recovery_output(
recovery_body,
include_reasoning=False,
- retry_session=retry_session,
+ execution=execution,
+ operation_kind=ProviderOperationKind.TOOL_REPAIR,
)
repair = accept_tool_json_repair(
repair_prefix,
diff --git a/src/free_claude_code/providers/openai_codex/auth.py b/src/free_claude_code/providers/openai_codex/auth.py
index b8dad057..bf46148e 100644
--- a/src/free_claude_code/providers/openai_codex/auth.py
+++ b/src/free_claude_code/providers/openai_codex/auth.py
@@ -35,6 +35,8 @@ from .login import (
)
REFRESH_EARLY_SECONDS = 5 * 60
+_UNAUTHORIZED_REFRESH_TOTAL_ATTEMPTS = 2
+_UNAUTHORIZED_REFRESH_RETRY_DELAY_SECONDS = 1.0
class OpenAIReconnectRequired(RuntimeError):
@@ -329,7 +331,7 @@ class OpenAIAuthManager:
)
async def recover_unauthorized(self, rejected_token: str) -> OpenAIAccess:
- """Reload cross-process state, then force at most one token refresh."""
+ """Reload cross-process state, then recover one rejected access token."""
async with self._state_lock:
current = self._credentials
@@ -347,11 +349,27 @@ class OpenAIAuthManager:
return OpenAIAccess(
current.access_token, current.account_id, current.fedramp
)
- refreshed = await self._refresh_locked(current)
+ refreshed = await self._recover_refresh_locked(current)
return OpenAIAccess(
refreshed.access_token, refreshed.account_id, refreshed.fedramp
)
+ async def _recover_refresh_locked(self, current: _Credentials) -> _Credentials:
+ """Retry one transient reactive refresh without repeating a provider call."""
+ for attempt in range(_UNAUTHORIZED_REFRESH_TOTAL_ATTEMPTS):
+ try:
+ return await self._refresh_locked(current)
+ except asyncio.CancelledError:
+ raise
+ except httpx.HTTPError as error:
+ if (
+ not _is_transient_refresh_error(error)
+ or attempt + 1 == _UNAUTHORIZED_REFRESH_TOTAL_ATTEMPTS
+ ):
+ raise
+ await asyncio.sleep(_UNAUTHORIZED_REFRESH_RETRY_DELAY_SECONDS)
+ raise RuntimeError("OpenAI refresh recovery ended without an outcome")
+
async def close(self) -> None:
"""Cancel login resources and close the owned HTTP client."""
diff --git a/src/free_claude_code/providers/openai_codex/provider.py b/src/free_claude_code/providers/openai_codex/provider.py
index 277e4cab..5ab57cb7 100644
--- a/src/free_claude_code/providers/openai_codex/provider.py
+++ b/src/free_claude_code/providers/openai_codex/provider.py
@@ -33,6 +33,9 @@ from free_claude_code.core.trace import trace_event
from free_claude_code.providers.admission import (
ProviderAdmissionController,
ProviderAttempt,
+ ProviderCorrectionAction,
+ ProviderExecution,
+ ProviderOperationKind,
)
from free_claude_code.providers.base import BaseProvider, ProviderConfig
from free_claude_code.providers.failure_policy import (
@@ -40,7 +43,11 @@ from free_claude_code.providers.failure_policy import (
classify_provider_failure,
is_retryable_stream_error,
)
-from free_claude_code.providers.stream_recovery import RecoveryController
+from free_claude_code.providers.http import maybe_await_aclose
+from free_claude_code.providers.stream_recovery import (
+ RecoveryController,
+ RecoveryFailureAction,
+)
from .auth import OpenAIAccess, OpenAIAuthManager, OpenAIReconnectRequired
from .login import OPENAI_CODEX_ORIGINATOR
@@ -104,26 +111,67 @@ class OpenAICodexProvider(BaseProvider):
async def list_model_infos(self) -> frozenset[ProviderModelInfo]:
"""Discover models visible to the currently connected ChatGPT account."""
+ return _model_infos(await self._list_models_payload())
- async def fetch() -> Any:
- access = await self._auth.access()
- response = await self._client.get(
- "models",
- params={"client_version": FCC_VERSION},
- headers={**self._client_headers, **_auth_headers(access)},
- )
- if response.status_code == 401:
- access = await self._auth.recover_unauthorized(access.access_token)
+ async def _list_models_payload(self) -> Any:
+ """Fetch one Codex model catalog with each provider GET admitted once."""
+ execution = self._admission.start_execution()
+ authentication_recovered = False
+ while execution.can_attempt:
+ response: httpx.Response | None = None
+ attempt: ProviderAttempt | None = None
+ try:
+ access = await self._auth.access()
+ attempt = await execution.open_attempt(
+ ProviderOperationKind.MODEL_DISCOVERY
+ )
response = await self._client.get(
"models",
params={"client_version": FCC_VERSION},
headers={**self._client_headers, **_auth_headers(access)},
)
- response.raise_for_status()
- return response.json()
+ if response.status_code == 401 and not authentication_recovered:
+ error = await _response_status_error(response)
+ correction = await attempt.correct(error)
+ if correction is ProviderCorrectionAction.FINAL:
+ await response.aclose()
+ response = None
+ await attempt.aclose()
+ attempt = None
+ raise error
+ await response.aclose()
+ response = None
+ await attempt.aclose()
+ attempt = None
+ await self._auth.recover_unauthorized(access.access_token)
+ authentication_recovered = True
+ continue
+ if not response.is_success:
+ raise await _response_status_error(response)
+ payload = response.json()
+ await attempt.accept()
+ execution.succeed()
+ return payload
+ except asyncio.CancelledError:
+ execution.abandon()
+ raise
+ except Exception as error:
+ if attempt is not None and not attempt.accepted:
+ decision = await attempt.fail(error)
+ if decision.retry_allowed:
+ continue
+ execution.fail(error)
+ raise
+ finally:
+ if response is not None:
+ await response.aclose()
+ if attempt is not None:
+ await attempt.aclose()
- payload = await self._admission.run_with_retry(fetch)
- return _model_infos(payload)
+ if execution.last_failure is not None:
+ raise execution.last_failure
+ execution.abandon()
+ raise RuntimeError("OpenAI model discovery ended without an attempt outcome")
def stream_response(
self,
@@ -171,7 +219,40 @@ class OpenAICodexProvider(BaseProvider):
response_model: str,
tool_names: OpenAIToolNameCodec,
) -> AsyncIterator[str]:
- retry_session = self._admission.new_retry_session(request_id=request_id)
+ execution = self._admission.start_execution(request_id=request_id)
+ provider_stream = self._run_stream_execution(
+ body,
+ input_tokens=input_tokens,
+ request_id=request_id,
+ response_model=response_model,
+ tool_names=tool_names,
+ execution=execution,
+ )
+ try:
+ async for event in provider_stream:
+ yield event
+ except asyncio.CancelledError:
+ raise
+ except Exception as error:
+ execution.fail(error)
+ raise
+ else:
+ execution.succeed()
+ finally:
+ await maybe_await_aclose(provider_stream)
+ execution.abandon()
+
+ async def _run_stream_execution(
+ self,
+ body: dict[str, Any],
+ *,
+ input_tokens: int,
+ request_id: str | None,
+ response_model: str,
+ tool_names: OpenAIToolNameCodec,
+ execution: ProviderExecution,
+ ) -> AsyncIterator[str]:
+ """Run one Codex execution while retaining Responses transport ownership."""
recovery = RecoveryController()
message_id = f"msg_{uuid.uuid4()}"
session_id = str(uuid.uuid4())
@@ -182,13 +263,14 @@ class OpenAICodexProvider(BaseProvider):
source="provider",
provider="openai",
request_id=request_id,
+ execution_id=execution.execution_id,
gateway_model=response_model,
downstream_model=body.get("model"),
item_count=len(body.get("input", [])),
tool_count=len(body.get("tools", [])),
)
- while retry_session.can_attempt:
+ while execution.can_attempt:
stream = ResponsesProviderStream(
message_id=message_id,
model=response_model,
@@ -205,7 +287,7 @@ class OpenAICodexProvider(BaseProvider):
stream_opened = False
try:
access = await self._auth.access()
- attempt = await self._admission.open_attempt(retry_session)
+ attempt = await execution.open_attempt(ProviderOperationKind.GENERATION)
response = await self._client.send(
self._client.build_request(
"POST",
@@ -221,23 +303,20 @@ class OpenAICodexProvider(BaseProvider):
stream=True,
)
if response.status_code == 401 and not authentication_recovered:
- await _read_bounded_body(response)
+ error = await _response_status_error(response)
+ correction = await attempt.correct(error)
+ await response.aclose()
+ response = None
+ await attempt.aclose()
+ attempt = None
+ if correction is ProviderCorrectionAction.FINAL:
+ raise error
+ recovery.discard()
await self._auth.recover_unauthorized(access.access_token)
- await attempt.retry_immediately()
authentication_recovered = True
- recovery.discard()
continue
if not response.is_success:
- body_bytes, body_truncated = await _read_bounded_body(response)
- try:
- response.raise_for_status()
- except httpx.HTTPStatusError as exc:
- attach_upstream_error_body(
- exc,
- body_bytes,
- truncated=body_truncated,
- )
- raise
+ raise await _response_status_error(response)
content_type = response.headers.get("content-type")
if content_type and "text/event-stream" not in content_type.lower():
body_bytes, body_truncated = await _read_bounded_body(response)
@@ -254,7 +333,7 @@ class OpenAICodexProvider(BaseProvider):
async for event_type, payload in _iter_sse(response):
if not attempt.accepted:
- await attempt.succeeded()
+ await attempt.accept()
for event in stream.feed(event_type, payload):
for held in recovery.push(event):
yield held
@@ -276,16 +355,24 @@ class OpenAICodexProvider(BaseProvider):
raise
except Exception as raw_error:
error = _effective_error(raw_error)
+ attempt_failure = None
if attempt is not None and not attempt.accepted:
- await attempt.retry(error)
- retryable_override = (
- attempt.failure_retryable
- if attempt is not None and attempt.failure_retryable is not None
- else None
- )
+ attempt_failure = await attempt.fail(error)
+ if attempt_failure is not None and attempt_failure.retry_allowed:
+ recovery.discard()
+ trace_event(
+ stage="provider",
+ event="provider.recovery.early_retry",
+ source="provider",
+ provider="openai",
+ request_id=request_id,
+ attempts_started=execution.attempts_started,
+ max_attempts=execution.max_attempts,
+ )
+ continue
retryable = (
- retryable_override
- if retryable_override is not None
+ attempt_failure.retryable
+ if attempt_failure is not None
else is_retryable_stream_error(error)
)
decision = recovery.advance_failure(
@@ -293,13 +380,9 @@ class OpenAICodexProvider(BaseProvider):
stream_opened=stream_opened,
generated_output=recovery.committed,
complete_tool_salvageable=False,
- attempts_remaining=retry_session.attempts_remaining,
+ attempts_remaining=execution.attempts_remaining,
)
- if (
- not decision.committed
- and decision.retryable
- and retry_session.can_attempt
- ):
+ if decision.action is RecoveryFailureAction.EARLY_RETRY:
recovery.discard()
trace_event(
stage="provider",
@@ -307,8 +390,8 @@ class OpenAICodexProvider(BaseProvider):
source="provider",
provider="openai",
request_id=request_id,
- attempts_started=retry_session.attempts_started,
- max_attempts=retry_session.max_attempts,
+ attempts_started=execution.attempts_started,
+ max_attempts=execution.max_attempts,
)
continue
@@ -336,7 +419,9 @@ class OpenAICodexProvider(BaseProvider):
if attempt is not None:
await attempt.aclose()
- raise RuntimeError("OpenAI retry session ended without a terminal result.")
+ if execution.last_failure is not None:
+ raise execution.last_failure
+ raise RuntimeError("OpenAI execution ended without a terminal result.")
async def _iter_sse(
@@ -394,6 +479,17 @@ async def _read_bounded_body(
return bytes(body[:limit]), truncated
+async def _response_status_error(response: httpx.Response) -> httpx.HTTPStatusError:
+ """Return one bounded, diagnostic-preserving HTTP status failure."""
+ body, truncated = await _read_bounded_body(response)
+ try:
+ response.raise_for_status()
+ except httpx.HTTPStatusError as error:
+ attach_upstream_error_body(error, body, truncated=truncated)
+ return error
+ raise RuntimeError("response status is successful")
+
+
def _auth_headers(access: OpenAIAccess) -> dict[str, str]:
headers = {
"Authorization": f"Bearer {access.access_token}",
diff --git a/src/free_claude_code/providers/vertex/client.py b/src/free_claude_code/providers/vertex/client.py
index 5e0169b7..5eb207e5 100644
--- a/src/free_claude_code/providers/vertex/client.py
+++ b/src/free_claude_code/providers/vertex/client.py
@@ -7,7 +7,10 @@ import httpx
from free_claude_code.application.model_metadata import ProviderModelInfo
from free_claude_code.core.anthropic import ReasoningReplayMode
-from free_claude_code.providers.admission import ProviderAdmissionController
+from free_claude_code.providers.admission import (
+ ProviderAdmissionController,
+ ProviderOperationKind,
+)
from free_claude_code.providers.base import ProviderConfig
from free_claude_code.providers.google_openai import (
GoogleOpenAIProvider,
@@ -121,7 +124,11 @@ class VertexProvider(GoogleOpenAIProvider):
raise
return response
- response = await self._admission.run_with_retry(request)
+ execution = self._admission.start_execution()
+ response = await execution.run_call(
+ request,
+ operation_kind=ProviderOperationKind.MODEL_DISCOVERY,
+ )
try:
try:
return response.json()
diff --git a/tests/providers/test_featherless.py b/tests/providers/test_featherless.py
index b5e4762c..3cf21f6d 100644
--- a/tests/providers/test_featherless.py
+++ b/tests/providers/test_featherless.py
@@ -1,7 +1,7 @@
"""Tests for the Featherless AI OpenAI-chat provider profile."""
from typing import Any
-from unittest.mock import AsyncMock
+from unittest.mock import AsyncMock, patch
import httpx2
import pytest
@@ -261,7 +261,8 @@ async def test_catalog_fetches_all_pages_filters_strictly_and_deduplicates_overl
]
)
- model_infos = await featherless_provider.list_model_infos()
+ with patch("free_claude_code.providers.admission.trace_event") as trace:
+ model_infos = await featherless_provider.list_model_infos()
assert model_infos == frozenset(
{
@@ -283,6 +284,19 @@ async def test_catalog_fetches_all_pages_filters_strictly_and_deduplicates_overl
"per_page": "1000",
"page": str(index),
}
+ attempt_rows = [
+ call.kwargs
+ for call in trace.call_args_list
+ if call.kwargs.get("event", "").startswith("provider.attempt.")
+ ]
+ assert [row["event"] for row in attempt_rows] == [
+ "provider.attempt.started",
+ "provider.attempt.resolved",
+ "provider.attempt.started",
+ "provider.attempt.resolved",
+ ]
+ assert {row["attempt"] for row in attempt_rows} == {1}
+ assert len({row["execution_id"] for row in attempt_rows}) == 2
@pytest.mark.parametrize(
diff --git a/tests/providers/test_groq_reasoning.py b/tests/providers/test_groq_reasoning.py
index c0b8b52e..7f4c63c6 100644
--- a/tests/providers/test_groq_reasoning.py
+++ b/tests/providers/test_groq_reasoning.py
@@ -11,6 +11,7 @@ import pytest
from free_claude_code.config.provider_catalog import GROQ_DEFAULT_BASE
from free_claude_code.core.anthropic.stream_contracts import parse_sse_text
from free_claude_code.core.reasoning import ReasoningEffort, ReasoningPolicy
+from free_claude_code.providers.admission import ProviderOperationKind
from free_claude_code.providers.groq import GroqProvider
from free_claude_code.providers.groq.client import (
_parse_reasoning_vocabulary,
@@ -347,7 +348,8 @@ async def test_exact_issue_retries_with_default_and_learns_model() -> None:
with patch.object(provider._client.chat.completions, "create", create):
_stream, used_body, attempt = await provider._create_stream(
body,
- provider._admission.new_retry_session(),
+ provider._admission.start_execution(),
+ ProviderOperationKind.GENERATION,
)
await attempt.aclose()
@@ -365,7 +367,8 @@ async def test_exact_issue_retries_with_default_and_learns_model() -> None:
with patch.object(provider._client.chat.completions, "create", next_create):
_stream, next_used_body, next_attempt = await provider._create_stream(
next_body,
- provider._admission.new_retry_session(),
+ provider._admission.start_execution(),
+ ProviderOperationKind.GENERATION,
)
await next_attempt.aclose()
assert next_create.await_count == 1
@@ -434,7 +437,8 @@ async def test_unknown_vocabulary_retries_without_effort_and_negative_caches() -
with patch.object(provider._client.chat.completions, "create", create):
_stream, used_body, attempt = await provider._create_stream(
body,
- provider._admission.new_retry_session(),
+ provider._admission.start_execution(),
+ ProviderOperationKind.GENERATION,
)
await attempt.aclose()
@@ -488,7 +492,8 @@ async def test_concurrent_first_requests_can_learn_without_state_corruption() ->
async def execute(body: dict):
_stream, used_body, attempt = await provider._create_stream(
body,
- provider._admission.new_retry_session(),
+ provider._admission.start_execution(),
+ ProviderOperationKind.GENERATION,
)
await attempt.aclose()
return used_body
@@ -523,7 +528,8 @@ async def test_stale_cache_self_heals_without_guessing_original_effort() -> None
with patch.object(provider._client.chat.completions, "create", create):
_stream, corrected_body, attempt = await provider._create_stream(
cached_body,
- provider._admission.new_retry_session(),
+ provider._admission.start_execution(),
+ ProviderOperationKind.GENERATION,
)
await attempt.aclose()
@@ -548,7 +554,11 @@ async def test_unrelated_400_propagates_without_cache_poisoning() -> None:
patch.object(provider._client.chat.completions, "create", create),
pytest.raises(_BadRequest, match="wizard"),
):
- await provider._create_stream(body, provider._admission.new_retry_session())
+ await provider._create_stream(
+ body,
+ provider._admission.start_execution(),
+ ProviderOperationKind.GENERATION,
+ )
assert create.await_count == 1
assert provider._model_reasoning_vocabularies == {}
@@ -574,7 +584,11 @@ async def test_nested_unrelated_allow_list_cannot_retry_or_poison_cache() -> Non
patch.object(provider._client.chat.completions, "create", create),
pytest.raises(_BadRequest),
):
- await provider._create_stream(body, provider._admission.new_retry_session())
+ await provider._create_stream(
+ body,
+ provider._admission.start_execution(),
+ ProviderOperationKind.GENERATION,
+ )
assert create.await_count == 1
assert create.await_args_list[0].kwargs["reasoning_effort"] == "none"
@@ -594,7 +608,11 @@ async def test_advertised_current_value_does_not_retry_or_cache() -> None:
patch.object(provider._client.chat.completions, "create", create),
pytest.raises(_BadRequest),
):
- await provider._create_stream(body, provider._admission.new_retry_session())
+ await provider._create_stream(
+ body,
+ provider._admission.start_execution(),
+ ProviderOperationKind.GENERATION,
+ )
assert create.await_count == 1
assert provider._model_reasoning_vocabularies == {}
@@ -611,7 +629,11 @@ async def test_last_attempt_learns_but_does_not_exceed_budget() -> None:
patch.object(provider._client.chat.completions, "create", create),
pytest.raises(_BadRequest),
):
- await provider._create_stream(body, provider._admission.new_retry_session())
+ await provider._create_stream(
+ body,
+ provider._admission.start_execution(),
+ ProviderOperationKind.GENERATION,
+ )
assert create.await_count == 1
assert provider._model_reasoning_vocabularies[_MODEL] == frozenset(
@@ -632,14 +654,18 @@ async def test_output_cap_and_reasoning_corrections_share_one_session() -> None:
)
cap_error = _BadRequest("max_completion_tokens must be less than or equal to 40960")
create = AsyncMock(side_effect=[cap_error, _vocabulary_error(), object()])
- session = provider._admission.new_retry_session()
+ execution = provider._admission.start_execution()
with patch.object(provider._client.chat.completions, "create", create):
- _stream, used_body, attempt = await provider._create_stream(body, session)
+ _stream, used_body, attempt = await provider._create_stream(
+ body,
+ execution,
+ ProviderOperationKind.GENERATION,
+ )
await attempt.aclose()
assert create.await_count == 3
- assert session.attempts_started == 3
+ assert execution.attempts_started == 3
assert create.await_args_list[1].kwargs["max_completion_tokens"] == 40_960
assert create.await_args_list[1].kwargs["reasoning_effort"] == "high"
assert create.await_args_list[2].kwargs["max_completion_tokens"] == 40_960
@@ -661,7 +687,11 @@ async def test_corrected_request_cannot_enter_vocabulary_retry_loop() -> None:
patch.object(provider._client.chat.completions, "create", create),
pytest.raises(_BadRequest),
):
- await provider._create_stream(body, provider._admission.new_retry_session())
+ await provider._create_stream(
+ body,
+ provider._admission.start_execution(),
+ ProviderOperationKind.GENERATION,
+ )
assert create.await_count == 2
diff --git a/tests/providers/test_nvidia_nim_degraded_retry.py b/tests/providers/test_nvidia_nim_degraded_retry.py
index 9284bf79..df64b781 100644
--- a/tests/providers/test_nvidia_nim_degraded_retry.py
+++ b/tests/providers/test_nvidia_nim_degraded_retry.py
@@ -11,6 +11,7 @@ from free_claude_code.core.failures import ExecutionFailure, FailureKind
from free_claude_code.providers.admission import (
UPSTREAM_TRANSIENT_TOTAL_ATTEMPTS,
ProviderAdmissionController,
+ ProviderOperationKind,
)
from free_claude_code.providers.base import ProviderConfig
from free_claude_code.providers.failure_policy import (
@@ -379,8 +380,9 @@ async def test_admission_override_preserves_raw_exception_after_exhaustion() ->
with (
pytest.raises(openai.BadRequestError) as exc_info,
):
- await admission.run_with_retry(
+ await admission.start_execution().run_call(
fail,
+ operation_kind=ProviderOperationKind.GENERATION,
provider_failure_override=override,
)
diff --git a/tests/providers/test_openai_chat_output_cap.py b/tests/providers/test_openai_chat_output_cap.py
index f2a06b94..16032f3e 100644
--- a/tests/providers/test_openai_chat_output_cap.py
+++ b/tests/providers/test_openai_chat_output_cap.py
@@ -10,6 +10,7 @@ from unittest.mock import AsyncMock, patch
import pytest
from free_claude_code.config.provider_catalog import GROQ_DEFAULT_BASE
+from free_claude_code.providers.admission import ProviderOperationKind
from free_claude_code.providers.groq import GroqProvider
from free_claude_code.providers.openai_chat.output_cap import (
clamp_output_tokens,
@@ -266,7 +267,8 @@ async def test_create_stream_clamps_and_learns_on_cap_rejection(groq_provider):
with patch.object(groq_provider._client.chat.completions, "create", create):
_stream, used_body, attempt = await groq_provider._create_stream(
body,
- groq_provider._admission.new_retry_session(),
+ groq_provider._admission.start_execution(),
+ ProviderOperationKind.GENERATION,
)
await attempt.aclose()
@@ -292,7 +294,8 @@ async def test_learned_cap_clamps_next_request_without_a_400(groq_provider):
with patch.object(groq_provider._client.chat.completions, "create", create):
_stream, used_body, attempt = await groq_provider._create_stream(
body,
- groq_provider._admission.new_retry_session(),
+ groq_provider._admission.start_execution(),
+ ProviderOperationKind.GENERATION,
)
await attempt.aclose()
@@ -318,7 +321,8 @@ async def test_unrelated_400_is_not_clamped_and_propagates(groq_provider):
):
await groq_provider._create_stream(
body,
- groq_provider._admission.new_retry_session(),
+ groq_provider._admission.start_execution(),
+ ProviderOperationKind.GENERATION,
)
assert create.call_count == 1
@@ -351,7 +355,8 @@ async def test_mixed_field_400_does_not_retry_or_poison_learned_cap(groq_provide
):
await groq_provider._create_stream(
body,
- groq_provider._admission.new_retry_session(),
+ groq_provider._admission.start_execution(),
+ ProviderOperationKind.GENERATION,
)
assert create.call_count == 1
diff --git a/tests/providers/test_openai_chat_usage.py b/tests/providers/test_openai_chat_usage.py
index d8c4bd3e..d87b5e63 100644
--- a/tests/providers/test_openai_chat_usage.py
+++ b/tests/providers/test_openai_chat_usage.py
@@ -12,6 +12,7 @@ from free_claude_code.core.anthropic import ReasoningReplayMode
from free_claude_code.core.anthropic.models import MessagesRequest
from free_claude_code.core.anthropic.stream_contracts import parse_sse_text
from free_claude_code.core.reasoning import DEFAULT_REASONING_POLICY, ReasoningPolicy
+from free_claude_code.providers.admission import ProviderOperationKind
from free_claude_code.providers.openai_chat import (
OpenAIChatProfile,
OpenAIChatProvider,
@@ -252,7 +253,8 @@ async def test_openai_chat_stream_retries_without_usage_when_option_is_rejected(
with patch.object(provider._client.chat.completions, "create", create):
_stream_obj, used_body, attempt = await provider._create_stream(
body,
- provider._admission.new_retry_session(),
+ provider._admission.start_execution(),
+ ProviderOperationKind.GENERATION,
)
await attempt.aclose()
diff --git a/tests/providers/test_openai_codex_auth.py b/tests/providers/test_openai_codex_auth.py
index 8146e9a7..63d26a48 100644
--- a/tests/providers/test_openai_codex_auth.py
+++ b/tests/providers/test_openai_codex_auth.py
@@ -13,6 +13,7 @@ import pytest
from free_claude_code.application.connected_accounts import (
ConnectedAccountLoginMode,
)
+from free_claude_code.providers.openai_codex import auth as openai_auth
from free_claude_code.providers.openai_codex import login as openai_login
from free_claude_code.providers.openai_codex.auth import (
OpenAIAuthManager,
@@ -206,6 +207,47 @@ async def test_proactive_transient_refresh_failure_preserves_unauthorized_recove
await client.aclose()
+async def test_unauthorized_recovery_retries_one_transient_refresh(
+ monkeypatch: pytest.MonkeyPatch,
+ tmp_path: Path,
+) -> None:
+ credential_path = tmp_path / "openai.json"
+ credential_path.write_text(
+ json.dumps(_credential_document(expires_at=4_000_000_000)),
+ encoding="utf-8",
+ )
+ current_access = _jwt({"exp": 4_000_000_000})
+ refreshed_access = _jwt({"exp": 4_100_000_000})
+ requests: list[httpx.Request] = []
+
+ def handler(request: httpx.Request) -> httpx.Response:
+ requests.append(request)
+ if len(requests) == 1:
+ return httpx.Response(503, request=request)
+ return httpx.Response(
+ 200,
+ json={"access_token": refreshed_access},
+ request=request,
+ )
+
+ monkeypatch.setattr(openai_auth, "_UNAUTHORIZED_REFRESH_RETRY_DELAY_SECONDS", 0)
+ client = httpx.AsyncClient(transport=httpx.MockTransport(handler))
+ manager = OpenAIAuthManager(
+ credential_path=credential_path,
+ lock_path=tmp_path / "openai.lock",
+ client=client,
+ )
+
+ access = await manager.recover_unauthorized(current_access)
+
+ assert access.access_token == refreshed_access
+ assert len(requests) == 2
+ assert all(request.url.path == "/oauth/token" for request in requests)
+ await manager.close()
+ await client.aclose()
+
+
@pytest.mark.asyncio
async def test_disconnect_revokes_refresh_token_and_always_deletes_credentials(
tmp_path: Path,
diff --git a/tests/providers/test_openai_codex_provider.py b/tests/providers/test_openai_codex_provider.py
index 4e65b90c..c4757721 100644
--- a/tests/providers/test_openai_codex_provider.py
+++ b/tests/providers/test_openai_codex_provider.py
@@ -1,6 +1,7 @@
import json
from collections.abc import AsyncIterator
from typing import Any
+from unittest.mock import patch
import httpx
import pytest
@@ -631,20 +632,75 @@ async def test_unauthorized_response_forces_one_auth_refresh() -> None:
@pytest.mark.asyncio
-async def test_transient_auth_refresh_failure_uses_coordinated_provider_retry() -> None:
- class TransientRecoveryAuth(_FakeAuth):
+async def test_model_discovery_re_request_is_a_second_admitted_attempt() -> None:
+ authorizations: list[str] = []
+
+ def handler(request: httpx.Request) -> httpx.Response:
+ authorizations.append(request.headers["authorization"])
+ if len(authorizations) == 1:
+ return httpx.Response(401, request=request)
+ return httpx.Response(
+ 200,
+ json={
+ "models": [
+ {
+ "slug": "gpt-visible",
+ "visibility": "list",
+ "supported_in_api": False,
+ "supported_reasoning_levels": [{"effort": "high"}],
+ }
+ ]
+ },
+ request=request,
+ )
+
+ client = httpx.AsyncClient(
+ base_url="https://chatgpt.com/backend-api/codex/",
+ transport=httpx.MockTransport(handler),
+ )
+ auth = _FakeAuth()
+ provider = OpenAICodexProvider(
+ _config(), auth=auth, admission=_admission(), client=client
+ )
+
+ with patch("free_claude_code.providers.admission.trace_event") as trace:
+ infos = await provider.list_model_infos()
+
+ assert {info.model_id for info in infos} == {"gpt-visible"}
+ assert authorizations == ["Bearer access_1", "Bearer access_2"]
+ assert auth.recovery_calls == 1
+ attempt_rows = [
+ call.kwargs
+ for call in trace.call_args_list
+ if call.kwargs.get("event", "").startswith("provider.attempt.")
+ ]
+ assert [row["event"] for row in attempt_rows] == [
+ "provider.attempt.started",
+ "provider.attempt.resolved",
+ "provider.attempt.started",
+ "provider.attempt.resolved",
+ ]
+ assert [row["attempt"] for row in attempt_rows] == [1, 1, 2, 2]
+ assert [row.get("outcome") for row in attempt_rows] == [
+ None,
+ "corrected",
+ None,
+ "accepted",
+ ]
+ assert len({row["execution_id"] for row in attempt_rows}) == 1
+ await client.aclose()
+
+
+async def test_auth_refresh_failure_does_not_repeat_rejected_provider_call() -> None:
+ class FailingRecoveryAuth(_FakeAuth):
async def recover_unauthorized(self, rejected_token: str) -> OpenAIAccess:
assert rejected_token == self.current_token
self.recovery_calls += 1
- if self.recovery_calls == 1:
- raise httpx.ReadError(
- "refresh interrupted",
- request=httpx.Request(
- "POST", "https://auth.openai.com/oauth/token"
- ),
- )
- self.current_token = "access_2"
- return OpenAIAccess(self.current_token, "account_1", False)
+ raise httpx.ReadError(
+ "refresh interrupted",
+ request=httpx.Request("POST", "https://auth.openai.com/oauth/token"),
+ )
authorizations: list[str] = []
@@ -664,26 +720,23 @@ async def test_transient_auth_refresh_failure_uses_coordinated_provider_retry()
base_url="https://chatgpt.com/backend-api/codex/",
transport=httpx.MockTransport(handler),
)
- auth = TransientRecoveryAuth()
+ auth = FailingRecoveryAuth()
provider = OpenAICodexProvider(
_config(), auth=auth, admission=_admission(), client=client
)
- body = await _collect(
- provider.stream_response(
- _request(),
- request_id="req_auth_transient",
- response_model="claude-opus-4",
+ with pytest.raises(ExecutionFailure, match="refresh interrupted") as exc_info:
+ await _collect(
+ provider.stream_response(
+ _request(),
+ request_id="req_auth_transient",
+ response_model="claude-opus-4",
+ )
)
- )
- assert authorizations == [
- "Bearer access_1",
- "Bearer access_1",
- "Bearer access_2",
- ]
- assert auth.recovery_calls == 2
- assert text_content(parse_sse_text(body)) == "recovered"
+ assert isinstance(exc_info.value.__cause__, httpx.ReadError)
+ assert authorizations == ["Bearer access_1"]
+ assert auth.recovery_calls == 1
await client.aclose()
@@ -723,6 +776,47 @@ async def test_second_unauthorized_response_is_terminal_without_refresh_loop() -
await client.aclose()
+async def test_final_attempt_unauthorized_preserves_the_provider_401() -> None:
+ attempts = 0
+
+ def handler(request: httpx.Request) -> httpx.Response:
+ nonlocal attempts
+ attempts += 1
+ status = 503 if attempts < 5 else 401
+ return httpx.Response(
+ status,
+ json={"error": {"message": f"attempt {attempts}"}},
+ request=request,
+ )
+
+ client = httpx.AsyncClient(
+ base_url="https://chatgpt.com/backend-api/codex/",
+ transport=httpx.MockTransport(handler),
+ )
+ auth = _FakeAuth()
+ provider = OpenAICodexProvider(
+ _config(), auth=auth, admission=_admission(), client=client
+ )
+
+ with pytest.raises(ExecutionFailure) as exc_info:
+ await _collect(
+ provider.stream_response(
+ _request(),
+ request_id="req_final_401",
+ response_model="claude-opus-4",
+ )
+ )
+
+ assert attempts == 5
+ assert auth.recovery_calls == 0
+ assert exc_info.value.status_code == 401
+ assert "Request ID: req_final_401" in str(exc_info.value)
+ assert isinstance(exc_info.value.__cause__, httpx.HTTPStatusError)
+ assert exc_info.value.__cause__.response.status_code == 401
+ await client.aclose()
+
+
@pytest.mark.asyncio
async def test_stream_failure_redacts_credentials_from_customer_diagnostic() -> None:
def handler(request: httpx.Request) -> httpx.Response:
diff --git a/tests/providers/test_provider_admission.py b/tests/providers/test_provider_admission.py
index 7c24e63f..cf975b00 100644
--- a/tests/providers/test_provider_admission.py
+++ b/tests/providers/test_provider_admission.py
@@ -14,7 +14,11 @@ from free_claude_code.core.failures import ExecutionFailure, FailureKind
from free_claude_code.providers.admission import (
UPSTREAM_TRANSIENT_TOTAL_ATTEMPTS,
ProviderAdmissionController,
- ProviderRetrySession,
+ ProviderAttempt,
+ ProviderCorrectionAction,
+ ProviderExecution,
+ ProviderExecutionState,
+ ProviderOperationKind,
_retry_after_seconds,
)
from free_claude_code.providers.failure_policy import ProviderRecoveryExhausted
@@ -58,6 +62,10 @@ def _status_error(
)
+async def _open(execution: ProviderExecution) -> ProviderAttempt:
+ return await execution.open_attempt(ProviderOperationKind.GENERATION)
+
+
@pytest.mark.parametrize(
("factory", "message"),
[
@@ -98,33 +106,120 @@ def test_admission_rejects_invalid_configuration(
factory()
-def test_retry_session_exposes_one_bounded_execution_budget() -> None:
- session = ProviderRetrySession(max_attempts=2)
-
- assert session.max_attempts == 2
- assert session.attempts_started == 0
- assert session.attempts_remaining == 2
- assert session.can_attempt
- assert session._claim_attempt() == 1
- assert session._claim_attempt() == 2
- assert not session.can_attempt
- assert session.attempts_remaining == 0
+async def test_execution_exposes_one_bounded_attempt_budget() -> None:
+ execution = _controller(max_attempts=2).start_execution()
+
+ assert execution.max_attempts == 2
+ assert execution.attempts_started == 0
+ assert execution.attempts_remaining == 2
+ assert execution.can_attempt
+ first = await execution.open_attempt(ProviderOperationKind.GENERATION)
+ await first.aclose()
+ second = await execution.open_attempt(ProviderOperationKind.GENERATION)
+ await second.aclose()
+ assert not execution.can_attempt
+ assert execution.attempts_remaining == 0
with pytest.raises(RuntimeError, match="exhausted"):
- session._claim_attempt()
+ await execution.open_attempt(ProviderOperationKind.GENERATION)
+
+
+async def test_execution_allows_only_one_active_physical_attempt() -> None:
+ execution = _controller().start_execution()
+ attempt = await _open(execution)
+
+ with pytest.raises(RuntimeError, match="active attempt"):
+ await _open(execution)
+
+ await attempt.aclose()
+
+
+async def test_attempt_outcome_and_close_are_idempotent() -> None:
+ controller = _controller(max_concurrency=1)
+ execution = controller.start_execution()
+ error = _status_error(400)
+
+ with patch("free_claude_code.providers.admission.trace_event") as trace:
+ attempt = await _open(execution)
+ first = await attempt.fail(error)
+ repeated = await attempt.fail(_status_error(503))
+ await attempt.accept()
+ assert await attempt.correct(error) is ProviderCorrectionAction.FINAL
+ await attempt.aclose()
+ await attempt.aclose()
+
+ assert not first.retryable
+ assert not first.retry_allowed
+ assert not repeated.retryable
+ assert not repeated.retry_allowed
+ resolved = [
+ call.kwargs
+ for call in trace.call_args_list
+ if call.kwargs.get("event") == "provider.attempt.resolved"
+ ]
+ assert len(resolved) == 1
+ assert resolved[0]["outcome"] == "rejected"
+
+ replacement = await asyncio.wait_for(
+ _open(controller.start_execution()),
+ timeout=1,
+ )
+ await replacement.accept()
+ await replacement.aclose()
+
+
+async def test_deterministic_correction_is_typed_and_final_at_exhaustion() -> None:
+ execution = _controller(max_attempts=2).start_execution()
+ error = _status_error(400)
+
+ first = await _open(execution)
+ assert await first.correct(error) is ProviderCorrectionAction.RETRY
+ await first.aclose()
+ assert execution.state is ProviderExecutionState.ACTIVE
+ assert execution.attempts_remaining == 1
+
+ final = await _open(execution)
+ assert await final.correct(error) is ProviderCorrectionAction.FINAL
+ await final.aclose()
+ assert execution.state is ProviderExecutionState.FAILED
+ assert execution.last_failure is error
+ assert execution.attempts_remaining == 0
+
+
+async def test_successful_run_call_is_terminal() -> None:
+ execution = _controller().start_execution()
+
+ assert (
+ await execution.run_call(
+ lambda: asyncio.sleep(0, result="ok"),
+ operation_kind=ProviderOperationKind.MODEL_DISCOVERY,
+ )
+ == "ok"
+ )
+ assert execution.state is ProviderExecutionState.SUCCEEDED
+ assert execution.attempts_remaining == 0
+ with pytest.raises(RuntimeError, match="terminal"):
+ await _open(execution)
@pytest.mark.asyncio
async def test_proactive_rate_limit_is_a_strict_rolling_window() -> None:
controller = _controller(rate_limit=1, rate_window=0.04)
- first = await controller.open_attempt(controller.new_retry_session())
- await first.succeeded()
+ first_execution = controller.start_execution()
+ first = await first_execution.open_attempt(ProviderOperationKind.GENERATION)
+ await first.accept()
await first.aclose()
started = time.monotonic()
- second = await controller.open_attempt(controller.new_retry_session())
+ second_execution = controller.start_execution()
+ second = await second_execution.open_attempt(ProviderOperationKind.GENERATION)
elapsed = time.monotonic() - started
- await second.succeeded()
+ await second.accept()
await second.aclose()
assert elapsed >= 0.03
@@ -133,22 +228,26 @@ async def test_proactive_rate_limit_is_a_strict_rolling_window() -> None:
@pytest.mark.asyncio
async def test_concurrency_bulkhead_limits_active_attempts() -> None:
controller = _controller(max_concurrency=2)
- first = await controller.open_attempt(controller.new_retry_session())
- second = await controller.open_attempt(controller.new_retry_session())
+ first = await controller.start_execution().open_attempt(
+ ProviderOperationKind.GENERATION
+ )
+ second = await controller.start_execution().open_attempt(
+ ProviderOperationKind.GENERATION
+ )
third_task = asyncio.create_task(
- controller.open_attempt(controller.new_retry_session())
+ controller.start_execution().open_attempt(ProviderOperationKind.GENERATION)
)
await asyncio.sleep(0)
assert not third_task.done()
- await first.succeeded()
+ await first.accept()
await first.aclose()
third = await asyncio.wait_for(third_task, timeout=1)
- await second.succeeded()
+ await second.accept()
await second.aclose()
- await third.succeeded()
+ await third.accept()
await third.aclose()
@@ -161,7 +260,12 @@ async def test_attempt_cancellation_releases_concurrency() -> None:
entered.set()
await asyncio.Event().wait()
- task = asyncio.create_task(controller.run_with_retry(never_finishes))
+ task = asyncio.create_task(
+ controller.start_execution().run_call(
+ never_finishes,
+ operation_kind=ProviderOperationKind.GENERATION,
+ )
+ )
await entered.wait()
task.cancel()
with pytest.raises(asyncio.CancelledError):
@@ -169,7 +273,10 @@ async def test_attempt_cancellation_releases_concurrency() -> None:
assert (
await asyncio.wait_for(
- controller.run_with_retry(lambda: asyncio.sleep(0, result="ok")),
+ controller.start_execution().run_call(
+ lambda: asyncio.sleep(0, result="ok"),
+ operation_kind=ProviderOperationKind.GENERATION,
+ ),
timeout=1,
)
== "ok"
@@ -177,7 +284,7 @@ async def test_attempt_cancellation_releases_concurrency() -> None:
@pytest.mark.asyncio
-async def test_run_with_retry_uses_one_five_attempt_budget() -> None:
+async def test_run_call_uses_one_five_attempt_budget() -> None:
controller = _controller()
attempts = 0
error = _status_error(503)
@@ -188,14 +295,17 @@ async def test_run_with_retry_uses_one_five_attempt_budget() -> None:
raise error
with pytest.raises(httpx.HTTPStatusError) as exc_info:
- await controller.run_with_retry(fail)
+ await controller.start_execution().run_call(
+ fail,
+ operation_kind=ProviderOperationKind.GENERATION,
+ )
assert attempts == UPSTREAM_TRANSIENT_TOTAL_ATTEMPTS
assert exc_info.value is error
@pytest.mark.asyncio
-async def test_run_with_retry_succeeds_without_multiplying_attempts() -> None:
+async def test_run_call_succeeds_without_multiplying_attempts() -> None:
controller = _controller(max_concurrency=1)
attempts = 0
@@ -207,7 +317,13 @@ async def test_run_with_retry_succeeds_without_multiplying_attempts() -> None:
return "recovered"
assert (
- await asyncio.wait_for(controller.run_with_retry(recover), timeout=1)
+ await asyncio.wait_for(
+ controller.start_execution().run_call(
+ recover,
+ operation_kind=ProviderOperationKind.GENERATION,
+ ),
+ timeout=1,
+ )
== "recovered"
)
assert attempts == 3
@@ -226,7 +342,10 @@ async def test_recovery_traces_keep_the_logical_request_id() -> None:
return "recovered"
with patch("free_claude_code.providers.admission.trace_event") as trace:
- result = await controller.run_with_retry(recover, request_id="req_trace")
+ result = await controller.start_execution(request_id="req_trace").run_call(
+ recover,
+ operation_kind=ProviderOperationKind.GENERATION,
+ )
assert result == "recovered"
recovery_rows = [
@@ -245,6 +364,52 @@ async def test_recovery_traces_keep_the_logical_request_id() -> None:
assert recovery_rows[-1]["outcome"] == "success"
+async def test_each_physical_call_has_one_correlated_trace_pair() -> None:
+ controller = _controller(max_attempts=2)
+ calls = 0
+
+ async def recover() -> str:
+ nonlocal calls
+ calls += 1
+ if calls == 1:
+ raise _status_error(503)
+ return "ok"
+
+ with patch("free_claude_code.providers.admission.trace_event") as trace:
+ assert (
+ await controller.start_execution(request_id="req_attempts").run_call(
+ recover,
+ operation_kind=ProviderOperationKind.MODEL_DISCOVERY,
+ )
+ == "ok"
+ )
+
+ attempt_rows = [
+ call.kwargs
+ for call in trace.call_args_list
+ if call.kwargs.get("event", "").startswith("provider.attempt.")
+ ]
+ assert [row["event"] for row in attempt_rows] == [
+ "provider.attempt.started",
+ "provider.attempt.resolved",
+ "provider.attempt.started",
+ "provider.attempt.resolved",
+ ]
+ assert [row["attempt"] for row in attempt_rows] == [1, 1, 2, 2]
+ assert [row.get("outcome") for row in attempt_rows] == [
+ None,
+ "retryable_failure",
+ None,
+ "accepted",
+ ]
+ assert {row["request_id"] for row in attempt_rows} == {"req_attempts"}
+ assert {row["operation_kind"] for row in attempt_rows} == {
+ ProviderOperationKind.MODEL_DISCOVERY.value
+ }
+ assert len({row["execution_id"] for row in attempt_rows}) == 1
+
+
@pytest.mark.asyncio
async def test_direct_exhaustion_trace_keeps_the_logical_request_id() -> None:
controller = _controller(max_attempts=1)
@@ -257,10 +422,15 @@ async def test_direct_exhaustion_trace_keeps_the_logical_request_id() -> None:
patch("free_claude_code.providers.admission.trace_event") as trace,
pytest.raises(httpx.HTTPStatusError),
):
- await controller.run_with_retry(fail, request_id="req_terminal")
+ await controller.start_execution(request_id="req_terminal").run_call(
+ fail,
+ operation_kind=ProviderOperationKind.GENERATION,
+ )
rows = [call.kwargs for call in trace.call_args_list]
assert {row["event"] for row in rows} == {
+ "provider.attempt.started",
+ "provider.attempt.resolved",
"provider.recovery.opened",
"provider.retry.exhausted",
}
@@ -278,7 +448,10 @@ async def test_non_retryable_error_is_attempted_once() -> None:
raise _status_error(400)
with pytest.raises(httpx.HTTPStatusError):
- await controller.run_with_retry(reject)
+ await controller.start_execution().run_call(
+ reject,
+ operation_kind=ProviderOperationKind.GENERATION,
+ )
assert attempts == 1
@@ -286,17 +459,18 @@ async def test_non_retryable_error_is_attempted_once() -> None:
@pytest.mark.asyncio
async def test_pre_response_protocol_failure_opens_coordinated_recovery() -> None:
controller = _controller()
- session = controller.new_retry_session()
- attempt = await controller.open_attempt(session)
+ execution = controller.start_execution()
+ attempt = await execution.open_attempt(ProviderOperationKind.GENERATION)
- assert await attempt.retry(
+ decision = await attempt.fail(
TruncatedProviderStreamError("stream ended before its first chunk")
)
- assert attempt.failure_retryable is True
+ assert decision.retryable
+ assert decision.retry_allowed
await attempt.aclose()
- probe = await controller.open_attempt(session)
- await probe.succeeded()
+ probe = await execution.open_attempt(ProviderOperationKind.GENERATION)
+ await probe.accept()
await probe.aclose()
@@ -322,8 +496,9 @@ async def test_provider_override_can_classify_retryable_semantics() -> None:
)
assert (
- await controller.run_with_retry(
+ await controller.start_execution().run_call(
degraded,
+ operation_kind=ProviderOperationKind.GENERATION,
provider_failure_override=classify,
)
== "healthy"
@@ -331,17 +506,44 @@ async def test_provider_override_can_classify_retryable_semantics() -> None:
assert attempts == 2
+async def test_discovery_and_generation_share_provider_recovery_health() -> None:
+ controller = _controller()
+ discovery = controller.start_execution()
+ discovery_attempt = await discovery.open_attempt(
+ ProviderOperationKind.MODEL_DISCOVERY
+ )
+
+ assert (await discovery_attempt.fail(_status_error(503))).retry_allowed
+ await discovery_attempt.aclose()
+
+ generation = controller.start_execution()
+ generation_wait = asyncio.create_task(
+ generation.open_attempt(ProviderOperationKind.GENERATION)
+ )
+ await asyncio.sleep(0)
+ assert not generation_wait.done()
+
+ probe = await discovery.open_attempt(ProviderOperationKind.MODEL_DISCOVERY)
+ await probe.accept()
+ await probe.aclose()
+
+ generation_attempt = await asyncio.wait_for(generation_wait, timeout=1)
+ await generation_attempt.accept()
+ await generation_attempt.aclose()
+
+
@pytest.mark.asyncio
async def test_one_leader_backs_off_while_followers_coalesce() -> None:
controller = _controller(base_delay=2.0, max_delay=60.0)
- leader_session = controller.new_retry_session()
- follower_session = controller.new_retry_session()
- leader = await controller.open_attempt(leader_session)
- follower = await controller.open_attempt(follower_session)
+ leader_execution = controller.start_execution()
+ follower_execution = controller.start_execution()
+ leader = await _open(leader_execution)
+ follower = await _open(follower_execution)
error = _status_error(429)
- assert await leader.retry(error)
- assert await follower.retry(error)
+ assert (await leader.fail(error)).retry_allowed
+ assert (await follower.fail(error)).retry_allowed
await leader.aclose()
await follower.aclose()
@@ -359,11 +561,9 @@ async def test_one_leader_backs_off_while_followers_coalesce() -> None:
"free_claude_code.providers.admission.asyncio.sleep",
side_effect=controlled_sleep,
):
- leader_probe_task = asyncio.create_task(controller.open_attempt(leader_session))
+ leader_probe_task = asyncio.create_task(_open(leader_execution))
await sleep_started.wait()
- follower_attempt_task = asyncio.create_task(
- controller.open_attempt(follower_session)
- )
+ follower_attempt_task = asyncio.create_task(_open(follower_execution))
await real_sleep(0)
assert len(delays) == 1
@@ -375,50 +575,50 @@ async def test_one_leader_backs_off_while_followers_coalesce() -> None:
await real_sleep(0)
assert not follower_attempt_task.done()
- await leader_probe.succeeded()
+ await leader_probe.accept()
await leader_probe.aclose()
resumed_follower = await asyncio.wait_for(follower_attempt_task, timeout=1)
- await resumed_follower.succeeded()
+ await resumed_follower.accept()
await resumed_follower.aclose()
@pytest.mark.asyncio
async def test_cancelled_follower_leaves_recovery_episode() -> None:
controller = _controller(base_delay=1.0, max_delay=1.0)
- leader_session = controller.new_retry_session()
- follower_session = controller.new_retry_session()
- leader = await controller.open_attempt(leader_session)
- follower = await controller.open_attempt(follower_session)
+ leader_execution = controller.start_execution()
+ follower_execution = controller.start_execution()
+ leader = await _open(leader_execution)
+ follower = await _open(follower_execution)
- assert await leader.retry(_status_error(503))
- assert await follower.retry(_status_error(503))
+ assert (await leader.fail(_status_error(503))).retry_allowed
+ assert (await follower.fail(_status_error(503))).retry_allowed
await leader.aclose()
await follower.aclose()
- follower_wait = asyncio.create_task(controller.open_attempt(follower_session))
+ follower_wait = asyncio.create_task(_open(follower_execution))
await asyncio.sleep(0)
episode = controller._episode
assert episode is not None
- assert follower_session in episode.waiters
+ assert follower_execution in episode.waiters
follower_wait.cancel()
with pytest.raises(asyncio.CancelledError):
await follower_wait
- assert follower_session not in episode.waiters
+ assert follower_execution not in episode.waiters
@pytest.mark.asyncio
async def test_cancelled_backoff_leader_transfers_to_a_waiter() -> None:
controller = _controller(base_delay=2.0, max_delay=60.0)
- leader_session = controller.new_retry_session()
- follower_session = controller.new_retry_session()
- leader = await controller.open_attempt(leader_session)
- follower = await controller.open_attempt(follower_session)
+ leader_execution = controller.start_execution()
+ follower_execution = controller.start_execution()
+ leader = await _open(leader_execution)
+ follower = await _open(follower_execution)
- assert await leader.retry(_status_error(503))
- assert await follower.retry(_status_error(503))
+ assert (await leader.fail(_status_error(503))).retry_allowed
+ assert (await follower.fail(_status_error(503))).retry_allowed
await leader.aclose()
await follower.aclose()
@@ -442,9 +642,9 @@ async def test_cancelled_backoff_leader_transfers_to_a_waiter() -> None:
"free_claude_code.providers.admission.asyncio.sleep",
side_effect=controlled_sleep,
):
- leader_task = asyncio.create_task(controller.open_attempt(leader_session))
+ leader_task = asyncio.create_task(_open(leader_execution))
await first_sleep_started.wait()
- follower_task = asyncio.create_task(controller.open_attempt(follower_session))
+ follower_task = asyncio.create_task(_open(follower_execution))
await real_sleep(0)
leader_task.cancel()
@@ -455,98 +655,96 @@ async def test_cancelled_backoff_leader_transfers_to_a_waiter() -> None:
release_second_sleep.set()
probe = await asyncio.wait_for(follower_task, timeout=1)
- await probe.succeeded()
+ await probe.accept()
await probe.aclose()
@pytest.mark.asyncio
async def test_stale_in_flight_success_cannot_close_recovery_episode() -> None:
controller = _controller()
- leader_session = controller.new_retry_session()
- stale_session = controller.new_retry_session()
- leader = await controller.open_attempt(leader_session)
- stale = await controller.open_attempt(stale_session)
+ leader_execution = controller.start_execution()
+ stale_execution = controller.start_execution()
+ leader = await _open(leader_execution)
+ stale = await _open(stale_execution)
- assert await leader.retry(_status_error(503))
+ assert (await leader.fail(_status_error(503))).retry_allowed
await leader.aclose()
- await stale.succeeded()
+ await stale.accept()
await stale.aclose()
- waiting = asyncio.create_task(
- controller.open_attempt(controller.new_retry_session())
- )
+ waiting = asyncio.create_task(_open(controller.start_execution()))
await asyncio.sleep(0)
assert not waiting.done()
- probe = await controller.open_attempt(leader_session)
- await probe.succeeded()
+ probe = await _open(leader_execution)
+ await probe.accept()
await probe.aclose()
resumed = await asyncio.wait_for(waiting, timeout=1)
- await resumed.succeeded()
+ await resumed.accept()
await resumed.aclose()
@pytest.mark.asyncio
async def test_abandoned_probe_transfers_leadership() -> None:
controller = _controller()
- leader_session = controller.new_retry_session()
- follower_session = controller.new_retry_session()
- leader = await controller.open_attempt(leader_session)
- follower = await controller.open_attempt(follower_session)
+ leader_execution = controller.start_execution()
+ follower_execution = controller.start_execution()
+ leader = await _open(leader_execution)
+ follower = await _open(follower_execution)
- assert await leader.retry(_status_error(503))
- assert await follower.retry(_status_error(503))
+ assert (await leader.fail(_status_error(503))).retry_allowed
+ assert (await follower.fail(_status_error(503))).retry_allowed
await leader.aclose()
await follower.aclose()
- abandoned_probe = await controller.open_attempt(leader_session)
- follower_probe_task = asyncio.create_task(controller.open_attempt(follower_session))
+ abandoned_probe = await _open(leader_execution)
+ follower_probe_task = asyncio.create_task(_open(follower_execution))
await asyncio.sleep(0)
assert not follower_probe_task.done()
await abandoned_probe.aclose()
follower_probe = await asyncio.wait_for(follower_probe_task, timeout=1)
- await follower_probe.succeeded()
+ await follower_probe.accept()
await follower_probe.aclose()
@pytest.mark.asyncio
async def test_cancelled_probe_resolution_remains_abandonable() -> None:
controller = _controller(max_concurrency=1)
- leader_session = controller.new_retry_session()
- follower_session = controller.new_retry_session()
- leader = await controller.open_attempt(leader_session)
+ leader_execution = controller.start_execution()
+ follower_execution = controller.start_execution()
+ leader = await _open(leader_execution)
- assert await leader.retry(_status_error(503))
+ assert (await leader.fail(_status_error(503))).retry_allowed
await leader.aclose()
- probe = await controller.open_attempt(leader_session)
+ probe = await _open(leader_execution)
await controller._condition.acquire()
- resolution = asyncio.create_task(probe.succeeded())
+ resolution = asyncio.create_task(probe.accept())
await asyncio.sleep(0)
resolution.cancel()
controller._condition.release()
with pytest.raises(asyncio.CancelledError):
await resolution
- follower = asyncio.create_task(controller.open_attempt(follower_session))
+ follower = asyncio.create_task(_open(follower_execution))
await probe.aclose()
replacement = await asyncio.wait_for(follower, timeout=1)
- await replacement.succeeded()
+ await replacement.accept()
await replacement.aclose()
@pytest.mark.asyncio
async def test_cancelled_probe_close_releases_slot_and_transfers_leadership() -> None:
controller = _controller(max_concurrency=1)
- leader_session = controller.new_retry_session()
- follower_session = controller.new_retry_session()
- leader = await controller.open_attempt(leader_session)
+ leader_execution = controller.start_execution()
+ follower_execution = controller.start_execution()
+ leader = await _open(leader_execution)
- assert await leader.retry(_status_error(503))
+ assert (await leader.fail(_status_error(503))).retry_allowed
await leader.aclose()
- probe = await controller.open_attempt(leader_session)
+ probe = await _open(leader_execution)
await controller._condition.acquire()
close = asyncio.create_task(probe.aclose())
await asyncio.sleep(0)
@@ -556,36 +754,37 @@ async def test_cancelled_probe_close_releases_slot_and_transfers_leadership() ->
await close
replacement = await asyncio.wait_for(
- controller.open_attempt(follower_session),
+ _open(follower_execution),
timeout=1,
)
- await replacement.succeeded()
+ await replacement.accept()
await replacement.aclose()
@pytest.mark.asyncio
async def test_non_retryable_probe_closes_episode_and_only_rejects_leader() -> None:
controller = _controller()
- leader_session = controller.new_retry_session()
- follower_session = controller.new_retry_session()
- leader = await controller.open_attempt(leader_session)
- follower = await controller.open_attempt(follower_session)
+ leader_execution = controller.start_execution()
+ follower_execution = controller.start_execution()
+ leader = await _open(leader_execution)
+ follower = await _open(follower_execution)
- assert await leader.retry(_status_error(503))
- assert await follower.retry(_status_error(503))
+ assert (await leader.fail(_status_error(503))).retry_allowed
+ assert (await follower.fail(_status_error(503))).retry_allowed
await leader.aclose()
await follower.aclose()
- probe = await controller.open_attempt(leader_session)
- follower_task = asyncio.create_task(controller.open_attempt(follower_session))
+ probe = await _open(leader_execution)
+ follower_task = asyncio.create_task(_open(follower_execution))
await asyncio.sleep(0)
assert not follower_task.done()
- assert not await probe.retry(_status_error(400))
- assert probe.failure_retryable is False
+ decision = await probe.fail(_status_error(400))
+ assert not decision.retryable
+ assert not decision.retry_allowed
await probe.aclose()
resumed = await asyncio.wait_for(follower_task, timeout=1)
- await resumed.succeeded()
+ await resumed.accept()
await resumed.aclose()
@@ -596,20 +795,20 @@ async def test_probe_exhaustion_fails_waiters_and_opens_after_cooldown() -> None
base_delay=0.02,
max_delay=0.02,
)
- leader_session = controller.new_retry_session()
- follower_session = controller.new_retry_session()
- leader = await controller.open_attempt(leader_session)
- follower = await controller.open_attempt(follower_session)
+ leader_execution = controller.start_execution()
+ follower_execution = controller.start_execution()
+ leader = await _open(leader_execution)
+ follower = await _open(follower_execution)
error = _status_error(429)
- assert await leader.retry(error)
- assert await follower.retry(error)
+ assert (await leader.fail(error)).retry_allowed
+ assert (await follower.fail(error)).retry_allowed
await leader.aclose()
await follower.aclose()
- follower_wait = asyncio.create_task(controller.open_attempt(follower_session))
- probe = await controller.open_attempt(leader_session)
- assert not await probe.retry(error)
+ follower_wait = asyncio.create_task(_open(follower_execution))
+ probe = await _open(leader_execution)
+ assert not (await probe.fail(error)).retry_allowed
await probe.aclose()
with pytest.raises(ProviderRecoveryExhausted) as waiter_error:
@@ -617,11 +816,11 @@ async def test_probe_exhaustion_fails_waiters_and_opens_after_cooldown() -> None
assert waiter_error.value.last_error is error
with pytest.raises(ProviderRecoveryExhausted):
- await controller.open_attempt(controller.new_retry_session())
+ await _open(controller.start_execution())
await asyncio.sleep(0.03)
- recovery = await controller.open_attempt(controller.new_retry_session())
- await recovery.succeeded()
+ recovery = await _open(controller.start_execution())
+ await recovery.accept()
await recovery.aclose()
@@ -632,28 +831,28 @@ async def test_exhausted_waiter_cannot_join_a_later_recovery_generation() -> Non
base_delay=0.01,
max_delay=0.01,
)
- leader_session = controller.new_retry_session()
- delayed_waiter_session = controller.new_retry_session()
- leader = await controller.open_attempt(leader_session)
- delayed_waiter = await controller.open_attempt(delayed_waiter_session)
+ leader_execution = controller.start_execution()
+ delayed_waiter_execution = controller.start_execution()
+ leader = await _open(leader_execution)
+ delayed_waiter = await _open(delayed_waiter_execution)
error = _status_error(503)
- assert await leader.retry(error)
- assert await delayed_waiter.retry(error)
+ assert (await leader.fail(error)).retry_allowed
+ assert (await delayed_waiter.fail(error)).retry_allowed
await leader.aclose()
await delayed_waiter.aclose()
- probe = await controller.open_attempt(leader_session)
- assert not await probe.retry(error)
+ probe = await _open(leader_execution)
+ assert not (await probe.fail(error)).retry_allowed
await probe.aclose()
await asyncio.sleep(0.02)
- later = await controller.open_attempt(controller.new_retry_session())
- await later.succeeded()
+ later = await _open(controller.start_execution())
+ await later.accept()
await later.aclose()
with pytest.raises(ProviderRecoveryExhausted) as exc_info:
- await controller.open_attempt(delayed_waiter_session)
+ await _open(delayed_waiter_execution)
assert exc_info.value.last_error is error
@@ -664,24 +863,26 @@ async def test_late_in_flight_failure_keeps_exhausted_generation_outcome() -> No
base_delay=0.01,
max_delay=0.01,
)
- leader_session = controller.new_retry_session()
- stale_session = controller.new_retry_session()
- leader = await controller.open_attempt(leader_session)
- stale = await controller.open_attempt(stale_session)
+ leader_execution = controller.start_execution()
+ stale_execution = controller.start_execution()
+ leader = await _open(leader_execution)
+ stale = await _open(stale_execution)
error = _status_error(503)
- assert await leader.retry(error)
+ assert (await leader.fail(error)).retry_allowed
await leader.aclose()
- probe = await controller.open_attempt(leader_session)
- assert not await probe.retry(error)
+ probe = await _open(leader_execution)
+ assert not (await probe.fail(error)).retry_allowed
await probe.aclose()
- assert await stale.retry(_status_error(503))
+ stale_decision = await stale.fail(_status_error(503))
+ assert stale_decision.retryable
+ assert not stale_decision.retry_allowed
await stale.aclose()
await asyncio.sleep(0.02)
with pytest.raises(ProviderRecoveryExhausted) as exc_info:
- await controller.open_attempt(stale_session)
+ await _open(stale_execution)
assert exc_info.value.last_error is error
@@ -701,7 +902,13 @@ async def test_retry_after_is_a_minimum_backoff() -> None:
"free_claude_code.providers.admission.asyncio.sleep",
return_value=None,
) as sleep:
- assert await controller.run_with_retry(recover) == "ok"
+ assert (
+ await controller.start_execution().run_call(
+ recover,
+ operation_kind=ProviderOperationKind.GENERATION,
+ )
+ == "ok"
+ )
sleep.assert_awaited_once()
await_args = sleep.await_args
@@ -725,19 +932,19 @@ def test_retry_after_accepts_http_date_and_rejects_invalid_values() -> None:
async def test_provider_controllers_do_not_share_recovery_state() -> None:
first = _controller(provider_name="FIRST")
second = _controller(provider_name="SECOND")
- first_session = first.new_retry_session()
- first_attempt = await first.open_attempt(first_session)
+ first_execution = first.start_execution()
+ first_attempt = await _open(first_execution)
- assert await first_attempt.retry(_status_error(503))
+ assert (await first_attempt.fail(_status_error(503))).retry_allowed
await first_attempt.aclose()
independent = await asyncio.wait_for(
- second.open_attempt(second.new_retry_session()),
+ _open(second.start_execution()),
timeout=1,
)
- await independent.succeeded()
+ await independent.accept()
await independent.aclose()
- probe = await first.open_attempt(first_session)
- await probe.succeeded()
+ probe = await _open(first_execution)
+ await probe.accept()
await probe.aclose()
diff --git a/tests/providers/test_streaming_errors.py b/tests/providers/test_streaming_errors.py
index 1e46a8ba..1fb167e7 100644
--- a/tests/providers/test_streaming_errors.py
+++ b/tests/providers/test_streaming_errors.py
@@ -21,7 +21,10 @@ from free_claude_code.core.anthropic.streaming import (
)
from free_claude_code.core.failures import ExecutionFailure
from free_claude_code.core.reasoning import DEFAULT_REASONING_POLICY, ReasoningPolicy
-from free_claude_code.providers.admission import UPSTREAM_TRANSIENT_TOTAL_ATTEMPTS
+from free_claude_code.providers.admission import (
+ UPSTREAM_TRANSIENT_TOTAL_ATTEMPTS,
+ ProviderOperationKind,
+)
from free_claude_code.providers.nvidia_nim import NvidiaNimProvider
from free_claude_code.providers.openai_chat.provider import (
_OpenAIChatStreamRunner,
@@ -233,7 +236,7 @@ class TestStreamingExceptionHandling:
async def test_stream_normalization_failure_closes_raw_stream(self):
provider = _make_provider()
stream = ClosableAsyncStreamMock([])
- retry_session = provider._admission.new_retry_session()
+ execution = provider._admission.start_execution()
with (
patch.object(
@@ -249,7 +252,11 @@ class TestStreamingExceptionHandling:
),
pytest.raises(ValueError, match="invalid stream wrapper"),
):
- await provider._create_stream({"messages": []}, retry_session)
+ await provider._create_stream(
+ {"messages": []},
+ execution,
+ ProviderOperationKind.GENERATION,
+ )
assert stream.closed
@@ -1078,12 +1085,15 @@ class TestStreamingExceptionHandling:
]
)
- with patch.object(
- provider._client.chat.completions,
- "create",
- new_callable=AsyncMock,
- side_effect=[*primary_streams, continuation],
- ) as create:
+ with (
+ patch.object(
+ provider._client.chat.completions,
+ "create",
+ new_callable=AsyncMock,
+ side_effect=[*primary_streams, continuation],
+ ) as create,
+ patch("free_claude_code.providers.admission.trace_event") as attempt_trace,
+ ):
events = await _collect_stream(provider, request)
assert create.await_count == UPSTREAM_TRANSIENT_TOTAL_ATTEMPTS
@@ -1105,6 +1115,19 @@ class TestStreamingExceptionHandling:
assert sum(event.event == "message_start" for event in parsed) == 1
assert sum(event.event == "message_delta" for event in parsed) == 1
assert sum(event.event == "message_stop" for event in parsed) == 1
+ starts = [
+ call.kwargs
+ for call in attempt_trace.call_args_list
+ if call.kwargs.get("event") == "provider.attempt.started"
+ ]
+ assert [row["operation_kind"] for row in starts] == [
+ ProviderOperationKind.GENERATION.value,
+ ProviderOperationKind.GENERATION.value,
+ ProviderOperationKind.GENERATION.value,
+ ProviderOperationKind.GENERATION.value,
+ ProviderOperationKind.CONTINUATION.value,
+ ]
+ assert len({row["execution_id"] for row in starts}) == 1
@pytest.mark.asyncio
async def test_clean_eof_after_text_continues_with_overlap_trim(self):
@@ -1192,7 +1215,7 @@ class TestStreamingExceptionHandling:
]
provider = _make_provider()
runner = _make_stream_runner(provider)
- retry_session = provider._admission.new_retry_session()
+ execution = provider._admission.start_execution()
with (
patch.object(
@@ -1206,7 +1229,8 @@ class TestStreamingExceptionHandling:
await runner._collect_recovery_output(
{"messages": []},
include_reasoning=True,
- retry_session=retry_session,
+ execution=execution,
+ operation_kind=ProviderOperationKind.CONTINUATION,
)
assert create.await_count == UPSTREAM_TRANSIENT_TOTAL_ATTEMPTS
@@ -1224,7 +1248,7 @@ class TestStreamingExceptionHandling:
]
provider = _make_provider()
runner = _make_stream_runner(provider)
- retry_session = provider._admission.new_retry_session()
+ execution = provider._admission.start_execution()
with (
patch.object(
@@ -1238,7 +1262,8 @@ class TestStreamingExceptionHandling:
await runner._collect_recovery_output(
{"messages": []},
include_reasoning=True,
- retry_session=retry_session,
+ execution=execution,
+ operation_kind=ProviderOperationKind.CONTINUATION,
)
assert create.await_count == UPSTREAM_TRANSIENT_TOTAL_ATTEMPTS
@@ -1255,7 +1280,7 @@ class TestStreamingExceptionHandling:
)
provider = _make_provider()
runner = _make_stream_runner(provider)
- retry_session = provider._admission.new_retry_session()
+ execution = provider._admission.start_execution()
with patch.object(
provider._client.chat.completions,
@@ -1266,7 +1291,8 @@ class TestStreamingExceptionHandling:
result = await runner._collect_recovery_output(
{"messages": []},
include_reasoning=True,
- retry_session=retry_session,
+ execution=execution,
+ operation_kind=ProviderOperationKind.CONTINUATION,
)
assert result.text == "world"
@@ -1279,7 +1305,7 @@ class TestStreamingExceptionHandling:
"""Provider semantics apply before the first recovery chunk as well."""
provider = _make_provider()
runner = _make_stream_runner(provider)
- retry_session = provider._admission.new_retry_session()
+ execution = provider._admission.start_execution()
request = httpx2.Request(
"POST", "https://test.api.nvidia.com/v1/chat/completions"
)
@@ -1310,7 +1336,8 @@ class TestStreamingExceptionHandling:
result = await runner._collect_recovery_output(
{"messages": []},
include_reasoning=True,
- retry_session=retry_session,
+ execution=execution,
+ operation_kind=ProviderOperationKind.CONTINUATION,
)
assert result.text == "world"
@@ -1383,14 +1410,14 @@ class TestStreamingExceptionHandling:
thinking="hidden reasoning more",
),
) as mock_collect:
- retry_session = runner._provider._admission.new_retry_session()
+ execution = runner._provider._admission.start_execution()
events = await runner._recovery_events(
body={"messages": [{"role": "user", "content": "hello"}]},
ledger=ledger,
error=TimeoutError("cutoff"),
tool_argument_alias_buffers={},
output_reasoning=True,
- retry_session=retry_session,
+ execution=execution,
)
assert events is not None
diff --git a/uv.lock b/uv.lock
index 14c75d5d..11ba32f8 100644
--- a/uv.lock
+++ b/uv.lock
@@ -607,7 +607,7 @@ wheels = [
[[package]]
name = "free-claude-code"
-version = "5.13.1"
+version = "5.13.2"
source = { editable = "." }
dependencies = [
{ name = "aiohttp" },