Macro is a unified workspace for teams: email, chat, docs, tasks, agents, calls, and CRM — @-linked together with shared AI memory.
TypeScript agentaiai-agentsall-in-onecrmcrm-systememaillinearmcpmessagingnotesnotionofficerustslackslack-alternativestartupstartup-toolstypescriptworkspace
feat(email): gate every Gmail request per mailbox (#8772)
teoc59227cadfb1
diff --git a/Cargo.lock b/Cargo.lockindex c1670e3cbc..2e7a5305db 100644--- a/Cargo.lock+++ b/Cargo.lock@@ -3120,6 +3120,15 @@ dependencies = ["tokio",]+[[package]]+name = "backon"+version = "1.6.0"+source = "registry+https://github.com/rust-lang/crates.io-index"+checksum = "cffb0e931875b666fc4fcb20fee52e9bbd1ef836fd9e9e04ec21555f9f85f7ef"+dependencies = [+ "fastrand 2.4.1",+]+[[package]]name = "base16ct"version = "0.2.0"@@ -7393,6 +7402,7 @@ dependencies = ["models_properties","notification","once_cell",+ "rand 0.9.4","redis","regex","reqwest 0.13.4",@@ -16214,12 +16224,15 @@ version = "1.2.1"source = "registry+https://github.com/rust-lang/crates.io-index"checksum = "72d32a1ac9123f0d84fda64bfc02a271d9868483162dd2d9099b5c362ece064c"dependencies = [+ "arc-swap","arcstr","async-lock",+ "backon","bytes","cfg-if","combine","crc16",+ "futures-channel","futures-util","itoa","log",diff --git a/crates/email/src/domain/mailbox/test.rs b/crates/email/src/domain/mailbox/test.rsindex 3accbfd03b..121dbba08b 100644--- a/crates/email/src/domain/mailbox/test.rs+++ b/crates/email/src/domain/mailbox/test.rs@@ -6,7 +6,7 @@ use std::time::Duration;fn provider_backoff_is_preserved_and_auth_failure_does_not_hot_loop() {let error = EmailApiError::RateLimited {retry_after: Some(Duration::from_secs(73)),- origin: RateLimitOrigin::Provider,+ origin: RateLimitOrigin::Provider { concurrency: false },};assert_eq!(retry_seconds(&error), 73);assert!(retry_seconds(&EmailApiError::AuthRequired) >= 300);diff --git a/crates/email_api_client/AGENTS.md b/crates/email_api_client/AGENTS.mdindex d826233a69..0f9f294351 100644--- a/crates/email_api_client/AGENTS.md+++ b/crates/email_api_client/AGENTS.md@@ -9,11 +9,12 @@ Outlook (Microsoft Graph) outbound adapters.## Start here- `src/domain/service.rs`: `EmailApiClientServiceImpl<R, T, L>` — the- orchestration layer, generic over a repository, token source, and rate- limiter port; submodules `attachments`/`blocklist`/`contacts`/`labels`/+ orchestration layer, generic over a repository, token source, and request+ gate port; submodules `attachments`/`blocklist`/`contacts`/`labels`/`messages`/`send`/`subscription`/`sync` hold one capability each.-- `src/domain/ports.rs`: `ProviderRateLimiter`, `ProviderTokenSource`, and the- repository trait adapters implement.+- `src/domain/ports.rs`: `ProviderRequestGate` (per-mailbox admission for the+ service above), `ProviderRateLimiter` (the Outlook `MailboxApiService`'s),+ `ProviderTokenSource`, and the repository traits adapters implement.- `src/outbound/gmail.rs`: the Gmail adapter (feature `outbound-gmail`).- `src/outbound/outlook.rs`: the Microsoft Graph adapter (feature`outbound-outlook`); every Graph call goes through `outlook/transport.rs`,@@ -47,13 +48,24 @@ Outlook (Microsoft Graph) outbound adapters.## Gotchas- `#![deny(missing_docs)]`.-- The service is generic over `<R, T, L>` (repository, token source, rate- limiter) — add a new dependency as a port + generic bound, not a concrete+- The service is generic over `<R, T, L>` (repository, token source, request+ gate) — add a new dependency as a port + generic bound, not a concretefield.+- Every provider call holds a `ProviderRequestGate` permit (`RequestPermit`)+ for its duration and finishes it with the call's error, so a provider rate+ limit cools the mailbox down for every caller. A permit dropped unfinished,+ by a cancelled future or an undispatched `PreparedSendMessage`, releases+ itself. New service methods go through `read_with_token` or+ `call_with_token`; a call made with a bare token is invisible to the+ in-flight cap. Their spans declare `gate_in_flight` and `gate_cap` as+ empty fields, which admission fills with the mailbox's load so it sits next+ to the call's outcome.- Link-based reads go through `read_with_token`, which retries once with afresh token when the provider answers `AuthRequired`; a cached token canoutlive a revoked grant, and only the fresh fetch lets the token source mark- the link for reauthorization. Writes use `prepare` and do not retry.+ the link for reauthorization. Contact listings use `retry_with_fresh_token`+ to keep their WARN-level logging. Writes use `call_with_token` and do not+ retry.- Graph answers a bodyless POST or PATCH without `Content-Length: 0` with 411Length Required. `transport.rs` adds the header, so new Graph calls must gothrough `request`/`conditional_request` rather than the raw client.diff --git a/crates/email_api_client/src/domain/models/error.rs b/crates/email_api_client/src/domain/models/error.rsindex 83b1695c6a..c0722f13c8 100644--- a/crates/email_api_client/src/domain/models/error.rs+++ b/crates/email_api_client/src/domain/models/error.rs@@ -72,15 +72,19 @@ impl EmailApiError {/// Where a rate-limit refusal originated.///-/// Callers use this to distinguish a local budget refusal (no provider quota-/// was consumed; a drop-and-wait policy may be appropriate) from the provider-/// itself throttling the request (retrying preserves delivery guarantees).+/// Callers use this to distinguish a local refusal (no provider request was+/// made, so the mailbox needs no cooldown) from the provider itself throttling+/// the request. Both defer the work; neither drops it.#[derive(Debug, Clone, Copy, PartialEq, Eq)]pub enum RateLimitOrigin {/// Refused by local quota policy before any provider call was made.Local,/// The provider itself throttled the request.- Provider,+ Provider {+ /// The provider refused because the mailbox had too many requests in+ /// flight at once, rather than for a quota.+ concurrency: bool,+ },}impl From<RateLimitRefusal> for EmailApiError {diff --git a/crates/email_api_client/src/domain/models/mod.rs b/crates/email_api_client/src/domain/models/mod.rsindex 3d6b2c7416..398a57bd43 100644--- a/crates/email_api_client/src/domain/models/mod.rs+++ b/crates/email_api_client/src/domain/models/mod.rs@@ -8,6 +8,7 @@ mod error;mod mailbox;mod mailbox_settings;mod operation;+mod permit;mod provider;mod send;@@ -19,6 +20,7 @@ pub use error::{EmailApiError, RateLimitOrigin, RateLimitRefusal};pub use mailbox::{CalendarPart, MessageWithCalendarParts, ProviderSubscription, ThreadListPage};pub use mailbox_settings::*;pub use operation::ApiOperationKind;+pub use permit::{MailboxLoad, RequestPermit};pub use provider::*;#[cfg(feature = "ports")]pub use send::PreparedSendMessage;diff --git a/crates/email_api_client/src/domain/models/permit.rs b/crates/email_api_client/src/domain/models/permit.rsnew file mode 100644index 0000000000..cbaf3f8df6--- /dev/null+++ b/crates/email_api_client/src/domain/models/permit.rs@@ -0,0 +1,72 @@+use uuid::Uuid;++type Release = Box<dyn FnOnce(Uuid, Uuid) + Send + Sync>;++/// How busy a mailbox was when the gate admitted a request.+#[derive(Debug, Clone, Copy, PartialEq, Eq)]+pub struct MailboxLoad {+ /// The mailbox's requests in flight, counting this one.+ pub in_flight: u32,+ /// The mailbox's in-flight cap when the request was admitted.+ pub cap: u32,+}++/// Admission for one provider request. A permit dropped before its gate+/// finishes it, by a cancelled caller or an abandoned send, hands itself to+/// the gate's release instead of holding its slot until the lease expires.+pub struct RequestPermit {+ link_id: Uuid,+ id: Uuid,+ load: Option<MailboxLoad>,+ release: Option<Release>,+}++impl RequestPermit {+ /// Creates a permit that runs `release` if it is dropped unfinished.+ pub fn new(+ link_id: Uuid,+ id: Uuid,+ release: impl FnOnce(Uuid, Uuid) + Send + Sync + 'static,+ ) -> Self {+ Self {+ link_id,+ id,+ load: None,+ release: Some(Box::new(release)),+ }+ }++ /// Records how busy the mailbox was when this request was admitted.+ pub fn with_load(mut self, load: MailboxLoad) -> Self {+ self.load = Some(load);+ self+ }++ /// How busy the mailbox was at admission, when the gate tracks it.+ pub fn load(&self) -> Option<MailboxLoad> {+ self.load+ }++ /// The mailbox this permit admits a request for.+ pub fn link_id(&self) -> Uuid {+ self.link_id+ }++ /// The gate's identifier for this admission.+ pub fn id(&self) -> Uuid {+ self.id+ }++ /// Records that the gate has finished this permit, so dropping it does nothing.+ pub fn disarm(mut self) {+ self.release = None;+ }+}++impl Drop for RequestPermit {+ fn drop(&mut self) {+ if let Some(release) = self.release.take() {+ release(self.link_id, self.id);+ }+ }+}diff --git a/crates/email_api_client/src/domain/models/send.rs b/crates/email_api_client/src/domain/models/send.rsindex e0ed6c91b2..18b38c86bf 100644--- a/crates/email_api_client/src/domain/models/send.rs+++ b/crates/email_api_client/src/domain/models/send.rs@@ -86,8 +86,12 @@ impl SendRequest {/// Locally prepared delivery. Tokens and message contents cannot be logged via/// `Debug`, and callers cannot construct it without completing preparation.+///+/// It holds the send's admission permit until dispatch finishes it; a delivery+/// dropped without dispatch releases the permit.#[cfg(feature = "ports")]pub struct PreparedSendMessage {+ pub(crate) permit: super::RequestPermit,pub(crate) access_token: AccessToken,pub(crate) mime: Vec<u8>,pub(crate) provider_thread_id: Option<String>,diff --git a/crates/email_api_client/src/domain/ports.rs b/crates/email_api_client/src/domain/ports.rsindex 87c3e8739d..b1e3b29d89 100644--- a/crates/email_api_client/src/domain/ports.rs+++ b/crates/email_api_client/src/domain/ports.rs@@ -13,8 +13,8 @@ use uuid::Uuid;use super::models::{AccessToken, ApiOperationKind, CalendarPart, ChangeBatch, EmailApiError,- MessageWithCalendarParts, ProviderSubscription, RateLimitRefusal, SendRequest, SentIds,- SyncCursor, ThreadListPage, TokenError, TokenFreshness,+ MessageWithCalendarParts, ProviderSubscription, RateLimitRefusal, RequestPermit, SendRequest,+ SentIds, SyncCursor, ThreadListPage, TokenError, TokenFreshness,};#[cfg(test)]@@ -315,6 +315,27 @@ pub trait ProviderRateLimiter: Send + Sync + 'static {) -> impl Future<Output = Result<(), RateLimitRefusal>> + Send;}+/// Admits provider requests for a linked mailbox. An admitted request holds a+/// permit until it completes. A permit dropped unfinished releases itself, and+/// a crashed process's permits expire on their own.+pub trait ProviderRequestGate: Send + Sync + 'static {+ /// Admits one operation, or reports when the mailbox may try again. An+ /// adapter may hold the caller briefly for room before refusing.+ fn acquire(+ &self,+ link_id: Uuid,+ operation: ApiOperationKind,+ ) -> impl Future<Output = Result<RequestPermit, RateLimitRefusal>> + Send;++ /// Releases a permit. A provider rate limit in `error` makes every caller+ /// for the mailbox wait out a shared cooldown.+ fn finish(+ &self,+ permit: RequestPermit,+ error: Option<&EmailApiError>,+ ) -> impl Future<Output = ()> + Send;+}+/// Mailbox adapter used when provider capabilities are intentionally unavailable.#[derive(Debug, Clone, Copy, Default)]pub struct NoOpMailboxClient;@@ -557,3 +578,17 @@ impl ProviderRateLimiter for AlwaysAllowRateLimiter {Ok(())}}++impl ProviderRequestGate for AlwaysAllowRateLimiter {+ async fn acquire(+ &self,+ link_id: Uuid,+ _: ApiOperationKind,+ ) -> Result<RequestPermit, RateLimitRefusal> {+ Ok(RequestPermit::new(link_id, Uuid::nil(), |_, _| {}))+ }++ async fn finish(&self, permit: RequestPermit, _: Option<&EmailApiError>) {+ permit.disarm();+ }+}diff --git a/crates/email_api_client/src/domain/service.rs b/crates/email_api_client/src/domain/service.rsindex 925fad096e..5076b841f7 100644--- a/crates/email_api_client/src/domain/service.rs+++ b/crates/email_api_client/src/domain/service.rs@@ -3,9 +3,10 @@use uuid::Uuid;use super::models::{- AccessToken, ApiOperationKind, EmailApiError, RateLimitOrigin, TokenError, TokenFreshness,+ AccessToken, ApiOperationKind, EmailApiError, RateLimitOrigin, RequestPermit, TokenError,+ TokenFreshness,};-use super::ports::{ProviderRateLimiter, ProviderTokenSource};+use super::ports::{ProviderRequestGate, ProviderTokenSource};mod attachments;mod blocklist;@@ -28,16 +29,16 @@ mod test_support;pub struct EmailApiClientServiceImpl<R, T, L> {repository: R,token_source: T,- rate_limiter: L,+ gate: L,}impl<R, T, L> EmailApiClientServiceImpl<R, T, L> {/// Creates an email API service from its provider and infrastructure ports.- pub fn new(repository: R, token_source: T, rate_limiter: L) -> Self {+ pub fn new(repository: R, token_source: T, gate: L) -> Self {Self {repository,token_source,- rate_limiter,+ gate,}}}@@ -45,7 +46,7 @@ impl<R, T, L> EmailApiClientServiceImpl<R, T, L> {impl<R, T, L> EmailApiClientServiceImpl<R, T, L>whereT: ProviderTokenSource,- L: ProviderRateLimiter,+ L: ProviderRequestGate,{/// Acquires an access token for health probes or provider-adjacent work.///@@ -64,71 +65,135 @@ where.map_err(map_token_error)}- /// Admits an operation and acquires its access token, logging a failure in- /// the calling operation's span.- async fn prepare(+ /// Runs one provider call with the cached token, without retrying.+ ///+ /// Admission and token failures are logged in the calling operation's+ /// span; the call's own errors are left to the caller.+ async fn call_with_token<O, F, Fut>(&self,link_id: Uuid,operation: ApiOperationKind,- ) -> Result<AccessToken, EmailApiError> {- // Check the cheap local limiter first: under throttling this avoids- // paying the full token dance (SELECT + Redis + possible auth-service- // refresh + health write) per refused attempt. The limiter does not- // consume provider quota on denial.- self.check_rate_limit(link_id, operation).await?;-- self.get_access_token(link_id, TokenFreshness::Cached)- .await- .inspect_err(log_operation_error)+ call: F,+ ) -> Result<O, EmailApiError>+ where+ F: FnOnce(AccessToken) -> Fut,+ Fut: Future<Output = Result<O, EmailApiError>>,+ {+ self.attempt(+ link_id,+ operation,+ self.get_access_token(link_id, TokenFreshness::Cached),+ call,+ )+ .await?}/// Runs a provider read with the cached token, retrying once with a fresh/// token if the provider rejects it, and logs a failure in the calling/// operation's span.///+ /// Callers declare `retry_count = 0` on their span; a retry sets it to 1.+ /// Like every provider call, they also declare the empty fields that+ /// [`Self::admit`] records.+ async fn read_with_token<O, F, Fut>(+ &self,+ link_id: Uuid,+ operation: ApiOperationKind,+ read: F,+ ) -> Result<O, EmailApiError>+ where+ F: Fn(AccessToken) -> Fut,+ Fut: Future<Output = Result<O, EmailApiError>>,+ {+ self.retry_with_fresh_token(link_id, operation, read)+ .await?+ .inspect_err(log_operation_error)+ }++ /// Runs a provider read with the cached token, retrying once with a fresh+ /// token if the provider rejects it.+ ////// A cached token can outlive its grant. Only a fresh fetch reaches the/// authentication service, which either renews the token or reports the/// revoked grant so the token source marks the link for reauthorization.///- /// Callers declare `retry_count = 0` on their span; a retry sets it to 1.- async fn read_with_token<O, F, Fut>(+ /// Like [`Self::attempt`], admission and token failures are logged and+ /// returned as the outer error; the provider's final answer is left to the+ /// caller.+ async fn retry_with_fresh_token<O, F, Fut>(&self,link_id: Uuid,operation: ApiOperationKind,read: F,- ) -> Result<O, EmailApiError>+ ) -> Result<Result<O, EmailApiError>, EmailApiError>whereF: Fn(AccessToken) -> Fut,Fut: Future<Output = Result<O, EmailApiError>>,{- let access_token = self.prepare(link_id, operation).await?;- let result = match read(access_token).await {+ let cached = self.get_access_token(link_id, TokenFreshness::Cached);+ match self.attempt(link_id, operation, cached, &read).await? {Err(EmailApiError::AuthRequired) => {tracing::Span::current().record("retry_count", 1);- self.check_rate_limit(link_id, operation).await?;- let fresh = self- .get_access_token(link_id, TokenFreshness::Fresh)- .await- .inspect_err(log_operation_error)?;- read(fresh).await+ let fresh = self.get_access_token(link_id, TokenFreshness::Fresh);+ self.attempt(link_id, operation, fresh, &read).await}- result => result,+ result => Ok(result),+ }+ }++ /// Makes one provider call under an admission permit.+ ///+ /// The gate is checked before `token` runs: under throttling this avoids+ /// paying the token dance (SELECT + Redis + possible auth-service refresh ++ /// health write) per refused attempt. An admission or token failure is+ /// logged and returned as the outer error; the provider's answer is the+ /// inner result, which also decides whether the mailbox cools down. If+ /// the caller is cancelled first, the dropped permit releases itself.+ async fn attempt<O, F, Fut>(+ &self,+ link_id: Uuid,+ operation: ApiOperationKind,+ token: impl Future<Output = Result<AccessToken, EmailApiError>>,+ call: F,+ ) -> Result<Result<O, EmailApiError>, EmailApiError>+ where+ F: FnOnce(AccessToken) -> Fut,+ Fut: Future<Output = Result<O, EmailApiError>>,+ {+ let permit = self.admit(link_id, operation).await?;+ let result = match token.await.inspect_err(log_operation_error) {+ Ok(token) => Ok(call(token).await),+ Err(error) => Err(error),};- result.inspect_err(log_operation_error)+ let provider_error = result.as_ref().ok().and_then(|call| call.as_ref().err());+ self.gate.finish(permit, provider_error).await;+ result}- /// Applies local quota policy, logging a refusal in the calling+ /// Takes an admission permit, logging a refusal in the calling/// operation's span.- async fn check_rate_limit(+ ///+ /// Calling operations declare `gate_in_flight` and `gate_cap` as empty+ /// span fields. They record the mailbox's load at admission next to the+ /// request's outcome, which is how Gmail's real concurrency limit can be+ /// measured.+ async fn admit(&self,link_id: Uuid,operation: ApiOperationKind,- ) -> Result<(), EmailApiError> {- self.rate_limiter- .check_rate_limit(link_id, operation)+ ) -> Result<RequestPermit, EmailApiError> {+ let permit = self+ .gate+ .acquire(link_id, operation).await.map_err(EmailApiError::from)- .inspect_err(log_operation_error)+ .inspect_err(log_operation_error)?;+ if let Some(load) = permit.load() {+ tracing::Span::current()+ .record("gate_in_flight", load.in_flight)+ .record("gate_cap", load.cap);+ }+ Ok(permit)}}@@ -136,7 +201,7 @@ where////// Unlike `#[instrument(err)]`, whose level is fixed, this distinguishes a/// local rate-limit refusal: expected backpressure where no provider request-/// was made and the caller decides whether to retry or drop. It is logged at+/// was made and the caller decides when to retry. It is logged at/// DEBUG; every other failure, including provider throttling, stays at ERROR.fn log_operation_error(error: &EmailApiError) {match error {@@ -144,6 +209,9 @@ fn log_operation_error(error: &EmailApiError) {origin: RateLimitOrigin::Local,..} => tracing::debug!(error = %error),+ EmailApiError::RateLimited { origin, .. } => {+ tracing::error!(error = %error, origin = ?origin)+ }_ => tracing::error!(error = %error),}}diff --git a/crates/email_api_client/src/domain/service/attachments.rs b/crates/email_api_client/src/domain/service/attachments.rsindex f138c3b256..5a8fb234f6 100644--- a/crates/email_api_client/src/domain/service/attachments.rs+++ b/crates/email_api_client/src/domain/service/attachments.rs@@ -1,17 +1,17 @@use uuid::Uuid;use super::super::models::EmailApiError;-use super::super::ports::{MailboxAttachmentClient, ProviderRateLimiter, ProviderTokenSource};+use super::super::ports::{MailboxAttachmentClient, ProviderRequestGate, ProviderTokenSource};use super::{ApiOperationKind, EmailApiClientServiceImpl};impl<R, T, L> EmailApiClientServiceImpl<R, T, L>whereR: MailboxAttachmentClient,T: ProviderTokenSource,- L: ProviderRateLimiter,+ L: ProviderRequestGate,{/// Downloads an attachment from a provider message.- #[tracing::instrument(skip(self), fields(retry_count = 0))]+ #[tracing::instrument(skip(self), fields(retry_count = 0, gate_in_flight = tracing::field::Empty, gate_cap = tracing::field::Empty))]pub async fn get_attachment(&self,link_id: Uuid,diff --git a/crates/email_api_client/src/domain/service/attachments/test.rs b/crates/email_api_client/src/domain/service/attachments/test.rsindex 3bfe8f65dc..42f598c1eb 100644--- a/crates/email_api_client/src/domain/service/attachments/test.rs+++ b/crates/email_api_client/src/domain/service/attachments/test.rs@@ -2,7 +2,7 @@ use std::sync::{Arc, Mutex};use uuid::Uuid;-use super::super::test_support::{Call, FakeRateLimiter, FakeTokenSource, call_log};+use super::super::test_support::{Call, FakeGate, FakeTokenSource, call_log};use super::*;use crate::domain::models::{AccessToken, TokenFreshness};@@ -36,7 +36,7 @@ async fn get_attachment_uses_attachment_quota_and_forwards_identifiers() {let service = EmailApiClientServiceImpl::new(repository,FakeTokenSource::new(calls.clone(), Ok(AccessToken::new("token"))),- FakeRateLimiter::new(calls.clone(), Ok(())),+ FakeGate::new(calls.clone(), Ok(())),);let result = service@@ -51,8 +51,9 @@ async fn get_attachment_uses_attachment_quota_and_forwards_identifiers() {assert_eq!(*calls.lock().unwrap(),vec![- Call::RateLimit(link_id, ApiOperationKind::GetAttachment),+ Call::Acquire(link_id, ApiOperationKind::GetAttachment),Call::Token(link_id, TokenFreshness::Cached),+ Call::Finish(link_id, None),]);}diff --git a/crates/email_api_client/src/domain/service/blocklist.rs b/crates/email_api_client/src/domain/service/blocklist.rsindex 4c2046c167..b548b26f4d 100644--- a/crates/email_api_client/src/domain/service/blocklist.rs+++ b/crates/email_api_client/src/domain/service/blocklist.rs@@ -1,58 +1,60 @@use uuid::Uuid;use super::super::models::EmailApiError;-use super::super::ports::{MailboxBlocklistClient, ProviderRateLimiter, ProviderTokenSource};+use super::super::ports::{MailboxBlocklistClient, ProviderRequestGate, ProviderTokenSource};use super::{ApiOperationKind, EmailApiClientServiceImpl, log_operation_error};impl<R, T, L> EmailApiClientServiceImpl<R, T, L>whereR: MailboxBlocklistClient,T: ProviderTokenSource,- L: ProviderRateLimiter,+ L: ProviderRequestGate,{/// Ensures messages from an email address are sent to trash.- #[tracing::instrument(skip(self, email_address))]+ #[tracing::instrument(skip(self, email_address), fields(gate_in_flight = tracing::field::Empty, gate_cap = tracing::field::Empty))]pub async fn block_sender(&self,link_id: Uuid,email_address: &str,) -> Result<(), EmailApiError> {- let access_token = self.prepare(link_id, ApiOperationKind::BlockSender).await?;-- self.repository- .block_sender(&access_token, email_address)- .await- .inspect_err(log_operation_error)+ self.call_with_token(link_id, ApiOperationKind::BlockSender, |token| async move {+ self.repository+ .block_sender(&token, email_address)+ .await+ .inspect_err(log_operation_error)+ })+ .await}/// Removes the blocked-sender rule for an email address, when present.- #[tracing::instrument(skip(self, email_address))]+ #[tracing::instrument(skip(self, email_address), fields(gate_in_flight = tracing::field::Empty, gate_cap = tracing::field::Empty))]pub async fn unblock_sender(&self,link_id: Uuid,email_address: &str,) -> Result<(), EmailApiError> {- let access_token = self- .prepare(link_id, ApiOperationKind::UnblockSender)- .await?;-- self.repository- .unblock_sender(&access_token, email_address)- .await- .inspect_err(log_operation_error)+ self.call_with_token(+ link_id,+ ApiOperationKind::UnblockSender,+ |token| async move {+ self.repository+ .unblock_sender(&token, email_address)+ .await+ .inspect_err(log_operation_error)+ },+ )+ .await}/// Lists email addresses covered by blocked-sender rules.- #[tracing::instrument(skip(self))]+ #[tracing::instrument(skip(self), fields(retry_count = 0, gate_in_flight = tracing::field::Empty, gate_cap = tracing::field::Empty))]pub async fn list_blocked_senders(&self, link_id: Uuid) -> Result<Vec<String>, EmailApiError> {- let access_token = self- .prepare(link_id, ApiOperationKind::ListBlockedSenders)- .await?;-- self.repository- .list_blocked_senders(&access_token)- .await- .inspect_err(log_operation_error)+ self.read_with_token(+ link_id,+ ApiOperationKind::ListBlockedSenders,+ |token| async move { self.repository.list_blocked_senders(&token).await },+ )+ .await}}diff --git a/crates/email_api_client/src/domain/service/blocklist/test.rs b/crates/email_api_client/src/domain/service/blocklist/test.rsindex d2da5a5f7e..3bf2a1d2b5 100644--- a/crates/email_api_client/src/domain/service/blocklist/test.rs+++ b/crates/email_api_client/src/domain/service/blocklist/test.rs@@ -2,7 +2,7 @@ use uuid::Uuid;use super::super::super::models::{AccessToken, EmailApiError, TokenFreshness};use super::super::super::ports::MailboxBlocklistClient;-use super::super::test_support::{Call, FakeRateLimiter, FakeTokenSource, call_log};+use super::super::test_support::{Call, FakeGate, FakeTokenSource, call_log};use super::EmailApiClientServiceImpl;#[derive(Clone)]@@ -77,9 +77,10 @@ async fn blocklist_methods_use_their_matching_costs_and_operations() {assert_eq!(*calls.lock().unwrap(),vec![- Call::RateLimit(Uuid::nil(), operation),+ Call::Acquire(Uuid::nil(), operation),Call::Token(Uuid::nil(), TokenFreshness::Cached),Call::Repository(repository_call),+ Call::Finish(Uuid::nil(), None),]);}@@ -87,12 +88,12 @@ async fn blocklist_methods_use_their_matching_costs_and_operations() {fn service(calls: super::super::test_support::CallLog,-) -> EmailApiClientServiceImpl<BlocklistClient, FakeTokenSource, FakeRateLimiter> {+) -> EmailApiClientServiceImpl<BlocklistClient, FakeTokenSource, FakeGate> {EmailApiClientServiceImpl::new(BlocklistClient {calls: calls.clone(),},FakeTokenSource::new(calls.clone(), Ok(AccessToken::new("access-token"))),- FakeRateLimiter::new(calls, Ok(())),+ FakeGate::new(calls, Ok(())),)}diff --git a/crates/email_api_client/src/domain/service/contacts.rs b/crates/email_api_client/src/domain/service/contacts.rsindex 20538678aa..ddd3ae3182 100644--- a/crates/email_api_client/src/domain/service/contacts.rs+++ b/crates/email_api_client/src/domain/service/contacts.rs@@ -2,63 +2,67 @@ use models_email::service::contact::{Contact, ContactList};use uuid::Uuid;use super::super::models::EmailApiError;-use super::super::ports::{MailboxContactsClient, ProviderRateLimiter, ProviderTokenSource};-use super::{ApiOperationKind, EmailApiClientServiceImpl, log_operation_error};+use super::super::ports::{MailboxContactsClient, ProviderRequestGate, ProviderTokenSource};+use super::{ApiOperationKind, EmailApiClientServiceImpl};impl<R, T, L> EmailApiClientServiceImpl<R, T, L>whereR: MailboxContactsClient,T: ProviderTokenSource,- L: ProviderRateLimiter,+ L: ProviderRequestGate,{/// Fetches the linked mailbox owner's contact record.- #[tracing::instrument(skip(self))]+ #[tracing::instrument(skip(self), fields(retry_count = 0, gate_in_flight = tracing::field::Empty, gate_cap = tracing::field::Empty))]pub async fn get_self_contact(&self, link_id: Uuid) -> Result<Contact, EmailApiError> {- let access_token = self- .prepare(link_id, ApiOperationKind::ListContacts)- .await?;-- self.repository- .get_self_contact(&access_token, link_id)- .await- .inspect_err(log_operation_error)+ self.read_with_token(+ link_id,+ ApiOperationKind::ListContacts,+ |token| async move { self.repository.get_self_contact(&token, link_id).await },+ )+ .await}/// Lists primary contacts, optionally continuing an incremental synchronization.////// Failures log at WARN: an expired sync token ([`EmailApiError::OutdatedCursor`])/// or a grant without the contacts scopes is routine, and callers recover or skip.- #[tracing::instrument(skip(self, sync_token), err(level = "warn"))]+ #[tracing::instrument(skip(self, sync_token), fields(retry_count = 0, gate_in_flight = tracing::field::Empty, gate_cap = tracing::field::Empty), err(level = "warn"))]pub async fn list_contacts(&self,link_id: Uuid,sync_token: Option<&str>,) -> Result<ContactList, EmailApiError> {- let access_token = self- .prepare(link_id, ApiOperationKind::ListContacts)- .await?;-- self.repository- .list_contacts(&access_token, link_id, sync_token)- .await+ self.retry_with_fresh_token(+ link_id,+ ApiOperationKind::ListContacts,+ |token| async move {+ self.repository+ .list_contacts(&token, link_id, sync_token)+ .await+ },+ )+ .await?}/// Lists automatically collected contacts, optionally continuing a synchronization.////// Failures log at WARN for the same reasons as [`Self::list_contacts`].- #[tracing::instrument(skip(self, sync_token), err(level = "warn"))]+ #[tracing::instrument(skip(self, sync_token), fields(retry_count = 0, gate_in_flight = tracing::field::Empty, gate_cap = tracing::field::Empty), err(level = "warn"))]pub async fn list_other_contacts(&self,link_id: Uuid,sync_token: Option<&str>,) -> Result<ContactList, EmailApiError> {- let access_token = self- .prepare(link_id, ApiOperationKind::ListContacts)- .await?;-- self.repository- .list_other_contacts(&access_token, link_id, sync_token)- .await+ self.retry_with_fresh_token(+ link_id,+ ApiOperationKind::ListContacts,+ |token| async move {+ self.repository+ .list_other_contacts(&token, link_id, sync_token)+ .await+ },+ )+ .await?}}diff --git a/crates/email_api_client/src/domain/service/contacts/test.rs b/crates/email_api_client/src/domain/service/contacts/test.rsindex 50a184308b..54df24189f 100644--- a/crates/email_api_client/src/domain/service/contacts/test.rs+++ b/crates/email_api_client/src/domain/service/contacts/test.rs@@ -3,7 +3,7 @@ use std::sync::{Arc, Mutex};use models_email::service::contact::{Contact, ContactList};use uuid::Uuid;-use super::super::test_support::{Call, FakeRateLimiter, FakeTokenSource, call_log};+use super::super::test_support::{Call, FakeGate, FakeTokenSource, call_log};use super::*;use crate::domain::models::{AccessToken, TokenFreshness};@@ -69,7 +69,7 @@ async fn assert_call<F, E>(operation: ApiOperationKind, expected_repository_callwhereE: FnOnce(Uuid) -> ContactCall,F: AsyncFnOnce(- &EmailApiClientServiceImpl<ContactsClient, FakeTokenSource, FakeRateLimiter>,+ &EmailApiClientServiceImpl<ContactsClient, FakeTokenSource, FakeGate>,Uuid,) -> Result<(), EmailApiError>,{@@ -80,7 +80,7 @@ wherelet service = EmailApiClientServiceImpl::new(repository,FakeTokenSource::new(calls.clone(), Ok(AccessToken::new("token"))),- FakeRateLimiter::new(calls.clone(), Ok(())),+ FakeGate::new(calls.clone(), Ok(())),);assert_eq!(invoke(&service, link_id).await, Err(expected_error()));@@ -91,8 +91,66 @@ whereassert_eq!(*calls.lock().unwrap(),vec![- Call::RateLimit(link_id, operation),+ Call::Acquire(link_id, operation),Call::Token(link_id, TokenFreshness::Cached),+ Call::Finish(link_id, Some(expected_error())),+ ]+ );+}++/// Answers a cached token the way Google answers a revoked or expired one.+struct RejectsCachedToken;++impl MailboxContactsClient for RejectsCachedToken {+ async fn get_self_contact(&self, _: &AccessToken, _: Uuid) -> Result<Contact, EmailApiError> {+ unreachable!("not used by this test")+ }++ async fn list_contacts(+ &self,+ access_token: &AccessToken,+ _: Uuid,+ _: Option<&str>,+ ) -> Result<ContactList, EmailApiError> {+ match access_token.expose_secret() {+ "cached" => Err(EmailApiError::AuthRequired),+ _ => Err(expected_error()),+ }+ }++ async fn list_other_contacts(+ &self,+ _: &AccessToken,+ _: Uuid,+ _: Option<&str>,+ ) -> Result<ContactList, EmailApiError> {+ unreachable!("not used by this test")+ }+}++#[tokio::test]+async fn a_rejected_cached_token_is_retried_fresh_for_contact_sync() {+ let calls = call_log();+ let link_id = Uuid::new_v4();+ let service = EmailApiClientServiceImpl::new(+ RejectsCachedToken,+ FakeTokenSource::new(calls.clone(), Ok(AccessToken::new("cached")))+ .with_fresh(Ok(AccessToken::new("fresh"))),+ FakeGate::new(calls.clone(), Ok(())),+ );++ let result = service.list_contacts(link_id, None).await;++ assert_eq!(result.err(), Some(expected_error()));+ assert_eq!(+ *calls.lock().unwrap(),+ vec![+ Call::Acquire(link_id, ApiOperationKind::ListContacts),+ Call::Token(link_id, TokenFreshness::Cached),+ Call::Finish(link_id, Some(EmailApiError::AuthRequired)),+ Call::Acquire(link_id, ApiOperationKind::ListContacts),+ Call::Token(link_id, TokenFreshness::Fresh),+ Call::Finish(link_id, Some(expected_error())),]);}diff --git a/crates/email_api_client/src/domain/service/labels.rs b/crates/email_api_client/src/domain/service/labels.rsindex d1e9dc2aa5..0e454a292f 100644--- a/crates/email_api_client/src/domain/service/labels.rs+++ b/crates/email_api_client/src/domain/service/labels.rs@@ -2,17 +2,17 @@ use models_email::email::service::label::Label;use uuid::Uuid;use super::super::models::EmailApiError;-use super::super::ports::{MailboxLabelClient, ProviderRateLimiter, ProviderTokenSource};+use super::super::ports::{MailboxLabelClient, ProviderRequestGate, ProviderTokenSource};use super::{ApiOperationKind, EmailApiClientServiceImpl, log_operation_error};impl<R, T, L> EmailApiClientServiceImpl<R, T, L>whereR: MailboxLabelClient,T: ProviderTokenSource,- L: ProviderRateLimiter,+ L: ProviderRequestGate,{/// Lists all labels for a linked mailbox.- #[tracing::instrument(skip(self), fields(retry_count = 0))]+ #[tracing::instrument(skip(self), fields(retry_count = 0, gate_in_flight = tracing::field::Empty, gate_cap = tracing::field::Empty))]pub async fn list_labels(&self, link_id: Uuid) -> Result<Vec<Label>, EmailApiError> {self.read_with_token(link_id, ApiOperationKind::ListLabels, |token| async move {self.repository.list_labels(&token, link_id).await@@ -21,33 +21,35 @@ where}/// Creates a user label.- #[tracing::instrument(skip(self))]+ #[tracing::instrument(skip(self), fields(gate_in_flight = tracing::field::Empty, gate_cap = tracing::field::Empty))]pub async fn create_label(&self,link_id: Uuid,label_name: &str,) -> Result<Label, EmailApiError> {- let access_token = self.prepare(link_id, ApiOperationKind::CreateLabel).await?;-- self.repository- .create_label(&access_token, link_id, label_name)- .await- .inspect_err(log_operation_error)+ self.call_with_token(link_id, ApiOperationKind::CreateLabel, |token| async move {+ self.repository+ .create_label(&token, link_id, label_name)+ .await+ .inspect_err(log_operation_error)+ })+ .await}/// Deletes a provider label.- #[tracing::instrument(skip(self))]+ #[tracing::instrument(skip(self), fields(gate_in_flight = tracing::field::Empty, gate_cap = tracing::field::Empty))]pub async fn delete_label(&self,link_id: Uuid,provider_label_id: &str,) -> Result<(), EmailApiError> {- let access_token = self.prepare(link_id, ApiOperationKind::DeleteLabel).await?;-- self.repository- .delete_label(&access_token, provider_label_id)- .await- .inspect_err(log_operation_error)+ self.call_with_token(link_id, ApiOperationKind::DeleteLabel, |token| async move {+ self.repository+ .delete_label(&token, provider_label_id)+ .await+ .inspect_err(log_operation_error)+ })+ .await}}diff --git a/crates/email_api_client/src/domain/service/labels/test.rs b/crates/email_api_client/src/domain/service/labels/test.rsindex 5a56a12b8a..e379360cf9 100644--- a/crates/email_api_client/src/domain/service/labels/test.rs+++ b/crates/email_api_client/src/domain/service/labels/test.rs@@ -3,7 +3,7 @@ use std::sync::{Arc, Mutex};use models_email::email::service::label::Label;use uuid::Uuid;-use super::super::test_support::{Call, FakeRateLimiter, FakeTokenSource, call_log};+use super::super::test_support::{Call, FakeGate, FakeTokenSource, call_log};use super::*;use crate::domain::models::{AccessToken, TokenFreshness};@@ -59,7 +59,7 @@ async fn list_labels_uses_list_labels_quota_and_forwards_link_id() {let service = EmailApiClientServiceImpl::new(repository,FakeTokenSource::new(calls.clone(), Ok(AccessToken::new("token"))),- FakeRateLimiter::new(calls.clone(), Ok(())),+ FakeGate::new(calls.clone(), Ok(())),);assert!(matches!(@@ -70,8 +70,9 @@ async fn list_labels_uses_list_labels_quota_and_forwards_link_id() {assert_eq!(*calls.lock().unwrap(),vec![- Call::RateLimit(link_id, ApiOperationKind::ListLabels),+ Call::Acquire(link_id, ApiOperationKind::ListLabels),Call::Token(link_id, TokenFreshness::Cached),+ Call::Finish(link_id, Some(expected_error())),]);}@@ -85,7 +86,7 @@ async fn create_label_uses_create_quota_and_forwards_name() {let service = EmailApiClientServiceImpl::new(repository,FakeTokenSource::new(calls.clone(), Ok(AccessToken::new("token"))),- FakeRateLimiter::new(calls.clone(), Ok(())),+ FakeGate::new(calls.clone(), Ok(())),);assert!(matches!(@@ -96,8 +97,9 @@ async fn create_label_uses_create_quota_and_forwards_name() {assert_eq!(*calls.lock().unwrap(),vec![- Call::RateLimit(link_id, ApiOperationKind::CreateLabel),+ Call::Acquire(link_id, ApiOperationKind::CreateLabel),Call::Token(link_id, TokenFreshness::Cached),+ Call::Finish(link_id, Some(expected_error())),]);}@@ -111,7 +113,7 @@ async fn delete_label_uses_delete_quota_and_forwards_provider_id() {let service = EmailApiClientServiceImpl::new(repository,FakeTokenSource::new(calls.clone(), Ok(AccessToken::new("token"))),- FakeRateLimiter::new(calls.clone(), Ok(())),+ FakeGate::new(calls.clone(), Ok(())),);assert_eq!(@@ -122,8 +124,9 @@ async fn delete_label_uses_delete_quota_and_forwards_provider_id() {assert_eq!(*calls.lock().unwrap(),vec![- Call::RateLimit(link_id, ApiOperationKind::DeleteLabel),+ Call::Acquire(link_id, ApiOperationKind::DeleteLabel),Call::Token(link_id, TokenFreshness::Cached),+ Call::Finish(link_id, Some(expected_error())),]);}diff --git a/crates/email_api_client/src/domain/service/messages.rs b/crates/email_api_client/src/domain/service/messages.rsindex 0623e60382..a9fc94702f 100644--- a/crates/email_api_client/src/domain/service/messages.rs+++ b/crates/email_api_client/src/domain/service/messages.rs@@ -3,7 +3,7 @@ use uuid::Uuid;use super::super::models::{CalendarPart, EmailApiError, MessageWithCalendarParts, ThreadListPage};use super::super::ports::{- MailboxCalendarClient, MailboxMessageClient, ProviderRateLimiter, ProviderTokenSource,+ MailboxCalendarClient, MailboxMessageClient, ProviderRequestGate, ProviderTokenSource,};use super::{ApiOperationKind, EmailApiClientServiceImpl, log_operation_error};@@ -11,11 +11,11 @@ impl<R, T, L> EmailApiClientServiceImpl<R, T, L>whereR: MailboxMessageClient,T: ProviderTokenSource,- L: ProviderRateLimiter,+ L: ProviderRequestGate,{/// Fetches and normalizes one provider message, including any calendar/// invitation parts, in a single provider read.- #[tracing::instrument(skip(self), fields(retry_count = 0))]+ #[tracing::instrument(skip(self), fields(retry_count = 0, gate_in_flight = tracing::field::Empty, gate_cap = tracing::field::Empty))]pub async fn get_message(&self,link_id: Uuid,@@ -30,7 +30,7 @@ where}/// Fetches the provider label identifiers attached to one message.- #[tracing::instrument(skip(self), fields(retry_count = 0))]+ #[tracing::instrument(skip(self), fields(retry_count = 0, gate_in_flight = tracing::field::Empty, gate_cap = tracing::field::Empty))]pub async fn get_message_label_ids(&self,link_id: Uuid,@@ -45,7 +45,7 @@ where}/// Lists message identifiers carrying all requested provider labels.- #[tracing::instrument(skip(self), fields(retry_count = 0))]+ #[tracing::instrument(skip(self), fields(retry_count = 0, gate_in_flight = tracing::field::Empty, gate_cap = tracing::field::Empty))]pub async fn list_messages(&self,link_id: Uuid,@@ -65,7 +65,7 @@ where}/// Lists message identifiers belonging to one provider thread.- #[tracing::instrument(skip(self), fields(retry_count = 0))]+ #[tracing::instrument(skip(self), fields(retry_count = 0, gate_in_flight = tracing::field::Empty, gate_cap = tracing::field::Empty))]pub async fn get_message_ids_for_thread(&self,link_id: Uuid,@@ -80,7 +80,7 @@ where}/// Fetches and normalizes every message in one provider thread.- #[tracing::instrument(skip(self), fields(retry_count = 0))]+ #[tracing::instrument(skip(self), fields(retry_count = 0, gate_in_flight = tracing::field::Empty, gate_cap = tracing::field::Empty))]pub async fn get_thread(&self,link_id: Uuid,@@ -95,7 +95,7 @@ where}/// Lists one page of provider threads.- #[tracing::instrument(skip(self), fields(retry_count = 0))]+ #[tracing::instrument(skip(self), fields(retry_count = 0, gate_in_flight = tracing::field::Empty, gate_cap = tracing::field::Empty))]pub async fn list_threads(&self,link_id: Uuid,@@ -112,7 +112,7 @@ where}/// Applies provider label additions and removals to one message.- #[tracing::instrument(skip(self))]+ #[tracing::instrument(skip(self), fields(gate_in_flight = tracing::field::Empty, gate_cap = tracing::field::Empty))]pub async fn modify_message_labels(&self,link_id: Uuid,@@ -120,19 +120,22 @@ wherelabels_to_add: &[String],labels_to_remove: &[String],) -> Result<(), EmailApiError> {- let access_token = self- .prepare(link_id, ApiOperationKind::ModifyMessageLabels)- .await?;-- self.repository- .modify_message_labels(- &access_token,- provider_message_id,- labels_to_add,- labels_to_remove,- )- .await- .inspect_err(log_operation_error)+ self.call_with_token(+ link_id,+ ApiOperationKind::ModifyMessageLabels,+ |token| async move {+ self.repository+ .modify_message_labels(+ &token,+ provider_message_id,+ labels_to_add,+ labels_to_remove,+ )+ .await+ .inspect_err(log_operation_error)+ },+ )+ .await}}@@ -140,7 +143,7 @@ impl<R, T, L> EmailApiClientServiceImpl<R, T, L>whereR: MailboxCalendarClient,T: ProviderTokenSource,- L: ProviderRateLimiter,+ L: ProviderRequestGate,{/// Finds calendar invitation parts in one provider message via a fresh/// provider fetch.@@ -148,17 +151,18 @@ where/// This charges a full message read; ingest paths that already fetched the/// message should consume [`MessageWithCalendarParts::calendar_parts`]/// instead. This lookup exists for durable re-extraction jobs.- #[tracing::instrument(skip(self))]+ #[tracing::instrument(skip(self), fields(retry_count = 0, gate_in_flight = tracing::field::Empty, gate_cap = tracing::field::Empty))]pub async fn get_calendar_parts(&self,link_id: Uuid,provider_message_id: &str,) -> Result<Vec<CalendarPart>, EmailApiError> {- let access_token = self.prepare(link_id, ApiOperationKind::GetMessage).await?;- self.repository- .get_calendar_parts(&access_token, provider_message_id)- .await- .inspect_err(log_operation_error)+ self.read_with_token(link_id, ApiOperationKind::GetMessage, |token| async move {+ self.repository+ .get_calendar_parts(&token, provider_message_id)+ .await+ })+ .await}}diff --git a/crates/email_api_client/src/domain/service/messages/test.rs b/crates/email_api_client/src/domain/service/messages/test.rsindex 1efc48a741..882c2d0988 100644--- a/crates/email_api_client/src/domain/service/messages/test.rs+++ b/crates/email_api_client/src/domain/service/messages/test.rs@@ -3,7 +3,7 @@ use std::sync::{Arc, Mutex};use models_email::email::service::message::Message;use uuid::Uuid;-use super::super::test_support::{Call, FakeRateLimiter, FakeTokenSource, call_log};+use super::super::test_support::{Call, FakeGate, FakeTokenSource, call_log};use super::*;use crate::domain::models::{AccessToken, TokenFreshness};@@ -138,7 +138,7 @@ async fn assert_call<F, E>(operation: ApiOperationKind, expected_repository_callwhereE: FnOnce(Uuid) -> MessageCall,F: AsyncFnOnce(- &EmailApiClientServiceImpl<MessageClient, FakeTokenSource, FakeRateLimiter>,+ &EmailApiClientServiceImpl<MessageClient, FakeTokenSource, FakeGate>,Uuid,) -> Result<(), EmailApiError>,{@@ -149,7 +149,7 @@ wherelet service = EmailApiClientServiceImpl::new(repository,FakeTokenSource::new(calls.clone(), Ok(AccessToken::new("token"))),- FakeRateLimiter::new(calls.clone(), Ok(())),+ FakeGate::new(calls.clone(), Ok(())),);assert_eq!(invoke(&service, link_id).await, Err(expected_error()));@@ -160,8 +160,9 @@ whereassert_eq!(*calls.lock().unwrap(),vec![- Call::RateLimit(link_id, operation),+ Call::Acquire(link_id, operation),Call::Token(link_id, TokenFreshness::Cached),+ Call::Finish(link_id, Some(expected_error())),]);}diff --git a/crates/email_api_client/src/domain/service/send.rs b/crates/email_api_client/src/domain/service/send.rsindex ca4a3f32e9..a38c928ced 100644--- a/crates/email_api_client/src/domain/service/send.rs+++ b/crates/email_api_client/src/domain/service/send.rs@@ -1,10 +1,10 @@use uuid::Uuid;use super::super::models::{- EmailApiError, PreparedSendMessage, SendRequest, SentIds, validate_message_id,+ EmailApiError, PreparedSendMessage, SendRequest, SentIds, TokenFreshness, validate_message_id,};use super::super::ports::{- MailboxSendClient, MailboxSendRecoveryClient, ProviderRateLimiter, ProviderTokenSource,+ MailboxSendClient, MailboxSendRecoveryClient, ProviderRequestGate, ProviderTokenSource,};use super::{ApiOperationKind, EmailApiClientServiceImpl, log_operation_error};@@ -12,22 +12,23 @@ impl<R, T, L> EmailApiClientServiceImpl<R, T, L>whereR: MailboxSendClient,T: ProviderTokenSource,- L: ProviderRateLimiter,+ L: ProviderRequestGate,{/// Sends a message through the linked mailbox.- #[tracing::instrument(skip(self, request))]+ #[tracing::instrument(skip(self, request), fields(gate_in_flight = tracing::field::Empty, gate_cap = tracing::field::Empty))]pub async fn send_message(&self,link_id: Uuid,request: &SendRequest,provider_thread_id: Option<&str>,) -> Result<SentIds, EmailApiError> {- let access_token = self.prepare(link_id, ApiOperationKind::SendMessage).await?;-- self.repository- .send_message(&access_token, request, provider_thread_id)- .await- .inspect_err(log_operation_error)+ self.call_with_token(link_id, ApiOperationKind::SendMessage, |token| async move {+ self.repository+ .send_message(&token, request, provider_thread_id)+ .await+ .inspect_err(log_operation_error)+ })+ .await}}@@ -35,11 +36,11 @@ impl<R, T, L> EmailApiClientServiceImpl<R, T, L>whereR: MailboxSendRecoveryClient,T: ProviderTokenSource,- L: ProviderRateLimiter,+ L: ProviderRequestGate,{- /// Completes quota, token and MIME preparation without dispatching a send.- /// Persist delivery-started state only after this method succeeds.- #[tracing::instrument(skip(self, request))]+ /// Completes admission, token and MIME preparation without dispatching a+ /// send. Persist delivery-started state only after this method succeeds.+ #[tracing::instrument(skip(self, request), fields(gate_in_flight = tracing::field::Empty, gate_cap = tracing::field::Empty))]pub async fn prepare_send(&self,link_id: Uuid,@@ -47,8 +48,20 @@ whereprovider_thread_id: Option<&str>,) -> Result<PreparedSendMessage, EmailApiError> {let mime = request.build_mime().inspect_err(log_operation_error)?;- let access_token = self.prepare(link_id, ApiOperationKind::SendMessage).await?;+ let permit = self.admit(link_id, ApiOperationKind::SendMessage).await?;+ let access_token = match self+ .get_access_token(link_id, TokenFreshness::Cached)+ .await+ .inspect_err(log_operation_error)+ {+ Ok(access_token) => access_token,+ Err(error) => {+ self.gate.finish(permit, None).await;+ return Err(error);+ }+ };Ok(PreparedSendMessage {+ permit,access_token,mime,provider_thread_id: provider_thread_id.map(str::to_owned),@@ -60,33 +73,36 @@ where#[tracing::instrument(skip_all)]pub async fn send_prepared(&self,- prepared: &PreparedSendMessage,+ prepared: PreparedSendMessage,) -> Result<SentIds, EmailApiError> {- self.repository+ let result = self+ .repository.send_prepared(&prepared.access_token,&prepared.mime,prepared.provider_thread_id.as_deref(),)- .await- .inspect_err(log_operation_error)+ .await;+ self.gate+ .finish(prepared.permit, result.as_ref().err())+ .await;+ result.inspect_err(log_operation_error)}/// Checks sent mail for a stable delivery identifier without sending again.- #[tracing::instrument(skip(self, message_id))]+ #[tracing::instrument(skip(self, message_id), fields(retry_count = 0, gate_in_flight = tracing::field::Empty, gate_cap = tracing::field::Empty))]pub async fn find_sent_message(&self,link_id: Uuid,message_id: &str,) -> Result<Option<SentIds>, EmailApiError> {validate_message_id(message_id).inspect_err(log_operation_error)?;- let access_token = self- .prepare(link_id, ApiOperationKind::ListMessages)- .await?;- self.repository- .find_sent_message(&access_token, message_id)- .await- .inspect_err(log_operation_error)+ self.read_with_token(+ link_id,+ ApiOperationKind::ListMessages,+ |token| async move { self.repository.find_sent_message(&token, message_id).await },+ )+ .await}}diff --git a/crates/email_api_client/src/domain/service/send/test.rs b/crates/email_api_client/src/domain/service/send/test.rsindex fc31102260..e8d0a137bc 100644--- a/crates/email_api_client/src/domain/service/send/test.rs+++ b/crates/email_api_client/src/domain/service/send/test.rs@@ -3,10 +3,10 @@ use models_email::email::service::message::MessageToSend;use uuid::Uuid;use super::super::super::models::{- AccessToken, EmailApiError, RateLimitRefusal, SendRequest, SentIds, TokenFreshness,+ AccessToken, EmailApiError, RateLimitRefusal, SendRequest, SentIds, TokenError, TokenFreshness,};use super::super::super::ports::{MailboxSendClient, MailboxSendRecoveryClient};-use super::super::test_support::{Call, FakeRateLimiter, FakeTokenSource, call_log};+use super::super::test_support::{Call, FakeGate, FakeTokenSource, call_log};use super::EmailApiClientServiceImpl;#[derive(Clone)]@@ -74,14 +74,14 @@ impl MailboxSendRecoveryClient for SendClient {}#[tokio::test]-async fn preparation_never_dispatches_and_dispatch_does_not_repeat_preparation() {+async fn preparation_holds_admission_until_the_single_dispatch_releases_it() {let calls = call_log();let service = EmailApiClientServiceImpl::new(SendClient {calls: calls.clone(),},FakeTokenSource::new(calls.clone(), Ok(AccessToken::new("access-token"))),- FakeRateLimiter::new(calls.clone(), Ok(())),+ FakeGate::new(calls.clone(), Ok(())),);let mut request = send_request();request.message_id = Some("[email protected]".into());@@ -92,13 +92,72 @@ async fn preparation_never_dispatches_and_dispatch_does_not_repeat_preparation()assert_eq!(*calls.lock().unwrap(),[- Call::RateLimit(Uuid::nil(), super::ApiOperationKind::SendMessage),+ Call::Acquire(Uuid::nil(), super::ApiOperationKind::SendMessage),Call::Token(Uuid::nil(), TokenFreshness::Cached),]);- service.send_prepared(&prepared).await.unwrap();- assert_eq!(calls.lock().unwrap().len(), 3);- assert_eq!(calls.lock().unwrap()[2], Call::Repository("send_prepared"));+ service.send_prepared(prepared).await.unwrap();+ assert_eq!(+ calls.lock().unwrap()[2..],+ [+ Call::Repository("send_prepared"),+ Call::Finish(Uuid::nil(), None),+ ]+ );+}++#[tokio::test]+async fn a_prepared_send_dropped_before_dispatch_releases_its_admission() {+ let calls = call_log();+ let service = EmailApiClientServiceImpl::new(+ SendClient {+ calls: calls.clone(),+ },+ FakeTokenSource::new(calls.clone(), Ok(AccessToken::new("access-token"))),+ FakeGate::new(calls.clone(), Ok(())),+ );+ let prepared = service+ .prepare_send(Uuid::nil(), &send_request(), Some("thread-1"))+ .await+ .unwrap();++ // The delivery coordinator lost its claim before submission.+ drop(prepared);++ assert_eq!(+ *calls.lock().unwrap(),+ [+ Call::Acquire(Uuid::nil(), super::ApiOperationKind::SendMessage),+ Call::Token(Uuid::nil(), TokenFreshness::Cached),+ Call::Abandon(Uuid::nil()),+ ]+ );+}++#[tokio::test]+async fn token_failure_during_preparation_releases_admission() {+ let calls = call_log();+ let service = EmailApiClientServiceImpl::new(+ SendClient {+ calls: calls.clone(),+ },+ FakeTokenSource::new(calls.clone(), Err(TokenError::ReauthRequired)),+ FakeGate::new(calls.clone(), Ok(())),+ );+ let error = service+ .prepare_send(Uuid::nil(), &send_request(), None)+ .await+ .err()+ .unwrap();+ assert_eq!(error, EmailApiError::AuthRequired);+ assert_eq!(+ *calls.lock().unwrap(),+ [+ Call::Acquire(Uuid::nil(), super::ApiOperationKind::SendMessage),+ Call::Token(Uuid::nil(), TokenFreshness::Cached),+ Call::Finish(Uuid::nil(), None),+ ]+ );}#[tokio::test]@@ -109,7 +168,7 @@ async fn quota_refusal_is_a_preparation_error_before_token_or_dispatch() {calls: calls.clone(),},FakeTokenSource::new(calls.clone(), Ok(AccessToken::new("access-token"))),- FakeRateLimiter::new(calls.clone(), Err(RateLimitRefusal { retry_after: None })),+ FakeGate::new(calls.clone(), Err(RateLimitRefusal { retry_after: None })),);let error = service.prepare_send(Uuid::nil(), &send_request(), None)@@ -119,7 +178,7 @@ async fn quota_refusal_is_a_preparation_error_before_token_or_dispatch() {assert!(matches!(error, EmailApiError::RateLimited { .. }));assert_eq!(*calls.lock().unwrap(),- [Call::RateLimit(+ [Call::Acquire(Uuid::nil(),super::ApiOperationKind::SendMessage)]@@ -134,7 +193,7 @@ async fn recovery_uses_read_quota_and_does_not_send() {calls: calls.clone(),},FakeTokenSource::new(calls.clone(), Ok(AccessToken::new("access-token"))),- FakeRateLimiter::new(calls.clone(), Ok(())),+ FakeGate::new(calls.clone(), Ok(())),);let sent = service.find_sent_message(Uuid::nil(), "[email protected]")@@ -145,9 +204,10 @@ async fn recovery_uses_read_quota_and_does_not_send() {assert_eq!(*calls.lock().unwrap(),[- Call::RateLimit(Uuid::nil(), super::ApiOperationKind::ListMessages),+ Call::Acquire(Uuid::nil(), super::ApiOperationKind::ListMessages),Call::Token(Uuid::nil(), TokenFreshness::Cached),Call::Repository("find_sent_message"),+ Call::Finish(Uuid::nil(), None),]);}@@ -160,7 +220,7 @@ async fn send_uses_send_cost_and_returns_ids_without_mutating_message() {calls: calls.clone(),},FakeTokenSource::new(calls.clone(), Ok(AccessToken::new("access-token"))),- FakeRateLimiter::new(calls.clone(), Ok(())),+ FakeGate::new(calls.clone(), Ok(())),);let request = send_request();@@ -177,9 +237,10 @@ async fn send_uses_send_cost_and_returns_ids_without_mutating_message() {assert_eq!(*calls.lock().unwrap(),vec![- Call::RateLimit(Uuid::nil(), super::ApiOperationKind::SendMessage),+ Call::Acquire(Uuid::nil(), super::ApiOperationKind::SendMessage),Call::Token(Uuid::nil(), TokenFreshness::Cached),Call::Repository("send_message"),+ Call::Finish(Uuid::nil(), None),]);}diff --git a/crates/email_api_client/src/domain/service/subscription.rs b/crates/email_api_client/src/domain/service/subscription.rsindex ec95ee3c78..d55bf2e4a1 100644--- a/crates/email_api_client/src/domain/service/subscription.rs+++ b/crates/email_api_client/src/domain/service/subscription.rs@@ -2,60 +2,69 @@ use models_email::service::link::Link;use uuid::Uuid;use super::super::models::{EmailApiError, ProviderSubscription, TokenFreshness};-use super::super::ports::{MailboxSubscriptionClient, ProviderRateLimiter, ProviderTokenSource};+use super::super::ports::{MailboxSubscriptionClient, ProviderRequestGate, ProviderTokenSource};use super::{ApiOperationKind, EmailApiClientServiceImpl, log_operation_error};impl<R, T, L> EmailApiClientServiceImpl<R, T, L>whereR: MailboxSubscriptionClient,T: ProviderTokenSource,- L: ProviderRateLimiter,+ L: ProviderRequestGate,{/// Registers or renews the linked mailbox's notification subscription.- #[tracing::instrument(skip(self))]+ #[tracing::instrument(skip(self), fields(gate_in_flight = tracing::field::Empty, gate_cap = tracing::field::Empty))]pub async fn register_subscription(&self,link_id: Uuid,) -> Result<ProviderSubscription, EmailApiError> {- let access_token = self.prepare(link_id, ApiOperationKind::Subscribe).await?;- self.repository- .subscribe(&access_token)- .await- .inspect_err(log_operation_error)+ self.call_with_token(link_id, ApiOperationKind::Subscribe, |token| async move {+ self.repository+ .subscribe(&token)+ .await+ .inspect_err(log_operation_error)+ })+ .await}/// Registers a subscription using a freshly fetched token.////// This is intended for mailbox initialization, where a cached access token/// could outlive a revoked provider grant.- #[tracing::instrument(skip(self, link), fields(link_id = %link.id))]+ #[tracing::instrument(skip(self, link), fields(link_id = %link.id, gate_in_flight = tracing::field::Empty, gate_cap = tracing::field::Empty))]pub async fn register_subscription_without_cache(&self,link: &Link,) -> Result<ProviderSubscription, EmailApiError> {- self.check_rate_limit(link.id, ApiOperationKind::Subscribe)- .await?;- let access_token = self- .token_source- .get_access_token_for_link(link, TokenFreshness::Fresh)- .await- .map_err(super::map_token_error)- .inspect_err(log_operation_error)?;-- self.repository- .subscribe(&access_token)- .await- .inspect_err(log_operation_error)+ let token = async {+ self.token_source+ .get_access_token_for_link(link, TokenFreshness::Fresh)+ .await+ .map_err(super::map_token_error)+ };+ self.attempt(+ link.id,+ ApiOperationKind::Subscribe,+ token,+ |token| async move {+ self.repository+ .subscribe(&token)+ .await+ .inspect_err(log_operation_error)+ },+ )+ .await?}/// Stops the linked mailbox's notification subscription.- #[tracing::instrument(skip(self))]+ #[tracing::instrument(skip(self), fields(gate_in_flight = tracing::field::Empty, gate_cap = tracing::field::Empty))]pub async fn stop_subscription(&self, link_id: Uuid) -> Result<(), EmailApiError> {- let access_token = self.prepare(link_id, ApiOperationKind::Unsubscribe).await?;- self.repository- .unsubscribe(&access_token)- .await- .inspect_err(log_operation_error)+ self.call_with_token(link_id, ApiOperationKind::Unsubscribe, |token| async move {+ self.repository+ .unsubscribe(&token)+ .await+ .inspect_err(log_operation_error)+ })+ .await}/// Stops the mailbox's subscription without token-health side effects.@@ -63,21 +72,26 @@ where/// This is intended for link teardown: a grant that died since the last/// probe must not set `needs_reauth` or fan out a reauthorization/// notification for an inbox that is being intentionally removed.- #[tracing::instrument(skip(self, link), fields(link_id = %link.id))]+ #[tracing::instrument(skip(self, link), fields(link_id = %link.id, gate_in_flight = tracing::field::Empty, gate_cap = tracing::field::Empty))]pub async fn stop_subscription_for_link(&self, link: &Link) -> Result<(), EmailApiError> {- self.check_rate_limit(link.id, ApiOperationKind::Unsubscribe)- .await?;- let access_token = self- .token_source- .get_access_token_health_neutral(link, TokenFreshness::Cached)- .await- .map_err(super::map_token_error)- .inspect_err(log_operation_error)?;-- self.repository- .unsubscribe(&access_token)- .await- .inspect_err(log_operation_error)+ let token = async {+ self.token_source+ .get_access_token_health_neutral(link, TokenFreshness::Cached)+ .await+ .map_err(super::map_token_error)+ };+ self.attempt(+ link.id,+ ApiOperationKind::Unsubscribe,+ token,+ |token| async move {+ self.repository+ .unsubscribe(&token)+ .await+ .inspect_err(log_operation_error)+ },+ )+ .await?}}diff --git a/crates/email_api_client/src/domain/service/subscription/test.rs b/crates/email_api_client/src/domain/service/subscription/test.rsindex 037669385d..a47818c2b4 100644--- a/crates/email_api_client/src/domain/service/subscription/test.rs+++ b/crates/email_api_client/src/domain/service/subscription/test.rs@@ -6,7 +6,7 @@ use super::super::super::models::{AccessToken, EmailApiError, ProviderSubscription, SyncCursor, TokenFreshness,};use super::super::super::ports::MailboxSubscriptionClient;-use super::super::test_support::{Call, FakeRateLimiter, FakeTokenSource, call_log};+use super::super::test_support::{Call, FakeGate, FakeTokenSource, call_log};use super::EmailApiClientServiceImpl;#[derive(Clone)]@@ -54,9 +54,10 @@ async fn registration_and_stop_use_matching_costs_and_operations() {assert_eq!(*calls.lock().unwrap(),vec![- Call::RateLimit(Uuid::nil(), operation),+ Call::Acquire(Uuid::nil(), operation),Call::Token(Uuid::nil(), TokenFreshness::Cached),Call::Repository(repository_call),+ Call::Finish(Uuid::nil(), None),]);}@@ -75,9 +76,10 @@ async fn registration_can_bypass_the_token_cache_for_initialization() {assert_eq!(*calls.lock().unwrap(),vec![- Call::RateLimit(Uuid::nil(), super::ApiOperationKind::Subscribe),+ Call::Acquire(Uuid::nil(), super::ApiOperationKind::Subscribe),Call::Token(Uuid::nil(), TokenFreshness::Fresh),Call::Repository("subscribe"),+ Call::Finish(Uuid::nil(), None),]);}@@ -92,9 +94,10 @@ async fn teardown_stop_uses_the_health_neutral_token_path() {assert_eq!(*calls.lock().unwrap(),vec![- Call::RateLimit(Uuid::nil(), super::ApiOperationKind::Unsubscribe),+ Call::Acquire(Uuid::nil(), super::ApiOperationKind::Unsubscribe),Call::HealthNeutralToken(Uuid::nil(), TokenFreshness::Cached),Call::Repository("unsubscribe"),+ Call::Finish(Uuid::nil(), None),]);}@@ -117,12 +120,12 @@ fn link() -> Link {fn service(calls: super::super::test_support::CallLog,-) -> EmailApiClientServiceImpl<SubscriptionClient, FakeTokenSource, FakeRateLimiter> {+) -> EmailApiClientServiceImpl<SubscriptionClient, FakeTokenSource, FakeGate> {EmailApiClientServiceImpl::new(SubscriptionClient {calls: calls.clone(),},FakeTokenSource::new(calls.clone(), Ok(AccessToken::new("access-token"))),- FakeRateLimiter::new(calls, Ok(())),+ FakeGate::new(calls, Ok(())),)}diff --git a/crates/email_api_client/src/domain/service/sync.rs b/crates/email_api_client/src/domain/service/sync.rsindex 51ebf593bb..b5e5984a81 100644--- a/crates/email_api_client/src/domain/service/sync.rs+++ b/crates/email_api_client/src/domain/service/sync.rs@@ -1,28 +1,26 @@use uuid::Uuid;use super::super::models::{ChangeBatch, EmailApiError, SyncCursor};-use super::super::ports::{MailboxSyncClient, ProviderRateLimiter, ProviderTokenSource};-use super::{ApiOperationKind, EmailApiClientServiceImpl, log_operation_error};+use super::super::ports::{MailboxSyncClient, ProviderRequestGate, ProviderTokenSource};+use super::{ApiOperationKind, EmailApiClientServiceImpl};impl<R, T, L> EmailApiClientServiceImpl<R, T, L>whereR: MailboxSyncClient,T: ProviderTokenSource,- L: ProviderRateLimiter,+ L: ProviderRequestGate,{/// Returns the number of threads reported by a linked mailbox.- #[tracing::instrument(skip(self))]+ #[tracing::instrument(skip(self), fields(retry_count = 0, gate_in_flight = tracing::field::Empty, gate_cap = tracing::field::Empty))]pub async fn get_thread_count(&self, link_id: Uuid) -> Result<u64, EmailApiError> {- let access_token = self.prepare(link_id, ApiOperationKind::GetProfile).await?;-- self.repository- .get_thread_count(&access_token)- .await- .inspect_err(log_operation_error)+ self.read_with_token(link_id, ApiOperationKind::GetProfile, |token| async move {+ self.repository.get_thread_count(&token).await+ })+ .await}/// Lists mailbox changes following `cursor`.- #[tracing::instrument(skip(self), fields(retry_count = 0))]+ #[tracing::instrument(skip(self), fields(retry_count = 0, gate_in_flight = tracing::field::Empty, gate_cap = tracing::field::Empty))]pub async fn list_changes(&self,link_id: Uuid,diff --git a/crates/email_api_client/src/domain/service/sync/test.rs b/crates/email_api_client/src/domain/service/sync/test.rsindex 57feb42077..55523a4a5c 100644--- a/crates/email_api_client/src/domain/service/sync/test.rs+++ b/crates/email_api_client/src/domain/service/sync/test.rs@@ -2,7 +2,7 @@ use std::sync::{Arc, Mutex};use uuid::Uuid;-use super::super::test_support::{Call, FakeRateLimiter, FakeTokenSource, call_log};+use super::super::test_support::{Call, FakeGate, FakeTokenSource, call_log};use super::*;use crate::domain::models::{AccessToken, InboxChanges, TokenFreshness};@@ -34,11 +34,11 @@ impl MailboxSyncClient for SyncClient {fn service(repository: SyncClient,calls: &super::super::test_support::CallLog,-) -> EmailApiClientServiceImpl<SyncClient, FakeTokenSource, FakeRateLimiter> {+) -> EmailApiClientServiceImpl<SyncClient, FakeTokenSource, FakeGate> {EmailApiClientServiceImpl::new(repository,FakeTokenSource::new(calls.clone(), Ok(AccessToken::new("token"))),- FakeRateLimiter::new(calls.clone(), Ok(())),+ FakeGate::new(calls.clone(), Ok(())),)}@@ -56,8 +56,9 @@ async fn get_thread_count_uses_profile_quota_and_forwards_link_id() {assert_eq!(*calls.lock().unwrap(),vec![- Call::RateLimit(link_id, ApiOperationKind::GetProfile),+ Call::Acquire(link_id, ApiOperationKind::GetProfile),Call::Token(link_id, TokenFreshness::Cached),+ Call::Finish(link_id, None),]);}@@ -79,8 +80,9 @@ async fn list_changes_uses_changes_quota_and_forwards_cursor() {assert_eq!(*calls.lock().unwrap(),vec![- Call::RateLimit(link_id, ApiOperationKind::ListChanges),+ Call::Acquire(link_id, ApiOperationKind::ListChanges),Call::Token(link_id, TokenFreshness::Cached),+ Call::Finish(link_id, None),]);}diff --git a/crates/email_api_client/src/domain/service/test.rs b/crates/email_api_client/src/domain/service/test.rsindex df81300114..3a2f937652 100644--- a/crates/email_api_client/src/domain/service/test.rs+++ b/crates/email_api_client/src/domain/service/test.rs@@ -1,15 +1,17 @@use std::sync::{Arc, Mutex};+use std::task::{Context, Waker};use std::time::Duration;use models_email::email::service::label::Label;use uuid::Uuid;use super::super::models::{- AccessToken, ApiOperationKind, EmailApiError, RateLimitRefusal, TokenError, TokenFreshness,+ AccessToken, ApiOperationKind, EmailApiError, RateLimitOrigin, RateLimitRefusal, TokenError,+ TokenFreshness,};use super::super::ports::MailboxLabelClient;use super::EmailApiClientServiceImpl;-use super::test_support::{Call, FakeRateLimiter, FakeRepository, FakeTokenSource, call_log};+use super::test_support::{Call, FakeGate, FakeRepository, FakeTokenSource, call_log};/// Rejects the listed tokens the way Gmail answers a 401.#[derive(Clone, Default)]@@ -29,6 +31,31 @@ impl MailboxLabelClient for TokenCheckingLabels {}}+ async fn create_label(+ &self,+ _: &AccessToken,+ _: Uuid,+ _: &str,+ ) -> Result<Label, EmailApiError> {+ Err(EmailApiError::RateLimited {+ retry_after: None,+ origin: RateLimitOrigin::Provider { concurrency: false },+ })+ }++ async fn delete_label(&self, _: &AccessToken, _: &str) -> Result<(), EmailApiError> {+ unreachable!("not used by these tests")+ }+}++/// Never answers, like a Gmail request still in flight.+struct UnansweredLabels;++impl MailboxLabelClient for UnansweredLabels {+ async fn list_labels(&self, _: &AccessToken, _: Uuid) -> Result<Vec<Label>, EmailApiError> {+ std::future::pending().await+ }+async fn create_label(&self,_: &AccessToken,@@ -43,6 +70,31 @@ impl MailboxLabelClient for TokenCheckingLabels {}}+#[tokio::test]+async fn a_caller_cancelled_mid_request_releases_its_admission() {+ let calls = call_log();+ let service = EmailApiClientServiceImpl::new(+ UnansweredLabels,+ FakeTokenSource::new(calls.clone(), Ok(AccessToken::new("access-token"))),+ FakeGate::new(calls.clone(), Ok(())),+ );+ let mut read = Box::pin(service.list_labels(Uuid::nil()));+ let poll = read.as_mut().poll(&mut Context::from_waker(Waker::noop()));+ assert!(poll.is_pending());++ // An HTTP client disconnected, or a lease-keeping select! gave up.+ drop(read);++ assert_eq!(+ *calls.lock().unwrap(),+ vec![+ Call::Acquire(Uuid::nil(), ApiOperationKind::ListLabels),+ Call::Token(Uuid::nil(), TokenFreshness::Cached),+ Call::Abandon(Uuid::nil()),+ ]+ );+}+async fn list_labels_with(rejected: Vec<&'static str>,fresh: Result<AccessToken, TokenError>,@@ -56,7 +108,7 @@ async fn list_labels_with(let service = EmailApiClientServiceImpl::new(repository,FakeTokenSource::new(calls.clone(), Ok(AccessToken::new("cached"))).with_fresh(fresh),- FakeRateLimiter::new(calls.clone(), Ok(())),+ FakeGate::new(calls.clone(), Ok(())),);let result = service@@ -68,22 +120,23 @@ async fn list_labels_with((result, calls, seen)}-fn retried_calls() -> Vec<Call> {- vec![- Call::RateLimit(Uuid::nil(), ApiOperationKind::ListLabels),- Call::Token(Uuid::nil(), TokenFreshness::Cached),- Call::RateLimit(Uuid::nil(), ApiOperationKind::ListLabels),- Call::Token(Uuid::nil(), TokenFreshness::Fresh),- ]-}-#[tokio::test]async fn a_rejected_cached_token_is_retried_with_a_fresh_one() {let (result, calls, seen) =list_labels_with(vec!["cached"], Ok(AccessToken::new("fresh"))).await;assert_eq!(result, Ok(0));- assert_eq!(calls, retried_calls());+ assert_eq!(+ calls,+ vec![+ Call::Acquire(Uuid::nil(), ApiOperationKind::ListLabels),+ Call::Token(Uuid::nil(), TokenFreshness::Cached),+ Call::Finish(Uuid::nil(), Some(EmailApiError::AuthRequired)),+ Call::Acquire(Uuid::nil(), ApiOperationKind::ListLabels),+ Call::Token(Uuid::nil(), TokenFreshness::Fresh),+ Call::Finish(Uuid::nil(), None),+ ]+ );assert_eq!(seen, ["cached", "fresh"]);}@@ -93,7 +146,17 @@ async fn a_revoked_grant_found_by_the_fresh_fetch_stops_the_read() {list_labels_with(vec!["cached"], Err(TokenError::ReauthRequired)).await;assert_eq!(result, Err(EmailApiError::AuthRequired));- assert_eq!(calls, retried_calls());+ assert_eq!(+ calls,+ vec![+ Call::Acquire(Uuid::nil(), ApiOperationKind::ListLabels),+ Call::Token(Uuid::nil(), TokenFreshness::Cached),+ Call::Finish(Uuid::nil(), Some(EmailApiError::AuthRequired)),+ Call::Acquire(Uuid::nil(), ApiOperationKind::ListLabels),+ Call::Token(Uuid::nil(), TokenFreshness::Fresh),+ Call::Finish(Uuid::nil(), None),+ ]+ );assert_eq!(seen, ["cached"]);}@@ -103,69 +166,104 @@ async fn a_rejected_fresh_token_is_not_retried_again() {list_labels_with(vec!["cached", "fresh"], Ok(AccessToken::new("fresh"))).await;assert_eq!(result, Err(EmailApiError::AuthRequired));- assert_eq!(calls, retried_calls());+ assert_eq!(+ calls,+ vec![+ Call::Acquire(Uuid::nil(), ApiOperationKind::ListLabels),+ Call::Token(Uuid::nil(), TokenFreshness::Cached),+ Call::Finish(Uuid::nil(), Some(EmailApiError::AuthRequired)),+ Call::Acquire(Uuid::nil(), ApiOperationKind::ListLabels),+ Call::Token(Uuid::nil(), TokenFreshness::Fresh),+ Call::Finish(Uuid::nil(), Some(EmailApiError::AuthRequired)),+ ]+ );assert_eq!(seen, ["cached", "fresh"]);}#[tokio::test]-async fn token_failure_stops_before_repository_after_the_quota_check() {+async fn token_failure_releases_admission_without_calling_the_provider() {let calls = call_log();let service = EmailApiClientServiceImpl::new(FakeRepository::new(calls.clone()),FakeTokenSource::new(calls.clone(), Err(TokenError::ReauthRequired)),- FakeRateLimiter::new(calls.clone(), Ok(())),+ FakeGate::new(calls.clone(), Ok(())),);let result = service- .prepare(Uuid::nil(), ApiOperationKind::GetMessage)+ .modify_message_labels(Uuid::nil(), "message-1", &[], &["INBOX".to_string()]).await;assert_eq!(result, Err(EmailApiError::AuthRequired));assert_eq!(*calls.lock().unwrap(),vec![- Call::RateLimit(Uuid::nil(), ApiOperationKind::GetMessage),+ Call::Acquire(Uuid::nil(), ApiOperationKind::ModifyMessageLabels),Call::Token(Uuid::nil(), TokenFreshness::Cached),+ Call::Finish(Uuid::nil(), None),]);}#[tokio::test]-async fn rate_limit_refusal_stops_before_the_token_dance_and_repository() {+async fn admission_refusal_stops_before_the_token_dance_and_repository() {let calls = call_log();let retry_after = Duration::from_secs(17);let service = EmailApiClientServiceImpl::new(FakeRepository::new(calls.clone()),FakeTokenSource::new(calls.clone(), Ok(AccessToken::new("access-token"))),- FakeRateLimiter::new(calls.clone(), Err(RateLimitRefusal::new(Some(retry_after)))),+ FakeGate::new(calls.clone(), Err(RateLimitRefusal::new(Some(retry_after)))),);- let result = service- .prepare(Uuid::nil(), ApiOperationKind::SendMessage)- .await;+ let result = service.get_thread_count(Uuid::nil()).await;assert_eq!(result,Err(EmailApiError::RateLimited {retry_after: Some(retry_after),- origin: crate::domain::models::RateLimitOrigin::Local,+ origin: RateLimitOrigin::Local,}));- // A refused attempt must not pay the token acquisition (SELECT + Redis +- // possible auth-service refresh + health write).+ // A refused attempt holds no permit and must not pay the token acquisition+ // (SELECT + Redis + possible auth-service refresh + health write).assert_eq!(*calls.lock().unwrap(),- vec![Call::RateLimit(Uuid::nil(), ApiOperationKind::SendMessage)]+ vec![Call::Acquire(Uuid::nil(), ApiOperationKind::GetProfile)]+ );+}++#[tokio::test]+async fn a_provider_rate_limit_is_reported_to_the_gate_when_releasing_admission() {+ let calls = call_log();+ let service = EmailApiClientServiceImpl::new(+ TokenCheckingLabels::default(),+ FakeTokenSource::new(calls.clone(), Ok(AccessToken::new("access-token"))),+ FakeGate::new(calls.clone(), Ok(())),+ );+ let throttled = EmailApiError::RateLimited {+ retry_after: None,+ origin: RateLimitOrigin::Provider { concurrency: false },+ };++ let result = service.create_label(Uuid::nil(), "Receipts").await;++ assert_eq!(result.err(), Some(throttled.clone()));+ assert_eq!(+ *calls.lock().unwrap(),+ vec![+ Call::Acquire(Uuid::nil(), ApiOperationKind::CreateLabel),+ Call::Token(Uuid::nil(), TokenFreshness::Cached),+ Call::Finish(Uuid::nil(), Some(throttled)),+ ]);}#[tokio::test]-async fn explicit_token_probe_honors_requested_freshness_without_quota_check() {+async fn explicit_token_probe_honors_requested_freshness_without_admission() {let calls = call_log();let service = EmailApiClientServiceImpl::new(FakeRepository::new(calls.clone()),FakeTokenSource::new(calls.clone(), Ok(AccessToken::new("access-token"))),- FakeRateLimiter::new(calls.clone(), Ok(())),+ FakeGate::new(calls.clone(), Ok(())),);let result = service@@ -203,7 +301,7 @@ async fn token_errors_preserve_transient_and_permanent_classification() {let service = EmailApiClientServiceImpl::new(FakeRepository::new(calls.clone()),FakeTokenSource::new(calls.clone(), Err(token_error)),- FakeRateLimiter::new(calls, Ok(())),+ FakeGate::new(calls, Ok(())),);assert_eq!(diff --git a/crates/email_api_client/src/domain/service/test_support.rs b/crates/email_api_client/src/domain/service/test_support.rsindex ad12f44305..d7666b3582 100644--- a/crates/email_api_client/src/domain/service/test_support.rs+++ b/crates/email_api_client/src/domain/service/test_support.rs@@ -7,19 +7,22 @@ use uuid::Uuid;use super::super::models::{AccessToken, ApiOperationKind, ChangeBatch, EmailApiError, ProviderSubscription,- RateLimitRefusal, SendRequest, SentIds, SyncCursor, ThreadListPage, TokenError, TokenFreshness,+ RateLimitRefusal, RequestPermit, SendRequest, SentIds, SyncCursor, ThreadListPage, TokenError,+ TokenFreshness,};use super::super::ports::{MailboxAttachmentClient, MailboxBlocklistClient, MailboxContactsClient, MailboxLabelClient,MailboxMessageClient, MailboxSendClient, MailboxSubscriptionClient, MailboxSyncClient,- ProviderRateLimiter, ProviderTokenSource,+ ProviderRequestGate, ProviderTokenSource,};#[derive(Debug, Clone, PartialEq, Eq)]pub(super) enum Call {Token(Uuid, TokenFreshness),HealthNeutralToken(Uuid, TokenFreshness),- RateLimit(Uuid, ApiOperationKind),+ Acquire(Uuid, ApiOperationKind),+ Finish(Uuid, Option<EmailApiError>),+ Abandon(Uuid),Repository(&'static str),}@@ -94,28 +97,41 @@ impl ProviderTokenSource for FakeTokenSource {}#[derive(Clone)]-pub(super) struct FakeRateLimiter {+pub(super) struct FakeGate {calls: CallLog,result: Result<(), RateLimitRefusal>,}-impl FakeRateLimiter {+impl FakeGate {pub(super) fn new(calls: CallLog, result: Result<(), RateLimitRefusal>) -> Self {Self { calls, result }}}-impl ProviderRateLimiter for FakeRateLimiter {- async fn check_rate_limit(+impl ProviderRequestGate for FakeGate {+ async fn acquire(&self,link_id: Uuid,operation: ApiOperationKind,- ) -> Result<(), RateLimitRefusal> {+ ) -> Result<RequestPermit, RateLimitRefusal> {self.calls.lock().unwrap()- .push(Call::RateLimit(link_id, operation));- self.result.clone()+ .push(Call::Acquire(link_id, operation));+ let calls = self.calls.clone();+ self.result.clone().map(|()| {+ RequestPermit::new(link_id, Uuid::nil(), move |link_id, _| {+ calls.lock().unwrap().push(Call::Abandon(link_id));+ })+ })+ }++ async fn finish(&self, permit: RequestPermit, error: Option<&EmailApiError>) {+ self.calls+ .lock()+ .unwrap()+ .push(Call::Finish(permit.link_id(), error.cloned()));+ permit.disarm();}}diff --git a/crates/email_api_client/src/outbound/gmail.rs b/crates/email_api_client/src/outbound/gmail.rsindex 2e53e51fff..3a0cd55315 100644--- a/crates/email_api_client/src/outbound/gmail.rs+++ b/crates/email_api_client/src/outbound/gmail.rs@@ -81,6 +81,10 @@ fn is_quota_403_body(body: &str) -> bool {QUOTA_403_REASONS.iter().any(|reason| body.contains(reason))}+/// Gmail's per-second 429 shares this one's `rateLimitExceeded` reason; only+/// the message tells the concurrency limit apart.+const CONCURRENCY_429_MESSAGE: &str = "Too many concurrent requests for user";+pub(crate) fn map_gmail_error(error: GmailApiHttpError) -> EmailApiError {match error {GmailApiHttpError::Http {@@ -91,14 +95,16 @@ pub(crate) fn map_gmail_error(error: GmailApiHttpError) -> EmailApiError {401 => EmailApiError::AuthRequired,403 if is_quota_403_body(&body) => EmailApiError::RateLimited {retry_after,- origin: RateLimitOrigin::Provider,+ origin: RateLimitOrigin::Provider { concurrency: false },},403 => EmailApiError::Forbidden,404 => EmailApiError::NotFound,409 => EmailApiError::Conflict,429 => EmailApiError::RateLimited {retry_after,- origin: RateLimitOrigin::Provider,+ origin: RateLimitOrigin::Provider {+ concurrency: body.contains(CONCURRENCY_429_MESSAGE),+ },},500..=599 => EmailApiError::Transient {message: format!("Gmail API returned {status}: {body}"),diff --git a/crates/email_api_client/src/outbound/gmail/test/error.rs b/crates/email_api_client/src/outbound/gmail/test/error.rsindex 3b26196c4e..c3e8acc5f9 100644--- a/crates/email_api_client/src/outbound/gmail/test/error.rs+++ b/crates/email_api_client/src/outbound/gmail/test/error.rs@@ -49,7 +49,33 @@ fn maps_provider_statuses_centrally() {)),EmailApiError::RateLimited {retry_after: Some(retry_after),- origin: RateLimitOrigin::Provider,+ origin: RateLimitOrigin::Provider { concurrency: false },+ }+ );+}++#[test]+fn only_the_concurrent_requests_429_is_a_concurrency_limit() {+ assert_eq!(+ map_gmail_error(http_error(+ StatusCode::TOO_MANY_REQUESTS,+ r#"{"error":{"code":429,"message":"Too many concurrent requests for user.","errors":[{"reason":"rateLimitExceeded"}]}}"#,+ None,+ )),+ EmailApiError::RateLimited {+ retry_after: None,+ origin: RateLimitOrigin::Provider { concurrency: true },+ }+ );+ assert_eq!(+ map_gmail_error(http_error(+ StatusCode::TOO_MANY_REQUESTS,+ r#"{"error":{"code":429,"message":"User-rate limit exceeded. Retry after 2026-10-25T10:00:00Z","errors":[{"reason":"rateLimitExceeded"}]}}"#,+ None,+ )),+ EmailApiError::RateLimited {+ retry_after: None,+ origin: RateLimitOrigin::Provider { concurrency: false },});}@@ -64,7 +90,7 @@ fn quota_403s_are_rate_limited_and_plain_403s_stay_forbidden() {)),EmailApiError::RateLimited {retry_after: None,- origin: RateLimitOrigin::Provider,+ origin: RateLimitOrigin::Provider { concurrency: false },});@@ -77,7 +103,7 @@ fn quota_403s_are_rate_limited_and_plain_403s_stay_forbidden() {)),EmailApiError::RateLimited {retry_after: Some(retry_after),- origin: RateLimitOrigin::Provider,+ origin: RateLimitOrigin::Provider { concurrency: false },});@@ -90,7 +116,7 @@ fn quota_403s_are_rate_limited_and_plain_403s_stay_forbidden() {)),EmailApiError::RateLimited {retry_after: None,- origin: RateLimitOrigin::Provider,+ origin: RateLimitOrigin::Provider { concurrency: false },},"reason {reason} should map to RateLimited",);diff --git a/crates/email_api_client/src/outbound/outlook/test.rs b/crates/email_api_client/src/outbound/outlook/test.rsindex a492bfc112..e7ec18fe5d 100644--- a/crates/email_api_client/src/outbound/outlook/test.rs+++ b/crates/email_api_client/src/outbound/outlook/test.rs@@ -163,7 +163,7 @@ async fn throttling_respects_provider_retry_after() {error,EmailApiError::RateLimited {retry_after: Some(Duration::from_secs(17)),- origin: RateLimitOrigin::Provider+ origin: RateLimitOrigin::Provider { concurrency: false }});}diff --git a/crates/email_api_client/src/outbound/outlook/transport.rs b/crates/email_api_client/src/outbound/outlook/transport.rsindex 74fe6a153f..664c01083d 100644--- a/crates/email_api_client/src/outbound/outlook/transport.rs+++ b/crates/email_api_client/src/outbound/outlook/transport.rs@@ -197,7 +197,7 @@ impl OutlookApiClientRepository {StatusCode::CONFLICT | StatusCode::PRECONDITION_FAILED => EmailApiError::Conflict,StatusCode::TOO_MANY_REQUESTS => EmailApiError::RateLimited {retry_after,- origin: RateLimitOrigin::Provider,+ origin: RateLimitOrigin::Provider { concurrency: false },},status if status.is_server_error() || status == StatusCode::REQUEST_TIMEOUT => {EmailApiError::Transient {diff --git a/crates/email_db_client/src/attachments/provider/upload.rs b/crates/email_db_client/src/attachments/provider/upload.rsindex 15314ce5c6..225a187691 100644--- a/crates/email_db_client/src/attachments/provider/upload.rs+++ b/crates/email_db_client/src/attachments/provider/upload.rs@@ -572,6 +572,19 @@ pub async fn new_email_media_atts(Ok(attachments)}+/// Releases an attachment's upload claim so the next claiming query picks it up again.+#[tracing::instrument(skip(db), err)]+pub async fn release_upload_claim(db: &Pool<Postgres>, attachment_id: Uuid) -> anyhow::Result<()> {+ sqlx::query!(+ "UPDATE email_attachments SET upload_claimed_at = NULL WHERE id = $1",+ attachment_id+ )+ .execute(db)+ .await?;++ Ok(())+}+pub async fn fetch_attachment_upload_metadata_by_id(db: &Pool<Postgres>,attachment_id: Uuid,diff --git a/crates/email_db_client/src/threads/insert.rs b/crates/email_db_client/src/threads/insert.rsindex 8a15d13cdb..4030d64236 100644--- a/crates/email_db_client/src/threads/insert.rs+++ b/crates/email_db_client/src/threads/insert.rs@@ -156,13 +156,14 @@ pub async fn insert_thread(Ok(existing_id)}-/// inserts a thread object into the database that has no metadata or rizz+/// Inserts a thread with no metadata, returning its id, or `None` when the+/// mailbox already has the thread, including when a concurrent insert won.#[tracing::instrument(skip(pool), err)]pub async fn insert_blank_thread(pool: &PgPool,thread_provider_id: &str,link_id: Uuid,-) -> anyhow::Result<Uuid> {+) -> anyhow::Result<Option<Uuid>> {let thread = thread::Thread {db_id: macro_uuid::generate_uuid_v7(),provider_id: Some(thread_provider_id.to_string()),@@ -180,5 +181,5 @@ pub async fn insert_blank_thread(let mut tx = pool.begin().await?;let id = insert_thread(&mut tx, &thread, link_id).await?;tx.commit().await?;- Ok(id)+ Ok((id == thread.db_id).then_some(id))}diff --git a/crates/email_db_client/src/threads/insert/test.rs b/crates/email_db_client/src/threads/insert/test.rsindex 2746eaadd6..42c6582ad5 100644--- a/crates/email_db_client/src/threads/insert/test.rs+++ b/crates/email_db_client/src/threads/insert/test.rs@@ -112,9 +112,9 @@ async fn blank_upsert_preserves_populated_timestamp(pool: Pool<Postgres>) -> any// A redelivered backfill blank insert must not wipe the timestamp or// rewrite the row.- let upserted_thread_id = insert_blank_thread(&pool, "prov1", link_id).await?;+ let inserted = insert_blank_thread(&pool, "prov1", link_id).await?;- assert_eq!(upserted_thread_id, existing_thread_id);+ assert_eq!(inserted, None, "the mailbox already had the thread");let after = get_thread_by_id_and_link_id(&pool, existing_thread_id, link_id).await?.expect("provider thread should still exist");@@ -137,7 +137,9 @@ async fn blank_upsert_preserves_populated_timestamp(pool: Pool<Postgres>) -> anyasync fn blank_insert_creates_new_thread(pool: Pool<Postgres>) -> anyhow::Result<()> {let link_id = Uuid::parse_str("00000000-0000-0000-0000-000000000001")?;- let thread_id = insert_blank_thread(&pool, "prov-brand-new", link_id).await?;+ let thread_id = insert_blank_thread(&pool, "prov-brand-new", link_id)+ .await?+ .expect("the thread is new");let thread = get_thread_by_id_and_link_id(&pool, thread_id, link_id).await?@@ -148,6 +150,28 @@ async fn blank_insert_creates_new_thread(pool: Pool<Postgres>) -> anyhow::ResultOk(())}+#[sqlx::test(+ migrator = "MACRO_DB_MIGRATIONS",+ fixtures(path = "../../../fixtures", scripts("links"))+)]+async fn only_one_of_two_concurrent_blank_inserts_creates_the_thread(+ pool: Pool<Postgres>,+) -> anyhow::Result<()> {+ let link_id = Uuid::parse_str("00000000-0000-0000-0000-000000000001")?;++ let (first, second) = tokio::join!(+ insert_blank_thread(&pool, "prov-raced", link_id),+ insert_blank_thread(&pool, "prov-raced", link_id)+ );++ assert_eq!(+ [first?, second?].iter().flatten().count(),+ 1,+ "only the winning insert backfills the thread's messages"+ );+ Ok(())+}+#[sqlx::test(migrator = "MACRO_DB_MIGRATIONS",fixtures(path = "../../../fixtures", scripts("links"))diff --git a/crates/gmail_client/AGENTS.md b/crates/gmail_client/AGENTS.mdindex 6edc3a93b9..137f5bad3b 100644--- a/crates/gmail_client/AGENTS.md+++ b/crates/gmail_client/AGENTS.md@@ -42,6 +42,9 @@ contacts, watch) behind a single public facade; most modules are- Any new error path should go through `sanitize_error_body` (or anequivalent) before logging — Gmail error bodies can contain email addresses.+- Requests time out after 60 s (`REQUEST_TIMEOUT`). The email service's+ Gmail request gate holds a permit for each call and sizes its lease above+ this (`PERMIT_MS` in `gmail_gate.rs`); keep the two in step.## Relateddiff --git a/crates/gmail_client/src/lib.rs b/crates/gmail_client/src/lib.rsindex 25efce93fd..685da92cb8 100644--- a/crates/gmail_client/src/lib.rs+++ b/crates/gmail_client/src/lib.rs@@ -16,8 +16,13 @@ mod test;use regex::Regex;use std::sync::LazyLock;+use std::time::Duration;const MAX_ERROR_BODY_LEN: usize = 1024;+// A stalled request fails instead of holding its worker, and its mailbox's+// request-gate permit, indefinitely. Callers size permit leases above this.+const REQUEST_TIMEOUT: Duration = Duration::from_secs(60);+const CONNECT_TIMEOUT: Duration = Duration::from_secs(10);static EMAIL_REGEX: LazyLock<Regex> =LazyLock::new(|| Regex::new(r"[a-zA-Z0-9._%+-]+@[a-zA-Z0-9.-]+\.[a-zA-Z]{2,}").unwrap());@@ -108,8 +113,14 @@ impl GmailClient {audience: String,) -> Self {Self {- inner: reqwest::Client::new(),+ inner: reqwest::Client::builder()+ .timeout(REQUEST_TIMEOUT)+ .connect_timeout(CONNECT_TIMEOUT)+ .build()+ .expect("Gmail HTTP client should initialize"),send_client: reqwest::Client::builder()+ .timeout(REQUEST_TIMEOUT)+ .connect_timeout(CONNECT_TIMEOUT).retry(reqwest::retry::never()).redirect(reqwest::redirect::Policy::none()).build()diff --git a/crates/sqs_client/src/email.rs b/crates/sqs_client/src/email.rsindex 2b9df57aa2..1306e11e70 100644--- a/crates/sqs_client/src/email.rs+++ b/crates/sqs_client/src/email.rs@@ -8,6 +8,10 @@ use models_email::service::pubsub::SFSUploaderMessage;use models_email::service::pubsub::ScheduledPubsubMessage;use uuid::Uuid;+/// Message attribute counting how often a backfill operation has been sent+/// back to its queue by [`SQS::defer_email_backfill_message`].+pub const BACKFILL_DEFERRALS_ATTRIBUTE: &str = "deferrals";+impl SQS {pub fn email_link_manager_queue(mut self, link_manager_queue: &str) -> Self {self.link_manager_queue = Some(link_manager_queue.to_string());@@ -66,6 +70,35 @@ impl SQS {Err(anyhow::anyhow!("email_backfill_queue is not configured"))}+ /// Sends a backfill operation back to the backfill queue as a new message+ /// that becomes visible after `delay_seconds`. Unlike waiting out the+ /// visibility timeout, the wait uses none of the operation's deliveries.+ /// `deferrals` is recorded in [`BACKFILL_DEFERRALS_ATTRIBUTE`].+ #[tracing::instrument(skip(self), err)]+ pub async fn defer_email_backfill_message(+ &self,+ message: &BackfillPubsubMessage,+ delay_seconds: i32,+ deferrals: u32,+ ) -> anyhow::Result<()> {+ let Some(email_backfill_queue) = &self.email_backfill_queue else {+ anyhow::bail!("email_backfill_queue is not configured");+ };+ let deferrals = aws_sdk_sqs::types::MessageAttributeValue::builder()+ .data_type("Number")+ .string_value(deferrals.to_string())+ .build()?;+ self.inner+ .send_message()+ .queue_url(email_backfill_queue)+ .message_body(serde_json::to_string(&message)?)+ .delay_seconds(delay_seconds)+ .message_attributes(BACKFILL_DEFERRALS_ATTRIBUTE, deferrals)+ .send()+ .await?;+ Ok(())+ }+/// Sends a message to the email crm cleanup queue#[tracing::instrument(skip(self), err)]pub async fn enqueue_email_crm_cleanup_message(diff --git a/crates/workspace-hack/Cargo.toml b/crates/workspace-hack/Cargo.tomlindex da66ea6e9a..6e869b1514 100644--- a/crates/workspace-hack/Cargo.toml+++ b/crates/workspace-hack/Cargo.toml@@ -120,7 +120,7 @@ prost = { version = "0.12", features = ["prost-derive"] }rand-274715c4dabd11b0 = { package = "rand", version = "0.9" }rand-c38e5c1d305a1b54 = { package = "rand", version = "0.8", features = ["small_rng"] }rand_core = { version = "0.6", default-features = false, features = ["std"] }-redis = { version = "1", features = ["cluster-async", "tokio-rustls-comp"] }+redis = { version = "1", features = ["cluster-async", "connection-manager", "tokio-rustls-comp"] }regex = { version = "1" }regex-automata = { version = "0.4" }regex-syntax = { version = "0.8" }@@ -284,7 +284,7 @@ quote = { version = "1" }rand-274715c4dabd11b0 = { package = "rand", version = "0.9" }rand-c38e5c1d305a1b54 = { package = "rand", version = "0.8", features = ["small_rng"] }rand_core = { version = "0.6", default-features = false, features = ["std"] }-redis = { version = "1", features = ["cluster-async", "tokio-rustls-comp"] }+redis = { version = "1", features = ["cluster-async", "connection-manager", "tokio-rustls-comp"] }regex = { version = "1" }regex-automata = { version = "0.4" }regex-syntax = { version = "0.8" }diff --git a/services/email_service/AGENTS.md b/services/email_service/AGENTS.mdindex 1dd25a9eb8..1eeb13a2c5 100644--- a/services/email_service/AGENTS.md+++ b/services/email_service/AGENTS.md@@ -67,6 +67,20 @@ the gateway `/email` prefix.calendar refresh so unrelated invitation updates retain teammates' OOO rows.- Spawn `run_transfers` regardless of the Outlook flags. Gmail inbox movesfinish there; with writes off it pauses only the Outlook retire step.+- Every `GmailApi` call takes a permit from `RedisGmailRequestGate`+ (`outbound/email_api/gmail_gate.rs`): per mailbox it caps requests in+ flight and quota units per window, and pauses after a Gmail 429 (live work+ for a second, backfill for at least 10 s). A "Too many concurrent requests"+ 429 also lowers that mailbox's in-flight cap, which then rises by one a+ minute back to five; backfill gets what the cap leaves above two. It fails+ closed, so a Redis outage surfaces as local rate limits. Consumers must+ requeue or redeliver a rate-limited message, never drop it. Up to twelve+ refused backfill requests per mailbox wait up to 20 s for room so backfill's+ slots don't idle; the rest are sent back as new, delayed messages+ (`pubsub/backfill/deferral.rs`), because waiting out the visibility timeout+ would spend the queue's 20 deliveries.+ Inbox sync releases the upload claims of a refused attachment fetch and+ requeues the upsert, because a claimed attachment is never offered again.- Header addresses that Gmail accepts can fail `MacroUserIdStr::try_from_email`.Skip the bad address with a WARN that omits it (the parse error echoes it)instead of failing the step; see `pubsub::util::build_contact_connections`.diff --git a/services/email_service/Cargo.toml b/services/email_service/Cargo.tomlindex 8eff8db927..edbc2cb25d 100644--- a/services/email_service/Cargo.toml+++ b/services/email_service/Cargo.toml@@ -144,8 +144,10 @@ models_pagination = { path = "../../crates/models_pagination", features = ["axummodels_permissions = { path = "../../crates/models_permissions" }once_cell = { workspace = true }models_properties = { path = "../../crates/models_properties" }+rand = { workspace = true }redis = { workspace = true, features = ["cluster-async",+ "connection-manager","tokio-rustls-comp",] }regex = { workspace = true }diff --git a/services/email_service/src/api/email/init.rs b/services/email_service/src/api/email/init.rsindex fb982885d1..eff3d7ae5a 100644--- a/services/email_service/src/api/email/init.rs+++ b/services/email_service/src/api/email/init.rs@@ -111,6 +111,10 @@ impl InitError {impl IntoResponse for InitError {fn into_response(self) -> Response {let status_code = self.status_code();+ let headers = match &self {+ InitError::ProviderError(error) => super::provider_error::provider_error_headers(error),+ _ => axum::http::HeaderMap::new(),+ };match self {InitError::SharedInboxConflict {@@ -142,7 +146,7 @@ impl IntoResponse for InitError {}),).into_response(),- other => (status_code, other.to_string()).into_response(),+ other => (status_code, headers, other.to_string()).into_response(),}}}diff --git a/services/email_service/src/api/email/init/test.rs b/services/email_service/src/api/email/init/test.rsindex 4d52361ea1..d16e2078c7 100644--- a/services/email_service/src/api/email/init/test.rs+++ b/services/email_service/src/api/email/init/test.rs@@ -99,11 +99,29 @@ fn unusable_provider_grants_classify_as_no_gmail_grant() {fn rate_limiting_classifies_as_http_429() {let error = classify_provider_init_error(EmailApiError::RateLimited {retry_after: None,- origin: email_api_client::domain::models::RateLimitOrigin::Provider,+ origin: email_api_client::domain::models::RateLimitOrigin::Provider { concurrency: false },});assert_eq!(error.status_code(), StatusCode::TOO_MANY_REQUESTS);}+#[test]+fn a_rate_limited_init_tells_the_client_when_to_retry() {+ let response = InitError::ProviderError(EmailApiError::RateLimited {+ retry_after: Some(std::time::Duration::from_millis(1_200)),+ origin: email_api_client::domain::models::RateLimitOrigin::Local,+ })+ .into_response();++ assert_eq!(response.status(), StatusCode::TOO_MANY_REQUESTS);+ assert_eq!(+ response+ .headers()+ .get(axum::http::header::RETRY_AFTER)+ .and_then(|value| value.to_str().ok()),+ Some("2")+ );+}+#[tokio::test]async fn provider_diagnostics_are_not_returned_to_clients() {let response = InitError::ProviderError(EmailApiError::Permanent {diff --git a/services/email_service/src/api/email/provider_error.rs b/services/email_service/src/api/email/provider_error.rsindex 4cbdb6b2f5..8c756b6d39 100644--- a/services/email_service/src/api/email/provider_error.rs+++ b/services/email_service/src/api/email/provider_error.rs@@ -5,15 +5,16 @@ use email_api_client::domain::models::EmailApiError;#[cfg(test)]mod test;-/// Response headers implied by a provider error: rate-limited requests carry-/// the provider's suggested delay as a `Retry-After` header when known.+/// Response headers implied by a rate limit: its suggested delay, when known,+/// as a `Retry-After` header rounded up to whole seconds, so a sub-second wait+/// is not sent as `0`.pub(crate) fn provider_error_headers(error: &EmailApiError) -> HeaderMap {let mut headers = HeaderMap::new();if let EmailApiError::RateLimited {retry_after: Some(delay),..} = error- && let Ok(value) = HeaderValue::from_str(&delay.as_secs().to_string())+ && let Ok(value) = HeaderValue::from_str(&delay.as_millis().div_ceil(1_000).to_string()){headers.insert(RETRY_AFTER, value);}diff --git a/services/email_service/src/api/email/provider_error/test.rs b/services/email_service/src/api/email/provider_error/test.rsindex eef819cfd6..a9a3a9661e 100644--- a/services/email_service/src/api/email/provider_error/test.rs+++ b/services/email_service/src/api/email/provider_error/test.rs@@ -10,7 +10,7 @@ fn maps_provider_errors_to_api_statuses() {(EmailApiError::RateLimited {retry_after: Some(Duration::from_secs(10)),- origin: RateLimitOrigin::Provider,+ origin: RateLimitOrigin::Provider { concurrency: false },},StatusCode::TOO_MANY_REQUESTS,),@@ -55,17 +55,27 @@ fn maps_outdated_cursor_to_conflict() {fn rate_limited_responses_carry_a_retry_after_header_when_known() {let headers = provider_error_headers(&EmailApiError::RateLimited {retry_after: Some(Duration::from_secs(42)),- origin: RateLimitOrigin::Provider,+ origin: RateLimitOrigin::Provider { concurrency: false },});assert_eq!(headers.get(RETRY_AFTER).and_then(|v| v.to_str().ok()),Some("42"));+ let headers = provider_error_headers(&EmailApiError::RateLimited {+ retry_after: Some(Duration::from_millis(950)),+ origin: RateLimitOrigin::Local,+ });+ assert_eq!(+ headers.get(RETRY_AFTER).and_then(|v| v.to_str().ok()),+ Some("1"),+ "a sub-second wait must not tell clients to retry immediately"+ );+assert!(provider_error_headers(&EmailApiError::RateLimited {retry_after: None,- origin: RateLimitOrigin::Provider,+ origin: RateLimitOrigin::Provider { concurrency: false },}).is_empty());diff --git a/services/email_service/src/bin/pubsub_workers/pubsub_workers.rs b/services/email_service/src/bin/pubsub_workers/pubsub_workers.rsindex 4499d79e24..f73e0d865a 100644--- a/services/email_service/src/bin/pubsub_workers/pubsub_workers.rs+++ b/services/email_service/src/bin/pubsub_workers/pubsub_workers.rs@@ -4,7 +4,7 @@ use document_storage_service_client::DocumentStorageServiceClient;use email_api_client::GmailApiClientRepository;use email_service::config::Config;use email_service::outbound::email_api::{- EmailServiceTokenSource, GmailApi, RateBudget, RedisProviderRateLimiter,+ EmailServiceTokenSource, GmailApi, RateBudget, RedisGmailRequestGate,};use email_service::pubsub::CrmMetadataResolver;use email_service::util::redis::RedisClient;@@ -30,15 +30,15 @@ fn compose_email_api(db: PgPool,subscription_topic: String,auth_service_client: authentication_service_client::AuthServiceClient,- redis_client: RedisClient,redis_conn: redis::aio::MultiplexedConnection,+ gate_conn: redis::aio::ConnectionManager,sqs_client: sqs_client::SQS,rate_budget: RateBudget,) -> GmailApi {GmailApi::new(GmailApiClientRepository::from_subscription_topic(subscription_topic),EmailServiceTokenSource::new(db, redis_conn, auth_service_client, sqs_client),- RedisProviderRateLimiter::new(redis_client, rate_budget),+ RedisGmailRequestGate::new(gate_conn, rate_budget),)}@@ -268,12 +268,7 @@ async fn main() -> anyhow::Result<()> {queue: ingress_queue,});- let redis_client = RedisClient::new(- redis_inner_client,- config.redis_rate_limit_reqs,- config.redis_rate_limit_reqs_backfill,- config.redis_rate_limit_window_secs,- );+ let redis_client = RedisClient::new(redis_inner_client, config.redis_rate_limit_window_secs);// One long-lived multiplexed connection shared by both token sources;// cloning it is an Arc bump, not a new TCP dial per provider call.@@ -282,13 +277,18 @@ async fn main() -> anyhow::Result<()> {.get_multiplexed_async_connection().await.context("failed to get multiplexed redis connection for token sources")?;+ let gate_conn = redis_client+ .inner+ .get_connection_manager()+ .await+ .context("failed to get reconnecting redis connection for the gmail request gates")?;let email_api_live = compose_email_api(db.clone(),config.gmail_gcp_queue.to_string(),auth_service_client.clone(),- redis_client.clone(),redis_conn.clone(),+ gate_conn.clone(),sqs_client.clone(),RateBudget::Live,);@@ -321,8 +321,8 @@ async fn main() -> anyhow::Result<()> {db_backfill.clone(),config.gmail_gcp_queue.to_string(),auth_service_client.clone(),- redis_client.clone(),redis_conn,+ gate_conn,sqs_client.clone(),RateBudget::Backfill,);diff --git a/services/email_service/src/config.rs b/services/email_service/src/config.rsindex 072c198d47..27bf31e0eb 100644--- a/services/email_service/src/config.rs+++ b/services/email_service/src/config.rs@@ -125,16 +125,7 @@ pub struct Config {#[macro_config_default(3)]pub sfs_uploader_workers: i32,- /// The number of requests we allow per window for backfilling. Less than redis_rate_limit_reqs- /// so we have room for normal gmail operations while backfilling is occurring- #[macro_config_default(13000)]- pub redis_rate_limit_reqs_backfill: u32,-- /// The number of requests we allow per window.- #[macro_config_default(14000)]- pub redis_rate_limit_reqs: u32,-- /// The size of the sliding window we use for rate limit.+ /// The size of the sliding window for the Google Calendar rate limit.#[macro_config_default(60)]pub redis_rate_limit_window_secs: u32,diff --git a/services/email_service/src/outbound/email_api/gmail_gate.rs b/services/email_service/src/outbound/email_api/gmail_gate.rsnew file mode 100644index 0000000000..fcd9562419--- /dev/null+++ b/services/email_service/src/outbound/email_api/gmail_gate.rs@@ -0,0 +1,445 @@+//! Redis admission shared by every Gmail request for a mailbox.++#[cfg(test)]+mod test;++use std::time::Duration;++use email_api_client::domain::models::{+ ApiOperationKind, EmailApiError, MailboxLoad, RateLimitOrigin, RateLimitRefusal, RequestPermit,+};+use email_api_client::domain::ports::ProviderRequestGate;+use models_email::gmail::operations::GmailApiOperation;+use redis::aio::ConnectionManager;+use uuid::Uuid;++// Gmail answers 429 "Too many concurrent requests for user" above an+// unpublished per-user limit that every client of the mailbox shares; it was+// 91% of our Gmail rate limits. Backfill leaves two of the mailbox's slots to+// sync and user writes.+const CONCURRENT_REQUESTS: u32 = 5;+const LIVE_RESERVED_REQUESTS: u32 = 2;+// Gmail's real limit may be lower, and other clients of the mailbox use it+// too. After that 429, the mailbox's cap drops to three quarters, at most once+// a second so requests already in flight don't count twice, then rises by one+// a minute back to five.+const LEARNED_CAP_HOLD_MS: u32 = 1_000;+const LEARNED_CAP_INCREASE_MS: u32 = 60_000;+// Gmail allows 250 quota units per user per second (moving average) and, under+// this project's grandfathered quota, 15,000 per minute. A ten-second window+// enforces both: 2,300 units averages 230 a second and caps a minute at 13,800.+const WINDOW_SECONDS: u32 = 10;+const UNITS_PER_WINDOW: u32 = 2_300;+const BACKFILL_UNITS_PER_WINDOW: u32 = 1_800;+// Longer than the Gmail client's 60-second request timeout. Finished and+// dropped permits release themselves; the lease frees the slots of a crashed+// process.+const PERMIT_MS: u32 = 90_000;+// Google asks clients to wait at least a second before retrying a 429.+const BUSY_RETRY_MS: u32 = 1_000;+// A refused backfill request defers its SQS message, and deferred work comes+// back in waves that leave backfill's slots idle. Up to twelve refused backfill+// requests per mailbox, about a second of work for its three slots, wait for+// room instead, so a freed slot is taken at once. The rest still defer, which+// keeps one mailbox from tying up the workers other mailboxes need.+const BACKFILL_WAITERS: u32 = 12;+// Well inside the backfill queue's 60-second visibility timeout.+const BACKFILL_WAIT: Duration = Duration::from_secs(20);+// How often a waiting request checks for a free slot. Reads take about 200 ms,+// and with several waiters polling a freed slot idles a few milliseconds.+const WAIT_POLL_MS: u32 = 50;+// A waiter's place outlives its sleep by this much, so a crashed waiter frees it.+const WAITER_LEASE_MARGIN_MS: u32 = 1_000;+// Gmail sends no Retry-After with its rate limits. The concurrency and+// per-second limits behind 97% of them clear within Google's one-second+// minimum, and a longer pause stalls every caller for the mailbox.+const DEFAULT_COOLDOWN: Duration = Duration::from_secs(1);+// Backfill stays out longer after Gmail rate-limits the mailbox, so sync and+// user writes get the room Gmail has left before backfill fills it again.+const BACKFILL_COOLDOWN: Duration = Duration::from_secs(10);+// Per-second and per-minute quotas recover within a minute. Longer Retry-After+// values come from daily sending or bandwidth limits, which must not pause the+// mailbox's other work for hours.+const MAX_COOLDOWN: Duration = Duration::from_secs(60);+const UNAVAILABLE_RETRY_AFTER: Duration = Duration::from_secs(5);++// Returns {delay, waiting, in_flight, cap}: {0, 0, in_flight, cap} admits the+// request, reporting the mailbox's requests in flight counting this one and the+// cap it was admitted under; {delay, 1, 0, 0} keeps it waiting for room and+// {delay, 0, 0, 0} refuses it.+const ACQUIRE: &str = r#"+local clock = redis.call('TIME')+local now = tonumber(clock[1]) * 1000 + math.floor(tonumber(clock[2]) / 1000)+local waiters = tonumber(ARGV[8])+local function refuse(delay, busy)+ local wait = delay+ if busy then wait = tonumber(ARGV[9]) end+ if waiters > 0 then+ redis.call('ZREMRANGEBYSCORE', KEYS[4], '-inf', now)+ if wait <= tonumber(ARGV[10]) and+ (redis.call('ZSCORE', KEYS[4], ARGV[1]) or redis.call('ZCARD', KEYS[4]) < waiters) then+ local lease = wait + tonumber(ARGV[11])+ redis.call('ZADD', KEYS[4], now + lease, ARGV[1])+ if redis.call('PTTL', KEYS[4]) < lease then redis.call('PEXPIRE', KEYS[4], lease) end+ return {wait, 1, 0, 0}+ end+ redis.call('ZREM', KEYS[4], ARGV[1])+ end+ return {delay, 0, 0, 0}+end+local cooldown = redis.call('PTTL', KEYS[3])+if ARGV[12] == '1' then cooldown = math.max(cooldown, redis.call('PTTL', KEYS[5])) end+if cooldown > 0 then return refuse(cooldown, false) end+local cap = tonumber(ARGV[2])+local learned = redis.call('HMGET', KEYS[6], 'cap', 'at')+if learned[1] then+ cap = math.min(cap, tonumber(learned[1]) + math.floor((now - tonumber(learned[2])) / tonumber(ARGV[13])))+end+redis.call('ZREMRANGEBYSCORE', KEYS[1], '-inf', now)+local in_flight = redis.call('ZCARD', KEYS[1])+if in_flight >= cap - tonumber(ARGV[14]) then return refuse(tonumber(ARGV[7]), true) end+local second = tonumber(clock[1])+local window = tonumber(ARGV[5])+local limit = tonumber(ARGV[3])+local cost = tonumber(ARGV[4])+local buckets = redis.call('HGETALL', KEYS[2])+local live = {}+local used = 0+for i = 1, #buckets, 2 do+ local bucket = tonumber(buckets[i])+ if bucket <= second - window then+ redis.call('HDEL', KEYS[2], buckets[i])+ else+ local units = tonumber(buckets[i + 1])+ used = used + units+ table.insert(live, {bucket, units})+ end+end+if used + cost > limit then+ table.sort(live, function(a, b) return a[1] < b[1] end)+ for _, entry in ipairs(live) do+ used = used - entry[2]+ if used + cost <= limit then+ return refuse(math.max(1, (entry[1] + window) * 1000 - now), false)+ end+ end+ return refuse(window * 1000, false)+end+redis.call('ZADD', KEYS[1], now + tonumber(ARGV[6]), ARGV[1])+redis.call('PEXPIRE', KEYS[1], ARGV[6])+redis.call('HINCRBY', KEYS[2], clock[1], cost)+redis.call('EXPIRE', KEYS[2], window + 1)+if waiters > 0 then redis.call('ZREM', KEYS[4], ARGV[1]) end+return {0, 0, in_flight + 1, cap}+"#;++const FINISH: &str = r#"+redis.call('ZREM', KEYS[1], ARGV[1])+for i = 2, 3 do+ local delay = tonumber(ARGV[i])+ if delay > 0 and redis.call('PTTL', KEYS[i]) < delay then+ redis.call('SET', KEYS[i], '1', 'PX', delay)+ end+end+if ARGV[4] == '1' then+ local clock = redis.call('TIME')+ local now = tonumber(clock[1]) * 1000 + math.floor(tonumber(clock[2]) / 1000)+ local default = tonumber(ARGV[5])+ local increase = tonumber(ARGV[7])+ local learned = redis.call('HMGET', KEYS[4], 'cap', 'at')+ if not learned[2] or now - tonumber(learned[2]) >= tonumber(ARGV[6]) then+ local cap = default+ if learned[1] then+ cap = math.min(cap, tonumber(learned[1]) + math.floor((now - tonumber(learned[2])) / increase))+ end+ cap = math.max(1, math.floor(cap * 3 / 4))+ redis.call('HSET', KEYS[4], 'cap', cap, 'at', now)+ redis.call('PEXPIRE', KEYS[4], (default - cap) * increase)+ end+end+return 1+"#;++/// Selects which share of a mailbox's Gmail allowance a service instance uses.+///+/// `Live` and `Backfill` are two thresholds over one shared Redis state per+/// mailbox, not two allowances: every request occupies the same in-flight set+/// and charges the same window, and the budget only decides the limit it is+/// compared against. Splitting the keys per budget would let live and backfill+/// together exceed Gmail's per-user limits. Don't "fix" the key sharing.+#[derive(Debug, Clone, Copy, PartialEq, Eq)]+pub enum RateBudget {+ /// Interactive, incremental, and user-initiated work, which may use the+ /// mailbox's whole allowance.+ Live,+ /// Mailbox backfills, held below the full allowance so live work keeps+ /// room while a backfill runs.+ Backfill,+}++impl RateBudget {+ /// The in-flight slots left to others, and the quota units per window.+ fn limits(self) -> (u32, u32) {+ match self {+ Self::Live => (0, UNITS_PER_WINDOW),+ Self::Backfill => (LIVE_RESERVED_REQUESTS, BACKFILL_UNITS_PER_WINDOW),+ }+ }++ /// Live callers handle their own retries, so only backfill waits.+ fn wait(self) -> Duration {+ match self {+ Self::Live => Duration::ZERO,+ Self::Backfill => BACKFILL_WAIT,+ }+ }+}++#[derive(Debug, PartialEq, Eq)]+enum Admission {+ Admitted(MailboxLoad),+ Wait(Duration),+ Refused(Duration),+}++/// Caps in-flight Gmail requests and quota units per mailbox, and pauses the+/// mailbox after Gmail rate-limits it.+///+/// It fails closed: when Redis cannot decide, the request is refused as a+/// local rate limit, so callers defer or requeue the work rather than send it+/// unmetered. Because of that it needs a reconnecting connection; a dropped+/// multiplexed connection would stop all Gmail work until the process restarts.+#[derive(Clone)]+pub struct RedisGmailRequestGate {+ connection: ConnectionManager,+ budget: RateBudget,+ wait: Duration,+}++impl RedisGmailRequestGate {+ /// Creates a gate that admits requests within the selected budget.+ pub fn new(connection: ConnectionManager, budget: RateBudget) -> Self {+ Self {+ connection,+ budget,+ wait: budget.wait(),+ }+ }+}++fn keys(link_id: Uuid) -> [String; 6] {+ // Hash tags colocate the Lua keys in Redis Cluster.+ let scope = format!("gmail:{{{link_id}}}");+ [+ format!("{scope}:inflight"),+ format!("{scope}:window"),+ format!("{scope}:cooldown"),+ format!("{scope}:waiting"),+ format!("{scope}:backfill_cooldown"),+ format!("{scope}:limit"),+ ]+}++/// What a finished request tells the gate about the mailbox.+#[derive(Default)]+struct Outcome {+ /// Pause for every caller.+ cooldown: Duration,+ /// Pause for backfill alone.+ backfill_cooldown: Duration,+ /// Gmail refused it for too many requests in flight.+ concurrency_limited: bool,+}++fn outcome(error: Option<&EmailApiError>) -> Outcome {+ match error {+ Some(EmailApiError::RateLimited {+ origin: RateLimitOrigin::Provider { concurrency },+ retry_after,+ }) => {+ let cooldown = retry_after.unwrap_or(DEFAULT_COOLDOWN).min(MAX_COOLDOWN);+ Outcome {+ cooldown,+ backfill_cooldown: cooldown.max(BACKFILL_COOLDOWN),+ concurrency_limited: *concurrency,+ }+ }+ _ => Outcome::default(),+ }+}++impl ProviderRequestGate for RedisGmailRequestGate {+ async fn acquire(+ &self,+ link_id: Uuid,+ operation: ApiOperationKind,+ ) -> Result<RequestPermit, RateLimitRefusal> {+ let permit = Uuid::now_v7();+ let cost = gmail_operation(operation).cost();+ let admission = match self.admit(link_id, permit, cost, self.wait).await {+ Ok(Admission::Wait(delay)) => self.wait_for_room(link_id, permit, cost, delay).await,+ admission => admission,+ };+ match admission {+ Ok(Admission::Admitted(load)) => {+ let gate = self.clone();+ Ok(RequestPermit::new(link_id, permit, move |link_id, permit| {+ // Drop cannot wait. Without a runtime, the lease frees the slot.+ if let Ok(runtime) = tokio::runtime::Handle::try_current() {+ runtime.spawn(async move {+ gate.release(link_id, permit, Outcome::default()).await+ });+ }+ })+ .with_load(load))+ }+ Ok(Admission::Wait(delay) | Admission::Refused(delay)) => {+ Err(RateLimitRefusal::new(Some(delay)))+ }+ Err(error) => {+ tracing::warn!(%link_id, %error, "Gmail request admission is unavailable");+ Err(RateLimitRefusal::new(Some(UNAVAILABLE_RETRY_AFTER)))+ }+ }+ }++ async fn finish(&self, permit: RequestPermit, error: Option<&EmailApiError>) {+ self.release(permit.link_id(), permit.id(), outcome(error))+ .await;+ permit.disarm();+ }+}++impl RedisGmailRequestGate {+ /// Runs one admission check. `wait` is how much longer the request may wait for room.+ async fn admit(+ &self,+ link_id: Uuid,+ permit: Uuid,+ cost: u32,+ wait: Duration,+ ) -> redis::RedisResult<Admission> {+ let [+ inflight,+ window,+ cooldown,+ waiting,+ backfill_cooldown,+ limit,+ ] = keys(link_id);+ let (reserved_requests, units_per_window) = self.budget.limits();+ let waiters = if self.wait.is_zero() {+ 0+ } else {+ BACKFILL_WAITERS+ };+ let (delay, waits, in_flight, cap): (u64, u8, u32, u32) = redis::Script::new(ACQUIRE)+ .key(inflight)+ .key(window)+ .key(cooldown)+ .key(waiting)+ .key(backfill_cooldown)+ .key(limit)+ .arg(permit.to_string())+ .arg(CONCURRENT_REQUESTS)+ .arg(units_per_window)+ .arg(cost)+ .arg(WINDOW_SECONDS)+ .arg(PERMIT_MS)+ .arg(BUSY_RETRY_MS)+ .arg(waiters)+ .arg(WAIT_POLL_MS)+ .arg(wait.as_millis().to_string())+ .arg(WAITER_LEASE_MARGIN_MS)+ .arg(u8::from(self.budget == RateBudget::Backfill))+ .arg(LEARNED_CAP_INCREASE_MS)+ .arg(reserved_requests)+ .invoke_async(&mut self.connection.clone())+ .await?;+ let delay = Duration::from_millis(delay);+ Ok(match (delay.is_zero(), waits == 1) {+ (true, _) => Admission::Admitted(MailboxLoad { in_flight, cap }),+ (false, true) => Admission::Wait(delay),+ (false, false) => Admission::Refused(delay),+ })+ }++ /// Holds a refused request until it is admitted or its wait runs out.+ #[tracing::instrument(+ name = "gmail.request_gate.wait",+ skip(self, cost),+ fields(admitted),+ err+ )]+ async fn wait_for_room(+ &self,+ link_id: Uuid,+ permit: Uuid,+ cost: u32,+ mut delay: Duration,+ ) -> redis::RedisResult<Admission> {+ let deadline = tokio::time::Instant::now() + self.wait;+ loop {+ tokio::time::sleep(delay).await;+ let wait = deadline.saturating_duration_since(tokio::time::Instant::now());+ match self.admit(link_id, permit, cost, wait).await? {+ Admission::Wait(next) => delay = next,+ admission => {+ tracing::Span::current()+ .record("admitted", matches!(admission, Admission::Admitted(_)));+ return Ok(admission);+ }+ }+ }+ }++ async fn release(&self, link_id: Uuid, permit: Uuid, outcome: Outcome) {+ let [inflight, _, cooldown_key, _, backfill_cooldown_key, limit] = keys(link_id);+ let result: redis::RedisResult<i32> = redis::Script::new(FINISH)+ .key(inflight)+ .key(cooldown_key)+ .key(backfill_cooldown_key)+ .key(limit)+ .arg(permit.to_string())+ .arg(outcome.cooldown.as_millis().to_string())+ .arg(outcome.backfill_cooldown.as_millis().to_string())+ .arg(u8::from(outcome.concurrency_limited))+ .arg(CONCURRENT_REQUESTS)+ .arg(LEARNED_CAP_HOLD_MS)+ .arg(LEARNED_CAP_INCREASE_MS)+ .invoke_async(&mut self.connection.clone())+ .await;+ if result.is_err() {+ // The provider call already happened and its outcome must reach the+ // caller even when releasing admission fails. The lease expires.+ tracing::warn!(%link_id, "failed to finish Gmail request admission");+ }+ }+}++fn gmail_operation(operation: ApiOperationKind) -> GmailApiOperation {+ match operation {+ ApiOperationKind::GetProfile => GmailApiOperation::UsersGetProfile,+ ApiOperationKind::ListChanges => GmailApiOperation::HistoryList,+ ApiOperationKind::ListLabels => GmailApiOperation::LabelsList,+ ApiOperationKind::CreateLabel => GmailApiOperation::LabelsCreate,+ ApiOperationKind::DeleteLabel => GmailApiOperation::LabelsDelete,+ ApiOperationKind::GetMessage => GmailApiOperation::MessagesGet,+ ApiOperationKind::ListMessages => GmailApiOperation::MessagesList,+ ApiOperationKind::ModifyMessageLabels => GmailApiOperation::MessagesModify,+ ApiOperationKind::GetAttachment => GmailApiOperation::MessagesAttachmentsGet,+ ApiOperationKind::SendMessage => GmailApiOperation::MessagesSend,+ ApiOperationKind::GetThread => GmailApiOperation::ThreadsGet,+ ApiOperationKind::ListThreads => GmailApiOperation::ThreadsList,+ // People API calls share this budget. Gmail has no corresponding quota+ // operation, so use a one-unit operation as the accounting proxy.+ ApiOperationKind::ListContacts => GmailApiOperation::UsersGetProfile,+ // Block and unblock are composite filter operations. One admission+ // charges the dominant write operation rather than every request.+ ApiOperationKind::BlockSender => GmailApiOperation::SettingsFiltersCreate,+ ApiOperationKind::UnblockSender => GmailApiOperation::SettingsFiltersDelete,+ ApiOperationKind::ListBlockedSenders => GmailApiOperation::SettingsFiltersList,+ ApiOperationKind::Subscribe => GmailApiOperation::Watch,+ ApiOperationKind::Unsubscribe => GmailApiOperation::Stop,+ }+}diff --git a/services/email_service/src/outbound/email_api/gmail_gate/test.rs b/services/email_service/src/outbound/email_api/gmail_gate/test.rsnew file mode 100644index 0000000000..72e3fca17c--- /dev/null+++ b/services/email_service/src/outbound/email_api/gmail_gate/test.rs@@ -0,0 +1,552 @@+use super::*;++async fn connection() -> ConnectionManager {+ redis::Client::open("redis://127.0.0.1:6379/")+ .unwrap()+ .get_connection_manager()+ .await+ .expect("start local Redis before running this integration test")+}++async fn cleanup(connection: &ConnectionManager, link_id: Uuid) {+ let _: i32 = redis::cmd("DEL")+ .arg(&keys(link_id))+ .query_async(&mut connection.clone())+ .await+ .unwrap();+}++fn refusal_delay(result: Result<RequestPermit, RateLimitRefusal>) -> Duration {+ result+ .err()+ .expect("the request should be refused")+ .retry_after+ .expect("a refusal reports when to retry")+}++#[tokio::test]+#[ignore = "requires local Redis on port 6379; uses only unique mailbox keys"]+async fn backfill_leaves_two_of_five_in_flight_slots_to_live_work() {+ let connection = connection().await;+ let live = RedisGmailRequestGate::new(connection.clone(), RateBudget::Live);+ let mut backfill = RedisGmailRequestGate::new(connection.clone(), RateBudget::Backfill);+ backfill.wait = Duration::ZERO;+ let link_id = Uuid::now_v7();++ let mut backfill_permits = Vec::new();+ for _ in 0..3 {+ backfill_permits.push(+ backfill+ .acquire(link_id, ApiOperationKind::GetMessage)+ .await+ .unwrap(),+ );+ }+ assert_eq!(+ refusal_delay(+ backfill+ .acquire(link_id, ApiOperationKind::GetMessage)+ .await+ ),+ Duration::from_secs(1)+ );+ let first_live = live+ .acquire(link_id, ApiOperationKind::ModifyMessageLabels)+ .await+ .unwrap();+ let _second_live = live+ .acquire(link_id, ApiOperationKind::ModifyMessageLabels)+ .await+ .unwrap();+ assert_eq!(+ refusal_delay(+ live.acquire(link_id, ApiOperationKind::ModifyMessageLabels)+ .await+ ),+ Duration::from_secs(1)+ );++ live.finish(first_live, None).await;+ let _third_live = live+ .acquire(link_id, ApiOperationKind::ModifyMessageLabels)+ .await+ .unwrap();+ backfill.finish(backfill_permits.remove(0), None).await;+ assert!(+ backfill+ .acquire(link_id, ApiOperationKind::GetMessage)+ .await+ .is_err(),+ "a released backfill slot must not let backfill exceed its share while live holds the rest"+ );+ cleanup(&connection, link_id).await;+}++#[tokio::test]+#[ignore = "requires local Redis on port 6379; uses only unique mailbox keys"]+async fn twelve_refused_backfill_requests_wait_for_room_and_the_rest_defer() {+ let connection = connection().await;+ let backfill = RedisGmailRequestGate::new(connection.clone(), RateBudget::Backfill);+ let link_id = Uuid::now_v7();+ let mut permits = Vec::new();+ for _ in 0..3 {+ permits.push(+ backfill+ .acquire(link_id, ApiOperationKind::GetMessage)+ .await+ .unwrap(),+ );+ }++ let waiters: Vec<Uuid> = (0..12).map(|_| Uuid::now_v7()).collect();+ for &waiter in &waiters {+ assert_eq!(+ backfill+ .admit(link_id, waiter, 5, BACKFILL_WAIT)+ .await+ .unwrap(),+ Admission::Wait(Duration::from_millis(50))+ );+ }+ let deferred = Uuid::now_v7();+ assert_eq!(+ backfill+ .admit(link_id, deferred, 5, BACKFILL_WAIT)+ .await+ .unwrap(),+ Admission::Refused(Duration::from_secs(1))+ );++ backfill.finish(permits.remove(0), None).await;+ assert_eq!(+ backfill+ .wait_for_room(link_id, waiters[0], 5, Duration::from_millis(50))+ .await+ .unwrap(),+ Admission::Admitted(MailboxLoad {+ in_flight: 3,+ cap: 5+ })+ );+ // The admitted waiter gave up its place.+ assert_eq!(+ backfill+ .admit(link_id, deferred, 5, BACKFILL_WAIT)+ .await+ .unwrap(),+ Admission::Wait(Duration::from_millis(50))+ );+ cleanup(&connection, link_id).await;+}++#[tokio::test]+#[ignore = "requires local Redis on port 6379; uses only unique mailbox keys"]+async fn a_backfill_request_takes_a_slot_freed_while_it_waits() {+ let connection = connection().await;+ let backfill = RedisGmailRequestGate::new(connection.clone(), RateBudget::Backfill);+ let link_id = Uuid::now_v7();+ let mut permits = Vec::new();+ for _ in 0..3 {+ permits.push(+ backfill+ .acquire(link_id, ApiOperationKind::GetMessage)+ .await+ .unwrap(),+ );+ }++ let mut waiting = tokio::spawn({+ let backfill = backfill.clone();+ async move {+ backfill+ .acquire(link_id, ApiOperationKind::GetMessage)+ .await+ }+ });+ assert!(+ tokio::time::timeout(Duration::from_millis(300), &mut waiting)+ .await+ .is_err(),+ "a full mailbox holds the backfill request instead of refusing it"+ );+ backfill.finish(permits.remove(0), None).await;+ let admitted = tokio::time::timeout(Duration::from_secs(5), waiting)+ .await+ .unwrap()+ .unwrap();+ assert!(admitted.is_ok());+ cleanup(&connection, link_id).await;+}++#[tokio::test]+#[ignore = "requires local Redis on port 6379; uses only unique mailbox keys"]+async fn a_backfill_request_defers_once_its_wait_runs_out() {+ let connection = connection().await;+ let mut backfill = RedisGmailRequestGate::new(connection.clone(), RateBudget::Backfill);+ backfill.wait = Duration::from_millis(200);+ let link_id = Uuid::now_v7();+ let mut permits = Vec::new();+ for _ in 0..3 {+ permits.push(+ backfill+ .acquire(link_id, ApiOperationKind::GetMessage)+ .await+ .unwrap(),+ );+ }++ assert_eq!(+ refusal_delay(+ backfill+ .acquire(link_id, ApiOperationKind::GetMessage)+ .await+ ),+ Duration::from_secs(1)+ );+ let waiting: u32 = redis::cmd("ZCARD")+ .arg(&keys(link_id)[3])+ .query_async(&mut connection.clone())+ .await+ .unwrap();+ assert_eq!(waiting, 0, "a request that stops waiting leaves its place");+ cleanup(&connection, link_id).await;+}++#[tokio::test]+#[ignore = "requires local Redis on port 6379; uses only unique mailbox keys"]+async fn a_gmail_rate_limit_pauses_live_work_briefly_and_backfill_longer() {+ let connection = connection().await;+ let live = RedisGmailRequestGate::new(connection.clone(), RateBudget::Live);+ let mut backfill = RedisGmailRequestGate::new(connection.clone(), RateBudget::Backfill);+ backfill.wait = Duration::ZERO;+ let link_id = Uuid::now_v7();++ let throttled = live+ .acquire(link_id, ApiOperationKind::GetMessage)+ .await+ .unwrap();+ let local_refusal = live+ .acquire(link_id, ApiOperationKind::GetMessage)+ .await+ .unwrap();+ live.finish(+ local_refusal,+ Some(&EmailApiError::RateLimited {+ retry_after: None,+ origin: RateLimitOrigin::Local,+ }),+ )+ .await;+ let _admitted = live+ .acquire(link_id, ApiOperationKind::GetMessage)+ .await+ .unwrap();++ live.finish(+ throttled,+ Some(&EmailApiError::RateLimited {+ retry_after: None,+ origin: RateLimitOrigin::Provider { concurrency: false },+ }),+ )+ .await;+ let live_delay = refusal_delay(live.acquire(link_id, ApiOperationKind::SendMessage).await);+ assert!(live_delay > Duration::from_millis(500) && live_delay <= DEFAULT_COOLDOWN);+ let backfill_delay =+ refusal_delay(backfill.acquire(link_id, ApiOperationKind::GetThread).await);+ assert!(+ backfill_delay > Duration::from_secs(9) && backfill_delay <= BACKFILL_COOLDOWN,+ "backfill leaves the room Gmail has left to live work for longer"+ );+ cleanup(&connection, link_id).await;++ let permit = live+ .acquire(link_id, ApiOperationKind::SendMessage)+ .await+ .unwrap();+ live.finish(+ permit,+ Some(&EmailApiError::RateLimited {+ retry_after: Some(Duration::from_secs(3_600)),+ origin: RateLimitOrigin::Provider { concurrency: false },+ }),+ )+ .await;+ let delay = refusal_delay(live.acquire(link_id, ApiOperationKind::GetMessage).await);+ assert!(delay > Duration::from_secs(59) && delay <= MAX_COOLDOWN);+ let delay = refusal_delay(+ backfill+ .acquire(link_id, ApiOperationKind::GetMessage)+ .await,+ );+ assert!(delay > Duration::from_secs(59) && delay <= MAX_COOLDOWN);+ cleanup(&connection, link_id).await;+}++#[tokio::test]+#[ignore = "requires local Redis on port 6379; uses only unique mailbox keys"]+async fn a_concurrency_429_lowers_the_mailbox_cap_until_it_recovers() {+ let connection = connection().await;+ let live = RedisGmailRequestGate::new(connection.clone(), RateBudget::Live);+ let mut backfill = RedisGmailRequestGate::new(connection.clone(), RateBudget::Backfill);+ backfill.wait = Duration::ZERO;+ let link_id = Uuid::now_v7();+ let [_, _, cooldown, _, backfill_cooldown, limit] = keys(link_id);+ let concurrency_429 = EmailApiError::RateLimited {+ retry_after: None,+ origin: RateLimitOrigin::Provider { concurrency: true },+ };++ let first = live+ .acquire(link_id, ApiOperationKind::GetMessage)+ .await+ .unwrap();+ let second = live+ .acquire(link_id, ApiOperationKind::GetMessage)+ .await+ .unwrap();+ live.finish(first, Some(&concurrency_429)).await;+ // Requests already in flight under the old cap don't lower it again.+ live.finish(second, Some(&concurrency_429)).await;+ let _: () = redis::cmd("DEL")+ .arg(&cooldown)+ .arg(&backfill_cooldown)+ .query_async(&mut connection.clone())+ .await+ .unwrap();++ let backfill_permit = backfill+ .acquire(link_id, ApiOperationKind::GetMessage)+ .await+ .unwrap();+ assert!(+ backfill+ .acquire(link_id, ApiOperationKind::GetMessage)+ .await+ .is_err(),+ "backfill keeps one of the three slots Gmail allows"+ );+ let mut permits = vec![backfill_permit];+ for _ in 0..2 {+ permits.push(+ live.acquire(link_id, ApiOperationKind::GetMessage)+ .await+ .unwrap(),+ );+ }+ assert_eq!(+ refusal_delay(live.acquire(link_id, ApiOperationKind::GetMessage).await),+ Duration::from_secs(1),+ "the mailbox's cap fell from five to three"+ );++ // Two minutes later the cap is back at five.+ let rewind = r#"+ redis.call('HINCRBY', KEYS[1], 'at', -120000)+ return 1+ "#;+ let _: i32 = redis::Script::new(rewind)+ .key(&limit)+ .invoke_async(&mut connection.clone())+ .await+ .unwrap();+ for _ in 0..2 {+ permits.push(+ live.acquire(link_id, ApiOperationKind::GetMessage)+ .await+ .unwrap(),+ );+ }+ cleanup(&connection, link_id).await;++ let permit = live+ .acquire(link_id, ApiOperationKind::GetMessage)+ .await+ .unwrap();+ live.finish(+ permit,+ Some(&EmailApiError::RateLimited {+ retry_after: None,+ origin: RateLimitOrigin::Provider { concurrency: false },+ }),+ )+ .await;+ let learned: Option<u32> = redis::cmd("HGET")+ .arg(&limit)+ .arg("cap")+ .query_async(&mut connection.clone())+ .await+ .unwrap();+ assert_eq!(learned, None, "a quota 429 leaves the cap alone");+ cleanup(&connection, link_id).await;+}++#[tokio::test]+#[ignore = "requires local Redis on port 6379; uses only unique mailbox keys"]+async fn an_admitted_request_reports_the_mailbox_load_it_joined() {+ let connection = connection().await;+ let live = RedisGmailRequestGate::new(connection.clone(), RateBudget::Live);+ let link_id = Uuid::now_v7();++ let first = live+ .acquire(link_id, ApiOperationKind::GetMessage)+ .await+ .unwrap();+ let second = live+ .acquire(link_id, ApiOperationKind::GetMessage)+ .await+ .unwrap();+ assert_eq!(+ first.load(),+ Some(MailboxLoad {+ in_flight: 1,+ cap: 5+ })+ );+ assert_eq!(+ second.load(),+ Some(MailboxLoad {+ in_flight: 2,+ cap: 5+ })+ );++ live.finish(+ first,+ Some(&EmailApiError::RateLimited {+ retry_after: None,+ origin: RateLimitOrigin::Provider { concurrency: true },+ }),+ )+ .await;+ let _: () = redis::cmd("DEL")+ .arg(&keys(link_id)[2])+ .query_async(&mut connection.clone())+ .await+ .unwrap();+ let third = live+ .acquire(link_id, ApiOperationKind::GetMessage)+ .await+ .unwrap();+ assert_eq!(+ third.load(),+ Some(MailboxLoad {+ in_flight: 2,+ cap: 3+ }),+ "the load reports the learned cap"+ );+ cleanup(&connection, link_id).await;+}++#[tokio::test]+#[ignore = "requires local Redis on port 6379; uses only unique mailbox keys"]+async fn quota_units_are_refused_until_the_oldest_second_leaves_the_window() {+ let connection = connection().await;+ let live = RedisGmailRequestGate::new(connection.clone(), RateBudget::Live);+ let mut backfill = RedisGmailRequestGate::new(connection.clone(), RateBudget::Backfill);+ backfill.wait = Duration::ZERO;+ let link_id = Uuid::now_v7();+ let fill = r#"+ local clock = redis.call('TIME')+ redis.call('HSET', KEYS[1], tonumber(clock[1]) - 2, ARGV[1])+ redis.call('HSET', KEYS[1], tonumber(clock[1]) - 1, ARGV[2])+ redis.call('HSET', KEYS[1], tonumber(clock[1]) - 60, 100000)+ return 1+ "#;+ let _: i32 = redis::Script::new(fill)+ .key(&keys(link_id)[1])+ .arg(200)+ .arg(2_000)+ .invoke_async(&mut connection.clone())+ .await+ .unwrap();++ // 2,200 units are in the window; the expired bucket no longer counts.+ let permit = live+ .acquire(link_id, ApiOperationKind::ListThreads)+ .await+ .unwrap();+ live.finish(permit, None).await;+ // A send fits once the oldest second leaves the window; backfill's lower+ // threshold also has to wait for the next one.+ let live_delay = refusal_delay(live.acquire(link_id, ApiOperationKind::SendMessage).await);+ let backfill_delay = refusal_delay(+ backfill+ .acquire(link_id, ApiOperationKind::GetMessage)+ .await,+ );+ assert!(live_delay > Duration::from_secs(6) && live_delay <= Duration::from_secs(8));+ assert!(backfill_delay > Duration::from_secs(7) && backfill_delay <= Duration::from_secs(9));+ assert!(backfill_delay > live_delay);+ cleanup(&connection, link_id).await;+}++#[tokio::test]+#[ignore = "requires local Redis on port 6379; uses only unique mailbox keys"]+async fn a_redis_failure_defers_the_request_instead_of_admitting_it() {+ let connection = connection().await;+ let live = RedisGmailRequestGate::new(connection.clone(), RateBudget::Live);+ let link_id = Uuid::now_v7();+ let _: () = redis::cmd("SET")+ .arg(&keys(link_id)[0])+ .arg("not a sorted set")+ .query_async(&mut connection.clone())+ .await+ .unwrap();++ assert_eq!(+ live.acquire(link_id, ApiOperationKind::ModifyMessageLabels)+ .await+ .err(),+ Some(RateLimitRefusal::new(Some(UNAVAILABLE_RETRY_AFTER)))+ );+ cleanup(&connection, link_id).await;+}++#[tokio::test]+#[ignore = "requires local Redis on port 6379; kills only its own connection"]+async fn admission_resumes_after_redis_drops_the_connection() {+ let connection = connection().await;+ let live = RedisGmailRequestGate::new(connection.clone(), RateBudget::Live);+ let link_id = Uuid::now_v7();+ let id: i64 = redis::cmd("CLIENT")+ .arg("ID")+ .query_async(&mut connection.clone())+ .await+ .unwrap();+ let _: redis::RedisResult<i32> = redis::cmd("CLIENT")+ .arg("KILL")+ .arg("ID")+ .arg(id)+ .query_async(&mut connection.clone())+ .await;++ // The first request after the drop may still meet the dead connection.+ let first = live+ .acquire(link_id, ApiOperationKind::GetMessage)+ .await+ .err();+ assert!(+ first.is_none() || first == Some(RateLimitRefusal::new(Some(UNAVAILABLE_RETRY_AFTER))),+ "{first:?}"+ );+ assert!(+ live.acquire(link_id, ApiOperationKind::GetMessage)+ .await+ .is_ok(),+ "the gate reconnects instead of refusing until a restart"+ );+ cleanup(&connection, link_id).await;+}++#[test]+fn contacts_use_a_one_unit_shared_budget_proxy() {+ assert_eq!(gmail_operation(ApiOperationKind::ListContacts).cost(), 1);+}++#[test]+fn composite_blocklist_operations_use_the_dominant_filter_write_cost() {+ assert_eq!(gmail_operation(ApiOperationKind::BlockSender).cost(), 5);+ assert_eq!(gmail_operation(ApiOperationKind::UnblockSender).cost(), 5);+}diff --git a/services/email_service/src/outbound/email_api/mod.rs b/services/email_service/src/outbound/email_api/mod.rsindex f9f39d6f04..892439240e 100644--- a/services/email_service/src/outbound/email_api/mod.rs+++ b/services/email_service/src/outbound/email_api/mod.rs@@ -1,12 +1,12 @@//! Email API infrastructure adapters.+mod gmail_gate;mod microsoft;-mod rate_limiter;mod token_source;pub use email_api_client::outbound::microsoft_gate::RedisMicrosoftRequestGate;+pub use gmail_gate::{RateBudget, RedisGmailRequestGate};pub use microsoft::{MicrosoftCredentialsClient, ProviderMailboxGateway};-pub use rate_limiter::{RateBudget, RedisProviderRateLimiter};pub use token_source::{EmailServiceTokenSource, StaticTokenSource};use email_api_client::GmailApiClientRepository;@@ -16,5 +16,5 @@ use email_api_client::domain::service::EmailApiClientServiceImpl;pub type GmailApi = EmailApiClientServiceImpl<GmailApiClientRepository,EmailServiceTokenSource,- RedisProviderRateLimiter,+ RedisGmailRequestGate,>;diff --git a/services/email_service/src/outbound/email_api/rate_limiter.rs b/services/email_service/src/outbound/email_api/rate_limiter.rsdeleted file mode 100644index a07f7bf62b..0000000000--- a/services/email_service/src/outbound/email_api/rate_limiter.rs+++ /dev/null@@ -1,114 +0,0 @@-//! Redis-backed provider rate-limiter adapter.--#[cfg(test)]-mod test;--use email_api_client::domain::models::{ApiOperationKind, RateLimitRefusal};-use email_api_client::domain::ports::ProviderRateLimiter;-use models_email::gmail::operations::GmailApiOperation;-use uuid::Uuid;--use crate::util::redis::RedisClient;-use crate::util::redis::rate_limit::RateLimitArgs;--/// Selects the shared provider quota budget used by a service instance.-///-/// `Live` and `Backfill` are two *thresholds over one shared Redis window*,-/// not two counters: every operation charges the same per-user key, and the-/// budget only decides the limit that charge is compared against. Splitting-/// the Redis key per budget would let live + backfill together exceed Gmail's-/// per-user quota — and would silently reset quota state already deployed-/// under the shared key. Don't "fix" the key sharing.-#[derive(Debug, Clone, Copy, PartialEq, Eq)]-pub enum RateBudget {- /// The normal threshold used by interactive and incremental operations.- Live,- /// The lower threshold reserved for mailbox backfills, leaving headroom- /// for live operations while a backfill runs.- Backfill,-}--impl RateBudget {- fn is_backfill(self) -> bool {- matches!(self, Self::Backfill)- }-}--/// Applies Gmail quota policy through the email service's Redis rate limiter.-///-/// Redis failures remain fail-open because [`RedisClient::is_rate_limited`]-/// reports them as an allowed request.-#[derive(Clone)]-pub struct RedisProviderRateLimiter {- redis_client: RedisClient,- budget: RateBudget,-}--impl RedisProviderRateLimiter {- /// Creates a provider rate limiter using the selected quota budget.- pub fn new(redis_client: RedisClient, budget: RateBudget) -> Self {- Self {- redis_client,- budget,- }- }-}--impl ProviderRateLimiter for RedisProviderRateLimiter {- async fn check_rate_limit(- &self,- link_id: Uuid,- operation: ApiOperationKind,- ) -> Result<(), RateLimitRefusal> {- let args = rate_limit_args(link_id, operation, self.budget);-- rate_limit_result(self.redis_client.is_rate_limited(args).await)- }-}--fn rate_limit_result(is_limited: bool) -> Result<(), RateLimitRefusal> {- if is_limited {- return Err(RateLimitRefusal::new(None));- }-- Ok(())-}--fn rate_limit_args(- link_id: Uuid,- operation: ApiOperationKind,- budget: RateBudget,-) -> RateLimitArgs {- RateLimitArgs {- user_id: link_id,- operation: gmail_operation(operation),- is_backfill: budget.is_backfill(),- }-}--fn gmail_operation(operation: ApiOperationKind) -> GmailApiOperation {- match operation {- ApiOperationKind::GetProfile => GmailApiOperation::UsersGetProfile,- ApiOperationKind::ListChanges => GmailApiOperation::HistoryList,- ApiOperationKind::ListLabels => GmailApiOperation::LabelsList,- ApiOperationKind::CreateLabel => GmailApiOperation::LabelsCreate,- ApiOperationKind::DeleteLabel => GmailApiOperation::LabelsDelete,- ApiOperationKind::GetMessage => GmailApiOperation::MessagesGet,- ApiOperationKind::ListMessages => GmailApiOperation::MessagesList,- ApiOperationKind::ModifyMessageLabels => GmailApiOperation::MessagesModify,- ApiOperationKind::GetAttachment => GmailApiOperation::MessagesAttachmentsGet,- ApiOperationKind::SendMessage => GmailApiOperation::MessagesSend,- ApiOperationKind::GetThread => GmailApiOperation::ThreadsGet,- ApiOperationKind::ListThreads => GmailApiOperation::ThreadsList,- // People API calls share this budget in v1. Gmail has no corresponding- // quota operation, so use a one-unit operation as the accounting proxy.- ApiOperationKind::ListContacts => GmailApiOperation::UsersGetProfile,- // Block and unblock are composite filter operations. One preflight- // check charges the dominant write operation rather than every request.- ApiOperationKind::BlockSender => GmailApiOperation::SettingsFiltersCreate,- ApiOperationKind::UnblockSender => GmailApiOperation::SettingsFiltersDelete,- ApiOperationKind::ListBlockedSenders => GmailApiOperation::SettingsFiltersList,- ApiOperationKind::Subscribe => GmailApiOperation::Watch,- ApiOperationKind::Unsubscribe => GmailApiOperation::Stop,- }-}diff --git a/services/email_service/src/outbound/email_api/rate_limiter/test.rs b/services/email_service/src/outbound/email_api/rate_limiter/test.rsdeleted file mode 100644index 8c43bf81d6..0000000000--- a/services/email_service/src/outbound/email_api/rate_limiter/test.rs+++ /dev/null@@ -1,66 +0,0 @@-use email_api_client::domain::models::ApiOperationKind;-use models_email::gmail::operations::GmailApiOperation;-use uuid::Uuid;--use super::{- RateBudget, RedisProviderRateLimiter, gmail_operation, rate_limit_args, rate_limit_result,-};--/// Redis being unreachable must fail open: quota accounting degrades, mail-/// flow does not.-#[tokio::test]-async fn unreachable_redis_fails_open() {- let redis_client = crate::util::redis::RedisClient::new(- redis::Client::open("redis://127.0.0.1:1/").expect("client construction is offline"),- 100,- 50,- 60,- );- let limiter = RedisProviderRateLimiter::new(redis_client, RateBudget::Live);-- let result = email_api_client::domain::ports::ProviderRateLimiter::check_rate_limit(- &limiter,- Uuid::new_v4(),- ApiOperationKind::GetMessage,- )- .await;-- assert!(- result.is_ok(),- "redis outage must not refuse provider calls"- );-}--#[test]-fn contacts_use_a_one_unit_shared_budget_proxy() {- assert_eq!(gmail_operation(ApiOperationKind::ListContacts).cost(), 1);-}--#[test]-fn composite_blocklist_operations_use_the_dominant_filter_write_cost() {- assert_eq!(gmail_operation(ApiOperationKind::BlockSender).cost(), 5);- assert_eq!(gmail_operation(ApiOperationKind::UnblockSender).cost(), 5);-}--#[test]-fn refusal_is_returned_only_for_a_denied_preflight_check() {- assert!(rate_limit_result(false).is_ok());-- let refusal = rate_limit_result(true).expect_err("denied request should be refused");- assert_eq!(refusal.retry_after, None);-}--#[test]-fn rate_budget_selects_live_or_backfill_limit() {- let link_id = Uuid::new_v4();-- let live = rate_limit_args(link_id, ApiOperationKind::GetMessage, RateBudget::Live);- assert_eq!(live.user_id, link_id);- assert_eq!(live.operation, GmailApiOperation::MessagesGet);- assert!(!live.is_backfill);-- let backfill = rate_limit_args(link_id, ApiOperationKind::GetMessage, RateBudget::Backfill);- assert_eq!(backfill.user_id, link_id);- assert_eq!(backfill.operation, GmailApiOperation::MessagesGet);- assert!(backfill.is_backfill);-}diff --git a/services/email_service/src/outbound/scheduled_delivery.rs b/services/email_service/src/outbound/scheduled_delivery.rsindex 92b9d00351..fc28f9c0b4 100644--- a/services/email_service/src/outbound/scheduled_delivery.rs+++ b/services/email_service/src/outbound/scheduled_delivery.rs@@ -282,7 +282,7 @@ impl ScheduledMessageSender<ScheduledClaim, SentDelivery> for ScheduledDeliveryA) -> Result<SentDelivery, SubmissionError> {let ids = self.email_api- .send_prepared(&prepared.provider)+ .send_prepared(prepared.provider).await.map_err(classify_submission_error)?;apply_sent_ids(&mut prepared.delivery.message, ids);diff --git a/services/email_service/src/outbound/scheduled_delivery/test.rs b/services/email_service/src/outbound/scheduled_delivery/test.rsindex 747f07210d..5658b38017 100644--- a/services/email_service/src/outbound/scheduled_delivery/test.rs+++ b/services/email_service/src/outbound/scheduled_delivery/test.rs@@ -48,7 +48,7 @@ fn explicit_provider_refusal_is_safe_to_restore() {},EmailApiError::RateLimited {retry_after: None,- origin: RateLimitOrigin::Provider,+ origin: RateLimitOrigin::Provider { concurrency: false },},] {assert!(matches!(diff --git a/services/email_service/src/pubsub/backfill/backfill_attachment.rs b/services/email_service/src/pubsub/backfill/backfill_attachment.rsindex ab5febe94e..6fffe8a61a 100644--- a/services/email_service/src/pubsub/backfill/backfill_attachment.rs+++ b/services/email_service/src/pubsub/backfill/backfill_attachment.rs@@ -41,10 +41,12 @@ pub async fn backfill_attachment(upload_attachment(ctx_upload, &attachment_args).await.map_err(|e| match e {- UploadAttachmentError::RateLimited => ProcessingError::Retryable(DetailedError {- reason: FailureReason::GmailApiRateLimited,- source: anyhow::Error::new(e).context("Failed to upload attachment"),- }),+ UploadAttachmentError::RateLimited(error) => {+ ProcessingError::Retryable(DetailedError {+ reason: FailureReason::GmailApiRateLimited,+ source: anyhow::Error::new(error).context("Failed to upload attachment"),+ })+ }_ => ProcessingError::NonRetryable(DetailedError {reason: FailureReason::GmailApiFailed,source: anyhow::Error::new(e).context("Failed to upload attachment"),diff --git a/services/email_service/src/pubsub/backfill/backfill_thread.rs b/services/email_service/src/pubsub/backfill/backfill_thread.rsindex 6be59c5549..e051b36e09 100644--- a/services/email_service/src/pubsub/backfill/backfill_thread.rs+++ b/services/email_service/src/pubsub/backfill/backfill_thread.rs@@ -53,8 +53,11 @@ pub async fn backfill_thread(.await.map_err(|error| map_email_api_error(error, "Failed to get provider thread message IDs"))?;- // insert thread object- let thread_db_id = email_db_client::threads::insert::insert_blank_thread(+ // The priority pass and the normal sweep both enqueue a thread, and both+ // copies can pass the check above while they wait for Gmail. Only the copy+ // that inserts the thread backfills its messages; counting them twice+ // would complete the thread early and strand the duplicates.+ let Some(thread_db_id) = email_db_client::threads::insert::insert_blank_thread(&ctx.db,&thread_provider_id,link.id,@@ -65,7 +68,11 @@ pub async fn backfill_thread(e,format!("Failed to insert thread shell for {}", thread_provider_id),)- })?;+ })?+ else {+ incr_completed_threads(ctx, link, scope.job_id).await?;+ return Ok(());+ };// create backfill pubsub message for each email messageenqueue_backfill_messages(ctx, scope, link, thread_db_id, message_ids).awaitdiff --git a/services/email_service/src/pubsub/backfill/deferral.rs b/services/email_service/src/pubsub/backfill/deferral.rsnew file mode 100644index 0000000000..0eae567272--- /dev/null+++ b/services/email_service/src/pubsub/backfill/deferral.rs@@ -0,0 +1,76 @@+//! When rate-limited backfill work goes back to Gmail.+//!+//! A rate-limited operation is sent back to the backfill queue as a new+//! message instead of waiting out the visibility timeout. The request gate+//! refuses most of a large backfill's operations many times before they get+//! one of the mailbox's backfill slots, and each wait on the visibility+//! timeout would spend one of the queue's 20 deliveries. A refusal must never+//! cost a backfill its work.++use std::ops::RangeInclusive;+use std::time::Duration;++use email_api_client::domain::models::EmailApiError;+use models_email::email::service::pubsub::{DetailedError, FailureReason};+use rand::Rng;+use sqs_client::email::BACKFILL_DEFERRALS_ATTRIBUTE;++#[cfg(test)]+mod test;++/// Ceiling of the first deferral's delay; each later deferral doubles it.+const FIRST_DEFERRAL: Duration = Duration::from_secs(1);+/// Longest wait between attempts. A mailbox's deferred backlog then comes+/// back faster than its three backfill slots drain it, so slots don't sit+/// idle, while long-waiting work retries about once a minute.+const MAX_DEFERRAL: Duration = Duration::from_secs(60);+/// SQS's longest message delay, which bounds a provider's longer Retry-After.+const MAX_SQS_DELAY: Duration = Duration::from_secs(900);++/// The wait a rate-limited operation was told to observe, when the failure is+/// a rate limit: the request gate's or Gmail's Retry-After, if either gave one.+pub(crate) fn rate_limit(error: &DetailedError) -> Option<Option<Duration>> {+ if error.reason != FailureReason::GmailApiRateLimited {+ return None;+ }+ Some(match error.source.downcast_ref::<EmailApiError>() {+ Some(EmailApiError::RateLimited { retry_after, .. }) => *retry_after,+ _ => None,+ })+}++/// How often the received operation has already been deferred.+pub(crate) fn deferrals(message: &aws_sdk_sqs::types::Message) -> u32 {+ message+ .message_attributes()+ .and_then(|attributes| attributes.get(BACKFILL_DEFERRALS_ATTRIBUTE))+ .and_then(|value| value.string_value())+ .and_then(|value| value.parse().ok())+ .unwrap_or(0)+}++/// Whole seconds to hold an operation deferred `deferrals` times before.+pub(crate) fn delay_seconds(deferrals: u32, retry_after: Option<Duration>) -> i32 {+ delay(deferrals, retry_after, |range| {+ rand::rng().random_range(range)+ })+}++/// Exponential backoff with equal jitter, so operations refused together come+/// back spread out, and never sooner than the refusal's Retry-After.+fn delay(+ deferrals: u32,+ retry_after: Option<Duration>,+ jitter: impl FnOnce(RangeInclusive<Duration>) -> Duration,+) -> i32 {+ let ceiling = 2u32+ .checked_pow(deferrals)+ .and_then(|factor| FIRST_DEFERRAL.checked_mul(factor))+ .map_or(MAX_DEFERRAL, |ceiling| ceiling.min(MAX_DEFERRAL));+ let delay = jitter(ceiling / 2..=ceiling)+ .max(retry_after.unwrap_or_default())+ .min(MAX_SQS_DELAY);+ // SQS counts whole seconds; rounding down would retry before Retry-After.+ let seconds = delay.as_secs() + u64::from(delay.subsec_nanos() > 0);+ i32::try_from(seconds.max(1)).unwrap_or(i32::MAX)+}diff --git a/services/email_service/src/pubsub/backfill/deferral/test.rs b/services/email_service/src/pubsub/backfill/deferral/test.rsnew file mode 100644index 0000000000..ff001f258b--- /dev/null+++ b/services/email_service/src/pubsub/backfill/deferral/test.rs@@ -0,0 +1,108 @@+use std::time::Duration;++use email_api_client::domain::models::{EmailApiError, RateLimitOrigin};+use models_email::email::service::pubsub::ProcessingError;++use super::*;+use crate::pubsub::backfill::db_error::map_db_error;+use crate::pubsub::backfill::email_api_error::map_email_api_error;++fn retryable(error: ProcessingError) -> DetailedError {+ match error {+ ProcessingError::Retryable(detail) => detail,+ ProcessingError::NonRetryable(detail) => {+ panic!("expected a retryable error, got {:?}", detail.reason)+ }+ }+}++fn longest(range: RangeInclusive<Duration>) -> Duration {+ *range.end()+}++fn shortest(range: RangeInclusive<Duration>) -> Duration {+ *range.start()+}++#[test]+fn a_refused_thread_listing_page_is_deferred_instead_of_spending_a_delivery() {+ let refused = retryable(map_email_api_error(+ EmailApiError::RateLimited {+ retry_after: Some(Duration::from_millis(2_500)),+ origin: RateLimitOrigin::Local,+ },+ "Failed to list provider threads",+ ));++ assert_eq!(+ rate_limit(&refused),+ Some(Some(Duration::from_millis(2_500)))+ );+}++#[test]+fn a_gmail_rate_limit_without_retry_after_is_deferred_too() {+ let throttled = retryable(map_email_api_error(+ EmailApiError::RateLimited {+ retry_after: None,+ origin: RateLimitOrigin::Provider { concurrency: false },+ },+ "Failed to fetch message",+ ));++ assert_eq!(rate_limit(&throttled), Some(None));+}++#[test]+fn other_retryable_failures_keep_waiting_out_the_visibility_timeout() {+ let database = retryable(map_db_error(+ anyhow::anyhow!("connection reset"),+ "Failed to insert thread",+ ));+ let transient = retryable(map_email_api_error(+ EmailApiError::Transient {+ message: "Gmail API returned 503".to_string(),+ },+ "Failed to fetch message",+ ));++ assert_eq!(rate_limit(&database), None);+ assert_eq!(rate_limit(&transient), None);+}++#[test]+fn deferrals_grow_from_a_second_to_a_minute() {+ assert_eq!(delay(0, None, shortest), 1);+ assert_eq!(delay(0, None, longest), 1);+ assert_eq!(delay(1, None, longest), 2);+ assert_eq!(delay(3, None, shortest), 4);+ assert_eq!(delay(3, None, longest), 8);+ assert_eq!(delay(6, None, longest), 60);+ assert_eq!(delay(40, None, shortest), 30);+ assert_eq!(delay(40, None, longest), 60);+}++#[test]+fn a_deferral_waits_at_least_its_retry_after_rounded_up() {+ assert_eq!(delay(0, Some(Duration::from_millis(4_200)), longest), 5);+ assert_eq!(delay(3, Some(Duration::from_millis(500)), longest), 8);+ assert_eq!(delay(0, Some(Duration::from_secs(3_600)), longest), 900);+}++#[test]+fn the_deferral_count_is_read_from_the_message() {+ let deferred = aws_sdk_sqs::types::Message::builder()+ .message_attributes(+ BACKFILL_DEFERRALS_ATTRIBUTE,+ aws_sdk_sqs::types::MessageAttributeValue::builder()+ .data_type("Number")+ .string_value("4")+ .build()+ .unwrap(),+ )+ .build();+ let fresh = aws_sdk_sqs::types::Message::builder().build();++ assert_eq!(deferrals(&deferred), 4);+ assert_eq!(deferrals(&fresh), 0);+}diff --git a/services/email_service/src/pubsub/backfill/email_api_error/test.rs b/services/email_service/src/pubsub/backfill/email_api_error/test.rsindex b0259a5037..6b22d6fcb7 100644--- a/services/email_service/src/pubsub/backfill/email_api_error/test.rs+++ b/services/email_service/src/pubsub/backfill/email_api_error/test.rs@@ -24,7 +24,9 @@ fn maps_rate_limits_to_retryable_rate_limit_failures() {assert_mapping(EmailApiError::RateLimited {retry_after: Some(Duration::from_secs(5)),- origin: email_api_client::domain::models::RateLimitOrigin::Provider,+ origin: email_api_client::domain::models::RateLimitOrigin::Provider {+ concurrency: false,+ },},true,FailureReason::GmailApiRateLimited,diff --git a/services/email_service/src/pubsub/backfill/error_handlers.rs b/services/email_service/src/pubsub/backfill/error_handlers.rsindex e6d40fb24d..8e15bf5cad 100644--- a/services/email_service/src/pubsub/backfill/error_handlers.rs+++ b/services/email_service/src/pubsub/backfill/error_handlers.rs@@ -1,3 +1,4 @@+use crate::pubsub::backfill::deferral;use crate::pubsub::backfill::increment_counters::{incr_completed_messages, incr_completed_threads,};@@ -9,6 +10,7 @@ use models_email::email::service::backfill::{};use models_email::email::service::pubsub::{DetailedError, FailureReason};use sqs_worker::cleanup_message;+use std::time::Duration;use uuid::Uuid;/// Handles non-retryable errors by updating the appropriate status in the database and cleaning up the SQS message@@ -107,6 +109,40 @@ async fn notify_job_failed(ctx: &PubSubContext, job_id: Uuid) {}}+/// Sends a rate-limited operation back to the queue after a growing delay+/// (see [`deferral`]), keeping its delivery budget for real failures. If the+/// send fails, the message is left for redelivery after its visibility timeout.+#[tracing::instrument(+ skip(ctx, message, data, e),+ fields(link_id = ?data.backfill_operation.link_id()),+ err+)]+pub async fn defer_rate_limited_operation(+ ctx: &PubSubContext,+ message: &aws_sdk_sqs::types::Message,+ data: &BackfillPubsubMessage,+ e: &DetailedError,+ retry_after: Option<Duration>,+) -> anyhow::Result<()> {+ let deferrals = deferral::deferrals(message);+ let delay_seconds = deferral::delay_seconds(deferrals, retry_after);+ if let Err(error) = ctx+ .sqs_client+ .defer_email_backfill_message(data, delay_seconds, deferrals + 1)+ .await+ {+ tracing::warn!(error = ?error, "Failed to defer a rate-limited backfill operation; it will be redelivered");+ return handle_retryable_error(data, e).await;+ }+ tracing::debug!(+ deferrals,+ delay_seconds,+ "Rate-limited backfill operation deferred"+ );+ cleanup_message(&ctx.sqs_worker, message).await?;+ Ok(())+}+/// Handles retryable errors by updating status to InProgress and adding the error message#[tracing::instrument(skip(data, _e),diff --git a/services/email_service/src/pubsub/backfill/mod.rs b/services/email_service/src/pubsub/backfill/mod.rsindex b56880458c..459a9edad3 100644--- a/services/email_service/src/pubsub/backfill/mod.rs+++ b/services/email_service/src/pubsub/backfill/mod.rs@@ -2,6 +2,7 @@ mod backfill_attachment;mod backfill_message;mod backfill_thread;pub(crate) mod db_error;+mod deferral;mod depopulate_crm_contact;mod depopulate_crm_for_user;pub(crate) mod email_api_error;diff --git a/services/email_service/src/pubsub/backfill/process.rs b/services/email_service/src/pubsub/backfill/process.rsindex afe3e41f0d..36c0684573 100644--- a/services/email_service/src/pubsub/backfill/process.rs+++ b/services/email_service/src/pubsub/backfill/process.rs@@ -1,5 +1,5 @@use crate::pubsub::backfill::{- backfill_attachment, backfill_message, backfill_thread, depopulate_crm_contact,+ backfill_attachment, backfill_message, backfill_thread, deferral, depopulate_crm_contact,depopulate_crm_for_user, error_handlers, increment_counters, init, list_threads,populate_crm_contact, populate_crm_for_user, seed_sent_contact, update_metadata,};@@ -45,9 +45,13 @@ pub async fn process_message(}// A temporary failure occurred. We log it and don't clean up the message, so it gets retried- Err(ProcessingError::Retryable(e)) => {- error_handlers::handle_retryable_error(&data, &e).await- }+ Err(ProcessingError::Retryable(e)) => match deferral::rate_limit(&e) {+ Some(retry_after) => {+ error_handlers::defer_rate_limited_operation(&ctx, message, &data, &e, retry_after)+ .await+ }+ None => error_handlers::handle_retryable_error(&data, &e).await,+ },}}diff --git a/services/email_service/src/pubsub/gmail_ops/email_api_error/test.rs b/services/email_service/src/pubsub/gmail_ops/email_api_error/test.rsindex 7172cb202b..f03d0b004b 100644--- a/services/email_service/src/pubsub/gmail_ops/email_api_error/test.rs+++ b/services/email_service/src/pubsub/gmail_ops/email_api_error/test.rs@@ -6,7 +6,7 @@ use super::*;fn rate_limits_use_queue_routing_policy() {let provider_rate_limit = EmailApiError::RateLimited {retry_after: Some(Duration::from_secs(30)),- origin: email_api_client::domain::models::RateLimitOrigin::Provider,+ origin: email_api_client::domain::models::RateLimitOrigin::Provider { concurrency: false },};assert_eq!(error_policy(&provider_rate_limit), ErrorPolicy::RateLimited);@@ -73,7 +73,7 @@ fn not_found_is_success_for_delete_label() {fn modify_message_labels_rate_limited_and_transient_failures_do_not_revert() {assert!(!is_permanent_mutation_error(&EmailApiError::RateLimited {retry_after: None,- origin: email_api_client::domain::models::RateLimitOrigin::Provider,+ origin: email_api_client::domain::models::RateLimitOrigin::Provider { concurrency: false },}));assert!(!is_permanent_mutation_error(&EmailApiError::Transient {message: "timeout".to_string(),diff --git a/services/email_service/src/pubsub/inbox_sync/email_api_error.rs b/services/email_service/src/pubsub/inbox_sync/email_api_error.rsindex 682629d318..dec1f7076d 100644--- a/services/email_service/src/pubsub/inbox_sync/email_api_error.rs+++ b/services/email_service/src/pubsub/inbox_sync/email_api_error.rs@@ -1,4 +1,4 @@-use email_api_client::domain::models::{EmailApiError, RateLimitOrigin};+use email_api_client::domain::models::EmailApiError;use models_email::gmail::inbox_sync::{InboxSyncOperation, InboxSyncPubsubMessage};use models_email::service::pubsub::{DetailedError, FailureReason, ProcessingError};@@ -50,25 +50,6 @@ pub(crate) async fn handle_operation_error(non_retryable(error)}-pub(crate) fn handle_gmail_message_error(error: EmailApiError) -> ProcessingError {- // Local budget refusals drop the notification by design: no provider- // quota was consumed and a later notification covers the same cursor- // range, avoiding fan-out backpressure. A provider-origin 429 retries- // (main's behavior) so an idle mailbox does not stay unsynced until its- // next change.- if matches!(- error,- EmailApiError::RateLimited {- origin: RateLimitOrigin::Local,- ..- }- ) {- return non_retryable(error);- }-- processing_error(error)-}-fn processing_error(error: EmailApiError) -> ProcessingError {if error.is_transient() || matches!(error, EmailApiError::RateLimited { .. }) {ProcessingError::Retryable(detail(error))diff --git a/services/email_service/src/pubsub/inbox_sync/email_api_error/test.rs b/services/email_service/src/pubsub/inbox_sync/email_api_error/test.rsindex bcf89f7df8..f90c8cc827 100644--- a/services/email_service/src/pubsub/inbox_sync/email_api_error/test.rs+++ b/services/email_service/src/pubsub/inbox_sync/email_api_error/test.rs@@ -1,5 +1,7 @@use std::time::Duration;+use email_api_client::domain::models::RateLimitOrigin;+use super::*;#[test]@@ -15,38 +17,6 @@ fn retry_worker_uses_visibility_timeout_for_rate_limits() {);}-#[test]-fn gmail_message_local_rate_refusal_is_non_retryable() {- let error = handle_gmail_message_error(EmailApiError::RateLimited {- retry_after: Some(Duration::from_secs(10)),- origin: RateLimitOrigin::Local,- });-- assert!(matches!(- error,- ProcessingError::NonRetryable(DetailedError {- reason: FailureReason::GmailApiRateLimited,- ..- })- ));-}--#[test]-fn gmail_message_provider_429_is_retried() {- let error = handle_gmail_message_error(EmailApiError::RateLimited {- retry_after: Some(Duration::from_secs(10)),- origin: RateLimitOrigin::Provider,- });-- assert!(matches!(- error,- ProcessingError::Retryable(DetailedError {- reason: FailureReason::GmailApiRateLimited,- ..- })- ));-}-/// Pins the full `EmailApiError` → `ProcessingError` policy table for/// inbox_sync operations (the non-rate-limit path of `handle_operation_error`/// and the retry-worker rate-limit branch both route through@@ -57,7 +27,7 @@ fn processing_error_policy_covers_every_variant() {(EmailApiError::RateLimited {retry_after: Some(Duration::from_secs(5)),- origin: RateLimitOrigin::Provider,+ origin: RateLimitOrigin::Provider { concurrency: false },},true,FailureReason::GmailApiRateLimited,@@ -125,16 +95,3 @@ fn processing_error_policy_covers_every_variant() {}}}--#[test]-fn outdated_cursor_is_a_terminal_history_failure() {- let error = handle_gmail_message_error(EmailApiError::OutdatedCursor);-- assert!(matches!(- error,- ProcessingError::NonRetryable(DetailedError {- reason: FailureReason::OutdatedHistoryId,- ..- })- ));-}diff --git a/services/email_service/src/pubsub/inbox_sync/error_handlers.rs b/services/email_service/src/pubsub/inbox_sync/error_handlers.rsindex 39bc31ce3d..52b1ccddfa 100644--- a/services/email_service/src/pubsub/inbox_sync/error_handlers.rs+++ b/services/email_service/src/pubsub/inbox_sync/error_handlers.rs@@ -12,8 +12,7 @@ pub async fn handle_non_retryable_error(e: &DetailedError,) -> anyhow::Result<()> {// A rate-limited operation is only non-retryable here once it has been- // re-enqueued to the retry queue, or when it is a notification that a- // later one supersedes, so deleting it loses nothing.+ // re-enqueued to the retry queue, so deleting it loses nothing.if e.reason == FailureReason::GmailApiRateLimited {tracing::debug!(error = %e, payload = format!("{:?}", data.operation), "Rate-limited inbox sync message deferred. The message will be deleted.");} else {diff --git a/services/email_service/src/pubsub/inbox_sync/operations/gmail_message.rs b/services/email_service/src/pubsub/inbox_sync/operations/gmail_message.rsindex e8fde95fbc..19fed01fb6 100644--- a/services/email_service/src/pubsub/inbox_sync/operations/gmail_message.rs+++ b/services/email_service/src/pubsub/inbox_sync/operations/gmail_message.rs@@ -1,12 +1,12 @@use crate::pubsub::context::PubSubContext;-use crate::pubsub::inbox_sync::email_api_error::handle_gmail_message_error;+use crate::pubsub::inbox_sync::email_api_error::handle_operation_error;use crate::util::process_pre_insert::sync_labels::sync_labels;use email_api_client::domain::models::{EmailApiError, SyncCursor};use models_email::email::service::backfill::BackfillJob;use models_email::email::service::backfill::{BackfillOperation, BackfillPubsubMessage, InitPayload, JobScopedPayload,};-use models_email::gmail::inbox_sync::GmailMessagePayload;+use models_email::gmail::inbox_sync::{GmailMessagePayload, InboxSyncOperation};use models_email::service::link::{Link, UserProvider};use models_email::service::pubsub::{DetailedError, FailureReason, ProcessingError};use sqlx::PgPool;@@ -59,14 +59,14 @@ pub async fn gmail_message(return Ok(());}- // Gmail message notifications intentionally refuse labels/history work when quota is- // exhausted. A later notification will cover the same cursor range, avoiding fan-out- // backpressure while preserving the separate retry queue for child operations.- let labels = ctx- .email_api- .list_labels(link.id)- .await- .map_err(handle_gmail_message_error)?;+ // A rate-limited notification is requeued like its child operations rather+ // than dropped: an idle mailbox may get no later notification to cover+ // this cursor range.+ let operation = || InboxSyncOperation::GmailMessage(payload.clone());+ let labels = match ctx.email_api.list_labels(link.id).await {+ Ok(labels) => labels,+ Err(error) => return Err(handle_operation_error(ctx, link.id, operation(), error).await),+ };sync_labels(&ctx.db, link.id, &labels).await.map_err(|e| {ProcessingError::Retryable(DetailedError {reason: FailureReason::DatabaseQueryFailed,@@ -82,9 +82,10 @@ pub async fn gmail_message(Ok(batch) => batch,Err(EmailApiError::OutdatedCursor) => {schedule_stale_cursor_backfill(ctx, link, payload.history_id).await?;- return Err(handle_gmail_message_error(EmailApiError::OutdatedCursor));+ let error = EmailApiError::OutdatedCursor;+ return Err(handle_operation_error(ctx, link.id, operation(), error).await);}- Err(error) => return Err(handle_gmail_message_error(error)),+ Err(error) => return Err(handle_operation_error(ctx, link.id, operation(), error).await),};ctx.gmail_historydiff --git a/services/email_service/src/pubsub/inbox_sync/operations/upsert_message.rs b/services/email_service/src/pubsub/inbox_sync/operations/upsert_message.rsindex 1e19b197ff..506cbe8585 100644--- a/services/email_service/src/pubsub/inbox_sync/operations/upsert_message.rs+++ b/services/email_service/src/pubsub/inbox_sync/operations/upsert_message.rs@@ -6,7 +6,9 @@ use crate::pubsub::util::{};use crate::pubsub::util::{cg_refresh_email, publish_email_event};use crate::util::process_pre_insert::{process_message_pre_insert, process_threads_pre_insert};-use crate::util::upload_attachment::{UploadAttachmentContext, upload_attachment};+use crate::util::upload_attachment::{+ UploadAttachmentContext, UploadAttachmentError, upload_attachment,+};use contacts::domain::ports::ContactsIngress;use email::domain::events::{EmailEventOrigin, EmailMacroEvent, MessageDraftSyncedMetadata, MessageReceivedMetadata,@@ -15,6 +17,7 @@ use email::domain::events::{use email::domain::models::{PreviewCursorQuery, PreviewView, PreviewViewStandardLabel};use email::domain::ports::EmailRepo;use email::outbound::EmailPgRepo;+use email_api_client::domain::models::EmailApiError;use email_db_client::attachments::provider::upload_filters::{attachment_is_document, attachment_is_media,};@@ -309,7 +312,7 @@ pub async fn upsert_message(publish_email_event(&ctx.macro_event_broker, &event);}- handle_attachment_upload(ctx, link, payload, &message.attachments).await?;+ let refused_upload = handle_attachment_upload(ctx, link, payload, &message.attachments).await?;handle_contacts_sync(ctx,@@ -349,24 +352,38 @@ pub async fn upsert_message(.await?;}+ // Requeued only after the steps above, which are safe to repeat: the retry+ // finds the message saved, so it doesn't notify again.+ if let Some(error) = refused_upload {+ return Err(handle_operation_error(+ ctx,+ link.id,+ InboxSyncOperation::UpsertMessage(payload.clone()),+ error,+ )+ .await);+ }+Ok(())}+/// Uploads the message's new attachments, returning the rate limit that+/// refused an attachment fetch, if one did.#[tracing::instrument(skip(ctx, attachments), err)]async fn handle_attachment_upload(ctx: &PubSubContext,link: &link::Link,payload: &UpsertMessagePayload,attachments: &[Attachment],-) -> result::Result<(), ProcessingError> {+) -> result::Result<Option<EmailApiError>, ProcessingError> {if cfg!(not(feature = "attachment_upload")) {- return Ok(());+ return Ok(None);}// Keep this parsed-attachment gate at least as permissive as the database claim filters.let eligibility = attachment_upload_eligibility(attachments);if !eligibility.documents && !eligibility.media {- return Ok(());+ return Ok(None);}// upload attachments to Macro@@ -424,6 +441,7 @@ async fn handle_attachment_upload(})})?;+ let mut uploads = Vec::with_capacity(attachments.len());for attachment in attachments {// get the email addresses of the recipients of the messagelet recipient_emails: Vec<String> = message_recipients@@ -444,13 +462,15 @@ async fn handle_attachment_upload(AttachmentUploadDestination::Dss};- let attachment_upload_args = AttachmentUploadArgs {+ uploads.push(AttachmentUploadArgs {recipient_emails,attachment_metadata: attachment,backfill: false,upload_destination,- };+ });+ }+ return upload_claimed_attachments(&ctx.db, uploads, |args| async move {let ctx_upload = UploadAttachmentContext {db: &ctx.db,email_api: &ctx.email_api,@@ -459,15 +479,56 @@ async fn handle_attachment_upload(system_properties_service: &ctx.system_properties_service,link,};+ upload_attachment(ctx_upload, &args).await+ })+ .await;+ }- // keep processing if it fails, best effort- if let Err(e) = upload_attachment(ctx_upload, &attachment_upload_args).await {- tracing::error!("Failed to upload attachment to Macro: {e}");+ Ok(None)+}++/// Uploads claimed attachments in order. Other upload failures are logged and+/// skipped, but a fetch the rate limit refused stops the run: the refused+/// attachment and the ones not yet tried lose their claims so a retry of the+/// upsert claims them again, and the refusal is returned.+async fn upload_claimed_attachments<F, Fut>(+ db: &PgPool,+ uploads: Vec<AttachmentUploadArgs>,+ mut upload: F,+) -> result::Result<Option<EmailApiError>, ProcessingError>+where+ F: FnMut(AttachmentUploadArgs) -> Fut,+ Fut: Future<Output = result::Result<String, UploadAttachmentError>>,+{+ let mut uploads = uploads.into_iter();+ while let Some(args) = uploads.next() {+ let attachment_id = args.attachment_metadata.attachment_db_id;+ match upload(args).await {+ Ok(_) => {}+ Err(UploadAttachmentError::RateLimited(error)) => {+ let unfinished = std::iter::once(attachment_id)+ .chain(uploads.map(|args| args.attachment_metadata.attachment_db_id));+ for attachment_id in unfinished {+ email_db_client::attachments::provider::upload::release_upload_claim(+ db,+ attachment_id,+ )+ .await+ .map_err(|e| {+ ProcessingError::Retryable(DetailedError {+ reason: FailureReason::DatabaseQueryFailed,+ source: e.context("Failed to release attachment upload claim"),+ })+ })?;+ }+ return Ok(Some(error));}+ // keep processing if it fails, best effort+ Err(e) => tracing::error!("Failed to upload attachment to Macro: {e}"),}}- Ok(())+ Ok(None)}#[derive(Debug, Default, PartialEq, Eq)]diff --git a/services/email_service/src/pubsub/inbox_sync/operations/upsert_message/test.rs b/services/email_service/src/pubsub/inbox_sync/operations/upsert_message/test.rsindex 5effdb84c4..9fdeb1e77d 100644--- a/services/email_service/src/pubsub/inbox_sync/operations/upsert_message/test.rs+++ b/services/email_service/src/pubsub/inbox_sync/operations/upsert_message/test.rs@@ -1,5 +1,8 @@use super::*;+use email_api_client::domain::models::RateLimitOrigin;+use email_db_client::attachments::provider::upload::new_email_media_atts;use macro_db_migrator::MACRO_DB_MIGRATIONS;+use std::time::Duration;fn id(s: &str) -> MacroUserIdStr<'static> {MacroUserIdStr::try_from(s.to_string()).unwrap()@@ -399,3 +402,76 @@ async fn inserting_into_a_concurrently_deleted_thread_is_retryable(pool: PgPool)other => panic!("expected a retryable insert failure, got {other:?}"),}}++#[sqlx::test(+ migrator = "MACRO_DB_MIGRATIONS",+ fixtures(+ path = "../../../../../../../crates/email_db_client/fixtures",+ scripts("upload_claim_tests")+ )+)]+async fn a_refused_attachment_fetch_is_claimed_again_by_the_retry(+ pool: PgPool,+) -> anyhow::Result<()> {+ let link_id = Uuid::parse_str("00000000-0000-0000-0000-000000000c01")?;+ let image = Uuid::parse_str("00000000-0000-0000-0000-00000c2a0201")?;+ let video = Uuid::parse_str("00000000-0000-0000-0000-00000c2a0202")?;+ let animation = Uuid::parse_str("00000000-0000-0000-0000-00000c2a0203")?;+ sqlx::query!(+ "INSERT INTO email_attachments (id, message_id, provider_attachment_id, filename, mime_type)+ VALUES ($1, '00000000-0000-0000-0000-0000000ec201', 'claim-test-media-att-3', 'animation.gif', 'image/gif')",+ animation+ )+ .execute(&pool)+ .await?;++ let claimed = new_email_media_atts(&pool, link_id, "claim-test-media-msg").await?;+ assert_eq!(+ claimed+ .iter()+ .map(|a| a.attachment_db_id)+ .collect::<Vec<_>>(),+ vec![image, video, animation]+ );+ let uploads = claimed+ .into_iter()+ .map(|attachment_metadata| AttachmentUploadArgs {+ recipient_emails: vec![],+ attachment_metadata,+ backfill: false,+ upload_destination: AttachmentUploadDestination::Sfs,+ })+ .collect();++ // The image uploads; the gate then refuses the video's fetch.+ let refusal = EmailApiError::RateLimited {+ retry_after: Some(Duration::from_secs(1)),+ origin: RateLimitOrigin::Local,+ };+ let mut attempted = Vec::new();+ let refused = upload_claimed_attachments(&pool, uploads, |args| {+ let attachment_id = args.attachment_metadata.attachment_db_id;+ attempted.push(attachment_id);+ let result = if attachment_id == image {+ Ok("sfs-image".to_string())+ } else {+ Err(UploadAttachmentError::RateLimited(refusal.clone()))+ };+ async move { result }+ })+ .await;++ assert!(matches!(refused, Ok(Some(ref error)) if *error == refusal));+ assert_eq!(attempted, vec![image, video]);+ // The retried upsert claims the refused and untried attachments, not the uploaded one.+ let reclaimed = new_email_media_atts(&pool, link_id, "claim-test-media-msg").await?;+ assert_eq!(+ reclaimed+ .iter()+ .map(|a| a.attachment_db_id)+ .collect::<Vec<_>>(),+ vec![video, animation]+ );++ Ok(())+}diff --git a/services/email_service/src/server.rs b/services/email_service/src/server.rsindex c6b4059918..c18b30a288 100644--- a/services/email_service/src/server.rs+++ b/services/email_service/src/server.rs@@ -9,7 +9,7 @@ use crate::calendar_refresh::ConnectionGatewayCalendarRefresh;use crate::calendar_request_gate::RedisCalendarRequestGate;use crate::calendar_tokens::CalendarTokenProviderAdapter;use crate::outbound::email_api::{- EmailServiceTokenSource, GmailApi, RateBudget, RedisProviderRateLimiter,+ EmailServiceTokenSource, GmailApi, RateBudget, RedisGmailRequestGate,};use anyhow::Context;use calendar_events::{@@ -121,8 +121,6 @@ pub async fn run() -> anyhow::Result<()> {let redis_client = crate::util::redis::RedisClient::new(redis_inner_client,- config.redis_rate_limit_reqs,- config.redis_rate_limit_reqs_backfill,config.redis_rate_limit_window_secs,);@@ -203,7 +201,12 @@ pub async fn run() -> anyhow::Result<()> {auth_service_client.as_ref().clone(),sqs_client.as_ref().clone(),),- RedisProviderRateLimiter::new(redis_client.clone(), RateBudget::Live),+ RedisGmailRequestGate::new(+ redis_client.inner.get_connection_manager().await.context(+ "failed to get reconnecting redis connection for the gmail request gate",+ )?,+ RateBudget::Live,+ ),);let redis_client = Arc::new(redis_client);let gmail_token_state = GmailTokenState::new(GmailTokenProviderImpl::new(diff --git a/services/email_service/src/util/redis.rs b/services/email_service/src/util/redis.rsindex c2230ded4a..299e4f37f7 100644--- a/services/email_service/src/util/redis.rs+++ b/services/email_service/src/util/redis.rs@@ -8,31 +8,14 @@ pub mod rate_limit;pub struct RedisClient {pub inner: redis::Client,- /// Maximum allowed Gmail API quota units within the rate limiting window, for normal operations.- /// Gmail API per-user quota is 15,000 units per minute.- pub rate_limit_units: u32,-- /// Maximum allowed Gmail API quota units within the rate limiting window, for backfill. Less- /// than redis_rate_limit_reqs so we have room for normal gmail operations while backfilling- /// is occurring- pub rate_limit_units_backfill: u32,-- /// Duration of the sliding window for Gmail API rate limiting in seconds.- /// Uses a 60-second window to align with Gmail's per-minute quota period.+ /// Duration of the sliding window for Google Calendar rate limiting in seconds.pub rate_limit_secs: u32,}impl RedisClient {- pub fn new(- inner: redis::Client,- rate_limit_units: u32,- rate_limit_units_backfill: u32,- rate_limit_secs: u32,- ) -> Self {+ pub fn new(inner: redis::Client, rate_limit_secs: u32) -> Self {Self {inner,- rate_limit_units,- rate_limit_units_backfill,rate_limit_secs,}}diff --git a/services/email_service/src/util/redis/rate_limit.rs b/services/email_service/src/util/redis/rate_limit.rsindex 47ccf044ff..fcee7aacd1 100644--- a/services/email_service/src/util/redis/rate_limit.rs+++ b/services/email_service/src/util/redis/rate_limit.rs@@ -1,100 +1,16 @@use crate::util::redis::RedisClient;-use models_email::gmail::operations::GmailApiOperation;use redis::Script;use std::time::{SystemTime, UNIX_EPOCH};use uuid::Uuid;-pub struct RateLimitArgs {- pub user_id: Uuid,- pub operation: GmailApiOperation,- pub is_backfill: bool,-}--impl RedisClient {- /// Checks if a Gmail API operation is rate-limited and returns the status.- /// Uses a 60-second sliding window implemented via a Lua script for efficiency and atomicity.- /// Using a Lua script provides several benefits: simpler logic compared to raw Redis commands, connection pooling- /// via async multiplexed connections rather than dedicated connections, reduced network round trips,- /// and guaranteed atomic execution. If anything goes wrong, return false and hope for the best- ///- /// # Arguments- /// * `user_id` - UUID of the user to check rate limiting for- /// * `operation` - The Gmail API operation to check rate limiting for- ///- /// # Returns- /// Returns `true` if the operation should be rate limited (blocked), `false` otherwise.- ///- pub async fn is_rate_limited(&self, args: RateLimitArgs) -> bool {- let RateLimitArgs {- user_id,- operation,- is_backfill,- } = args;- // If we're backfilling, use a lower rate limit than if we are not. This is to allow some room- // for normal inbox operations while backfill is occurring.- let rate_limit_units = if is_backfill {- self.rate_limit_units_backfill- } else {- self.rate_limit_units- };-- let cost = operation.cost();- let mut con = match self.inner.get_multiplexed_async_connection().await {- Ok(conn) => conn,- Err(e) => {- tracing::warn!(- "Failed to get Redis connection for rate limiting user_id {}: {}",- user_id,- e- );- return false;- }- };-- let lua_script = get_rate_limit_script_with_usage();-- let redis_key = format!("gmail-ratelimit:log:{}", user_id);- let now_micros = SystemTime::now()- .duration_since(UNIX_EPOCH)- .expect("Time went backwards")- .as_micros() as u64;- let member_id = format!("{}:{}", cost, Uuid::new_v4());-- let script = Script::new(lua_script);-- let (is_limited, _current_units): (i32, u32) = match script- .key(&redis_key)- .arg(rate_limit_units)- .arg(self.rate_limit_secs * 1_000_000)- .arg(now_micros)- .arg(cost)- .arg(&member_id)- .invoke_async(&mut con)- .await- {- Ok(res) => res,- Err(e) => {- tracing::warn!(- "Failed to execute rate limit script for user_id {}: {}",- user_id.to_string(),- e- );- return false;- }- };-- is_limited == 1- }-}-/// Google Calendar's per-user quota is 600 queries per minute; cap our/// background sync at half of it so interactive calendar features keep room.const CALENDAR_RATE_LIMIT_QUERIES: u32 = 300;impl RedisClient {- /// Checks if a Google Calendar API request is rate-limited, using the- /// same atomic sliding-window Lua script as the Gmail limiter but with a- /// calendar-specific key and Calendar's flat one-query-per-request cost.+ /// Checks if a Google Calendar API request is rate-limited, using an+ /// atomic sliding-window Lua script with a calendar-specific key and+ /// Calendar's flat one-query-per-request cost.pub async fn is_calendar_rate_limited(&self, email_link_id: Uuid) -> bool {let mut con = match self.inner.get_multiplexed_async_connection().await {Ok(conn) => conn,diff --git a/services/email_service/src/util/upload_attachment.rs b/services/email_service/src/util/upload_attachment.rsindex 593909855e..5e6d602c70 100644--- a/services/email_service/src/util/upload_attachment.rs+++ b/services/email_service/src/util/upload_attachment.rs@@ -28,7 +28,7 @@ mod test;#[derive(Error, Debug)]pub enum UploadAttachmentError {#[error("Gmail API rate limit exceeded")]- RateLimited,+ RateLimited(EmailApiError),#[error("Failed to fetch attachment data from Gmail: {0}")]GmailFetchFailed(EmailApiError),@@ -213,7 +213,7 @@ async fn fetch_gmail_attachment_data(.get_attachment(link.id, &p.email_provider_id, &p.provider_attachment_id).await.map_err(|error| match error {- EmailApiError::RateLimited { .. } => UploadAttachmentError::RateLimited,+ error @ EmailApiError::RateLimited { .. } => UploadAttachmentError::RateLimited(error),// Preserve the provider error so API handlers can map it to the// right status instead of collapsing everything to 500.error => UploadAttachmentError::GmailFetchFailed(error),